RCTS
RCTS
Randomized Trials and Other Research
All of this research was by outside scholars
1. Can Education be Standardized? Evidence from Kenya
Guthrie Gray-Lobe, Anthony Keats, Michael Kremer, Isaac Mbiti, Owen Ozier
September 2022
Context:
Jay Kimmelman and Shannon May created Bridge. Some of the grittiest work you can imagine. In May 2013, the New York Times described their efforts here. They hired me 2 months later as a CAO. They already had an academic team and 134 schools, but wanted to use data to drive improvement as they grew.
We overhauled curriculum, training, and other systems – testing things along the way. Once we were confident things were humming, we enlisted some of the world’s best economists to measure us independently. The effects sizes, they wrote, were in the 99th percentile of any interventions ever studied.
New York Times coverage of that study is here.
PML Daily in Uganda is here.
Education Next is here.
2. Exploring Mechanisms of Effective Teacher Coaching: A Tale of Two Cohorts From a Randomized Experiment
David Blazar and Matthew A. Kraft
Educational Evaluation and Policy Analysis
Context
NSNO was a venture philanthropy that invited Match to help their larger effort in New Orleans. Erica was an amazing talent working in our teacher prep start-up, and wanted to move home to New Orleans from Boston. So together she and I assembled an intervention. My “condition” to NSNO was that they’d pay extra money to allow for a randomized trial of our effort. It took a little persuasion but they agreed.
David and Matt found a nice effect size at first, but as the program grew (and Erica had to coach less and manage other coaches more), the effect faded. This finding pushed me to believe: To get outlier gains, often you need outlier talent, and that makes scaling difficult.
This failure to scale well is a topic that comes up frequently in education. See the failure of the USA effort to “Scale What Works.”
3. Informing the Debate: Comparing Boston's Charter, Pilot, and Traditional Schools January 2009
Susan Dynarski, Atila Abdulkadiroglu, Joshua Angrist, Sarah Cohodes, Jon Fullerton, Thomas Kane, Parag Pathak
Context:
Back in 2002, I enlisted Josh Angrist to study our charter school as a randomized trial. We tried for years to get access to our “lottery loser” data, but couldn’t get permission from the state’s Department of Education. Finally, Thomas Kane cracked the approval code, and this team did a massive study.
They found Boston charter schools had very large positive effects. Ultimately Stanford CREDO found that these charters had larger gains than those in any other city in the USA.
I was also on the board of a pilot school, called Lee Academy. This study found that Pilot schools had no effects. They were like charters in having more flexibility, but still unionized. They also tended to embrace more “progressive” pedagogy and tended to be more skeptical of standardized tests.
Could the successful Boston charter schools scale? Glad you asked. At Match, we decided to open a new school, called Match Community Day. Many other Boston charters also added a second or third school.
This study examined the results of this second wave of Boston charters.
Can Successful Schools Replicate? Scaling Up Boston's Charter School Sector
Sarah R. Cohodes, Elizabeth M. Setren, Christopher R. Walters
American Economic Journal: Economic Policy
February 2021
However, a cautionary tale. Charter schools (not in Boston, but nationally) did not seem to help kids with earnings. This was a distressing finding, since the premise of many charters was to lift kids out of poverty. If test scores rise and later life is no better, what have we accomplished?
Charter Schools and Labor Market Outcomes
Will S. Dobbie & Roland G. Fryer, Jr
August 2016
It’s that work which inspired me to push my charter school friends to care more about the long-term results (do the poor kids get out of poverty as adults), even though that seems a long time after they graduate. It’s why I created 1Up Coaching with Geordie in Philly, which counsels first-in-family college graduates who are struggling in the labor market.
My last point on Boston charter schools: much has changed since the Kane/Angrist landmark “Informing the Debate” study. I fear that the large learning gains kids they found in 2009 have disappeared in 2024. There is non RCT evidence along those lines here.
4. Closing the income-achievement gap? Experimental evidence from high-dosage tutoring in Dutch primary education.
Joppe de Ree, Mario A. Maggioni, Bowen Paulle, Domenico Rossignoli, Nienke Ruijs, Dawid Walentek.
This research showed that the Match tutoring approach could work in another country, though it took incredible attention to detail. My friend Bo led this effort.
5. How much does teacher quality vary across teacher preparation programs? Re-analyses from six states.
Context:
Match built a small teacher preparation program. Did our teachers outperform those prepared at other universities? We thought so, but weren’t sure.
A researcher named Paul Von Hippel looked at all the teacher prep programs in Massachusetts.
All the data was de-identified in the paper, but this is the key slide.
Our teacher prep grads are the ones in blue. So they were, in fact, creating outlier gains for their students.
Was it because we were better at recruiting and selecting novice teachers? Or training? Or perhaps finding ways to get them hired into good charter schools?
We couldn’t know for sure.
How to find out? An RCT! We approached the Arnold Foundation and they generously earmarked $5 million for a randomized trial of our teacher prep program – it would have been the first one ever of a Graduate School of Education’s teacher prep program. I was so excited!
But it was hard to solve the recruiting issue (getting future teachers to agree to be randomized between our training program and a traditional university program). We tried some hacks but couldn’t get there. The study was never done.
Despite the large gains for students taught by the teachers we trained, sadly, there was no public push to “Scale what works.” In fact, our little program continued to be hobbled by regulators trying to block us from doing things differently.
6. The Effect of Teacher-Family Communication on Student Engagement: Evidence from a Randomized Field Experiment
By Matt Kraft and Shaun Dougherty
Charlie Sposato, the Match founding principal, was a maniac about proactively connecting with families. He pushed all our teachers to do the same.
Eventually I wondered: does this really help kids? The result of this small RCT was a resounding yes.
7. Experiments Inside Of Bridge (from Education Next)
Sean Geraghty led many RCTs inside our Learning Lab. Sometimes we’d collaborate with outside researchers.
Again, the philosophy here is published research should give you ideas, but then you need to test what works in your context. Some “proven” ideas don’t work in a particular context, and some do. There’s no way to find out without measurement.
