Lecture 10: Is It Possible to Deliver Quality Education to the Poor-The Pratham-JPAL Partnership

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality, educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

PROFESSOR: So today, we're showing just a very short video I just wanted you to hear. And we're going to be talking about Pratham today. We've had a very long relationship with Pratham over the years, we being J-PAL. That is me and Professor Banerjee and several graduate students.

And one of the leaders of is this person, Rukmini Banerji And it's sort of useful to hear this. It's a very short segment where she explains a little bit what she sees the education problem being about. And then we'll take it from there.

[VIDEO WITHHELD]

PROFESSOR: So I kind of wanted to take one lecture to go in detail to all of our work with Pratham, not only because I think Pratham is an amazing organization, but also because it will expose you to how we learn about the problem, how we build from one evolution to the next, one project to the next, how one project is bringing some more questions then we started with, and how the next project tries to get at these questions, also how the partnership with the local partners develops and works and all that.

And to this, we are going to be talking about this quality problem, where we are trying to address the basic point that she makes. Kids are in school.

At the moment, she talked at the Clinton Global initiative, that was 90%. But the enrollment rates are even higher now, close to 100%. In some districts, it's more than 100%, because some of the five-year-olds also go to school, because that's kind of a convenient way to get babysitting.

But despite that, the achievement is low. She did point out the results from the ASER survey. So every year, since 2005, Pratham puts together teams of people, local people, who go from local universities, et cetera. And they go in every single district in India.

How many districts did she say there was in India? She mentioned it at some point. You see that everything [INAUDIBLE] if you are listening to the video. She said it at some point, how many district they work in and how many districts they have.

If I'm not mistaken, she said something like 600. So they are sending people to every district in India, teams of college students, local volunteers, et cetera. They have a sampling. Of course they don't go to every village. But they do a sample in each district. And they administer a very simple test of reading skills and math skills.

So the reading skill test looks like a page. On the left side, you have a very simple story. Like she said, I go to school. My brother goes to school. We like going to school. On the right side, you will have single words and then single letters.

First, you start the child with a very simple story. If they can do that, you bring them to a slightly longer story and try to measure their comprehension. If they come to the very simple story, you ask them to read a word. If they can't read a word, you ask them to read a letter. So you, in this way, classify your child as a nothing reader, a letter reader, a word reader, a small paragraph reader, or a story reader.

And the simple, small paragraph reader is what you should be able to do at the end of the first grade. And what they find is, on average, in 2005, that about 35,000 children, aged 7 to 14, most of them are in school and could not read a grade 1 paragraph and 60% could not read a grade 2 story.

So these are numbers on one of the ASER survey. So blue is better in the maps I'm going to show you. And this is the fraction of kids who are out of school. And so that's quite a blue map, because there are not very many kids who are out of school.

And the colors here are the same. This is now the reading ability. This is the fraction of kids who can read a standard one text. So it is the fraction of kids who can read or understand one text among the kids who are enrolled in standard three to five.

So among children who are enrolled in standard three to five, how many of them can read this grade one paragraph? And you can see that now we have a lot of red in this card. The red is a bit distributed in different places, but mostly it's quite red.

If you know India, does this map look like what you would expect for any social indicator within India? For anyone who knew India or about India, because this is a question that you might not know the answer to.

So it doesn't really, because the folklore in India-- and if you look at immunization, that's really what you get-- is that sort of the bad states are these ones, Uttar Pradesh, Bihar, Rajasthan, Jharkhand.

These are what they call the BIMARU state, Bihar, Rajasthan, Uttar Pradesh. This is like where you have a lot of boys and not very many girls, suggesting infanticide or selective abortion. This is where you have very low immunization rate, bad outcomes, of course, as I mentioned.

And these are the good states, Andhra Pradesh, Karnataka, Tamil Nadu, Kerala. Kerala is, in particular, one of Amartya Sen and John Rawls' favorite states for its successful education. Kerala does well here, almost everyone can read.

But what is surprising is you have Tamil Nadu. They're very proud of themselves. A lot of the kids are in school, et cetera. It's very red. Less than half the kids enrolled in [INAUDIBLE] three to five cannot read a sentence.

So that upset the government of Tamil Nadu a great deal. So when the first ASER report came out, they didn't believe the result. That is not possible. They went ahead and did their own test, but they found the same thing again. So this suggests that there is something different.

It's not only like disorganized places that are unable to deliver to their kids. Their is something maybe deeper than that. And Bihar did surprisingly well in not being at the absolute bottom of the barrel in this graph. So Bihar was happy.

Another thing that's is maybe even more troubling is this survey has been done from 2005 to 2010, every year. It's been released in January every year, on the Republic Day in India. And there is no progress.

So India is progressing across all these dimensions, becoming richer, there are fewer poor people. We're seeing they're eating less and less, which is not progress, per se. But education, nothing, no progress.

Also troubling is it's not an Indian exception. Surveys very similar to ASER have been done in Kenya and Uganda and Tanzania and in Pakistan. And they all find similar kind of things. A lot of children are in school, most of them can't read anything. Yep?

AUDIENCE: Is there still progress across the board on average, like in the nation? Or the states are static as well?

PROFESSOR: So you're exactly right. That's a very good question. It's no progress across the board in the nation, but the states are not static. Some states start doing well for a while, and means some states are moving down to compensate.

So, for example, in the last report, 2010, Punjab had a big progress. So that's one of the things they are trying to look at is why is Punjab suddenly progressing? What are they doing that is different? And I'm going to be able to talk a bit about that.

And in fact, in the Punjab example, what they seem to be doing that is different is really to try to insist on these core competencies in the schools.

They have a period of time, every school day, two hours a day, which are devoted to teaching the kids at whatever level they are forgetting the curriculum. And that is what they think is the reason why Punjab is progressing.

So what we are going to do today is-- do we have evidence to suggest that, in fact, they're right? And that is the key problem and, therefore, the key of the success of Punjab. Meanwhile, some other state, like Uttar Pradesh, would be going down. Ben?

AUDIENCE: I have a question. Is the student to teacher ratio more or less? I know these states are huge. But are they more or less the same across the different states?

PROFESSOR: So it's a very good question. I think, they are, probably, generally more or less the same. And they are not horrible. Because, actually, being a teacher is not a bad job. So there are a number of teachers who are available.

So you don't have the very, very, large class size that you have, for example, in a country like Kenya, where, after free primary education, the class size in grade one was easily 70 to 80 children per classroom. You have less of that in India. Yeah?

AUDIENCE: So it seems like the enrollment is high, but do they also track that the attendance is high?

PROFESSOR: So that's a very good question. And the answer is that, yes, they check attendance. And attendance is low. So attendance is low, both for teachers and for children. So in Bihar, for example, we have a survey of children's attendance.

The rate of absence is between 30 to 50% for kids. And we've done the same thing in several countries of the world. Generally, children's absence is very, very high. So kids are enrolled, but they don't show up. And it's not that they are ghost enrollees. This 30% attendance is not concentrated across 30% of kids who never come.

It's every child who is absent one day, coming one day, absent on day, coming one day. And that is one problem. So we have a lot of absence of children. And we have a lot of absence of teachers as well. So a survey was done in four to five developing countries, by the World Bank, including India. They found a very large teacher absence rate.

So in India, it was about 25% absence rate. And that's about what ASER found as well, of teachers. So you show up to a school during school hours, and you have three of four teachers who are actually in the school.

The second problem is that once they are in the school, it is not a given that they are actually teaching. So another one out of these four teachers is actually in school but not teaching. There was a funny report, a few years ago, describing what they were doing, ranging from drinking tea to drawing political posters.

The reason being that a lot of these teaching jobs are patronage jobs, which are filled by the political parties, so they need to draw their posters sometimes. So that means that the teachers are present teaching half the time. The students are, themselves, present about half the time.

That means the students get about a quarter of the time that they should get in front of a teacher. This is a where, [? Hookman's ?] point, that you need a bit of time of student-teacher interaction to get teaching. It actually becomes quite pertinent. Yep, sorry?

AUDIENCE: I have a question about the standard curriculum. What kind of things does it teach? Is there any [INAUDIBLE] benefits from it? Are there any kids who are up to the curriculum?

PROFESSOR: So that's a very good question. What is the curriculum? So the curriculum in India-- and that actually is true in the countries I know-- tends to be very ambitious. And there are various reasons why those curriculum are very ambitious.

They are probably more ambitious than what is being taught in the US. Actually, significantly more ambitious than what is being taught in US primary schools. We once showed up to a school. We were with the Pratham team. The visit was announced. The teacher wanted to look good.

And his idea of looking good was to draw things like that on the board, in this grade three class, so all the kids were like looking around. And it doesn't mean that this is what he actually does on a day in day out basis, but it means it is what they suggest to do.

So it's not necessarily a bad curriculum. In fact, India has a lot of good, very good, very serious education experts who are thinking about educating the child to a real, true understanding of science. But for most of the children, it's completely above their head, very quickly.

And then some of the children benefit from it. We can see an example of that very quickly. We'll see an example of that very quickly. Yes, [INAUDIBLE]?

AUDIENCE: Is their proficiency in math as bad as in [INAUDIBLE]?

PROFESSOR: It tends to be worse. So math is a bit worse than reading. For example, if you look at division, which is the equivalent of your standard two kid, you have like 20% of kids can do division.

So that's where we are. So this quality of education, I was saying the other day, has not concerned the policy establishment, very much, for many years. The policy establishment has been more concerned about getting as many children as possible in schools, enrolled in schools. With the understanding that, oh, if we can manage to get them in schools, somehow learning will happen.

And the Turkey experiment is, in a sense, an archetypal example of that, which is let's make education compulsory, let's have boarding schools, let's have buses that are going to bring the kids to the boarding school. And then black box, learning happens. You have an educated child.

But it's been about 10 years since academics have been a bit puzzled at that. And looking at the quality of education saying, how do we make education actually work? What's important? And the first thing that jumps at you, when you go to a school in India, in Kenya, even in rural Morocco, which is a richer country, is that there is nothing in the school.

The school is usually very bare. You have fewer desks than students. People are bunching like three to a desk. They will usually have a blackboard, very few textbooks, and, of course, nothing like computers and things like that.

So the first generation of studies was about, well, they have none of that. How can they learn? Maybe what they need is more material.

And in fact, when the idea of randomized evaluation started to be applied to development, Michael Kremer, who is a professor at Harvard, wanted to do a demonstration project, wanted to show that it is possible to do a randomized evaluation. And it's interesting. And he wanted, for demonstration purposes, an example where it would clear that it has an effect.

And so he asked around, et cetera. And he found out that what would be obvious is textbooks. In Kenya, no one has textbooks. If only we give textbooks to the kids, obviously, they are going to learn better. So that was his first example that he chose, which was chosen specifically to demonstrate success.

And running his experiment, he was learning how to do an experiment at the same time. So his first experiment was very small, 14 schools, 7 treatment, 7 control, and found no effect.

He thought about it, and he said, oh well, of course, it's because I have so few schools. And you remember, when you have schools, you need to take into account, when you calculate your standard error, that all these children are similar.

So he said, oh well, my experiment is not big enough. So let's take a bigger sample of schools. He took 100 schools. He took half of them and again found no effect. So there he thought, well, maybe the test is too hard. I'm using the test, the normal test. Maybe that is too hard. Let me use a different test that is going to be able to discriminate progress even at a lower level.

And he got his experiment started. And again, he found no effect. So there he started saying maybe there is no effect, even in a larger sample, even with a better test. So maybe there is no effect.

So eventually, what he found is that, if you focus on the children, who were already doing well at the baseline, then they benefit from the textbooks. So that's a very long winded answer to your question, which is there are children who benefit from the curriculum and, therefore, benefit from the textbooks. But it's a small minority.

And the explanation he gave is the textbooks are in English, which makes sense, because the curriculum in Kenyan schools is in English. But a lot of children don't even speak English, because English is their third language.

They first speak their mother tongue, the local language. Then they learn Swahili in the first few grades. And then English is introduced in grade one as a language, and then as a language of instruction from grade three or grade four.

But the problem is that because the kids aren't actually learning effectively, by the time they reach grade five or six, they actually don't know English, so the textbooks are of no use.

So exposed, it could be understood what was going on here. And then they tried a bunch of stuff. Flip charts also have no effect. Cutting the class size in two, if you make no other differences, also has no effect.

And a little bit worrisome, things like the deworming program, which increases attendance, but it didn't increase test scores. So it seemed that these extra school days also had no effect.

So changing inputs just didn't work. So maybe it's a problem that is just incredibly difficult. So we were there a few years ago. So what this common with all this intervention is that they are just changing the inputs.

They are doing more of the same, adding more textbooks, adding more teachers, adding more resources. But they are no change to the pedagogy and no change to the incentives. So no one is given incentives. The teachers are not given incentives. The students are not given incentives. The parents are not given incentives. And the pedagogy and the curriculum stays the same.

So that's kind of where we were about five, six years ago, a little bit depressed. Picture us, this was our first set of randomized experiments and nothing works. So we were thinking, maybe, this is going to be the end of randomized experiments, because you can't be a doomsayer forever, otherwise people really hate you.

So that's why we were a little bit sad. Rukmini came to visit us, shortly after that happened, a few years after the establishment of Pratham, came here to MIT. Because a former MIT student had gone to work for them as an intern. A former undergraduate student of ours had a relationship with them.

So Pratham started in 1994. At that time, it was established by UNICEF to help some kids in Bombay. It would be your typical, small NGO doing some work in Bombay, particularly what they call bridge classes. Which is take a kid who is out of school, give them a course for a few months so that they can try and go back to their regular school. That's what they were doing.

But they were ambitious. So there was the Rukmini that you heard and Madhav Chavan wanted to make a real difference. So since then, they're grown substantially. They've reached millions and millions of children. I think there are about 38 million children, who are reached by the Pratham program, one way or the other, so about half the population of France, overall population of France, just in terms of scale.

And so that's the largest non-governmental organization to do education in India, probably in the world. And their motto is "every child in school"-- that half of the world would agree with them-- "and learning well"-- is where, maybe, they have a little difference.

So they came to us towards the beginning. And that's the reading that read for today. They wanted to evaluate the Balsakhi program, which was the flagship program at the time. Balsakhi means the friend of the child.

So the Balsakhi is a young woman, from the community, so an adult. Like she said, you need some adult time. Barely an adult, some 18-year-old, usually having only high school education, so grade 10 to 12.

And Pratham would give them a very short training and then dump them into the school, with which they had an agreement with the teachers that the school would let them pull out the kids who were lagging behind in grade three of in grade four.

So why is this a good demographic, the 18-year-old, grade 10 educated woman? Why is that a good goal to work with? Yep?

AUDIENCE: She's more familiar with the community, so she can probably relate to [INAUDIBLE].

PROFESSOR: So that is one reason is that she is more known to the community, because she's local. She's not much older. And she's not intimidating or scary for the parents, who, maybe, can work better with the kids. That's a very good reason. What's another reason?

AUDIENCE: I would say for the teacher's as well. Maybe the teachers are not as intimidated that the balsakhi will go and teach these students some really innovative thing that would make [INAUDIBLE].

PROFESSOR: Yes. So for the teachers, she might be not very intimidating, which will cut both ways. On the one hand, it's more easily acceptable for the teachers. On the other hand, there is a tendency that teacher would use the balsakhi to make the tea or other activities like that.

And we are going to see some of that happening, not actually in Baroda. Yep?

AUDIENCE: So she's locally trained, which makes sense in kind of [INAUDIBLE] perspective.

PROFESSOR: Yes. She is locally trained, only for two weeks. This is very cheap. And there is another aspect of the cost. Yeah?

AUDIENCE: Not about the cost, but she might be more motivated to help the [INAUDIBLE] children, because it's not like this is a second job she can perform. It's not a fallback option. It's something that she wants to do that is a period in her life. It seems a bit time sensitive.

PROFESSOR: Yes. So she might be more motivated for this reason. Also for what you said earlier, that she's close to them, et cetera. But also this is something that she effectively chose to do at this point and not something that is just putting money in a bank account.

AUDIENCE: She's not highly qualified, so the [INAUDIBLE].

PROFESSOR: Right. She's not highly qualified, so she's cheap. What is another reason why she is cheap? Why don't these people go and do something else?

AUDIENCE: Also there's a high turnover in these [INAUDIBLE] make a certain income to teach, so you're not really relying on this thing. It's [INAUDIBLE] quality of the teaching. It doesn't have to be like a small, really enthusiastic [INAUDIBLE]. So it's easy to get [INAUDIBLE].

PROFESSOR: Yeah. They have high turnover, which could be a plus or a minus. On the minus side, of course, you don't get the experience. On the plus side, you don't get tired. And you don't rely on people who have a huge vocation. You rely on people getting enthusiastic for one year.

So very much along the model of Teach For America, where you get MIT, graduating from school, energetic about wanting to do this. A bit of what you're saying, which is it's what I want to do at this point in my life. And then by the time you don't want to do it any more, you stop.

And there is one more thing about them, specifically, which makes them cheap.

AUDIENCE: Everything about them. I mean I think there's like a good chance that they have families or that they're part of a family, and maybe that's why they're stuck in the area. And they don't necessarily have the ability to go somewhere else.

PROFESSOR: Exactly. That's a very important point. Most of these women are unmarried. They are living at home. They are sort of waiting to be married. And there is actually not much they can do. Because the parents won't let them leave.

And mostly, their parents won't let them work, because working is not becoming for a certain, sort of lower, middle class category woman, waiting to be married. You shouldn't be like working. You should be able to be-- it's not a good signal for the family that they're not able to provide for you and all that.

But this is hardly seen as working. This is more like volunteering and helping your community and things like that. So they can do that. So they are sort of available.

So one part of the genius of Pratham is that they identify a group of people who are actually sort of available for free. Here, they pay them, but in future programs, that I'm going to talk to you a minute after, they're not even paid.

They are just unpaid. They do that for a while. And that get some, presumably, other rewards of doing that, which are more in the form of intrinsic reward of seeing the kid progress, and, therefore, you also select people who are motivated in that dimension as one of you pointed out.

So that's all of the pluses. The minuses you already mentioned. The turnover is one of them. You can't get them to get experienced.

The second minus is that they're, of course, much less educated. Maybe there is some good reasons why a teacher needs a college degree. Maybe it's actually terrible to take the kid out of the classroom to put them in front of someone who knows nothing. So that's kind of the risk that would potentially be there.

So that's the program. And when they came to see us, we were wondering how to evaluate this program, with them, in an experiment, in a way that is not going to cause a problem for them in the city, that's not going to hinder their work too much.

And that's experimental design that we decided to adopt in a new place. So Vadodara and Gujarat, those were places where they had not worked at all. Had about 100 schools. A little more than 100 schools divided in two groups, randomly, with the computer, group A schools and group B schools.

The group A schools got the balsakhi for grade three children. The group B schools got the balsakhi for grade four children. So what's an advantage of this design from the point of view of political acceptability? Why did we go like this, instead of doing the more standard thing, which would be to say, well, in group A treatment, and grade three and grade four get the balsakhi and group B, no one gets a balsakhi?

AUDIENCE: Well, in a way, if you don't give one group any treatment at all, you're almost sabotaging your education [INAUDIBLE].

PROFESSOR: Sabotaging is a strong word. But exactly the idea is that this allows you to be present in every school. So Pratham is not infinitely rich. They can easily argue-- and I think that was very true at that time-- that look, we can give you one balsakhi per school. We can't afford more.

So you either put it in three or you put it in four. We can't give you both. But at least every school is being engaged with. So they are engaged with the system as a whole. And no school is disadvantaged.

That makes it easier for the school system to say, yes, they are just working with us. They are part of us. It also makes data collection much easier, because you're involved with everyone.

What is a potential danger of this design? The plus is what I just said. What's the potential minus? Since you have a balsakhi in every school, what's the potential minus?

AUDIENCE: They might not just be teaching grade three. The might be overlapping grade three and four or something.

PROFESSOR: Exactly. The balsakhi might take both groups, so that your control group becomes partly treated. Or even if it's not the balsakhi, the head teacher might say, oh great, we have a balsakhi for grade three. So we're going to take the teacher out of grade three and use it to divide grade four in two. And so grade four would be associated.

Or they could say, well, since the grade three got the balsakhi, when some other organization comes with computers, well, we'll put them in grade four. So there is a danger of resources being reallocated across grades in a way that contaminates your treatment.

So fortunately, that was not really an issue there, because in Vadodara, the way the school system is organized, a school is made of one teacher per grade, regardless. Sometimes in the same school building, you have more than one school. And that happens sometimes in the US as well. You have this charter school and the regular school sharing the same school building.

In Vadodara what happened is, if it's a big school, it's actually a big school building, but under totally separate administration. So a school is made of five teachers and a head teacher. That's it. So there is less scope for reallocating, because you have your one teacher anyway.

And they're also not very imaginative in terms of using the resources in the most optimal way. So that turned out not to be a problem.

So the way we evaluated the program is then by comparing grade three, group A's treatment compared with control. And in grade four, that's the opposite. We also worked in Bombay. So that's the second year we have asked a group.

So if you were a kid who was in grade three, in group A, you are treated in grade three and then, again, you are treated in grade four, because you would have moved one grade.

If you were a kid who entered in grade four, you would never be treated. Or if you were a kid who entered grade three in group B, you also would never be treated. So we have kids treated one year, kids treated two years, and kids treated one year.

And then in Bombay, we did something similar. This is Shobhini She used to work at Pratham. And now she's one of the executive director of J-PAL South Asia. You can say, hi, to her.

And in Bombay, we did something similar, except we started in grade two, three, and then moved it to three, four. The reason why we switched is that they realized that they didn't have right pedagogy for engaging with the grade two children, that the grade two children were too small, and they didn't know how to deal with them.

The initial plan was to do two, three and then reverse again, two, three. And then they turned out that they weren't happy with they're grade two, so they moved three, four.

So that's kind of the idea. So this is the design. So in principle, now, we could say, well, they're good. We have the designs. Now, all we need to do is to compare treatment and control after one year and after two years, and we're going to be done.

But unfortunately, when you run real experiments, you get real problems. I've yet to meet an experiment where there was no problems. And this was, at least, my first big one. So we had a lot of problems that I want to talk about.

The first one is the way to evaluate the program was to administer a test in the school. And we had some issue with the test. The first one is that, as I was saying, a lot of kids are absent. And if we only tested the kids who were present in school, we have some attrition.

Maybe in urban India, absenteeism is less bad, but maybe 20% of kids are absent on a given day, so they don't take the test. So why is that potentially a problem to have missing kids in the test? Yep?

AUDIENCE: Maybe the kids that are absent are those that are more likely to not [INAUDIBLE] class, and so they either chose or their parents chose that, maybe, they could better use that time working from home. And so all the results are biased. Then you're only really testing the kids that understand more and you're ignoring the kids that understood the least.

PROFESSOR: Exactly. So that could be the first problem, which is those kids, whom we are testing, they are not representative, because maybe they're the one who understand the most.

And that could lead to an even bigger problem, because suppose the intervention is effective and most kids now understand better. Then what could happen with the attrition in the treatment group versus the control group?

Now you all know very well, it's the kids who don't understand who don't come. And now the kids who understand better are in the treatment group. What might happen to attrition?

AUDIENCE: The attrition would be less for the students who are in the treatment group.

PROFESSOR: The attrition would be less for students who are in the treatment group. And the kids who are coming tends to be the weaker kids. So now we are comparing the strong kids in the control group to the strong kids plus some of the weak kids in the treatment group.

So that will tend to make our estimate look smaller relative to what it should be. Because now we have a different population of students in the treatment and control groups. So we have a bias here. It goes in the direction of not finding an effect. So if we still find an effect, we're happy. But it's not the right estimate.

So to solve this problem, what did we do? Did we rely, uniquely, on the test in the school? Yeah?

AUDIENCE: You found the kids, even if they happened to be at home or elsewhere and administrated the test [INAUDIBLE].

PROFESSOR: We went and looked for them, wherever they were, except if they'd left for the village. So the attrition ended up being low, less than 10%, and similar, now, in treatment and control school. Because it's not because people don't understand that they leave for the village. It's because they have something to do there.

So that was the first thing. The second problem we had is that the testing instrument-- actually, not in year 1 but in year 0-- we designed a test which was at grade level. We designed a test to test the curriculum.

So it's like, if this is what should be tested, what's on the board, here, we tested at that level. So if they were supposed to learn Euclidean geometry, that's what was tested. And we had a big problem, which is that the teachers cheated like crazy.

So the reason we saw that is that all of the tests were the same. So clearly, the teacher had written the answer on the board, and all the students were copying down. In one class, all the students had the same name. So the teacher clearly wrote the name, like a sample name, on the board. And all the students dutifully wrote down the name that they saw on the broad.

So we said that's not working. There is a paper by Steve Levitt, who shows very subtle things, how you can detect cheating. That was easy. It didn't require a lot of imagination. It was pretty obvious that cheated like crazy to make their kids look better.

So we thought about how we can solve this problem. We are going to administer a new test without the teacher, that some special Pratham staff is going to come to administer the test. And we did that. And then we realized that most of the students now had 0 on the test.

So this test was way too hard, which is why the teacher had been cheating, because they were a bit embarrassed. So the solution here was to develop a much easier test that covered the competencies starting from, can you write your own name, to-- I don't know-- 2 plus 2, finishing with a few questions such as as, Nancy goes to the market. And she buys three onions at 15 rupees. And how much change does she have, that kind of thing.

So that's the problems that we had. Further problems we had, we had a problem in Bombay. Because the second year of the evaluation turned out to be 2002 to 2003.

Before that, September 2001 happened. We were walking in a quite Muslim neighborhood of Bombay. A lot of kids were named like Osama and things like that. People were pretty upset about the US at the time.

And so Pratham, maybe because of us, maybe just it would have been anywhere, was somehow seen, maybe, as an American presence. So some schools said that they don't want to work with the program. So about 30% of schools said that they did not want to work with the program. Or, in another set of schools, someone was identified, but they couldn't read, themselves, so they couldn't really be entrusted to teach reading to the kids. So they also didn't do it.

So can we just drop all the schools that refuse to participate from the program? Can we just drop them from the analysis?

AUDIENCE: Well, that's a selective bias. There is a some reasons for which they dropped, which make them inherently different from the other schools.

PROFESSOR: Exactly. So we can't do that, because that's a selection bias. Maybe these are the weaker schools who are refusing the program. Maybe they are the stronger schools, because they are independently minded. Who knows?

But what is clear is that's it's certainly not random. So we lose random control comparison. So what we do? Well, the first thing you can do is to say, well, I'm going to not measure the effect of the balsakhi program.

I'm going to measure the effect of my intention to treat the schools with the balsakhi program. So we called that intention to treat. What an intention to treat is a Pratham official coming to the headmaster and saying, hey, we have a balsakhi for you.

So this, we can do in an unbiased way, right? Because if we compare all the test scores in the school where someone has come and offered the program to the test scores in the school where no one has offered the program, there is no selection here.

But it's not the effect of the program. It's the effect of my attempt to provide the program. So that's why we call it an intention to treat. It's not the treatment effect. It's the wishful thinking. It's like my attempt to treat you.

Now, I wrote, here, on the board, if we wanted to know the effect of the program itself not the intention to treat, we can divide the intention to treat effect by the fraction of schools who accept the program.

So, for example, if 70% of schools accept the program, and we find an effect of 10 on the test scores, a gain of 10 points in the intention to treat, the intention to treat is 10 points. That's due to only 3/4 of the schools.

So we can blowup the estimate by dividing by 3/4, which is about multiplying by 1.25. So instead of 10, the effect is 12.5. So this is what's called a Wald estimate. When you divide your intention to treat by the fraction of the take-up of the program, which, here, we call that the first stage.

So in the first stage, you offer a program, and then some fraction of people take it. In the second stage, that has an effect. And the combination of the first stage and the second stage produces the intention to treat.

So now to go from the intention to treat the effect of the program, you do a scale-up by dividing by the take-up. So this is, in five sentences, you're instrumental viable estimate,

We could spend about a whole semester on it. There are conditions under which this is valid to do this. There are conditions under which this is not valid. And this has interpretations which vary. So Josh Angrist, in the econ department, is the one who did most of the work on this.

So the point it that-- but think of it in this simple way, for this kind of noncompliance problem-- is let's just normalize the estimate by dividing by the take-up. It would also work if you have some school in the control group that managed to get the balsakhi anyway.

Then you could divide by the difference between the take-up in the intention to treat group and the intention not to treat group. So if 10% of schools manage to get the program in the control group, you would divide your intention to treat by 27% minus 10%, that's 65%. And you would blow it up that way.

What it amounts to doing is to say, this is my estimate of the effect of the program, assuming that the only reason why they're in the intention to treat effect is because those who might try to treat are more likely to be treated.

So here are the results. The results are expressed in standard deviation of the unique test scores. What this means is that we take the mean of the control group and divide by the standard deviation. So the advantage of doing that-- maybe you can see that more in recitation-- is that every education program does that. So we can compare our results to what other people did.

For year one, this is the standard deviation. This is the test score in treatment and the test score in control. That's the difference, 0.17. That's the standard error of this difference below. So you divide one by the other. You get a t-stat of about 1.7. So that's significant but at a 10% level.

In year two, kids progressed. And the difference is much larger. So you get an effect of 0.4 standard deviation in math and a 0.25 standard deviation in language and something similar for the standard four classes.

So those are a large effect. To scale that up, one of the most famous education experiments is the Tennessee STAR class-size reduction experiment, where that reduced class-size in Tennessee from 20 to 8. It was very, very expensive program that had an effect of 0.2 standard deviation.

So 0.2 is considered to be a reasonably large effect. Here, in the second year of the program, we are way above that for math. Yep?

AUDIENCE: How big is the standard deviation relative to the average score?

PROFESSOR: So this is all normalized. So this is, in a sense, already standardized. This is already standardized to the average and to the standard deviation.

AUDIENCE: But there is like a wide distribution in the classroom?

PROFESSOR: You mean, is it wide? Yeah, so you can see, it's not very wide, in this case. Because the students are so weak that, I think, the distribution of test scores, in terms of points, is not that large compared to what you could find elsewhere. It's kind of a relatively tight distribution.

So in terms of points, it's useful. There are other ways to look at this than standard deviation. For example, the fraction of kids who can do these kinds of things, so in percentage points. How many more kids can do division? How many more kids can do addition? Which are also useful to look at, which I don't have with me. That's a good point.

So these estimates are large, but they're not very precise, because there's a lot of differences between kids. And one thing that really helps to control this noise is to control for how good where they before. Because test scores of children are extremely stable over time.

The biggest predictor, sadly, of how a child does is how they did last year. And so, when we control for that, we get much more precise results, which are here. So these are a bunch of results.

So this is the improvement in year 1, for both cities together, where we get an improvement of 0.2 in math, 0.3 in language-- sorry, opposite, 0.2 in math in year 1, 0.3 in math in year 2, for verbal, 0.7, not significant in math, for verbal, 0.7 for year 1, 0.15 for year 2.

And then we can look at various aspects of it, where we get a slightly bigger effect for Vadodara and Bombay but, generally, quite similar results for math. And language has bigger effect in Vadodara than in Bombay. Bombay starts from a higher level. But across the broad, except with Bombay year two, you find pretty large effects, quite large effects.

So this is very different from what we had with the test scores. Now, you might still wonder that these are for all the kids involved in the program. And you might still wonder whether the effect comes from helping the lower achieving kids, which is what you're trying to do, or it's coming from helping the higher achieving kids by removing these disturbing, lower achieving kids from their classroom for two hours a day.

That would be fine. But the distributional effect would be very different. If the idea is that you just send the kids to recess, and they kind of hang-out, and learn strictly nothing, meanwhile, your 20, high achieving kids progress, that could give us those mean effects. But that is not necessarily something that we would be very excited about.

So for that, what is important is then to look at what is the effect by sub-group? And what was the finding when we looked at sub-groups, who benefited?

AUDIENCE: The lowest one third benefited the most.

PROFESSOR: The lowest one benefited the most. And who was more likely to go to the balsakhi? Also the lowest one, fortunately. So when you look at the lowest one, you start finding a pretty large effect for the bottom third and a much weaker for the top third.

For example, if you look at year 2, the effect for the bottom 1/3 is 0.5 standard deviation. That's starting to be a huge effect, 0.5 standard deviation progress. 0.3 for the middle one, and 0.03 for the top one, no effect.

And this is your chance to go to the balsakhi. So your chance to go to the balsakhi declines with the group. So the point estimate of the effect declines at about the same rate as the point estimate for the balsakhi.

So the next thing we did is a little bit the same exercise as adjusting the Bombay estimate, to go from this estimate of the effect of putting a balsakhi in the classroom, to are you going to the balsakhi?

So basically, the effect, this 0.5 standard deviation, is due to about 20% of kids who are actually going to the balsakhi. Because there doesn't seem to be an effect for the other children, which we see from looking at the pattern declining as the pattern of take-up declines.

So now you can divide this 0.5 by 0.2. And you get the effect for people who go to the balsakhi of one standard deviation, which are very, very, very large effects, in the education literature. And if you divide 0.32 by 0.16, you'll also find about 1.

And this is, of course, unsignificant. But if you divided 0.4 by 0.06, taking the estimate seriously, again, you would find the same type of effect.

So this is a program that was highly effective, for kids who were sent to the program, but had no effect on the kids who, in principle, should have also benefited from the reduction in class size.

So what did we learn? We learned that it is possible to make a lot of progress, with a grade 10 educated woman trained for two weeks. And we also learned that teachers are not very good at exploiting freed-up resources. Yep?

AUDIENCE: How much variability was there between the different thirds? If the top third isn't that much better off than then the bottom third, then that would say something different than, if the top third is very well educated whereas the bottom third is really not.

PROFESSOR: So the bottom third, basically, knows nothing at all, for starters. Like they can't do anything at the beginning. They can't recognize letters, can't recognize numbers. Whereas the top third is not at grade level but can at least do something.

So what I'm saying is not that we shouldn't care about the top third. We should care about them. But that program was really targeted toward the bottom.

And so the big difference, in the context of this program, is that the top third didn't get the benefit from the program, because they weren't sent to the program. Except that there could have been indirect benefits from the fact that the teacher's now have a much smaller class size. Instead of 40 kids of very heterogeneous levels, they now have 20 of better level. So they could have adjusted their teaching to do better by these kids. And it seems that they didn't.

So in terms of the whether this is an optimistic or pessimistic conclusion, it's kind of the glass is half-full or half-empty. The half-full part is you can do this. It's reasonably easy to make a lot of progress. Perhaps because kids start from such a low level, to go back to your question, initially.

If the standard deviation is low, because everyone is at 0, then it's easy to make some progress. And you're going to see it very quickly. On the other hand, the teachers seem to not really be using the resources in a way that allows them to take advantage of this.

So that's where we were at the end of this. And we had a bunch more questions that we wanted to ask. What do we still need to know? So suppose we would want to go from this program to, say, well, let's do a policy for all of India. What else do we need to know before we move further? Yeah?

AUDIENCE: The thing, for me, is to understand what is the relation between attendance and how much those kids, those programs are actually effective? Because, I mean, you showed that enrollment, as a whole, doesn't have too much of an influence. But maybe attendance would help.

PROFESSOR: Right. So we could say, well, maybe there are other things to look at, children's attendance, teacher's attendance. Maybe the big difference with the balsakhi is that they were there. Maybe that's just it. Can we change their learning just with incentives?

AUDIENCE: Also, maybe look at the curriculum. Maybe the reason why the children are learning is because the material is more basic, whereas if you just stay in the classroom, the material is already too hard, and you're not able to do it.

PROFESSOR: That's the other big contender. So you have both contenders. One is, OK, the balsakhi are actually there. The kids are going. That's why they are learning. The second is that's not-- the incentive to this is really the pedagogy, which is about the learning material. And if we could train the teachers to train at that level, we would have the same effects.

AUDIENCE: The cost effectiveness of the program, [INAUDIBLE].

PROFESSOR: Right, so we could say, what's the cost effectiveness in the city? We could look at the cost effectiveness elsewhere.

AUDIENCE: Also, it could be interesting to see the effects of having more strict, I guess, passing policy. Because a lot of kids are advancing to the next grade without actually being prepared. And maybe making that a little bit more strict, so that they don't go to next level until they're actually prepared. That could improve.

PROFESSOR: Yeah. It's a very good point. In fact, the policy is not strict at all at the moment. Rukmini says it the short video, everybody passes, regardless. So maybe having something where, actually, repeating is allowed or remedial education of some kind is provided would help.

The other question one might ask is, is it only an urban phenomenon or would it also work in rural schools? Do you need to pay the people or could you have volunteers? Would it be sufficient to distribute materials, if it's a material question?

Would you be able to motivate the teachers to do it, to focus on the students? Can you concentrate even more on the basics and make even more progress on the basics, by not trying to do remedial of everything, but just focusing on learning?

So those were the questions that Pratham had. Those were the questions we had. Another question you could ask is, on the basics, you could ask the opposite question.

You were saying that the students at the top, it's not that they are exactly ready to come to MIT. They are still way below grade level. Is there something that can be done for them? Or does this very simple pedagogy that Pratham has work only for to very low achieving kids?

So we had all these questions. But it was many years ago. We had a lot of time. So we started looking at them. So the first thing we did was a new evaluation in Jaunpur, in Uttar Pradesh.

In the first program, Pratham was running this program. And they called us to evaluate it. And we went and evaluated what it is they were doing. But with the Jaunpur program, we were working together.

So this study is, actually, authored by Rukmini Banerji, along with the rest of us. We were now all working together to try to figure out, can we learn more what's going on?

So Pratham renamed the balsakhi program as the Read India Program. And as they renamed it, they shifted to focus it on something even more basic and simple, on reading, and tried to spread out much more everywhere, not focusing on cities but also working in rural areas and working on a much larger scale.

So we worked in rural Uttar Pradesh. We had three groups of villages, here, in Jaunpur. One group, where we just went to see all the parents, had village meetings, the women and Pratham.

We went to see the parents and say, you know what? There are things you can do. You can advocate. We can lobby for more resources for your schools. You can get an extra teacher that the government will have to pay for. You can get scholarships, et cetera. That's the first treatment, to see whether parents would be able to engage with the system.

The second thing we had is we realized that parents were overestimating how much their kids knew. So a first step, that Pratham has discovered, to get people excited about reading, is to train parents to administer the small Pratham test. So you go and say, Ben, read this. And Ben is like staring blankly. And then they realize, oh, my god, the kids can't read.

And they have been giving their kids to the schools, faithfully, for years, and assuming that something would come out of it. And then they realize in this exercise that actually, not really. So the parents prepare a report card for the village. And then there was a lot of discussion.

And the third was the Read India volunteer. At the end of this process, Pratham asked anybody, are their volunteers to learn Read India? In this case, they were not paid. People just came up, boys and girls, usually youngish boys and girls. They got trained by Pratham and stared running these camps for the students, Read India camps.

And what did we find? That's kind of the result in one graph, kids who, at baseline, could not read anything. And this their result at end-line. These three lines look the same, control, information only, information plus test provided no significant difference.

But there's a somewhat bigger jump for the Read India program. That jump, you might think, is really not that big. So first, I have to tell you, it's statistically significant. This is different than that.

But second is, this jump is only due to 13% of kids who actually showed up. So only 13% of kids who couldn't read went to the reading camp. So we can do the same exercise, the same Josh Angrist exercise that we did for Bombay or we did for looking at the effect of the balsakhi, to look at the effect of Read India.

So we divide this little bar by 13%. What is it going to do to my little bar. It's going to make it look much bigger. So this is what we have here. And that's what we find.

So these are kids who couldn't read letters at baseline. And now I'm adding to the control group the little bar that have now become bigger. And we get to exactly 100%. So this suggests that 100% of the kids who actually attended the camp are able to read letters at end-line.

So this is a program which is trying to get the kids to read. And it doesn't get them to read full paragraphs, but it put them one level up. And you can do the same exercise. Kids who were at letter levels are able to read paragraphs and kids who were at paragraph level are able to read stories.

So this is a program that, as a program, is tremendously effective. But there were still some issues, which is that very few kids attended the camp. Why did only 13% of kids attend the camp? And that is something that was a puzzle for us.

Next time, we'll spend much more time on why did only 13% of kids come to the camp. Therefore, the overall effect was low. Next time, we'll try to understand that.

But today, I'll tell you what happened when we saw this result, which is we saw an effective pedagogy, where, if I managed to grab Ben by the collar and try to inculcate him how to read with Pratham technique, I can do it. He can learn letters.

But on the other hand, if it's left to volunteers and left to the effort of the parents, we only managed to get 13% of the kids. So the next step was is it possible to integrate this within the school system? Because enrollment is already high. Students are already a captive audience in the school. If we could use the teachers to do this, then that would work better.

So off we went. Pratham got a bunch of money, by the Gates and Hewlett Foundations, to expand the program in about 100 districts. That's how they reach 38 million kids today. But they felt that they should try and work through the states.

And so here the trade-off was, well, if it does work, if we can make it work with regular teachers, then we will have many more kids reached. So that's the advantage. The 13% will become 80%.

On the other hand, the treatment effect might go from this very large treatment effect we have to a much lower treatment effect if the teachers are not willing or able to carry out the program. So that's the question.

So the next experiment, we worked in two states, Bihar and Uttarakhand. In the meantime, UP got cut in two. And Uttarakhand is one of the big, mountainous states, a former part of UP, very beautiful place with mountains and stuff.

So in Bihar, we contrasted four models in different state, a summer camp taught by government teachers, trained teachers to implement Pratham as part of their regular teaching, trained volunteers roughly like the Jaunpur model, and distribute only the material. Can you just distribute the material, the textbooks, and get an effect from that? That would be the best, because that's very cheap.

In Uttarakhand, we tried this same teacher training as in Bihar and the volunteers. But the volunteers were put in the school, with the idea that, maybe, in this way, the kids would come. And what did we find?

What we found is that, again, when you have the volunteer intervention, that again works extremely well, with effects similar to what we had in Jaunpur. So what we found in Jaunpur, we find the same thing in Bihar. It worked in the same way, with not very many kids going, but the kids who went really benefited.

And what is interesting here is that it's not only basic reading level any more. It also covers more advanced skills. And the gains were felt at all levels. So the same approach of trying to teach what the kids don't know, it works at reading, but it also works at more advanced levels.

So great. We had a big effect on learning. We were happy. However, the teacher intervention had very, very little effect. So training the teachers had very, very little effect. There is one effect on Hindi written tests. So if you squint, you find a little bit of an effect of the teachers but, basically, not much.

And moreover, when you put the volunteers in the schools, the volunteers had no effect. In fact, what we found is when you put the volunteers in schools, the teacher is absent more. So they just stay home more. And the volunteers start teaching the kids. And then it has no impact.

So what do we conclude? Is it that the teachers are just horrible, and they can absolutely, never teach anything to the kids? Well, we don't think that's the case, because of the summer camp. In the summer camp program, it's the teachers who were trained to teach. A teacher had to volunteer, and they were paid extra to teach summer camps.

But they were still teachers. And they were trained. And they did it. And the summer camp effect was as large as the volunteer treatment. It means that teachers are able to teach kids to read if they want to. So the problem is that it's not that they can't. It's that usually they choose not to.

And why do they choose not to? That's what we are going to see next time. Where are we? We know that we can improve the quality of education. Because you can take a high school graduate, train them for two weeks, and they can do it.

And the puzzle is that, why is it not taken up more? Why is it not taken up more by the school system? When the teacher's are trained, it doesn't work. Why is not taken more by teachers themselves? Why is it not taken more by parents, who are not sending their kids to the reading camp when they have a chance? I mean some do but not that many.

And finally, they are private schools. A lot of these kids are going to private schools. The private schools are somewhat more effective than the public schools but not tremendously more. So why aren't the private schools using those techniques?

After all, nothing stops them. They can fire the teacher if they don't do what they do. They are more flexible. So why aren't the private school adopting that?

So that's kind of where we are, which is there is an effective methodology, of very effective, you know. You can get everybody to read at least one more level, from where they were, in three months of a volunteer. We can increase their skills by one standard deviation.

And yet this is not taken up. So this is not the technology. The technology exists. But it's not adopted, neither at the individual level nor at the system level. So we'll see where we are next time. And we'll conclude with you.

AUDIENCE: How would you [INAUDIBLE]?

PROFESSOR: Teachers are very well educated. Teachers have a BA in teaching plus a discipline. And it's a pretty competitive job to get. And it's very well paid. But it's very well paid at entry, and then you don't increase much. And you can never, ever be fired. So it's a very desirable job.