RETHINKING EDUCATION

INNOVATIONS, POLICY, & PRACTICE

Aditi Bhutoria
Email: [email protected]
Twitter: @AditiBhutoria
LinkedIn: Aditi K. Bhutoria

Indian Institute of Management Calcutta

Term IV 2020
Example:
SPDI for the Education Sector
Example taken from: Andrabi, Tahir, Jishnu Das, and Asim Ijaz Khwaja. 2017.

"Report Cards: The Impact of Providing School and Child Test Scores on Educational
Markets." American Economic Review, 107 (6): 1535-63.

Problem Statement:

“Pakistani schools are of low quality, on average, as evidenced by low learning

levels and growth”

Scope: In a 2016 standardized test of Grade 3 students in rural Pakistan:

• 71.7% couldn’t do 2-digit subtraction
• 85.3% couldn’t read a sentence in English
• 58.4% couldn’t read a sentence in Urdu/Pashto/Sindhi

Target Population:
• Villages in Punjab province with at least one private school and one public
school
• Exclude: Villages w/ no private schools (increasingly less salient)

Source: https://openknowledge.worldbank.org/handle/10986/29065
Pitfalls in Step 1: Identifying Problem
Avoid these!
• Problem statement that is too broad
• Ex. “The education system in Pakistan has many
problems.”

• Solutions in search of a problem

• Ex. “Schools in Pakistan are low quality because they
don’t have enough computers.”
2. Diagnose Causes
2. Diagnose Causes
• Underlying causes

• Demand Side: Parents/students have limited information about school quality

• Observing quality in education is difficult since schooling only pays off
years later
• Although parents seem to know something about quality, their estimates
are noisy and imprecise

• Supply Side: Political Economy such that it leads to weak incentives for
teachers in public sector schools
• Data indicate performance is not linked to pay or job security
• Public schools pay more competent teachers less
• Higher absenteeism among public school teachers
• Politicians feel Little political pressure to improve teachers; instead get
more political rents by posting teachers to desirable places
SPDI Diagnosis Tree -- Example
3. Design (I)
EDUCATION REPORT CARDS

• Provide Child Information

• Scores and rank in math,
Urdu & English
• Comparison with average
child in their school and in
their village
• Provide Village Schools
Information
• Name, number of tested
children, scores, and rank
of each school in village
3. Design (II)
Program
Underlying Feature Theory of Change (In words or using
(Addressing
Cause arrows)
underlying
cause)
Child report card à Parents have better
information about whether their children are
Child report card learning in school à Parents (i) work with
children and (ii) put pressure on schools to help
their child
Lack of
Information Village schools report card à Parents have
(Parents & better information about the quality of their
students) child’s school relative to other options à Parents
Village schools
can (i) switch to better school (ii) pressure
report card
current school à Greater competition/pressure
leads to schools increasing quality and/or
dropping fees
Implementation:
Who are the relevant actors and what are their
incentives?

Decision
Government Planners Implementer Beneficiaries
Makers
4. Implementation
Actors Roles Incentives
(Which (What are the (Is the program incentive compatible
actors will actors’ roles?) for all actors and for people excluded
be or harmed?)
involved?)
Parents invest significantly in their
Attend meetings to children’s education and this information
Parents
receive report cards helps them better measure return on their
investment
High quality private schools want to signal
Private quality and if low quality private schools
Consent to testing
schools opt-out, they reveal themselves as low
quality
Public Work with government to get permissions
Consent to testing
schools & cooperation
Education Ed dept interested in testing results &
Support project
Dept impact of score dissemination
4. Implementation: Process Monitoring:
continuous monitoring
Intermediate Outcomes Final Outcome
Program TOC Stage 1 TOC Stage 2 TOC Final Stage
Feature
Child report card Parents receive report Parents have greater Parents pressure school
cards information about school to improve quality or
quality lower cost
Indicator Meeting attendance Meeting discussion & Test scores and fees
parent surveys
Timeline After summer break During meeting Next spring
Who receives Field & Research team Field team Research team
the data?
Actions taken Calculate scores and rank Explain cards in a neutral Analysis to determine
accurately; monitor manner; redesign/re-think impact, Refinements
attendance & improve as if understanding poor
needed
5. Test: Data and Analytics: Impact Evaluation
• Work in 112 villages (randomly selected)

• Use Randomized Controlled Trial:

• Half the villages (56) – the treatment group - randomly selected to
received report cards (no change in other half - comparison group)
• Assumption: Balance between T and C groups (concern given small
sample)
• Feasibility:
• Admin – tested all schools; data available and delivery done by
existing team
• Polit – permission by government, private schools buy-in
• Fin – cost-effective delivery at schools
6. Refine
• Group responsible:
• Research team

• Action Set:
• Delay/Change project design
• Identify potential for future work as revealed by evaluation
results
• Build policy/scale-up support

• Next Areas of focus:

• Finance
• Educational support services
Vision Action
} Become a Learning } Start from the First
Organisation Principles
} Think Like a Last Mover } Find the Monopolistic
} Find the Monopolistic Advantage
Advantage } Think Like a Last Mover
} Start from the First } Become a Learning
Principles Organisation
QUALITY OF EDUCATION IN INDIA
PROBLEM & SOLUTIONS – PART I
 Basic Learning Improvement Program
Scale up in Bihar, India
Pratham-Bihar Government Partnership
2012-2013 and 2013-14
Past History of PRATHAM & Bihar Govt
Collaborations
Sankalp Program:
• Pratham and Bihar Government had worked together from 2007 to 2010
• A multi-year collaboration for bringing children to school
• Led to impressive decline in the numbers of children out of school
Other initiatives:
Adult literacy missions, ICDS – anganwadi/pre-school education, and training of community
volunteers of educationally backward areas.

Pratham materials have been used in mass education programs, in school programs and school
libraries as well as in pre-school centres (anganwadis).

Over the years, ASER reports have been discussed seriously at different levels of Bihar Government

As enrolment in Bihar had risen to over 95%, concern with attendance and learning
started increasing in government and in the media. Since 2011 there were
discussions between Pratham & Bihar Government about what could be done.
The PRATHAM-JEHANABAD experience

https://www.youtube.com/watch?v=wKHUW_CAxVM
ACTIVE PRATHAM
PARTNERSHIP IN
12 DISTRICTS in
BIHAR
Total districts in Bihar = 38

Total districts in which Pratham is

partnering = 12

All government elementary schools in

East Champaran, Darbhanga, Supaul,
Katihar, Rohtas, Gaya, Nalanda,
Kishanganj, Purnia, Jehanabad, Jamui.
In Patna district (all urban blocks + 2
rural blocks) and Munger (urban).

In the districts where Pratham is actively

partnering:

Total number of schools in these 12

districts = 24131
Total number of CRCCs trained = 1676
Total number of teachers trained =
49342
Total number of children reached =
1.64 million
Exercise

9 groups of 5 people each

Choose a Moderator (those who haven’t been
moderators before)

Break Out Rooms 1-2: Topic 1

Break Out Rooms 3-4: Topic 2
Break Out Rooms 5-6: Topic 3
Break-Out Rooms 7-9: Topic 4
Topic 1: Learning Intervention:

What were the key features of the Pratham-

Jehanabad Learning Intervention?
List at least five features.
KEY ELEMENTS OF LEARNING INTERVENTION
§ CLEAR GOALS: Clear learning goals for Std 3, 4 and 5. By the end of
the school year, children should be reading basic text fluently and be
able to do the basic arithmetic operations. Periodic reviews of progress
at district & block level.

§ DESIGNATED TIME: 1.5 to 2 hours after midday meal put aside daily
for this work.

§ SIMPLE ASSESSMENT: Children assessed one-on-one using simple

tools. Based on their level, children placed in groups. This grouping is
not on the basis of grade but by level. Progress was reviewed using
similar tools periodically.

§ GROUPING BY LEVEL FOR INSTRUCTION: Each school had at least

3 groups into with children from Std 3, 4 and 5 were assigned. One
teacher assigned to each group. The number of groups depended on
the number of teachers available.

§ APPROPRIATE ACTIVITIES & MATERIALS: Activities and materials

appropriate for each group/ level was used. A child could move to the
next group if she or he was ready.
MATERIALS
For
each
child
For teachers
and for the
groups of
children

Instruction Small booklet of

manual for short simple stories
teachers in large font

Child’s
own
phonetic
Phonetic chart chart &
(barahkhadi) & number
number chart chart
READING IS ASSESSED USING THE ASER
READING TOOL Assessment of each child
enrolled in Grade 3, 4 and 5 is
done using these basic set of
reading tasks. Each child is
marked at the highest level the
child can read.

The “story level” is text at Grade

2 level of difficulty. “Para level”
is text at Grade 1 level of
difficulty.

Teachers in the school assess

the children. The Cluster
Coordinator guides and
monitors them.

The assessment is used to

group children for instruction.
Children in Grade 3-5 were
grouped by level rather than by
grade for the “special class”
during the school day.
BASIC ARITHMETIC IS ASSESSED USING
A SIMPLE TOOL

Assessment of each child

enrolled in Std 3, 4 and 5 is
done using these basic set of
arithmetic tasks. Each child is
marked at the highest level the
child can reach.

The “subtraction level” is the

level expected of children in Std
2 (Includes knowledge of
numbers till 100, place value
and ability to do basic
operations.)

Teachers in the school assess

the children. The Cluster
Coordinator guides and
monitors them.
Topic 2: Training the Teachers

Which stakeholders were involved by Pratham for

training teachers? Discuss 2-3 advantages and
challenges associated with Pratham’s teacher
training policy.
CREATING A CADRE WITHIN THE SCHOOL SYSTEM
TO LEAD LEARNING IMPROVEMENT
• A CRCC (Cluster Resource Centre Coordinator) is a government functionary who is responsible for
10-15 schools. It was decided that this should be the cadre who will lead the learning
improvement initiative.

• Each CRCC (Cluster Resource Centre Coordinator) was trained for 4 days by Pratham teams. After
their 4 day training, the CRCCs conducted daily “practice classes” for 15-20 days before the
implementation rolled out in their districts.

• 2-3 CRCCs worked together in one school for the practice class.

• All children in Grade 3, 4 and 5 were assessed using simple reading and math tools.

• Based on this assessment children were grouped by levels rather than by grade. In each school
there were at least 3 groups.

• For the purposes of conducting the 15-20 day “practice class” for 2 hours a day. 2-3 CRCCs
worked together in one school with each CRCC responsible for one group of children in that
school.

• Challenges: https://www.youtube.com/watch?v=J-laQ7FOdeY (8.07)

PATHWAYS: ROLL OUT/SCALE UP OF INTERVENTION

District & Block Level

Learning improvement classes begin in school.
Orientation
CRCCs and Pratham team members visit schools,
provide demonstration and support. CRCC expected
CRCC Training by to spend 20 days in the field.
Pratham Periodic review of progress at different levels.
4 days
\
CRCC practice classes
15-20 days
Children assessed in reading and arithmetic by
Monitoring and support by teachers
Pratham Groups formed by level instead of by grade.
Instructional time - about 2 hours a day
Teachers assigned to specific groups,
Head teachers’ orientation Materials distributed to children in school
1 day

Teaching-learning materials Teachers trained by CRCCs. 4 days Assisted by

Pratham
printed
Topic 3: Impact

Discuss 1-2 methodological benefits and

limitations of each of the 3 evaluation studies
undertaken to measure the impact of the
Pratham–Jehanabad Program?
Data and Evaluation:
• Baseline and endline program data from schools
• Pre-post comparison but what about other changes that may have
happened in between that influenced learning?
• Data from a one-time study of student achievement
• Don’t know if they increased or decreased; can’t comment on the
program effectiveness
• Comparison of ASER data from 2013 and 2014
• Treatment and control groups + Pre-Post
• For the 12 districts, 37.8% of children who were in Std IV in 2013 could
do subtraction problems. This figure rose to 53% by the same time
next year when these children had reached Std V. Similar figures in the
other districts were 38.4% (Std IV in 2013) to 49% (Std V in 2014).
• We are not sure appropriate is the choice of control group and which
factors are different across them? – RCT would’ve addressed this
problem
Topic 4: What worked?

Discuss 3-4 factors that led to a reasonably

successful implementation of the Pratham-
Jehnabad Model. Which enabling factor do you
think was most important?
Way forward
There have been important learnings from the work of 2013-14:

§ Evidence from 2013-14 shows that substantial improvement in

children’s learning is possible even in a short period of time if there is
focussed attention and action devoted to the problem.

§ CRCCs are able to lead the entire effort but they need to be supported to
do this kind of academic work as opposed to many other administrative
tasks.

§ In 2013-14 there were many delays in getting implementation work

moving at the district level. There were delays in teaching training and in
printing of teaching-learning materials. Such delays can be avoided in
the future.

§ Several studies of student achievement have been carried out in 2013-

2014 in the state. These should help in planning way forward.

§ Strong leadership by DM and clear priority for improving basic learning for
children needs to continue and sustain at the state level. This greatly
helps district level action. A multi-year vision for learning improvement is
also needed.
Thank You!