Learning and Educational Achievements in Pakistan Schools (LEAPS) Program

The LEAPS project is a large-scale longitudinal study exploring how to improve learning outcomes in Pakistan. It is aimed at exploring the educational landscape of Pakistan and providing a framework for an evidence-based debate about education performance and policy in Pakistan. LEAPS is experimenting with offering schools innovative financial products that can help them meet the needs of their students and evaluating if it has an impact on student test scores.

The LEAPS team has been pursuing transformational research with the aim of improving educational attainment among Pakistan’s children. In this regard, the team has identified and is working on four areas with system-level failure, which include: (1) labor market failures, (2) information asymmetries, (3) lack of access to finance and (4) innovation failures.

In the context of improving educational quality, the LEAPS project expands the research focus beyond teacher training and textbooks, taking a more comprehensive approach of examining the education ecosystem, especially the demand and supply side barriers impeding educational attainment for children in Pakistan. A large-scale data collection exercise was carried out, spanning across 112 villages throughout Punjab and data was collected on 850 schools, 12,000 children, 5500 teachers and 800 head teachers. CERP oversees the implementation of randomized controlled trials under the LEAPS project in Pakistan. The project’s research agenda has now been expanded to include experiments focused on catalyzing innovation in education to achieve improved learning outcomes.

The LEAPS team is seeking ways to improve educational outcomes by testing the impact of:

• Providing all players in a closed educational ecosystem equal access to information regarding the quality of education provided by schools.
• Giving schools access to the financial resources necessary to enable quality enhancements.
• Addressing knowledge and innovation failures by facilitating the growth of a robust market for educational support services.
• Alleviating frictions in the labor market in order to support the recruitment, training, and retention of the highest performing teachers in both public and private school settings.

We address our questions with three large scale randomized controlled trials.

1. Increasing Funds to Public Schools:

The first evaluates the effects of a government reform that led to a large increase in the funds available to locally run public school management committees. Work on this is ongoing.

2. Unconditional Cash Grants to Low Cost Private Schools:

Our second study examines the impact of unconditional cash grants to low cost private schools (LCPS). In collaboration with the Aman Foundation and Tameer MicroFinance Bank (TMFB), we offered cash grants worth roughly 500 USD each to 855 schools in rural Punjab. This grant value was equivalent to about two months’ worth of revenues and half a year worth of profits for the median school. The grants were offered in two forms: “low saturation”, where only one (randomly selected) school in a selected village was offered the grant, and “high saturation,” where all LCPS in the village were offered the grants. We find that when all schools in a village receive financing (as opposed to just one school) they begin to invest in quality of education rather than fixed investments, to remain competitive in the village’s educational marketplace.

3. Financial Products for Low Cost Private Schools:

For our third study in education financing, we partnered with Telenor Bank (previously Tameer Microfinance Bank) to design and evaluate promising financial products for low cost private schools, ultimately the only sustainable way to make financial resources accessible to schools at scale in Pakistan. We piloted a loan product and a micro-equity product, both specifically tailored to the needs of low cost schools. We aim to examine the effect of school financing on test scores, enrolment and financial metrics. This study is ongoing. Key outcomes to be examined include increased investment, increased enrolment, hiring of additional teachers and staff and/or improvements in quality of education.

Findings
LEAPS is a longitudinal large-scale study, possibly the first of its kind in Pakistan, that tracks 10,000 students over 15 years. It seeks to look at the impact of schooling on early adult labour force outcomes, occupational choice, and family formation stratified by gender and access to schooling heterogeneity by tracking a large sample of individuals from childhood into young adulthood in the context of a low-income country.

Tests of over 22,000 children in grades 3, 4, and 5 (ages 8–10) in Urdu, English, and Mathematics collected between 2003 and 2007 identified substantial deficits. LEAPS surveys in rural Punjab showed that by the end of grade 3, many children had not mastered grade 1 curriculum, and the majority had not mastered grade 2 curriculum. 

• There is a learning crisis: Even as per-child spending has increased, test scores remain low.
• Distance matters: Distance is a key factor that impacts school attendance, particularly for girls.
• Low-cost private schools are pervasive: Today, 42% of all children in Pakistan are enrolled in a private school.
• Education is an active marketplace: In a typical village in Punjab, parents can choose between 7–8 schools.
• Debunking the madrassa myth: Religious education is small and not growing.

Learning Levels

• While research shows that private schools have consistently out-performed public schools in terms of student test scores, absolute learning levels in Pakistan remain shockingly low.
• When data on third grade test scores in Urdu, English, and Math from the LEAPS study in 2008 is compared to the nationwide ASER survey conducted in 2016, it becomes evident that student performance has not budged in nearly a decade. Both tests indicate that a student who drops out in Grade 3 is very likely to be functionally illiterate.
• 73% Grade 3 students cannot read a simple sentence. 78% Grade 3 students cannot do a simple arithmetic sum.

Access to Finance and Modality of Distribution

We conducted an experiment in which we offered 342 private schools in 189 villages an unconditional grant of $500 to explore whether schools react differently when other schools also receive the grant. We gave the grant to only one school in some villages (one-financed/ low saturation) while in other villages we gave grants to all private schools (all-financed/ high saturation).

• The method of financing matters just as much as the grant money itself. The all-financed model leads to larger total social returns for the entire village compared to the one financed model, where school owners benefit most. 
• All-financed schools were more likely to pay their teachers more and spent on a variety of things while one-financed schools spent more on fixed items and were less likely to increase teacher pay.
• While enrolment (thus revenue) increased by 12% in one-financed villages as opposed to 5% in all-financed schools, tuition fee and test scores only increased in all-financed villages.
• Tuition increased in all-financed schools because schools signalled to parents that their quality had improved and were able to charge on average 7.9% higher tuition fees the next year.
• Revenue and quality of education both increase when all schools have more resources
• Intensity of financing in a given village matters – revenue and quality both increase when all private schools receive grants as opposed to only one private school

Information Asymmetry

LEAPS conducted a randomised controlled trial in 112 villages across Punjab province. Students were tested and village families were provided report cards with information about their child’s test scores as well as a comparison of the average test scores in the school their child attends versus those in all the different schools in their village. Our groundbreaking report cards study has proven to be one of the most cost-effective and impactful interventions according to the “Smart Buys” report.

Village education markets are a very tight knit community. Information flows fast and competitive parents demand better quality. Even public-school outcomes improve.

Providing report cards to parents improves average test scores in all schools. Families make the majority of a child’s schooling choices. The composition and dynamics of a household can have large impacts on educational choices.

Treatment villages saw average student test scores increase by about 42% (0.11 Standard Deviations) more than control villages at a cost of $1 per child. Even public schools improved due to parental pressure.

Easily understandable Information about school quality is a cost-effective and sustainable way of improving quality through parental involvement.

Teacher Pay and Performance

We know that teacher quality matters to student learning and that teachers should ideally be compensated on the basis of their ‘value-add’. We analysed a dataset on teachers in Punjab from 2003-2007 to calculate a Teacher Value Added (TVA) score – an estimate of the increase in test score points a student would receive if assigned to a particular teacher. Teacher productivity (value-add) impacts student test scores but is not currently always linked to their pay, especially in the public sector.

• Teacher quality matters – a 1 Standard Deviation increase in a teacher’s TVA score led to a 0.21 Standard Deviation increase in their students’ test scores. This implies that moving a student from a low-performing teacher to a high-performing teacher can lead to a massive gain, nearly equivalent to one full year of schooling, in learning outcomes.
• The conventional public-school teacher pay system (that pays based on qualification and number of years of teaching) does not necessarily reward the value a teacher adds.
  • Only 5% of TVA is accounted for by observable characteristics, i.e., teacher content knowledge and experience. Other characteristics such as teacher training or having a bachelor’s degree do not impact TVA significantly.
  • Similarly, lower wages do not necessarily lead to poor teaching. This is evident from the performance of contract teachers (paid 35% less than permanent teachers) hired in the public sector, and from data that shows that private school teachers, who are paid significantly less (about one-fourth) than their public-school peers, perform better.

Past Findings

Labour Market Failures:

• The adequate supply of quality teachers is a critical input for high-quality education.

Since teachers in low-cost private schools are almost entirely women, as a result in villages with secondary schools for girls there are twice as many educated women and private schools are three times as likely to open.

• We find that student test scores improve by 0.64 standard deviations when teacher quality improves. We also find that there is no correlation between pay and productivity.

Information Asymmetries:

• Spending $1 per child on information can massively increase test scores
• We carried out an RCT across 112 villages where we provided treatment village families with report cards that not only gave information about their child’s test scores, but also let parents know how that school was performing compared to other schools in their village. We found:
  • Learning improved
  • The worst schools closed
  • Enrolment increased
  • School became more affordable

Lack of Access to Finance:

• The intensity of financing provided in a given village matters.
• Our team studied the effects of unconditional cash grants transferred to private schools across more than 250 villages in 850 schools in rural Pakistan. Our findings suggest that when all schools in a village receive financing (as opposed to just one school) they begin to invest in education quality rather than physical infrastructure in order to remain competitive in the village’s education marketplace.

Innovation Failures:

• While low-cost schools have access to markets for “hard” investments (such as desks and chairs), they have limited access to markets for “soft” investments (such as curriculum development and teacher training)

FIndings have been summarised from the LEAPS Brochure.

Key Findings from 2008

The fallacy of measuring by enrolment:

While enrolment in Pakistan primary schools has increased dramatically since 2000, especially for girls and the poor, LEAPS research showed that access does not guarantee sufficient learning to participate in a swiftly globalising world. Our surveys in rural Punjab showed that by the end of grade 3 (around 8 years old), many children had not mastered grade 1 curriculum (6-year-old level), and the majority had not mastered grade 2.

The rise of low cost private schools:
Secular, low cost private schools have become widespread in both urban and rural areas across Pakistan. The LEAPS survey found that by the end of 2005, one in every three enrolled children at the primary level was in a private school. LEAPS research showed that within the same village, children in private schools have significantly better learning outcomes than those in government schools. The learning gap between a child in a private school and one in a public school was 8 to 18 times wider than the gap between a child who is relatively rich and one who is relatively poor.

Distance matters:

Distance is a key factor that impacts school attendance, particularly for girls. Every additional 500 metres increase in distance to the closest school results in a large drop in enrolment. Girls living 500 metres away from a school were 15 percentage points less likely to attend than those living next door.

Education is an active marketplace:

Pakistani parents are highly motivated to invest in their children’s education. Government investment in education has also increased substantially. The new educational landscape in Pakistan – both rural and urban – is best described as an active marketplace. A typical village in the LEAPS sample has 7-8 schools for parents to choose from.

Findings Extracted from LEAPS website

Findings from original LEAPS report

The findings shed light on the relative strengths and weaknesses of private and government schooling. Driven by higher teacher salaries, government schools require twice the resources to educate a child compared to private schools. Furthermore, children studying in private schools report higher test-scores in all subjects—partly because their teachers exert greater effort. Private schooling alone, however, cannot be the solution. Access to private schools is not universal. Private schools choose to locate in richer villages and richer settlements within villages, limiting access for poor households. In contrast, a laudable feature of the government school system is that it ensures equal geographical access to schools for all. Since children who receive less attention and educational investments at home are also more likely to be enrolled in government schools (if they are enrolled at all), government school reform could ensure that no child is left behind. 

Findings have been extracted from the LEAPS Landmark World Bank Report