research report: impact study pilot literacy...

Literacy program initiated by:

Study conducted by: Oasis Insights (Pvt) Limited

Report submitted on September 23th 2013

(Revised on September 25th)

Research Report: Impact Study

Pilot Literacy Program

Research Background1

Research Objectives2

Research Design3

Methodology4

Report outline

Mode of Teaching & Sample Split5

Study Findings6

Research Background• The Dawood Foundation (TDF) in collaboration with Engro Foods Limited (EFL), The Citizen

Foundation (TCF) and Literate Pakistan Foundation (LPF) have planned to start a literacy program for children aged 12-21 who have no formal or less than primary schooling. Before launching this social initiative at large scale, a pilot literacy program was initiated in selected villages of Sahiwal district and adjoining areas.

• Pilot literacy program was executed by LPF during April 23 to Aug 22, 2013 on 14 Milk Collection Centres (MCCs) of EFL and a TCF school.

• Recruitment of participants for this pilot study was conducted by Oasis Insights, before starting execution of literacy program a baseline study of literacy level was conducted using LPF test papers.

• To gauge the impact of this social initiative a post literacy test was also conducted by Oasis Insights at completion of 3 month literacy program.

• In this report; findings of impact study conducted for this literacy program are presented.

Research ObjectiveTo gauge the impact of Pilot Literacy Program on literacy level of children aged 12 - 21

who have no formal or less than primary schooling through conducting pre and post

literacy tests.

Success Criteria (As per Client brief)

◦ 90% students pass the LPF tests ◦ Technology can be managed in the field ◦ Core operation of the location is not affected ◦ Low drop out rate (%)

• Considering research objective, a research design of conducting pre & post literacy tests among “Test” and “Control”

panels of target respondents was agreed. • Test Panel: A group of students who failed in pre program literacy test (obtained <20 marks) and participated in the

literacy program executed by facilitators of LPF. • Control Panel: A group of students who failed in pre program literacy test (obtained <20 marks) and did not participated

in the literacy program. • Control Panel was suggested to gauge the “spill over” or extended impact of study on literacy level. • Five “Combos” of education materials were tested to further gauge the most impactful “Combo”.

• T1= Test panel, Pre activity “Level of Literacy”. • T2= Test panel, Post activity “Level of Literacy”. • C1= Control panel, Pre activity “Level of Literacy” • C2= Control panel, Post activity “Level of Literacy”.

Literacy Test Test Panel Control Panel

Pre Program T1 C1

Post Program T2 C2 DT = T2 - T1

Activity ImpactDC = C2 - C1

Natural Shift

Magnitude of Impact (MI) = DT – DC ✓If MI > 0, Impact is positive ✓If MI < 0, Impact is negative ✓If MI = 0, No Impact of activity

Research Design & Impact Benchmarks

Methodology

Recruitment Survey by OASIS

(April 18th to May 5th 2013)

•Recruitment Survey conducted through door to door interviews using screening questions from all 15 selected villages for getting reported literacy level.

•Recruited those respondents who have no less than primary education.

Post Training Literacy Test

(July 23rd to Aug 22nd 2013)

After completion of training program, post assessment test was conducted for knowing

the impact of pilot activity.

Literacy Training Program by LPF

(April 23rd to Aug 22nd 2013)

Training given to selected respondents for period of three months.

Pre Assessment Literacy Test by OASIS

(April 22nd to May 8th 2013)

Test conducted and after analyzing the test scores decision was made regarding participation of respondent in training program (Test & Control panels).

Combo-B: Mobiles supported by teachers, 3 villages

• 35/4-L Sahiwal

• 112/9L Sahiwal

• 109/9L Sahiwal

Combo-E: 10 inch Tablet with Internet, only 1 village

• 31/4L Koain walay

Combo-C: 7 inch Tablets, 2 villages

• 126/15-L, TCF School Mianchannu (Excluded from analysis as per client recommendation)

• Chaah Suian Wala (near nor pur Daira Sukheray)

• 73/D Pahoran Wala (near Noor Pur east side)

Combo-D: 10 inch Tablets, 5 villages

• 140/9L near adda shabeel sahiwal to arif road

• 141/9L near Adda Shabeel sahiwal to arif road

• 96/D (Near noorpur)

• 126/9L near shree moor, to arif road

• 34/4L Thakara near skandar chowk

Mode of Teaching Pilot Literacy program was conducted in 14 selected villages

Combo-A: Books only, 3 villages

• 29/4L (Bokhian)

• 37-38/4L Suka chak

• 95/D (near Noor pur west side)

Sample SplitMode of Teaching

Pre Wave Post Wave

Test Panel Control Panel Test Panel Control Panel

Pilot A: Books (3 Villages) 137 18 59 20

Pilot B: Mobiles (3 Villages) 119 18 77 14

Pilot C: 7 inch Tablets (2 Villages) 79 17 53 10

Pilot D: 10 inch Tablets (5 Villages) 231 30 129 11

Pilot E: Internet (1 Villages) 41 8 23 6

Achieved Sample 607 91 341 61

Target Sample 450 90 375 75

Around 56% of all selected respondents for the literacy program actually participated in this program.

Students Average Attendance (As Per Client Record)

Combo-A: Books only, 3 villages

May

June July 3-

M

• 29/4L (Bokhian) 22 23 20 22

• 37-38/4L Suka chak 16 17 17 17

• 95/D (near Noor pur west side) 15 18 15 16

Combo-B: Mobiles supported by teachers, 3 villages

May

June July 3-

M

• 35/4-L Sahiwal (Females) 33 28 28 30

• 112/9L Sahiwal 19 17 15 17

• 109/9L Sahiwal 30 27 23 27

Combo-E: 10 inch Tablet with Internet, only 1 village

May

June July 3-

M

• 31/4L Koain walay 18 18 18 18

Combo-C: 7 inch Tablets, 2 villages

May

June July 3-

M

• Chaah Suian Wala (near nor pur Daira Sukheray) 24 19 19 21

• 73/D Pahoran Wala (near Noor Pur east side) 33 33 29 32

Combo-D: 10 inch Tablets, 5 villages

May

June July 3-

M

• 140/9L near adda shabeel sahiwal to arif road 20 18 16 18

• 141/9L near Adda Shabeel sahiwal to arif road 36 37 30 34

• 96/D (Near noorpur) 31 30 26 27

• 126/9L near shree moor, to arif road 25 31 24 26

• 34/4L Thakara near skandar chowk 21 25 23 23

Attendance Average for 3 months May

June July 3-

M

14 selected villages 31 30 26 29

Student Score criteria (Both Waves) Urdu Test Mathematics Test

Student Selection criteria (At pre stage student was selected if total marks obtained were less than 20) Student Passing criteria recommended by LPF (At post stage if student total marks obtained more than 25, it should be considered passed)

Research Findings

7

18

29

39

50

0

13

25

38

50

Pre Ac/vity Post Ac/vity Overall Books. Mobiles.

Test Control Magnitude of Impact

15

811

33

7

43

67

33

6

43

Mean Scores

Pre Activity Post Activity Difference Magnitude of Impact

Test Control Test Control Test ControlT1 C1 T2 C2 DT = T2 -‐ T1 DC = C2 -‐ C1 MI = DT -‐ DC

Overall 6 7 43 33 37 26 11Books 5 39 34 8Mobiles 3 44 41 157 inch Tablets 6 42 36 1010 inch Tablets 7 43 36 10

Change in Literacy Level

Impact of Literacy program is positive • Literacy program has significantly increased literacy level in Test panel vis-‐a-‐vis. control Panel at 95% confidence level.

• Teaching material of Mobile & Tablets both are significantly better than Books.

Difference = +10 30% enhancement

Difference = -‐1

Figures are Mean Scores Out of 50

Change in Literacy Level

Zero 01 -‐ 05 06 -‐ 10 11 -‐ 15 16 -‐ 20 21 -‐ 24 25 -‐ 30 31 -‐ 35 36 -‐ 40 41 -‐ 45 46 -‐ 50

64%

10%7%5%5%3%2%2%0%1%1% 0%0%0%0%0%0%10%14%14%

31%31%

Pre Literacy Test Score Post Literacy Test Score

• After attending literacy program, 91% of participants have obtained passing score 24+ out of 50. • Literacy program has also caused a positive spill over impact, 65% of control panel respondents obtained score of 24+ out of 50.

Test

Pa

nel

(n =

34

1)

Con

tro

l Pa

nel

(n =

58

)

Zero 01 -‐ 05 06 -‐ 10 11 -‐ 15 16 -‐ 20 21 -‐ 24 25 -‐ 30 31 -‐ 35 36 -‐ 40 41 -‐ 45 46 -‐ 50

31%

18%

3%3%10%12%

8%5%2%8%

0% 0%0%0%0%0%0%

19%

9%

20%

32%

20%

Activity Impact

Spill over

Percentage

Figures are percentages of respondents

91% Students Passed

65% Students Passed

Mean Score

5

11

18

24

30

0

8

15

23

30



967

17

2

24

22

17

2

24

Pre Activity Post Activity Difference Magnitude

of ImpactTest Control Test Control Test ControlT1 C1 T2 C2 DT = T2 -‐ T1 DC = C2 -‐ C1 MI = DT -‐ DC


Change in Literacy Level: Urdu

Literacy level has significantly improved in

Urdu. However magnitude of impact is

higher in students who were educated using

mobiles and tablets as education material. Difference = +7 41% enhancement

Difference = 0


Mean Score

2

7

11

16

20

0

5

10

15

20



533

17

4

19

34

17

3

19




Change in Literacy Level: Math's

Difference = +2 12% enhancement

Difference = 1

Comparative analysis of Pre and post activity shows that literacy level has significantly improved in Math's. Magnitude of impact is higher in students who had educated by Mobiles method.


Mean Score

0

13

25

38

50

0

13

25

38

50

Pre Ac/vity Post Ac/vity Overall


11

33

7

43

67

33

6

43

Pre Activity Post Activity Difference Magnitude of

ImpactTest Control Test Control Test ControlT1 C1 T2 C2 DT = T2 -‐ T1 DC = C2 -‐ C1 MI = DT -‐ DC

Overall 6 7 43 33 37 26 11Male 6 7 43 33 37 26 11Female 4 7 43 33 39 26 13

Change in Literacy Level: Gender wise

Literacy level significantly improved it in

both gender. However there is no

significant difference on gender wise but

magnitude of impact is good in female

students compare to male.

Difference = 10

Difference = 1

Mean Score

10

20

30

40

50

0

13

25

38

50

Pre Ac/vity Post Ac/vity Overall 12 -‐ 15 Years.


1211

33

7

43

67

33

6

43



Overall 6 7 43 33 37 26 1112 – 15 Years 5 7 43 33 38 26 1216 – 18 Years 7 7 43 33 36 26 1019 – 21 Years 8 7 42 33 34 26 8

Change in Literacy Level: Age wise

Comparative analysis of Pre and post activity

shows that literacy program has good impact

on the literacy level and leads to significantly

improved it across all the student’s age

group. However there is no significant

difference among students marks on age

wise but magnitude of impact is good in

student of lower age as compare to older age

group.

Difference = 10

Difference = 1

Significantly High at 95%

Significantly Lower at 95%

Conclusion✓Impact of literacy program is very positive; beside being supported by statistical indicators qualitative feedback from participants is overwhelming. ✓There is a need to invest in this type of literacy program as it can bring a real change. ✓People of target group (aged 12-‐21 with no or less than primary education) have a sense of loss that they have missed a very important thing in life therefore they look for opportunity from where they can get basic literacy level. ✓Participants had a life time experience which will impact their life positively.

✓Learning to read and write simple Urdu text doing some basic numeric calculation is a big achievement. After attending literacy program, 91% of Students have obtained passing score 24+ out of 50.

✓Spill over impact of this literacy program is very high; indicating an interest generated in peer group and participants sharing their knowledge with other friends not participating in the program. Spill over impact, 65% of control panel students obtained score of 24+ out of 50.

✓Teaching material of Mobile & Tablets both are significantly better than “Books only”. ✓Based on analysis we can rank the teaching material as below:

1st – Mobiles (Books of LPF Adult Literacy + Use mobile phones issued to each student to facilitate learning) 2nd – Tablets (Books of LPF Adult Literacy + Use tablets which have content and let group of students to use it) 3rd – Books only (Books of LPF Adult Literacy )

✓Magnitude of impact is better in lower age group (12 – 15 years) and female students; indicating a potential of improvement in literacy

Recommendations

✓It is recommended that focus of literacy program should be on aged 12-‐15, focus and priority of people 16-‐21 change towards earning a livelihood.

✓Girls are more receptive to this type of literacy program, there is a need to design and execute literacy program for girls only (co-‐education of target age is not a workable idea in underprivileged / rural population).

✓Teaching through “Books Only” is not as impactful as a combo of electronic educational devices such as mobiles tablets and computers.

✓Using electronic devices (like Mobile, Laptop or Tablet) along with books to teach reading and writing skills is more beneficial as compared to books only. Most important is the learning to use these devices which can help them access lot more knowledge through mobile networks / Internet.

✓Exploring Internet as literacy enhancement program is an exciting idea but we do not have sufficient data about its impact; therefore it is recommended that before embarking on this “Combo” a special pilot study is conducted.

Appendix

Marks Obtained: Different Parameters

Marks Obtained (Internet Questions)

0

6

13

19

25

Total Marks Obtained Search picture of famous place. Search picture of famous animal / bird.

0333

5

15

Total Marks 25 (Mean Score)Q

uest

ion

wis

e

* Very low base for analysis

2

4

6

8

10

0

2

5

7

9


Test Panel Control Panel Magnitude of Impact

322

7

2

9

22

7

2

9

Reading & Writing of Urdu Alphabet:

Reading and Writing of Urdu alphabet

After e-‐education program score of students

in reading and writing of Urdu alphabet has

significantly increased in test & control

panel across all type of teaching material.




Difference = 2

Difference = 0


0

3

5

8

10

0

3

5

8

10



3

12

5

0

7

00

5

0

7

Reading & Writing of Small Urdu Words




Reading & Writing of Urdu Words After program score of reading and writing of Urdu words has significantly better in test and control panel. Teaching material Mobile and Tablets has significantly obtained better marks than books at 95% LOC.

Difference = 2

Difference = 0


2

4

6

8

10

0

3

5

8

10

Pre AcTvity Post AcTvity Overall Books. Mobiles.


32

3

5

0

8

00

5

0

8

Reading & Writing of Urdu Paragraph




Reading & Writing of Urdu Paragraph Reading and writing of Urdu paragraph has significantly increased in both panels. However teaching material Tablets has significantly obtained better marks than Books.

Difference = 3

Difference = 0


0

3

5

8

10

0

3

5

8

10



4

11

9

4

9

34

9

3

9

Reading & Writing of Numeric Numbers




After literacy program score of students in reading and writing of math’s numbers has significantly increased in test panel & control panel.

Difference = 0

Difference = 1


0

3

5

8

10

0

3

5

8

10



222

8

1

9

01

8

0

9

Adding & Subtracting



Overall 0 1 9 8 9 7 2Books 0 1 9 8 9 7 2Mobiles 0 1 9 8 9 7 27 inch Tablets 1 1 9 8 8 7 110 inch Tablets 0 1 9 8 9 7 2

Marks obtained out of 10 (Mean Score)

There is no significantly difference among

different types of teaching material used in

literacy program at this parameter.

Difference = 1

Difference = 1



Marks Obtained: Question wise

0

1

3

4

5

0

1

3

4

5



222

4

2

5

1

2

4

1

5

Can Read Urdu Alphabet:





After literacy program score of students in

reading Urdu alphabet has significantly

improved in test panel & control panel

and across all type of teaching material.

Difference = 1

Difference = 1

0

1

3

4

5

0

1

3

4

5



3

1

2

3

1

5

11

3

1

5

Can Write Urdu Alphabet:





Writing Urdu alphabet has significantly

increased in test panel & control panel

across all type of teaching material.

Difference = 2

Difference = 0

0

1

3

4

5

0

1

3

4

5



2

1

22

0

4

00

2

0

4

Can Read Small Urdu Words:






reading small Urdu words has significantly

improved in test panel, control panel and


Difference = 2

Difference = 0

0

1

3

4

5

0

1

3

4

5



2

1

22

0

4

00

2

0

4

Can Write Small Urdu Words:





Teaching material Mobile & Tablets has

significantly obtained better marks than

Books.

Difference = 2

Difference = 0

0

1

3

4

5

0

1

3

4

5



2

1

22

0

4

00

2

0

4

Can Read a Urdu Paragraph:





Reading Urdu paragraph has significantly

improved in test panel & control panel.

There is no significantly difference among

different types of teaching material .

Difference = 2

Difference = 0

0

1

3

4

5

0

1

3

4

5



2

1

22

0

4

00

2

0

4

Can Write a Urdu Paragraph:






writing Urdu paragraph has significantly

improved across all type of teaching

material. Difference = 2

Difference = 0

0

1

3

4

5

0

1

3

4

5



2

11

5

3

5

2

3

5

2

5

Can Read Numbers:





Math’s reading numbers has significantly

improved across all type of teaching

material.

Difference = 0

Difference = 1



0

1

3

4

5

0

1

3

4

5



2

0

1

4

1

5

11

4

1

5

Can Write Numbers:





Across all type of teaching material. there

is no significantly difference in writing

numbers has significantly improved

Difference = 1

Difference = 0

0

1

3

4

5

0

1

3

4

5



2

1

2

4

1

5

0

1

4

0

5

Can Add Numbers:






adding of numbers has significantly

increased in test & control panel. There is

no significantly difference among different

types of teaching material used in literacy

program.

Difference = 1

Difference = 1

0

1

3

4

5

0

1

3

4

5



1

0

1

4

0

5

00

4

0

5

Can Subtract Numbers:





After literacy program score of students

in subtracting numbers has significantly

improved in test panel, control panel


Difference = 1

Difference = 0

39

Thank You

research report: impact study pilot literacy...

Documents