national assessment findings (grade 4 & 8) national education assessment system (neas) ministry...

National Assessment Findings National Assessment Findings (Grade 4 & 8)(Grade 4 & 8)

National Education Assessment System (NEAS)National Education Assessment System (NEAS)

Ministry of Education (Pakistan)Ministry of Education (Pakistan)

South Asia Regional Conferenceon

Quality Education for All

(October 24, 2007)

22

Vision & Mission StatementVision & Mission Statement

VisionVision

To provide fair and valid Assessments to promote To provide fair and valid Assessments to promote

the quality of children's learning in Pakistanthe quality of children's learning in Pakistan

MissionMissionTo carry out fair and valid National Assessments To carry out fair and valid National Assessments

along with related activities with the overall along with related activities with the overall

objectives of enhancing quality, equity & access to objectives of enhancing quality, equity & access to

educationeducation

33

Purpose of National Assessment Purpose of National Assessment in Pakistanin Pakistan

Monitoring the performance of the education systemMonitoring the performance of the education system

Providing feed back to improve the quality of services in Providing feed back to improve the quality of services in education system (curriculum, textual materials, education system (curriculum, textual materials, teachers’ training and delivery)teachers’ training and delivery)

Setting and disseminating realistic standards of students Setting and disseminating realistic standards of students achievementachievement

Identifying correlates of achievementIdentifying correlates of achievement

Providing information to policy makers for effective Providing information to policy makers for effective interventionsinterventions

3

44

National Assessment ScheduleNational Assessment Schedule

LevelLevel : : Grade 4 and 8Grade 4 and 8Subjects: Languages, Maths, General Science and Social StudiesSubjects: Languages, Maths, General Science and Social Studies

YearYear Grade and SubjectGrade and Subject Test TypeTest Type

20042004 GradeGrade 4 (Mathematics, Languages 4 (Mathematics, Languages Pilot TestingPilot Testing

20052005Grade 4 (Mathematics, Languages)Grade 4 (Mathematics, Languages)

GradeGrade 4 (General Science, Social Studies) 4 (General Science, Social Studies)

Large Scale TestingLarge Scale Testing

Pilot TestingPilot Testing

20062006

Grade 4 (Mathematics, Languages, General Grade 4 (Mathematics, Languages, General Science, Social Studies)Science, Social Studies)

GradeGrade 8 (Mathematics, Languages) 8 (Mathematics, Languages)




GradeGrade 8 (General Science, Social Studies) 8 (General Science, Social Studies)




Grade 8 (General Science, Social Studies)Grade 8 (General Science, Social Studies)


Large Scale TestingLarge Scale Testing 4

55

National Sample of NEASNational Sample of NEAS

NEAS uses a two-stage stratified random cluster sample NEAS uses a two-stage stratified random cluster sample for G-4 and G-8.for G-4 and G-8.

Explicit strata are :Explicit strata are :• Gender: male-female – two explicit strata.Gender: male-female – two explicit strata.• Location: urban-rural – two explicit strata.Location: urban-rural – two explicit strata.• Province/Area: four provinces and four areas – eight explicit Province/Area: four provinces and four areas – eight explicit

strata.strata.

Total proposed explicit strata: 2x2x8=32. Total proposed explicit strata: 2x2x8=32.

The implicit strata are school type and districtThe implicit strata are school type and district

Contd…

66

National Sample of NEASNational Sample of NEAS

The basic sampling frames are taken from the NEMIS The basic sampling frames are taken from the NEMIS database.database.

Within a stratum, all schools are included, with selection Within a stratum, all schools are included, with selection probability proportional to size (PPS).probability proportional to size (PPS).

Clustering occurs only at the school level. A maximum Clustering occurs only at the school level. A maximum of 20 and minimum 6 students are selected from each of 20 and minimum 6 students are selected from each school.school.

In some rural areas, up to 2% of students in the smallest In some rural areas, up to 2% of students in the smallest

schools are eliminated from frames, and additional schools are eliminated from frames, and additional schools are sampled.schools are sampled.

77

Geographical Distribution of NEAS Geographical Distribution of NEAS Primary Sampling UnitsPrimary Sampling Units

88

Schools Sampled – Planned and Schools Sampled – Planned and Achieved Achieved

Province/AreaProvince/Area

Grade-4 (2006)Grade-4 (2006)Planned Achieved Planned Achieved

% %

Grade-8 (2007)Grade-8 (2007)Planned Achieved Planned Achieved

%%

BalochistanBalochistan 110110 8181 100100 100100

NWFPNWFP 100100 9999 101101 100100

PunjabPunjab 240240 9696 226226 100100

SindhSindh 165165 9393 125125 100100

AJKAJK 7070 98.598.5 6767 98.598.5

FANAFANA 6969 9090 6262 100100

FATAFATA 7272 7878 6969 9191

ICTICT 3030 100100 2020 100100

Total Achieved SchoolsTotal Achieved Schools 856856 9292 770770 9999

The outcome for achieved school sample at G-8 level(99%) is better than G-4(92%)

99

Students Sampled – Planned and Students Sampled – Planned and AchievedAchieved

Province/AreaProvince/AreaGrade-4 (2006)Grade-4 (2006)

Planned Achieved %Planned Achieved %Grade-8 (2007)Grade-8 (2007)

Planned Achieved %Planned Achieved %

BalochistanBalochistan 22002200 56.556.5 20002000 9494

NWFPNWFP 20002000 8282 20202020 9999

PunjabPunjab 48004800 8282 45204520 9898

SindhSindh 33003300 6363 25002500 9090

AJKAJK 14001400 6969 13401340 8989

FANAFANA 13801380 5757 12801280 8787

FATAFATA 14401440 5050 12001200 6868

ICTICT 600600 9797 400400 9797

Total Achieved SchoolsTotal Achieved Schools 1712017120 7070 1526015260 9393

Students’ participation rate at G-8 level is higher than G-4

1010

National Assessment InstrumentsNational Assessment Instruments

Basic competencies in the following core subjects are Basic competencies in the following core subjects are assessed at both grades levelsassessed at both grades levels

• LanguageLanguage• MathematicsMathematics• Science Science • Social StudiesSocial Studies

The focus of NEAS assessment is to determine the The focus of NEAS assessment is to determine the extent which students have acquired reading, writing, extent which students have acquired reading, writing, problem solving, and critical thinking skills, and not the problem solving, and critical thinking skills, and not the rote learning from textbooks rote learning from textbooks

1111

Development of National Assessment Development of National Assessment InstrumentsInstruments

The instruments used for national assessment were based on The instruments used for national assessment were based on National CurriculumNational Curriculum

These instruments were developed through a long process of These instruments were developed through a long process of consultation among national and provincial subject experts consultation among national and provincial subject experts therefore, they reflect broad national consensus on both the therefore, they reflect broad national consensus on both the structure and contentstructure and content

The instruments are based on assessment The instruments are based on assessment FrameworksFrameworks covering three ability strands in each subject. These covering three ability strands in each subject. These frameworks are reviewed refined and updated on regular basis frameworks are reviewed refined and updated on regular basis to reflect the desired goals of assessmentto reflect the desired goals of assessment

These instruments undergo rigorous expert review as well as These instruments undergo rigorous expert review as well as empirical and psychometric evaluation before acquiring the empirical and psychometric evaluation before acquiring the final shapefinal shape

Several Several spot/pilot testsspot/pilot tests are conducted in active collaboration are conducted in active collaboration with provincial/area assessment centres to ascertain the with provincial/area assessment centres to ascertain the reliability of the NEAS assessment instrumentsreliability of the NEAS assessment instruments

1212

Students’ Achievement in National Students’ Achievement in National AssessmentAssessment

Students’ achievement is assessed on the basis of their Students’ achievement is assessed on the basis of their scores. scores.

These scores provide a measure of how well the students can These scores provide a measure of how well the students can answer items (questions) based on the National Curriculum. answer items (questions) based on the National Curriculum.

Scaled scores are used to prevent item selection having an Scaled scores are used to prevent item selection having an undue influence on student scores. undue influence on student scores.

The scale used in NEAS is similar to TIMSS, SAT, TOEFL etc. The scale used in NEAS is similar to TIMSS, SAT, TOEFL etc. The range of scaled scores is between The range of scaled scores is between 0-1000.0-1000.

The achievement scales are constructed so that a student The achievement scales are constructed so that a student achieving 50% correct marks receives a achieving 50% correct marks receives a scaled scorescaled score of of 500500 and the and the Standard DeviationStandard Deviation of the scale is set at of the scale is set at 100.100.

These scores are used to report trends over time as well as the These scores are used to report trends over time as well as the average and relative performance of different groups of average and relative performance of different groups of students. students.

1313

National Scores of Grade 4 & Grade 8National Scores of Grade 4 & Grade 8

Subject Subject Year/ Year/ Grade Grade

Scaled Mean (500)Scaled Mean (500)

Score Range (0 -1000) Score Range (0 -1000)

Significance of Significance of DifferenceDifference

MathematicsMathematics2005 2005

Grade 4Grade 4

421421 Mathematics score Mathematics score significantly higher significantly higher than Language than Language score score

LanguageLanguage 369369

General Science General Science 2006 2006

Grade 4Grade 4

467467 Social Studies Social Studies score significantly score significantly higher than General higher than General Science scoreScience score

Social StudiesSocial Studies 496496

MathematicsMathematics2007 2007

Grade 8Grade 8

457457 Language score Language score significantly higher significantly higher than Mathematics than Mathematics scorescore

LanguageLanguage 512512

1414

Achievement of Competencies-Maths (G-4)Achievement of Competencies-Maths (G-4)

Difficult

Average

Easy

Measurement, Procedural Knowledge, and Practical Reasoning: Mutual conversion of units of length; Adding/subtracting expressions using problems from every day life involving units of length, mass and capacity.Geometry Procedural Knowledge : Finding perimeter of a square and rectangle.

Number sense, Practical Reasoning, and Information Handling : reading and interpreting a line graph Measurement Conceptual Understanding: relationship among units of length; Geometry, Conceptual Understanding: measuring and drawing line segments in MM and CM;

Number sense; conceptual understanding, procedural knowledge, : Concept of an improper, mixed and equivalent fractions. The highest common fraction of two or more than natural numbers.

1515

Reading for literary experience: Example: Poems and storiesReading for information: Example: essay General understanding : Global knowledge of text, providing title of text, explaining purpose or theme of text

Reading to perform a task: Reading to accomplish something , using purpose and structure of text; Example: Time TableKnowledge of Language and Vocabulary: Know and distinguish noun, verb, and adjectives; understand the use of idioms

Narrative writing: Use of creativity and power of observation, for example writing a storyPersuasive Writing: Using factual information, reasoning and comparison, developing and organizing ideas, using language effectivelyExample: Letter to Father

Difficult Average Easy

Achievement of Achievement of Competencies-Urdu Competencies-Urdu

(G-4)(G-4)

1616

Content Domains achieved in LanguageContent Domains achieved in Language

Subtest Subtest

Number of Number of ItemsItems Scaled MeanScaled Mean

ReadingReading 4646 520520

GrammarGrammar 1111 507507

WritingWriting 66 484484

Language Total Language Total 6363 512512

• Scores are significantly higher on reading as compared to

other domains of Language

• Students got the lowest scores on writing domain

1717

Location–Wise Achievement ScoresLocation–Wise Achievement Scores

LocationLocation

Scaled Scaled Mean Mean

(Grade-4)(Grade-4) (Grade-8)(Grade-8)

MathsMaths LanguageLanguage General General ScienceScience

Social Social StudiesStudies

MathsMaths LanguageLanguage

RuralRural 418418 360360 465465 491491 471471 503503

UrbanUrban 430430 399399 475475 513513 439439 525525

DifferenceDifference n. sn. s Highly Highly SignificantSignificant

n. sn. s SignificantSignificant SignificantSignificant SignificantSignificant

1818

Gender-Wise PerformanceGender-Wise Performance

GenderGender

Scaled Scaled Mean Mean

(G – 4) 2005(G – 4) 2005 (G – 4) 2006(G – 4) 2006 (G – 8) 2007(G – 8) 2007

MathsMaths LanguageLanguageGeneral General ScienceScience

Social Social StudiesStudies

MathsMaths LanguageLanguage

BoysBoys 420420 350350 459459 489489 464464 503503

GirlsGirls 424424 405405 479479 506506 446446 529529

DifferenceDifference n. sn. sHigh High

SignificantSignificant SignificantSignificant n. sn. s SignificantSignificant SignificantSignificant

1919

Earthquake effects on Students’ Earthquake effects on Students’ Achievement (Grade 4)Achievement (Grade 4)

Did you have to move to a new Did you have to move to a new place after the October 2005 place after the October 2005 earthquakeearthquake %%

Scaled ScoresScaled Scores

MathsMaths UrduUrdu ScienceScience S. StudiesS. Studies

YesYes 77NoNo 9393

356356400400

355355386386

430430471471

455455500500

DifferenceDifference SignificantSignificant SignificantSignificant SignificantSignificant SignificantSignificant

Earthquake significantly affected students’ achievement in all the four Earthquake significantly affected students’ achievement in all the four subjects.subjects.

Students having to migrate after earthquake scored significantly lower on all Students having to migrate after earthquake scored significantly lower on all the four subjects the four subjects

Achievement in Relation to Achievement in Relation to Background FactorsBackground Factors

2121

Background Data GatheringBackground Data Gathering

During every cycle of assessment, background data are During every cycle of assessment, background data are gathered by questionnaire and analysis of fieldworkers’ gathered by questionnaire and analysis of fieldworkers’ reportsreports

Questionnaires are given to:Questionnaires are given to:• students (including a section for parents), students (including a section for parents), • head teachers and head teachers and • teachersteachers

Conditions, facilities etc. in schools are observed and Conditions, facilities etc. in schools are observed and recorded by Test Administratorsrecorded by Test Administrators

Background data are correlated with learning achievement to Background data are correlated with learning achievement to identify needs for improvement and factors associated with identify needs for improvement and factors associated with achievementachievement

2222

Background Data Gathering FrameworkBackground Data Gathering Framework

Background data are classified into five areas following Background data are classified into five areas following the Craig-Heneveld Framework:the Craig-Heneveld Framework:

• The StudentsThe Students• Supporting Inputs from Outside the SchoolSupporting Inputs from Outside the School• Teaching-Learning ProcessTeaching-Learning Process• Enabling Conditions Enabling Conditions • School ClimateSchool Climate

2323

The StudentsThe Students

Data is gathered from students about:Data is gathered from students about:

Socio-economic background (e.g. family size, Socio-economic background (e.g. family size, father’s/mother’s education and occupation)father’s/mother’s education and occupation)

Factors in the home environment related to achievement Factors in the home environment related to achievement (e.g. help in homework, space for reading, need to work (e.g. help in homework, space for reading, need to work outside)outside)

Attitudes towards educationAttitudes towards education

2424

Example of Student Background Example of Student Background Question:Question:

VariableVariable % %Scaled ScoresScaled Scores

MathsMaths UrduUrdu General General ScienceScience Social StudiesSocial Studies

YesYes 2323 NoNo 7777

383383403403

370370389389

455455473473

481481503503


Do you work outside the home to earn money? Do you work outside the home to earn money?

ConclusionConclusion:: about 23% of Grade 4 students reported having to work about 23% of Grade 4 students reported having to work outside the home. Their achievement is significantly lower than others’. outside the home. Their achievement is significantly lower than others’.

2525

Supporting Inputs from OutsideSupporting Inputs from Outside

Background Information is gathered from different Background Information is gathered from different sources about:sources about:

Parent and community support (e.g. frequent school-Parent and community support (e.g. frequent school-parent meetings, financial and other support, parental parent meetings, financial and other support, parental involvement in homework) involvement in homework)

Effective support from the education system (e.g. Effective support from the education system (e.g. supervision, availability of teaching materials and supervision, availability of teaching materials and supplementary materials)supplementary materials)

Adequate material support (e.g. facilities, furniture in Adequate material support (e.g. facilities, furniture in school, school budget for supplies)school, school budget for supplies)

2626

Use of Blackboard and AchievementUse of Blackboard and Achievement

Use of Use of BlackboardBlackboard%%

Scaled ScoresScaled Scores

MathsMaths UrduUrdu G. ScienceG. Science S. StudiesS. Studies

YesYes9797NoNo0303

39 939 9350350

384384341341

469469431431

497497457457


Teachers’ use of black board significantly increased Teachers’ use of black board significantly increased students’ achievement in all the four subjects students’ achievement in all the four subjects

Ninety-seven percent teachers reported they Ninety-seven percent teachers reported they dailydaily use use black boardsblack boards

2727

Multi-grade Teaching and Multi-grade Teaching and AchievementAchievement

Multi-grade Multi-grade Teaching %Teaching %

Students’ Scaled ScoresStudents’ Scaled ScoresMaths UrduMaths Urdu

Teachers’ Scaled ScoresTeachers’ Scaled Scores Maths UrduMaths Urdu

YesYes 21 21NoNo 78 78

442442466466

495495516516

616616644644

644644661661


Lower frequency of Lower frequency of Multi-grade Teaching is linked with increased Multi-grade Teaching is linked with increased performance for both students and teachersperformance for both students and teachers

Twenty- one percent teachers reported they have to teach more than on Twenty- one percent teachers reported they have to teach more than on classes in one periodclasses in one period

2828

Rewards and Punishment in SchoolRewards and Punishment in School

RewardReward % %Scaled ScoresScaled Scores

MathsMaths UrduUrdu General General ScienceScience S. StudiesS. Studies

Yes 85Yes 85NoNo 14 14

403403

372372

390390

354354

474474

440440

504504

464464


PunishmentPunishment % %

FrequentlyFrequently .03.03Occasionally 38Occasionally 38NeverNever 58 58

351351388388405405

345345376376391391

444444458458476476

452452490490505505


Student who Never got punished by their teachers performed significantly better Fifty-eight Student who Never got punished by their teachers performed significantly better Fifty-eight percent students reported they never got punished. percent students reported they never got punished.

Only .03 % students reported frequent punishmentOnly .03 % students reported frequent punishment Rewarding students’ performance increased achievement all the subjectsRewarding students’ performance increased achievement all the subjects

2929

PTA/ SMC in School: PTA/ SMC in School: Head Teachers’ ReportHead Teachers’ Report

0

10

2030

40

50

60

7080

90

100

PTA/SMCSchool

Need ForPTA/SMC in

Schools

Funding ofPTA/SMCfrom Govt.

CopperativeRole of

PTA/SMC

Pe

rce

nt

Yes

No

• Around 80% head teachers reported presence of PTA/ SMCs’ in School

•Presence, need, funding, or role of PTA/SMC did not increase students performance in any of the NEAS tested school

•Only 39% head teachers reported getting funds from Govt.

•Ninety-seven percent head teachers reported need for SMCs in Schools

3030

Strengthening Links between NEASStrengthening Links between NEASand the Educational Planning Processand the Educational Planning Process

NEAS can Support Educational Planning by:NEAS can Support Educational Planning by:

• providing data about the actual conditions in schools; andproviding data about the actual conditions in schools; and

• indicating the possible impact of policy actions on student indicating the possible impact of policy actions on student achievementachievement..

Educational Planners can make NEAS more Effective by:Educational Planners can make NEAS more Effective by:

• proposing topics to be added to background data gathering; proposing topics to be added to background data gathering; andand

• suggesting relationships to be explored.suggesting relationships to be explored.

3131

The Presentation SummaryThe Presentation Summary

NEAS has assessed the achievement of Grade 4 & 8 NEAS has assessed the achievement of Grade 4 & 8 students in four subjects: students in four subjects: Urdu, Mathematics, Science and Social studies.Urdu, Mathematics, Science and Social studies.

The assessment reveals large differences in average The assessment reveals large differences in average performance among provinces and between students performance among provinces and between students from differing backgrounds.from differing backgrounds.

These differences in average performance can be traced These differences in average performance can be traced back to differences in student and family characteristics, back to differences in student and family characteristics, school processes and inputs. school processes and inputs.

3232

Institutionalization of NEAS and Institutionalization of NEAS and Future Vision of National Assessment Future Vision of National Assessment in Pakistan in Pakistan

Institutionalization of NEAS:Institutionalization of NEAS:

Government of Pakistan declared NEAS as a permanent institution of the Government of Pakistan declared NEAS as a permanent institution of the Ministry of Education, from July 2007Ministry of Education, from July 2007

Future Vision of NEAS:Future Vision of NEAS:

Recommendations by National NEAS Select Committee:Recommendations by National NEAS Select Committee:

• NEAS future planning till 2020 comprises National Assessment at Grade 4, NEAS future planning till 2020 comprises National Assessment at Grade 4, 5, 8, 10 and 12 5, 8, 10 and 12

• Private schools are included in NEAS sample on Pilot basis in 2008 Private schools are included in NEAS sample on Pilot basis in 2008 National AssessmentNational Assessment

• Listening and speaking skills of student assessment would be part of Listening and speaking skills of student assessment would be part of NEAS future activitiesNEAS future activities

• Pakistan can explore the possibility to participate in the next TIMSS in 2012Pakistan can explore the possibility to participate in the next TIMSS in 2012

3333

Thank You…Thank You…

national assessment findings (grade 4 & 8) national education assessment system (neas) ministry...

Documents

languages grade

social studies grade

education education

quality education

implicit strata

languages pilot testing

total proposed explicit

social studies year