Standards, Evaluation and AccountabilityState-of-Play and Challenges from Latin America

Gilbert A. Valverde, Ph.D.Comparative and International Education Policy ProgramUniversity at Albany – State University of New Yorkvalverde@uamail.albany.edu

Working Group on Standards and AssessmentProgram to Promote Educational Reform in Latin America

A World-Wide Shift of Policy Focus• We are experiencing a global shift in education

policy priorities▫ Not only is it important for children to have access

to education.▫ The quality of what transpires within the

classroom also matters.• Prioritizing quality, means focusing on the

content of schooling▫ Global interest in standards, evaluation and

accountability is a signal of this policy shift.

Latin America• Much discussion of the importance of the content of

schooling• Important and persistent problems of quality▫ National, regional, and international testing programs

have thoroughly documented the poverty of educational outcomes in the region.

• The evidence has conclusively demonstrated that the traditional repertoire of policy instruments is not sufficient to address this problem.

• A growing consensus that standards, evaluation and associated policy instruments merit serious consideration.

Goals for this talk

• Discuss how initiatives in standards, assessment and accountability are forwarding important policy conversations on educational quality in Latin America.

• Consider challenges and lessons from the Latin American experience that may be of interest to other parts of the world.

Basic premises

• Educational reform based on principals of promotion of educational quality requires:▫ A clear vision of what is to be accomplished▫ A system to monitor how these goals are being

met.▫ A commitment to act in accordance with the

results of monitoring efforts

Elements of Quality of Vision

• Research and policy experience indicate that accomplishing high levels of quality in educational outcomes requires the articulation of a clear vision of pedagogical objectives that operationalize goals that are▫ rigorous,▫ challenging,▫ have well defined priorities and foci▫ evidence-based

Challenges for Latin America

• A critical examination of the instruments of curriculum policy in the region suggests an urgent need to correct problems of▫ Ambiguity▫ Dispersion and lack of priorities▫ Lack of focus▫ Absence of academic rigor

An example of dispersion and lack of focus

0 5 10 15 20 25 30 35 40

18

17

16

15

14

13

12

11

10

9

8

7

6

5Japan

0 5 10 15 20 25 30 35 40

5

6

7

8

9

10

11

12

13

14

15

16

17

18

Mexicoviejo

viejo / foco

nuevo

nuevo / foco

11

1982

1993

1996

1996

1996

1996

1998

1991

1993

2002

19862005

1996

1991

1990

1996

1996

1998

1997

Source: Sistemas de evaluación de aprendizajes en América Latina. Balances y desafíos. PREAL, 2006.

First National Assessments

RAPID GROWTH•In 1980 – none•In 1989 – 2 (Chile, Costa Rica)•Today – universal

Consequences of rapid growth• Introduction was precipitous, without much preparatory work

with schools or other actors.▫ Many claim that introduction was forced or imposed.

• In most cases the initial measurement model was not at all aligned with the curriculum reforms that often were taking place simultaneously.

• Results were quickly and extensively disseminated, most often accompanied by condemnations of the quality of teaching (which is never measured).

• Important questioning of the technical quality of testing efforts▫ Much suspicion that testing systems are not committed to

technical quality.• Tepid government commitment to testing.▫ Often testing systems do not have budgets until a few months prior

to test administration.▫ Low levels of investment, with unclear commitment to technical

quality.▫ Reticence to act in accordance with test results

Continuing progress• Growing participation in large-scale cross-national

assessments.• Increasing investments in state-of-the-art technologies• Awareness that disappointing outcomes measured by

tests imply need to explore new policy instruments• Setting of more specific improvement targets• Emerging “culture of evaluation”• Capacity building, growing experience and exchange• Openness to revision and improvement of approaches,

methods, analyses and reporting• Teachers growingly interested in potential of properly

utilized assessments for professional development and school improvement purposes

• More interest in returning results to schools and non-traditional educational stakeholders

Nascent efforts in Accountability• If the accountable unit is the school, requires a

census of schools, if the student, a census of students

• No high-stakes testing in the Region• “Low-Stakes” Tests in Costa Rica, El Salvador and

the Dominican Republic▫ In the Dominican Republic, for example, the school-

leaving examination or “Prueba Nacional” contributes only 30% of a student’s grade

▫ Students have 3 or 4 opportunities to pass the exam▫ Few students fail to pass based on the test alone

Schools = primary units of accountability• Chile▫ Yearly publication of league tables ranking all

schools• El Salvador▫ Publication of lists of highest performing schools

• Colombia (Bogotá)▫ “Excellence” awards based on school test scores.

School Choice• Extremely uncommon• Chile is the notable exception

Ministry distributes school-by-school test results for each region, including average scores in each subject area, changes in scores since previous testing round,

and comparisons with schools serving similar

socioeconomic groups Special reports for parents since 2003 Recent research has demonstrated conclusively

that despite the intentions, parents do not make significant use of test results in choosing schools.

New efforts to promote accountability

• Teacher as the unit of accountability▫ Chile: Awards for teachers in the best schools according to the

national test A new voluntary teacher test (including video, teacher

portfolio and content assessment) to compete for “pedagogical excellence” designation

Efforts to put together a required teacher certification test for graduating teachers

▫ Mexico: Teacher bonuses based on complex formula that includes

student test results

However….

Still too early to say whether, in most Latin American countries, institutionalized and permanent mechanisms have been firmly established which enable continuous setting and revising of learning goals and measuring their achievement.

In only few countries, student testing and other kinds of outcomes assessment and reporting are clearly framed or aligned within a set of policies and norms about the kind of utilization which will be given to the resulting information, as part of a well integrated, clear and balanced set of policies for the improvement of learning and for adequate accountability.

New Policy Routes• The most important challenge is to increase mean levels

of achievement in all schools.• Increasing quality in outcomes requires▫ Operationalization of a concrete vision of what can be

considered quality outcomes▫ Assignment of responsibilities▫ Monitoring▫ Commitment to act according to monitoring results.

• A new policy instrument increasingly of interest: standards▫ There are educational standards or projects underway in:

Guatemala, Honduras, Panama, Colombia, Argentina and Chile

▫ Following diverse models from the United States and Australia

Main challenges

• Clarification and social legitimation of assessment purposes and processes

• Clarification of learning goals• Selection of what will be measured and how, according to

purposes • Selection of appropriate institutional framework • Improvement of testing instruments and processes• More and better data analyses • More and better diversified reports• More and better dissemination of results • Appropriate use of results for decision making

Key question looking to the future• Is it possible or desirable to compensate for the early

shortcomings in the introduction of assessment systems by greater efforts in transparency and genuine consultation?

• Can the region learn from failures to lay sufficient policy groundwork in the introduction of assessment systems, in order to avoid pushback against nascent efforts to introduce standards that could lead to similar questioning of their political legitimacy, pedagogical value, and technical soundness?

• Can research and development efforts be made more transparent so that a broad spectrum of actors will be persuaded that governments are sufficiently concerned with the technical quality of standards and the validity and reliability of measures used in their assessment systems?

• Will it be possible to introduce or maintain a concern for benchmarking Latin American standards against “world-class standards” when the likelihood is that Latin American countries will continue to be a the bottom of PISA, TIMSS and other global league tables?

• Can the development of assessment systems in the region be used to introduce a concern for evidenced-based policy making in the region, to temper enthusiasm for theoretical innovations with weak evidentiary bases in the development of curriculum policy?

• Can assessment and standards be brought into alignment with a commitment to act in accordance with evaluation findings in order to develop accountability systems with greater potential for stimulating educational quality?