SAM case history: from school level to regional level

A. Vorontsov

Goal:

to demonstrate different opportunities and potential of SAM results usage for different parties of education (policy makers, school principals, teachers, students and parents).

Questions to discuss:

1)Why school teacher needs SAM?

2)How does SAM influence on improving of quality of educational process and it’s results?

3)How to conduct SAM monitoring in school?

4)How can SAM influence educational policy at regional and federal level?

1. Teachers construct tool – professional development

Goal: adoption of SAM building principals and functioning.

•1st stage – developing of matrix of subject content, that gives compact idea about complex of cognitive methods of actions (terms, principals, schemes, algorithms and so on), that compose the content of main parts of some school subject.

Samples from content matrixes of SAM tests

Content area Mode of action Means of action

Mathematical literacy

Measurement of Values

•Direct measurement•Indirect measurement

The ratio between number, value and unit"part and whole“ relationFormula for the area of a rectangle

Scientific literacy

Processes and their conditions

Establishing temporal relations between events

process scheme experimental scheme

Language literacy

Word and its meaning

Construction an array of words and word forms with alternating sounds

The ratio of the "sound-letter"Audible word model

Structure of test tasks

Mode of action(task)

Task 1.1.

Task 1.2.

Task 1.3.

1 level -Formal

2 level -Object-related

3 level -Functional

Structure of SAM test

Task 1 Task 2 Task N…Task 1.1

Task 1.2

Task 1.3

Task 2.1

Task 2.2

Task 2.3

Task n.1

Task n.2

Task n.3

Test subject is a measure of cultural appropriation of means and their respective modes of action

Content base of a test – technological matrix (System of the tools / methods of action orientation in a given

subject area)

Characteristics of the tasks of each level within the same test item

Levels Characteristics of the task

Formal - Description of tasks shows its affiliation with particular class with standard procedures for solution (standard, model, typical tasks);

- Relations, essential for task performance, are related to external characteristics of the situation that don’t need determination

Reflexive - Tasks that don’t have standard solutions. Types of tasks:A) tasks that can be solved by typical solution need changing of taks’ situation ;B) tasks are presented in abstract form to avoid manipulation of data;C) involving the treatment of typical circuits action (movement from the result to

conditions) when performing tasks by other people

Functional A) reconstruction and testing options with a choice of one action, meet some requirements;B) on the analysis given circuits (formulas) steps to select adequate; C) to adapt the formula and steps to the specifics of the object; D) involving edge detection method and the main attraction of additional resources;E) in which the main method is a means of implementing another action.

Basic test results

3 key group of indexes:

•Integrated scores (primary and test) – characterize general success of adoption of knowledge;

•3D profile – fixes the measure of adoption of cultural ways and relevant actions;

•Levels of achievement – based on integrated scores of test participants and benchmarks that divide all participants into groups, that relevant to different quality levels

THREE-DIMENSIONAL PROFILES

  Achievements (different sections of curriculum)

Content area

Averageraw score

Level I,

% Level II,

% Level III,

% Elements of geometry

. . . . .

1 level 2 level 3 level

0

10

20

30

40

50

80

60

70

90 100

D

ifficu

lty, %

Class 4V Class 4G

1 Level 2 Level 3 Level

0

10

20

30

40

50

80

60

70

90

100

Difi

culty

, %

10th grade 8th grade 6th grade 4th grade

Interpretation of childs’ profile: determination of zone of proximal

development (ZPD)

› Levels shows the leading type of childs’ orientation in subject;

› Lets teacher see and take into account zone of proximal development:

1 level - ZPD includes additional understanding of general points of curriculum.

2 level – the broaden of the experience of solving of not standard tasks with exit to the boarders on this subject area is important.

3 level – motivation and possibilities for deepen in content of curriculum in needed.

Proficiency level interpretation: individual progress (Delta)

Second proficiency level is a norm for primary school leavers

Distribution of students of different age by proficiency levels

3. Comparative assessment of student educational achievements -

establishment of statistical norms

• Assessment of test results in frameworks of SAM based on 4 things:

• Average norm – average index of test accomplishment (students/schools);

• Socio-cultural norm – average index of leading schools;

• Average norm for the end of the primary school – 2nd level (reflexive);

• Absolute cultural norm – adoption of knowledge at 3rd (functional) level.

 

City level:

› At the level of regional management entities the following set of test indexes can be interesting:

› Test score - class /school /city

› Distribution of students at levels of achievement - class /school /city

› Test score for each part of the program - city

› Distribution of students at levels of achievements for each part of the program – city

Krasnoyarsk region - monitoring of leading schools

Group of leaders is schools that have the highest indicators. Its and orienteer for the educational society. It sets socio-cultural norm as a standard for the future and additional base for assessment of test results.

Leading group of Krasnoyarsk region (schools)

4th grade5th grade6th grade7th grade8th grade8th grade9th grade10th grade11th grade

Below 1st proficiency level1st proficiency level2nd proficiency level3rd proficiency level

Proficiency level distribution

Profile is more important than percents!

Evaluation of educational strategies

31 2

80%

60%

40%

20%

0%

100%

Regional monitoring The Yamal-Nenets

Autonomous District

Level 0 Level 1 Level 2 Level 37th grade5th grade

Level 0 Level 1 Level 2 Level 3

Secondary general education school, 5th grade

SGES Regional average

Regional average

Gymnasium, 7th grade

Gymnasium

Level 1 Level 2 Level 3

Analysis of possible mistakes

Hypothesis:

1) Not enough level of adoption of some parts of the curriculum in mathematics.

2) General reason for mistakes is fail in general competencies

Analysis of possible mistakes

Test items difficulty diagram

0

10

20

30

40

50

60

70

80

90

100

М-01 М-03 М-06 М-08 М-11 C-01 C-03 C-05 C-08 D-04 D-05 R-04 R-05 G-01 G-02

Analysis of possible mistakes

0

10

20

30

40

50

60

70

80

90

100

Measurement Calculations RegularitiesDependencies Geometry

Test items difficulty (different content area)

Analysis of possible mistakesTest item difficulty (level 2 items)

0

10

20

30

40

50

60

70

80

90

100

М-01 М-03 М-06 М-08 М-11 C-01 C-03 C-05 C-08 D-04 D-05 R-04 R-05 G-01 G-02

Analysis of reasons for mistakes

These data allow to give preference to the second hypothesis:

General reason for mistakes is fail in general competencies and skills.

First of all following skills:

- Text analysis, use of information presented in different forms;

- To mind a few conditions, including those that conflict;

- To use modelling to determine essential relation in a task;

- To identify patterns in structured objects;

- To implement tentative steps in finding a solution;

- To monitor the progress and the result of solving the problem.

Summary and recommendations

For primary and post-primary school (5-6 grades):

a)To include into educational process different tasks that are relevant to 2nd and 3rd levels.

b) To analyze of test tasks and use different tools for modeling (graphic, etc);

c) To engage problem solving when working on educational material including substandard situations;

d) To pay more attention at forming control actions: formal and profound

Why SAM is needed and who needs SAM?

Who? Test indicators Purpose and use of the results

Student Integrated score

3D profile

Level of achievements

- Lets determine «problem zones» in curriculum; to rate a student;- Lets determine individual progress of a student

- Level of achievement – Zone of Proximal Development

Teacher Integrated score

3D profile

Level of achievements

- Lets rate the class (school level); determine “problem spots” of students;

- Shows leading Zone of Proximal Development, lets build individual programs for students

- Shows level of adoption of curriculum of different pupils

Three-dimentional profile and teacher interpretation

Levels Teacher’s action

1 Formal Zone of proximal development includes additional understanding of general ideas of subjects

2 Reflexive Increasing of atypical test items quantity that are on a border of different content areas

3 Functional Organizing of motivation and possibilities to deepen into the context of curriculum.

Why SAM is needed and who needs SAM?

Who? Test indicators Purpose and use of the results

School principal Integrated score

3D profile

Level of achievements

- Lets assess classes on quality of adoption of parts of curriculum - Lets to determine teaching approach in different classes;- To assess quality of teachers work in dynamics

Regional manager Integrated score3D profile

- Monitoring on quality characteristics of topical area, determination of special “clusters” of schools

Minister Monitoring researches

- Determination of age norms of adoption of curriculum: for the primary school leavers the assimilation of the curriculum at the second level is a norm;- Determination of educational technologies , textbooks, that demonstrate best results at 2nd and 3rd levels in dynamics

Tasks for discussion

1 2 3

4 5

Level 0 Level 1 Level 2 Level 3 Level 0 Level 1 Level 2 Level 3 Level 0 Level 1 Level 2 Level 3

Level 0 Level 1 Level 2 Level 3 Level 0 Level 1 Level 2 Level 3

SGES #1 (Muravlenko) , 5th grade SGES #2 (Purovsky), 5th grade

Boarding-school (Harshaimskaya), 5th grade SGES (Raduga), 5th grade

SGES #1 Regional average Regional average

Regional averageBoarding-school 5A 5B 5V

5th grade7th grade SGES #2

Summary:• At the core of results' assessment using SAM instrument is the

age and culture norm taken in their absolute form. That fact contributes to the assessment being more multidimensional 

• SAM can change approaches to teaching methods through inclusion of teachers in the design of test materials for formative assessment;

• SAM can be used as an effective tool for pedagogical feedback and as a basis for formative assessment;

• SAM provides information on the educational process - its efficiency, targeting, target orientation;

• In addition to the possibility of measuring quantitative increase of competence (linear progress)

• SAM allows to fix qualitative increase (level progress);

• SAM allows you to influence education policy at all levels

Thank you for attention!