iccs / iv conference, prague 2011. franz schott: measuring the outcome of education measuring the...

ICCS / IV Conference, Prague 2011. Franz Schott: Measuring the outcome of educationICCS / IV Conference, Prague 2011. Franz Schott: Measuring the outcome of education

MEASURING THE OUTCOME OF EDUCATIONFRANZ SCHOTT

UNIVERSITY OF TECHNOLOGY, DRESDEN, GERMANY


Introduction• The topic of this conference is:

“Employability - Mobility - Flexibility. European demands challenging Religion and Education”. We read in the announcement of this conference:• “The aim of education and training is focused on a concept of flexible skills

and competencies that serve employability, mobility and flexibility.”• “The Conference will encompass and discuss the European perspectives of

education …”• One central perspective on education since the last decade is the view on the

efficiency of the educational system• This leads to demand more output-orientation of the educational system

instead of input-orientation (input is for example: curriculum, training of teachers, teaching materials)

• The output of an educational system is what the addressees of this system have learnt, in other words: it is the outcome of education

• International comparative studies of the outcome of education (e.g. Programme for International Student Assessment: PISA) brought questions about the efficiency of national educational systems and about measuring the outcome of education


We will discuss the topic: measuring the outcome of education as follows:1. On measuring2. On the outcome of education3. Purposes of measuring the outcome of education4. Problems of measuring the outcome of education5. SummaryLiterature

Introduction


We have to differentiate measuring in a narrow sense and in a broad sense:•Measurement in a narrow sense

• This concept is used in the physical sciences and defines: Measurement is the process or the result of determining the magnitude of a quantity, such as length or mass, relative to a unit of measurement, such as a meter or a kilogram (see e.g.: Deutsche Norm DIN 1319-1, 1995).

•Measurement in a broad sense• In social sciences there are no generally binding definitions of units of

measurement such as meter or kilogram• Therefore we find in social sciences a broader concept of measurement:• Measurement is "the assignment of numerals to objects or events

according to some rule“ (Stevens, 1946)• Stevens differentiated different levels of measurement:

• Ordinal scale => the numbers represent a rank order (1st, 2nd, 3rd, etc.)

• Interval scale => the interval between the numbers is equal• Ratio scale => like interval scale plus a well-defined zero point

We do not find ratio scales in social sciences

1. On measuring


Stevens´ broad definition of measurement is accepted in all social sciences.According to this broad definition of measurement, people use often measurements in everyday lifeAn example of a measurement in everyday lifeIn a conversation Elisabeth says: “Peter is much more self-confident than Mary”.•According to the above-mentioned broad definition of measurement as the assignment of numerals to objects or events according to some rule Elizabeth has made a measurement on the level of an ordinal scale:•Regarding the criterion of self-confidence, she has compared Peter and Mary:

• Peter got rank 1st and Maria got the rank 2nd•What was Elizabeth´s rule of this measurement? Perhaps she will explain:

• I observed Peter and Mary in situations which demand a self-confident behavior. In most of the cases, Peter performed better than Mary.

•To generalize:•A personal trait or ability can be measured by assessing the behavior in question in a certain situation according to some rules

• This measurement is possible and is actually proceeded in many domains , for example: ability in math or tennis, trust in God, etc.

1. On measuring


Criteria for good measurement The most important criteria for good measurement are:•Objectivity => the evaluation is independent of the assessing person•Reliability => a repeated evaluation has the same result •Validity=> the evaluation reflects what is the intention of measurement

1. On measuring


1. On measuring


• The outcome of education is what is learnt in a certain educational system, for example what students learn in a school

• Important differentiations of outcome: • Various domains of the outcome:

• for example outcomes concerning: • Mathematics• Geography• Religion• Employability

• Achieved outcome: outcome as description, what is learnt• Intended outcome: outcome as norm, what should be learnt

• Names used for intended outcomes:• Learning goal or instructional objective• Intended Competence• Educational standard

• Short-term outcome: the outcome immediately after the lesson• Long-term outcome: the sustainable outcome, which outlasts the short-

term outcome

2. On the outcome of education

Increasing generality


There a several purposes of measuring the outcome of education, for example:•International comparison of the quality of schools (e.g. PISA: Programme for International Student Assessment) •Certification of the learner (e.g. diploma, university degree)•Feedback to the learner•Feedback for the teacher•Maintaining and improving the quality of instruction

The measurement of the outcome is essential to control and improve education

3. Purposes of measuring the outcome of education


To evaluate the success of instruction, it is necessary to compare the intended outcome with the achieved outcome The quality of this comparison depends on how precise the achieved outcome

and the intended outcome can be analyzed Therefore the assessment quality of the outcome depends on how

precise the description of the intended outcome is There is no chance to improve instruction without a sufficient

precise description and measurement of the intended outcome!

3. Purposes of measuring the outcome of educationThe measurement of the outcome is essential to maintain quality control and

improve education

INSTRUCTIONINSTRUCTION MEASUREMENT OF THE ACHIEVED

OUTCOME

MEASUREMENT OF THE ACHIEVED

OUTCOME

COMPARISON OF THEINTENDED OUTCOME

WITH THE ACHIEVED OUTCOME

COMPARISON OF THEINTENDED OUTCOME

WITH THE ACHIEVED OUTCOME

FEEDBACK


We stated: There is no chance to improve instruction without a sufficient precise description and measurement of the intended outcome! Here an example:We asked:How can teachers ensure that their students have a good chance to learn during class the intended outcome of education?To answer this question, we developed approaches (Schott & Seidl, 1994; Schott & Azizi Ghanbari, 2008, 2010, 2011) called PLANA and GovI (Goal-valid Instruction) Here some aspects of these approaches:•Step 1: Describe the indented outcome of education (that is an instructional goal) as accurately as necessary by specifying the sets of tasks that can be solved by the learners if they have reached the instructional goal. To specify the sets of task we have developed a special method of task analysis.•Step 2: These task sets must occur in an appropriate manner during class•Step 3: These task sets must occur in an appropriate manner when measuring the outcome of education•These three steps ensure that both the lesson and the control of the learning success is content-valid to the intended outcome of education

• We speak of “parallel content validity”



GovI a proposal on how to teach and measure the intended outcome of education:=> 3 Steps of an outcome-oriented instruction and measurement


The three steps ensure that both the lesson and the control of the learning success is content-valid to the intended outcome of education

Step 1PRECISE DEFINITION OF THE

INTENTED OUTCOME

Step 2OUTCOME-ORIENTED

INSTRUCTION

Specifying sets of tasks describing the intended outcome

of educationt

The task sets describing the intended outcome of education must occur in an

appropriate manner during class

Step 3OUTCOME-ORIENTED

MEASUREMENTThe task sets describing the intended

outcome of education must occur in an appropriate manner in the exam

PARALLEL CONTENT VALIDITY


4. Problems of measuring the outcome of education


Arguments against the measuring of the output of education:•Objection: Measuring the outcome of education prevents demanding lessons and sophisticated teaching, it leads to teaching to the test

This argument must be taken seriously. The teacher must make sure that students are not prepared only for the

test• Objection: If at all, only the short-term outcome is measurable. Long-term

outcomes are not measurable This argument must be taken seriously too. We usually cannot examine how

the learners will behave after graduation But we can do something to promote sustainability of the outcome of

education



A careful task analysis can help to promote sustainability and measurability of the outcome of educationAn example:In a research project concerning outcome-oriented instruction we developed with university students a lesson for ethics education at school. The topics of the lesson were ethical concepts like Kant's categorical imperative or concepts of utilitarianism. When we began to specify the intended outcome of this lesson, my students insists that one of the most relevant outcomes of this lesson should be:• The learners should recognize the importance of the topics•The learners should see the sense of the lessonThe challenging question was: •How can one distinguish a learner who has achieved this outcome from a learner who did not achieve it? • Which set of tasks can measure that?



A careful task analysis can help to promote sustainability and measurability of the outcome of educationExample continuedVery relevant outcome of the lesson: •The learners should recognize the importance of the topics; they should see the sense of the lesson.•Which set of tasks can measure this outcome?

After a long discussion we came to the following solution: •A learner who has achieved this outcome is able to solve the following 3 tasks: •Explain: If there is no controversy on moral norms, there are no ethical problems to solve.•Explain: If there is a disagreement concerning a certain moral norm, then ethical rules are necessary to solve this ethical problem•Explain: If there are various ethical rules (such as Kant's categorical imperative and utilitarianism), then philosophical methods are necessary to compare these different rules.We decided: If a learner can solve the set of tasks consisting of these 3 tasks he or she has understood the sense of the lesson.Specifying in this way may help to make an outcome of education, which is challenging because of its compexity, measurable and sustainable.



The concept “measurement”• Measurement in a narrow sense

• is used in the physical sciences and defined as the process or the result of determining the magnitude of a quantity relative to a unit of measurement

• is not applicable to the outcome of education• Measurement in a broad sense

• is defined as the assignment of numerals to objects or events according to some rule• is applicable to the outcome of education

• Important criteria for good measurement • Objectivity• Reliability• Validity

• According to the rules of assignment of numerals to objects or events we differentiate:•Informal measuring•Semi-formal measuring•Formal measuring

5. Summary


The concept “outcome of education”• Different types of outcome of education•Various domains of the outcome•Achieved outcome: what is learnt•Intended outcome: what should be learnt•Short-term outcome•Long-term outcome

5. Summary


Purposes of measuring the outcome of education• International comparison of the quality of schools (e.g. PISA) • Certification of the learner (e.g. diploma, university degree)• Feedback to the learner• Feedback for the teacher• Maintaining and improving the quality of instruction Problems of measuring the outcome of education• We discussed arguments against the measuring of the output of education• Measuring the output of education

• is incompatible with human dignity• is not possible in the case of

• advanced skills• emotions, attitudes and values• long-term outcomes of education

• leads to teaching to the test• We suggest that appropriate procedures of outcome-oriented instruction can help to

teach and test the outcome of education• A careful task analysis supports measuring the outcome of education

=> Measuring the outcome of education is in many cases useful and humanitarian

5. Summary


Schott, F. & Azizi Ghanbari, S. (2008). Kompetenzdiagnostik, Kompetenzmodelle, kompetenzorientierter Unterricht. Zur Theorie und Praxis überprüfbarer Bildungsstandards. Münster: Waxmann Verlag.

Schott, F & Azizi Ghanbari, S. (2010). Zur Theorie und Praxis kompetenzorientierten Lehrens und Lernens: Probleme und Lösungsmöglichkeiten. Report Psychologie. 11/12 2010, 480-490.

Schott, F & Azizi Ghanbari, S. (2011, in Vorbereitung). Der Beitrag von Bildungsstandards, kompetenzorientiertem Unterricht und Kompetenzdiagnostik zur Qualitätssicherung des Bildungswesens in der Schule: Eine problemorientierte Einführung. Münster: Waxmann Verlag.

Schott, F. & Seidl, P. (1994): PLANA - an ID model focussing on instructional task analysis. In: R. Tennyson, F. Schott, N. Seel & S. Dijkstra (Eds.) (1997): Instructional Design: International Perspectives. Volume I: Theory, Research, and Models. pp. 395 – 413. Lawrence Erlbaum Associates, Inc., Publishers: Hillsdale, New Jersey.

Stevens, S. S. (1946). On the theory of scales of measurement. Science, 103 (2684): 677–80

Literature

iccs / iv conference, prague 2011. franz schott: measuring the outcome of education measuring the...

Documents

outcome of education

measuring slide

measuring iccs

aim of education

interval scale

social sciences

unit of measurement

equal ratio scale