Paul Kleiman PALATINE Lancaster University

Chaos and Crystals: negotiating assessment

orAssessing at the edge of chaos

Chaos and Complexity

Performance Theory

Personal Experience

Research Paradigms

Assessment

Chaos and Complexity

Performance Theory

Personal Experience

Research Paradigms

Assessment

SOME STARTING POINTS

Performer/Teacher

Text/Curriculum

Audience/ Students

Performance/ Learning Space

Assessment

c h a o s a n d c o m p l e x i t y

Self-organisation

Paradox

Emergence

The edge of

chaos

operating on the edge of chaos

STASIS

Close to certainty

Close to agreement

Far from agreement

Far from certainty(Based on Stacey 2000)

zone of complexity

CHAOSEdge of Chaos

– tension AND

creativity

Simple

Complicated

Complex

Students

Failure

Finance

Staff

RealityHEI

AdministrationResits Paperwork

generator

Learning &Teaching

Understanding

Research

ExternalExaminer

GraduatesStress

Knowledge

Skills

Students

Failure

Finance

Staff

RealityHEI

AdministrationResits Paperwork

generator

Learning &Teaching

Understanding

Research

ExternalExaminer

GraduatesStress

Knowledge

Skills

Closed system

Knowledge as external and transmitted

Predictability

Linear causality

Deterministic

the problem• The more assessment involves qualitative information, the

more subjectivity is involved• Stricter assessment criteria + more structured and proscribed

content = improved reliability• BUT…..• above would “obliterate the essence of qualitative assessment

in terms of flexibility, personal orientation and authenticity”.

Driessen et al (2005) Medical Education 39: 214–220

•Positivist

AimExplanation, control

KnowledgeObjective, measurable, value-free, universal, generalisable, external, quantifiable, can be transmitted and

acquired

Teacher/Researcher Expert, independent

CriteriaRigour via ‘holy trinity’ of validity, reliability and generalisability

There’s a reality ‘out there’ that can be studied, captured and understood

•Interpretive

AimUnderstanding

KnowledgeSubjective, contextualised, value-dependent, multiple-

realities

Teacher/Researcher Independent participant

Criteria‘Rigour’ (?) via credibility, transferability, dependability,confirmability

‘Truth’ is a matter of consensus amongst informed and sophisticated constructors, not of correspondence with an objective reality.

• Interpretive

‘Facts’ have no meaning except within some value framework; hence there cannot be an ‘objective’ assessment of any proposition.

Phenomena can be understood only within the context in which they are studied; findings from one context cannot be be generalised to another; neither problems nor their solutions can be generalised from one setting to another.

Note: the problem with ‘evidence-based practice’ , ‘best practice’, ‘transferable skills’, ‘benchmarks’, ‘national standards’, etc.

Therefore: Find and replace

• Replace

• Validity with Credibility, Coherence,

Consistency, Trustworthiness, Authenticity

• Certainty with Relativity

• Generalised Explanation with Local

Understanding

• Source Data with Empirical Materials

(which can become data…. and then

evidence)

• Is it true? with Does it work?

• Single Point Perspective with Multiple

Perspective

•The Triangle with the Crystal

The complex and

multidimensional nature of

learning cannot be captured

effectively and

comprehensively by any

single instrument or

analytical procedure.

The complex and

multidimensional nature of

learning cannot be captured

effectively and

comprehensively by any

single instrument or

analytical procedure.

The use of multiple methods

reflects an attempt to secure an

in-depth understanding of the

phenomenon in question.

However…..

from TRIANGULATION to CRYSTALLISATION

Triangulation carries with it the

image of a mathematical

procedure adding rigour and

discipline in one sense, but in

another sense restricting the

research to one of scientific

method and a positivist framework

where variables are few and can

be controlled or manipulated.

(Chien, 2004)

A

BC

P

Ad sedere / to sit down

together

"You've got to involve students

actively, not just view them as

objects of assessment, but as

agents of assessment. This can

be done in many ways. One is

that you ask students

systematically what they have

learned. It's a simple idea; it's

rarely done.....You find that

students say some remarkable

things.”

Walt Haney, Professor of Education,

Center for the Study of Testing, Evaluation and Educational

Policy, Boston, USA.

negotiating assessment

negotiating assessment

Presentation and Performance: Negotiated assessment

• Students engaged in creative practice work at different levels AND different ways.

• The products they create will be different, as will the processes and methods utilised, and the disciplines they represent

• That assessment should operate and be perceived as an integral part of the learning process rather than 'bolted-on' to the end of that process.

• That the form, content and implementation of the assessment process should be commensurable with the discourse and practices of the field

negotiating assessment

Six assessment fields:

• Presentation/Production i.e. the finished product presented to an audience

• Process i.e. the journey that led to the product

• Idea i.e. the ideas that informed both the process and the product.

• Technical i.e. the quality and utility of the technical features of the product and the skills with which they were assembled and/or operated

• Documentation i.e. research, design, sketches, planning, evaluation, analyses, portfolio, etc.

• Interview i.e. the student's ability to articulate their understanding, utilisation and application and use of any of the above.

negotiating assessment

• Learning contract – negotiated

• Regular meetings/tutorials

• Assessors see the performance/presentation + student compiles ‘portfolio of evidence’

• Assessment tutorial c. 30 min; individual / group + at least 2 assessors

• Work through the criteria - moving upwards

• Reach a point of maximum information, optimal achievement

• Agree a grade band

• Sense of ‘completeness’, ‘accomplishment

• A learning experience for all concerned

negotiating assessment

?p.kleiman@lancaster.ac.uk