Assessment of Higher Education

Learning Outcomes – What comes

after the AHELO-Project?

Professor Hamish Coates

hamishc@unimelb.edu.au

OECD

AHELO

Stocktake of

initiatives

Optimal

strategies

Making

progress that

counts

OECD

AHELO

Are valid and reliable comparisons of learning outcomes scientifically and practically feasible?

Conceived ~2005, announced 06, scoping 06-09, work 10-13

An international feasibility study – not a ‘pilot’

Testing 23000 ‘bachelor degree’ students, with data from 5000 faculty, 25 national managers, 250 institutional coordinators, 17 policymakers, and hundreds of advisors and stakeholders

Designed, developed, adapted, translated validated and evaluated frameworks, tests, infrastructure and processes, etc.

HEI reports (not for systems), along with international reports and materials, and recommendations for a main study

Linguistically diverse… Arabic Dutch English Finish Flemish Italian Japanese Korean Norwegian Russian Slovak Spanish

• Outcome specification

• Document analysis

• Consultation

• Synthesis, review

Framework creation

• Gather existing materials

• Item workshops

• Technical review

• Framework mapping

• Adaptation, translation

• Verification

Item creation

• Qualitative testing

• Quantitative testing

• Operationalisation

Instrument validation

Framework and item development

Tuning AHELO / QAA frameworks Curriculum documents Accreditation systems Discipline research

Authentic, hybrid item types ‘Above content’ reasoning

Non-engineering

Basic/Engineering sciences

Practice

Engineering processes

Analysis

Design

Problem context

Item situation

Engineering competence

Branch-specific

General

Branch-specific

General

Generic skills

Engineering

Engineering Assessment Framework

MCQ-oriented

Emphasis on constructed

response

Engineering Assessment

Robin King Australia (Chair) Giuliano Augusti Italy Mario Gomez Mexico Michael Hoffman Germany Kikuo Kishimoto Japan Johan Malmqvist Sweden Nobutoshi Masuda Japan Jim Melsa United States Engineering Expert Lueny Morell United States Group

Julian Fraillon (Director)

Constructed response

tasks

Representative sample of authentic and engaging Civil Engineering contexts

Photographs, diagrams and charts used to stimulate interest and minimise text

Students need to exercise components of Engineering competency

Can they think like an Engineer?

Competencies demonstrated in student responses

Disc

Multiple choice item development

Based on Japanese licensing examinations

Focused on Basic and Engineering Sciences

40 items selected, revised and internationalised

Items panelled and mapped

Source version

Translation 1

Translation 2

Final check

National review

Reconciliation

Verification: verifier (linguist) and domain specialist

Designed to maintain cross-national comparability of assessment materials

A holistic, robust and flexible approach, linked with item production and validation

Adaptations managed as a continuous process

Native speakers of target language trained to detect specific pitfalls

Economists or engineers who are speakers of target language

Translation, adaptation

and verification

Project implementation

Leadership: International project management and supporting national teams

Operationalisation: Preparing tests and context instruments for secure online delivery, and training makers

Sampling: Engaging institutions, sampling faculty and students, and quality assurance

Assessment: Supporting national training, managing testing, and managing marking quality

Reporting: Compiling data products, psychometric and statistical analysis, and system, institution and stakeholder reports

Evaluating: Scientific and practical feasibility, recommendations for full-scale study

Implementation architecture

Pre

pa

rati

on

Establish management centre

Translate and validate instruments

Engage institutions

Train institution coordinators

Prepare for testing Asse

ssm

en

t

Select students

Administer secure tests

Score responses

Verify and provide data

Re

po

rtin

g

Prepare multilevel benchmarking reports

Distribute reports to management centres

Interpretation for monitoring and improvement

0

10

20

30

40

50

60

70

80

90

100

50 150 250 350 450 550 650 750 850 950

Stu

de

nts

(c

um

ula

tiv

e p

er c

en

t)

Generic Skills Score

This institution Mean of all institutions All institutions

Defined engagement cycle established to support systems, institutions and students (…but not faculty)

Designed sustainable business models for such work (…but other models)

Frameworks and tests developed to support multidimensional test/context instrumentation (…but greater consultation)

Established methods for test design, development, translation/adaptation and validation

Defined operational workflow and quality control procedures required to support global testing

Forming awareness of how such work is positioned globally

Broad insights

Stocktake of

initiatives

CALOHEE

supertest

Learning gains

Taking stock of trends • Assessment innovation unfolding with broader

market/governance changes in higher education

• Developments at a range of levels: international, regional, national, cross-institutional, institutional

• Field-specific initiatives to ensure substantive relevance / authenticity

• Experimentation with bottom-up / top-down / mixed methodologies

• Innovation of new practices and technologies

• New capabilities and organisations developing

• New interests/resources emerging

Optimal

strategies

Building an optimal strategy

• Look at changing socio-political economies of higher

education

• Step back from stakeholders, technologies and histories

• Design optimised assessment strategy

– VCA + BPE

– Who (asst proximities) X what (asst processes)

– Account for levels (students, teachers, HEIs, nations)

and nature (substantive, technical, practice) of change

• Gather momentum around breakthrough progression

• Looking at “generalised learning outcomes assessment”

Proximities

Planning

• Governance

• Leadership

• Management

Development

• Mapping resources

• Specifying outcomes

• Selecting formats

• Drafting materials

• Qualitative review

• Quantitative review

• Material production

Implementation

• Designing administration

• Organising facilities

• Managing students

• Administering assessment

• Resolving problems

Analysis

• Collation of results

• Marking

• Data production

• Cross-validation

Reporting

• Production of grades

• Analysis and commenting

• Reporting

• Assessment review and improvement

Processes

• Strengthening collegial

forms of governance

• Adopting more

advanced management

• Improve assessment task design

• More collaborative and scalable

approaches

• Better material validation and

production

• Apply new technologies

• Increased independence,

collaboration and outsourcing

Sample

Generic options

Sample initiatives

‘Optimal’ scenarios

Student Academic Institution Industry Government Company

Planning

Development

Implementation

Analysis

Reporting

Unpacking ‘optimal’

1980s 1990s 2000s 2010s 2020s 2030s

Traditional ways

Faculty collaborations

Definitional projects

Proprietary products

Policy projects

Optimised approaches

Making

progress that

counts

1st wave: initiations (1990s)

2nd wave: foundations

(2000s)

3rd wave: experimen-

tations (2010s)

Nascent but maturing field

4th wave: consolid-

ations (2020s)

“technologies distil insights

from measurement science

into systems that make

good assessment easy for

time-poor and non-expert

academics” (Coates,

2016:12).

Working beyond barriers • Workforce changes

– Retirements – New internationalisations

– Industrial changes

• Changing institutions

– New leaders and leaderships – New strategic priorities

– Greater mission alignments

• New transparencies

– New open markets disclosures

– Academic work more visible

• New technologies

– New hybrid academic teams – New security solutions

– Specialist organisations – New seamless items

Assessment of Higher Education

Learning Outcomes – What comes

after the AHELO-Project?

Professor Hamish Coates

hamishc@unimelb.edu.au