ASSESSMENT 2009

IMPACT ASSESSMENT – ACTION RESEARCH –

PLEs – OLEs – CGOs – Kirkpatrick – ACCESS –

Bertelsmann – Schön – Reflective Practice – Societal Impact – Personal Impact – Operative Attention

SCARED OR INSPIRED??

TIME TO REALLY ASSESS VALUE…

THE OBLIGATORY JOKE How many Academic Administrators does it

take to change a light bulb?

CHANGE !!??

GENERATION PONG – TIME TO EVOLVE

THESE NEW-FANGLED IDEAS… 1998 Bertelsmann report:

• Cognitive tools empower learners to design their own representations of knowledge rather than absorbing representations preconceived by others.

• Cognitive tools can be used to support the deep reflective thinking that is necessary for meaningful learning.

• Ideally, tasks or problems for the application of cognitive tools will be situated in realistic contexts with results that are personally meaningful for learners.

• Using multimedia programs as cognitive tools engages many skills in learners such as: project management skills, research skills, organization and representation skills, presentation skills, and reflection skills.

LEARNERS AS DESIGNERS

Spoehr (1993) reports that students who build and use hypermedia develop a proficiency in organizing knowledge about a subject in a more expert-like fashion. They are able to represent multiple linkages between ideas and organize concepts into meaningful clusters.

Salomon et al. (1991) - Educators should empower learners with cognitive tools and assess their abilities in conjunction with the use of these tools. Such a development will entail a new conception of ability as an intellectual partnership between learners and the tools they use.

ASKING THE RIGHT QUESTIONS.

Tools –

Questions – tbd

START with Action research : a reflective process of progressive problem solving led by individuals working with others in teams or as part of a "community of practice" to improve the way they address issues and solve problems.

Dr. Joan McMahon mcmahon@towson.edu John Sener – jsener@senorlearning.com

SO WHAT?

Impact Assessment basically asks the question “So what?” After anything you do or try, simply asking that question gets you closer to the “impact” issue. –

Dr Joan McMahon

KIRKPATRICK’S EVALUATION OF T+L

Kirkpatrick's 1975 book Evaluating Training Programs defined his originally published ideas of 1959, increasing awareness of them, so that his theory has now become arguably the most widely used and popular model for the evaluation of training and learning.

Kirkpatrick's four-level model is considered an industry standard across the HR and training communities.

The four levels of Kirkpatrick's evaluation model essentially measure:

KIRKPATRICK’S FOUR LEVELS L1 – Teaching or reaction of student - what they

thought and felt about the training L2 - Learning - the resulting increase in

knowledge or capability L3 – Results - extent of behavior and capability

improvement and implementation/application L4 – Impact - the effects on the business or

environment resulting from the trainee's performance

All these measures are recommended for full and meaningful evaluation of learning in organizations, although their application broadly increases in complexity, and usually cost, through the levels from level L1-4.

WHAT QUESTIONS CAN YOU ASK TO

GET TO “IMPACT”?

T = I taught it.

TL = I taught it but did they learn it?

TLR = I taught it but did they learn it AND RETAIN IT?

TLRI = I taught it but did they learn it and retain it AND SO WHAT?

EXAMPLES – BETTER QUESTIONS!!

The teacher encouraged interaction – Yes / No 1 2 3 4 5

The teacher was tasked to encourage interaction among students – what would have motivated you personally to interact more with your peers?

Did you meet the Learning Goals or the course as in the syllabus?

What has been the impact of this course in your work or personal life? What new skills have you used to affect change?

THE EVOLUTION OF ASSESSMENT

This slide attributed to Bobby Elliot - Scottish Qualifications Authoritybobby.elliott@sqa.org.uk

ASSESSMENT 1.0

Paper-based Classroom based Formalised Synchronised Controlled Industrialised

Changed little since early 20th Century

This slide attributed to Bobby Elliot - Scottish Qualifications Authoritybobby.elliott@sqa.org.uk

ASSESSMENT 1.5

Computer-based assessment Characteristics

E-testing E-portfolios Simulations

Embedded in most VLEs Stand-alone systems Familiar to students and teachers

This slide attributed to Bobby Elliot - Scottish Qualifications Authoritybobby.elliott@sqa.org.uk

STUDENT PERCEPTIONS (OF 1.0/1.5)

Artificial and contrived

Something that is done to them

Doesn’t measure anything important

(It’s a) Hurdle to be jumpedNot part of their learningSole purpose of their learning

This slide attributed to Bobby Elliot - Scottish Qualifications Authoritybobby.elliott@sqa.org.uk

WEB 2.0

User-generated content Architecture of participation Network effects Openness Data on an epic scale Power of the crowd

ASSESSMENT 2.0 NEEDS TO BE…

Authentic Personalised Negotiated Collaborative Recognising existing skills

WEB 2.0 SERVICES

Web Service Example Functions

Personal Portal Netvibes Evidence Organization

E-mail Gmail Evidence Storage

Blog Wordpress E-portfolio, diary, log-book

RSS Bloglines Evidence Discovery

Social Bookmarking Del.icio.us Evidence Capture

Instant Messaging MSN Discussion Groups / Collaboration

VOIP Skype Audio Evidence /

Wiki Wikispaces Collaboration / Groupwork

Video Upload YouTube Create / Store Video Evidence

Social Networking Facebook Evidence discovery / Collaboration

This slide attributed to Bobby Elliot - Scottish Qualifications Authoritybobby.elliott@sqa.org.uk

REASONS TO OPEN THINGS UP

“Because you’re pouring money into a black hole that students don’t like, which is unnatural to them, which can’t possibly keep up with developments on the Web, and which is little more than a comfort blanket to teachers who can’t or won’t embrace the 21st Century.”

Bobby Elliot

A LITTLE MORE CONSERVATIVELY

It’s crude but it’s an important evolutionary step

Not every student is a digital native Not every teacher can use Web 2.0

“I can’t get my staff to use the quiz in Moodle so what chance is there that they’ll embrace Web 2.0?”

MELD THE RIGHT QUESTIONS TO THE APPROPRIATE (STUDENT-SELECTED MEDIA)

Ask the right questions – Open vs Closed / Fat vs. Skinny / IMPACT Assessment = SO WHAT?!

Review objectives to encourage multiple means of achieving success.

Even when you are in a Blackboard/Moodle (confined) world, be open to a variety of deliverables, and look to nurture that creativity.

Accept OLEs (Online Learning Environment) / PLEs (Personal Learning Environment) – Voicethread / Skitch …

(Impactful) QUESTIONS??