consistency of assessment (validation) webinar – part 2 renae guthridge wa training institute...

Consistency of Assessment(Validation)

Webinar – Part 2

Renae GuthridgeWA Training Institute (WATI)

RQI - 321

What three things do you recall about yesterday’s webinar?

What two questions do you have about yesterday’s?

What was one interesting point about yesterday’s webinar?

Foxes and a Vixen

Principles of Assessment

Rules of Evidence

FourVeryRampantFoxes

ASatisfiedCunningVixen

Ensuring Validity - Unpacking the Unit of Competence

Mandatory unitrequirements include:• Pre-requisites and/or co-

requisites• Elements • Required Skills• Required Knowledge• Range Statement (‘must’)• Critical Aspects of

Evidence• Other “must”

requirements

Mandatory unit requirements include:

• Pre-requisite(s)• Elements • Foundation Skills• Performance Evidence • Knowledge Evidence

These can be listed in a Unit Requirements List…

Unit Requirements List

Unit of competency: SHOP123A Review stock

Mandatory unit requirementsElements of competency1. Identify stock requirements2. List stock in store3. List stock shortages4. Prepare stock order formsFoundation skills1. Literacy skills (reading and writing)2. Numeracy skillsPerformance evidence1. Accurate list of stock on display2. Accurate list of stock in store-room3. Accurate list of stock to order4. Two categories of stockKnowledge evidence1. Organisation procedures2. Range of stock management processesPre-requisites:SHOP101A


Mandatory unit requirementsElements of competency1. Identify stock requirements2. List stock in store3. List stock shortages4. Prepare stock order formsRange Statement (musts)1. Stock on display2. Stock in storeroomCritical Aspects of EvidenceCAE 1 Accuracy and timelinessCAE 2 Full review of stockRequired SkillsRS 1 Communicates with staffRS 2 Correct forms selectedRS 3 Correct calculationsRequired Knowledge (See list)Pre-requisites: SHOP101AOther (specify): Two contexts

Old Unit Structure New Unit Structure

Ensuring Validity -Identifying Mandatory Unit Requirements

• Highlight all mandatory requirements in a unit of competency.

• Prepare a list of all the mandatory requirements.

• Validate your list with a colleague by comparing lists.

• Prepare a flexible and fair evidence-gathering plan.

• Validate your plan with a colleague.

Planning to Gather Evidence

The Unit Requirements List can now be extended to show the relationship between the unit requirements and evidence sources.


Mandatory unit requirementsSources of evidence

WP1 WP2 SIM1 SIM2 Q/AElements of competency 1. Identify stock requirements 2. List stock in store 3. List stock shortages 4. Prepare stock order forms Range Statement 1. Stock on display 2. Stock in storeroom Critical Aspects of Evidence CAE 1 Accuracy and timeliness CAE 2 Full review of stock Required Skills RS 1 Communicates with staff RS 2 Correct forms selected RS 3 Correct calculations Required Knowledge (see list) Pre-requisites: SHOP101A Other (specify): Two contexts

WP1: Context of workplace assessment

WP2: Context of workplace assessment

SIM1: Context of simulation

SIM2: Context of simulation

Assessment Planning Matrix

Shade the cells that don’t apply.

The blank cells in a column provide a space to record evidence later on when it has been gathered.

Developing Evidence-gathering Tools - Assessing Required Knowledge / Knowledge Evidence

Assessing ‘know-how’…

…tell me…

…written or oral short answer tests.

Ensuring Validity, Reliability and Sufficiency -Assessing Required Knowledge / Knowledge Evidence

Is the AQF level of the questions appropriate?

Is the context and content of the questions relevant to the unit?

Do the questions extend beyond the elements and demonstrate transfer?

Is ALL required knowledge covered?

If other knowledge is covered, has it been distinguished from the required knowledge?

Is ALL required knowledge demonstrated before signing off on the unit record?

Developing Evidence-Gathering Tools -Assessing Elements and Required Skills / Performance Evidence

Assessing can do…

…show me…

…observation in the workplace or a simulation of a workplace.

Unit of competency: SHOP123A Review stock Name of student: Joe Bloggs

Mandatory unit requirements Performance Criteria Comments Result

E1. Identify stock requirements 1.1 Contextualised for third party observer.

The observer would write a narrative of what was observed for the assessor to review.

1.21.31.4

E2. List stock in store 2.1

2.22.3

RK1 Stock on display Display stock not reviewed N/a

RK2 Stock in storeroom CAE1 Accuracy and timeliness Review completed to company standards

CAE2 Full review of stock All storeroom stock reviewed

RS1 Communicates with staff RS2 Correct forms selected Assessment context: Two different contexts

requiredFrozen foods

Observer’s comments:

Observer’s name: M. Ployer Observer’s position: SupervisorSignature and date: M. Ployer 5/8/10 Student signature and date: Joe Bloggs 5/8/10

Workplace Observation Tool 1 (WP1) (3rd Party Observer)



E3. List stock shortages 3.1

3.2E4. Prepare stock order forms 4.1

4.24.3

RK1 Stock on display RK2 Stock in storeroom Storeroom stock not reviewed N/aCAE1 Accuracy and timeliness Met our company standards

CAE2 Full review of stock Full range of display stock reviewed

RS1 Communicates with staff Did not ask for advice or report problems to supervisor

RS2 Correct forms selected RS3 Correct calculations Assessment context:

Two different contexts required

Dry goods


Observer’s name: Sam Smith Observer’s position: RTO AssessorSignature and date: Sam Smith 7/9/10 Student signature and date: Joe Bloggs 7/9/10

Workplace Observation Tool 2 (WP2) (RTO Observer)



E1. Identify stock requirements 1.1

1.21.3 1.4

E2. List stock in store 2.1

2.22.3

E3. List stock shortages 3.1 Incorrect stock list used.

3.2E4. Prepare stock order forms 4.1

4.24.3

RK1 Stock on display

RK2 Stock in storeroom N/aCAE1 Accuracy and timeliness CAE2 Full review of stock RS2 Correct forms selected RS3 Correct calculations Assessment context: Two different contexts

requiredDry goods



Simulation Observation Tool (SIM)


Required knowledge

itemsQuestions Verbatim answers or comments Correct Score Criterion Result

RK1. 1 2 2

3 RK2. 2

2 3 6

Hot water is mixed with soap powder to make a warm nourishing drink

7 8

RK3. 4 1 1 RK4. 5

4 3

9 10 11

13 Record the incorrect answer here ….

RK5. 12 1 1 RK6. 14

2 2 15

RK7. 16 2 2 17

18 Overall unit result Extra knowledge Questions EK1. 19

EK2.20

Sandals can be worn in the cool room

EK3. 21

EK4. 22


Question & Answer Tool (Q/A)

Ensuring Validity, Reliability and Sufficiency - Assessing Elements and Required Skills / Performance Evidence

Are clear instructions provided to the observer?

Are clear instructions provided to the candidate?

Are all four dimensions of competency (task skills, task management skills, contingency management skills, and job/role environment skills) met?

If third-party evidence-gatherers are used, have the performance criteria been contextualised by writing them in familiar language (plain English) and with examples, and has space been provided for narrative to explain the or assigned to each criterion?

Recording evidence Use the Assessment Planning Matrix as a

record of individual evidence by adding the name of the student and a ‘Result’ column and recording evidence from the tests and observation checklists into the blank cells.



WP1 WP2 SIM1 SIM2 Q/AElements of competency 1. Identify stock requirements 2. List stock in store 3. List stock shortages 4. Prepare stock order forms Range Statement 1. Stock on display N/a 2. Stock in storeroom N/a N/a N/a Critical Aspects of Evidence CAE 1 Accuracy and timeliness CAE 2 Full review of stock Required Skills RS 1 Communicates with staff RS 2 Correct forms selected RS 3 Correct calculations Required Knowledge (See list) Pre-requisites: SHOP101A Pre-requisite unit achieved in 2007 at XYZ CollegeOther (specify): Two contexts Frozen

foodsDry goods Dry goods Dry goods

WP1: Frozen foods in storeroom

WP2: Dry goods on display

SIM1: Dry goods on display

SIM2: Supplementary assessmentName of assessor: Sam Smith Student signature and date:Signature and date: Result reported date:

Assessment Record

Meeting the Rules of Evidence -Recording evidence of competency

Are checklists used to record evidence as it happens?

Is poor quality evidence discarded?

Is evidence gathered and collated from checklists?

Is evidence from different units in a cluster distributed to appropriate unit records?

Making the Assessment Judgement

The Assessment Record can now become a Judgement Tool showing which requirements had been met in the ‘Results’ column.



ResultWP1 WP2 SIM1 SIM2 Q/A

Elements of competency

1. Identify stock requirements 2. List stock in store 3. List stock shortages 4. Prepare stock order forms Range Statement

1. Stock on display N/a 2. Stock in storeroom N/a N/a N/a Critical Aspects of Evidence

CAE 1 Accuracy and timeliness CAE 2 Full review of stock Required Skills

RS 1 Communicates with staff RS 2 Correct forms selected RS 3 Correct calculations Required Knowledge (See list) Pre-requisites: SHOP101A Pre-requisite unit achieved in 2007 at XYZ College Other (specify): Two contexts Frozen

foodsDry goods Dry

goodsDry

goods

Result WP1: Frozen foods in storeroom



SIM2: Supplementary assessment

Name of assessor: Sam Smith Student signature and date: Joe Bloggs 6/11/15Signature and date: Sam Smith 6/11/15 Result reported date: 14/11/15

Judgement Tool

Meeting the Foxes and Vixen -Making the assessment judgement

Has sufficient evidence been gathered for each mandatory requirement?

Is the evidence authentic and current?

Have alternative arrangements been offered for candidates with specific needs?

Has each mandatory requirement been demonstrated?

Has the record been signed off by person(s) with assessor qualifications and vocational expertise?

Have other uses of the record been quarantined from the competency judgement?

Training and Assessment Plan

The Assessment Planning Matrix, Assessment Record and Judgement Tool can now morph into a Training Delivery and Assessment Plan by listing the learning resources for each unit (and any extra) requirements.


Mandatory unit requirementsLearning resources Sources of evidence

ResultSession Reference WP1 WP2 SIM1 SIM2 Q/A

Elements of competency 1. Identify stock requirements 1 - 3 pp 15 - 23 2. List stock in store 4 pp 37 - 39 3. List stock shortages 5 - 7 Handout

1

4. Prepare stock order forms 9 Handout 3

Range Statement 1. Stock on display 8 Handout

2N/a

2. Stock in storeroom 8 Handout 2

N/a N/a N/a

Critical Aspects of Evidence CAE 1 Accuracy and timeliness 4 p. 38 CAE 2 Full review of stock 4 p. 39 Required Skills RS 1 Communicates with staff 7 Handout

1

RS 2 Correct forms selected 8 Handout 2

RS 3 Correct calculations 7 Handout 1

Required Knowledge (See list) 1 - 9 All Pre-requisites: SHOP101A Student records Pre-requisite unit achieved in 2007 at XYZ College Other (specify): Two contexts 1 - 9 All Frozen foods Dry

goodsDry goods Dry

goods

Result WP1: Frozen foods in storeroom



SIM2: Supplementary assessment

Workplace or regulatory requirements: OH&S: Use of steps & cool room access

Name of assessor (Assessment): Sam Smith Name of Assessor (Vocational): Sam SmithStudent signature & date: Joe Bloggs

6/11/15Signature and date: Sam Smith 6/11/15 Signature and date: Sam Smith 6/11/15 Result reported date: 14/11/15

Training and Assessment Record

Thoughts…What do you think?Would you use this approach?What do you like / dislike about this approach?

Strategies for Systematically Validating Assessment Procedures

Review assessment strategies

Compare assessment evidence

Evaluate assessment outcomes

How Can We Review Assessment Strategies?

Reviewing involves the inspection of assessment processes and products to determine the ‘face validity’ of the assessment strategy…‘does it look right’?

What criteria could be used to guide the review of your assessment processes?

Review Criteria…

How Can We Compare Assessment Evidence?

Comparing assessment processes involves comparing assessment documentation and comparing assessment outcomes to determine content validity, reliability, consistency, and reproducibility of the assessment strategy.

What criteria could be used to guide the comparison of your assessment processes with those of another assessor?

Comparison Criteria…

How Can We Evaluate Assessment Outcomes?

Evaluating assessment processes involves gathering stakeholder feedback to determine the predictive validity, impact, effectiveness and credibility of the assessment strategy.

What criteria could be used to guide the evaluation of your assessment processes.

Evaluation Criteria…

How Can We Improve Assessment Processes?

The quality cycle

How can you follow this cycle to improve your assessment processes?

What Records of Validation Should We Keep and Why?

Records of Validation…Create a separate file for each unit of competency. Each file to contain:

• UoC;• Assessment plan;• Current assessment instruments and instructions;• Validation form;• A schedule for the administration of validation forms

showing past and future activities;• A summary of evidence gathered to date from each

validation source;• A summary of problems identified; and• A summary of actions taken to remedy them.

Before You Go…

Answer the following with a yes, no or maybe…1. I have learned something about validation

during these two sessions. 2. I will change some of the assessment

processes I currently use. 3. I am more confident about validating with

others and sharing my assessment processes.

Is there anything else you need to know about validation?

Thank you and watch out forFoxes and Vixens!

All pictures in these slides were taken from http://search.creativecommons.org/

consistency of assessment (validation) webinar – part 2 renae guthridge wa training institute...

Documents