validation and moderation workshop session 1
DESCRIPTION
TRANSCRIPT
Good morning
June 2013
Validation and
Moderation Workshop
Workshop overview
The quality of learning – what do we assess ?
Assessment is ongoing – when do we assess ?
Approaches to assessment – how do we assess ?
What is Validation and ModerationValidation is a quality review process of assessment. • producing valid, reliable, sufficient, current and authentic evidence • enabling reasonable judgements to be made • reviewing and making recommendations for future improvements
Moderation is the process of bringing assessment judgements and standards into alignment. • ensuring the same standards are applied to judge the quality of
performance (i.e. the assessment decision making rules)• including administration, recording and reporting requirements• it is an active process
Validation versus moderation • have been used interchangeably in the VET sector; and whilst each are
based on similar processes, there are a number of distinctive features.
Features Validation Moderation
Assessment Quality Management Type
Quality Review Quality Control
Primary Purpose Continuous improvement Bring judgements and standards into alignment.
Timing On-going Prior to the finalisation of candidate results
Focus Assessment Tools and Candidate Evidence (including assessor judgements) (desirable only)
Assessment tools and Candidate Evidence, including assessor judgements (mandatory)
Type of Approaches
Assessor Partnerships Consensus Meetings External (validators or panels)
Consensus Meetings External (moderators or panels) Statistical
Outcomes Recommendations for future improvements
Recommendations for future Improvements and adjustments to assessor judgements (if required)
What is Assessment?• purposeful process of systematically gathering, interpreting,
recording and communicating to stakeholders, information on student performance.
• assessment for learning occurs when teachers use inferences about student progress to inform their teaching (formative)
• assessment as learning occurs when students reflect on and monitor their progress to inform their future learning goals (formative)
• assessment of learning occurs when teachers use evidence of student learning to make judgements on student achievement against goals and standards (summative)
Why participate in validation and moderation processes?• to ensure assessments align with current industry practice
and benchmarks remain within acceptable limits
• to ensure the quality and consistency of assessment
• confirms the consistency of assessor/s judgement
• to gain a comprehensive understanding of the depth of students’ knowledge and skills
• VET Quality Framework compliance
Outcomes of validation
Recommendations for future improvements:
• Review of context and conditions for the assessment
• Task/s to be administered to the candidates
• Administration instructions
• Criteria used for judging the quality of performance (e.g. the decision making rules, evidence requirements etc.)
• Guidelines for making reasonable adjustments to ensure that the expected standard of performance specified within the Unit(s) of Competency has not been altered
• Recording and reporting requirements
Outcomes of moderation
• recommendations for future improvement and adjustments to assessor judgements (if required)
• recommendations for improvement to the assessment tools • adjusting the results of a specific cohort of candidates prior to
the finalisation of results and• requesting copies of final candidate assessment results in
accordance with recommended actions.
Approaches to Validate and Moderate Assessments
Statistical Requires some form of common assessment
Strength strongest form of quality control
Weakness lacks face validity, may have limited content validity
External Site visits
Strength ‘expert’ interpretations of standards/ advice to RTO/ assessors on assessment approaches and procedures / observe actual assessment processes in real time
Weakness can be expensive and there is less control than a statistical approach
Approaches to Validate and Moderate Assessments
Partnerships Sharing, discussing and/or reviewing one another’s tools and/or judgements
Strength Low costs, personally empowering, non-threatening , easily organised
Weakness Potential to reinforce misconceptions and mistakes
Consensus Reviewing own & colleagues assessment tools and judgements as a group
Strength professional development, networking, promotes collegiality and sharing
Weakness less quality control than external and statistical approaches as they can also be influenced by local values and expectations and requires a culture of sharing
Systematic ValidationIndicators
is there a plan for assessment validation (including validation of RPL assessment) in place?
• are units of competency to be validated over a set period of time• provide dates for proposed validation activities• include details about who will participate in assessment validation• include a strategy to ensure that all relevant staff are involved• identify what processes and materials for implementing and recording the outcomes of
assessment validation
does the RTO have validation materials (policy, procedure, forms) in place that cause participants to engage effectively in validation?
does the RTO have a process for monitoring the action taken as a result of validation?
does the RTO have a process and plan in place for reviewing the effectiveness of assessment validation?
Assessment Quality Management
Quality Assurance (Input approach)
Quality Control(Outcome approach)
Quality Review(Retrospective approach)
• Industry competency standards as the benchmarks of varying levels of performances
• National assessment principles
• Minimum qualifications for assessors
• Standardisation of reporting formats
• Assessment tool banks• Common assessment tasks• Exemplar assessment tools• Professional development
programs/workshops for assessors
Examples include:Moderation in which adjustments to assessor judgements are made to overcome differences in the difficulty of the assessment tool and/or severity of the judgement.
Examples Include:• Monitoring and auditing
of organisation• Review and validation of
assessment tools, processes and outcomes to identify future improvements.
• Follow-up surveys with key stakeholders (e.g., student destination surveys, employer feedback on how well the assessment outcomes predicted workplace performance).
Key Stages in developing assessment tools
• identify and describe the purposes for the assessment
• identify the assessment information that can be used as evidence of competence/learning
• identify a range of possible methods that might be used to collect assessment information
• define the contexts for interpreting assessment information in ways that are meaningful for both assessor and candidate
• determine the decision making rules
• define procedures for coding and recording assessment information
• identify stakeholders in the assessment and define their reporting needs.
Essential Characteristics –Assessment Tool
An assessment tool includes the following components: • the context and conditions for the assessment• the tasks to be administered to the candidate• an outline of the evidence to be gathered from the candidate• the evidence criteria used to judge the quality of performance
(i.e., the assessment decision making rules); as well as the • the administration, recording and reporting requirements.
Ideal Characteristics• the context• competency mapping• the information to be provided to the candidate• the evidence to be collected from the candidate• decision making rules• range and conditions• materials/resources required• assessor intervention• reasonable adjustments • validity evidence• reliability evidence • recording requirements• reporting requirements
Where do you start the process?
Training and Assessment StrategyOutlines the overall validation and moderation processes that will be used for the qualification. Includes a description of the types of documentation that will be kept as evidence.
Validation and Moderation ScheduleProvides a schedule and description of processes for when the various validation and moderation activities will occur. • Industry consultation of the TAS• Industry consultation of assessment tools• Validation activities with other assessors• Moderation activities with other assessors• Internal moderation of judgements• Internal review & Internal audit
Record of Industry Consultation• Industry consultation of TAS• Industry consultation of assessment tools
Provides documented evidence of validation activities with relevant industry stakeholders. This needs to be signed by the industry stakeholder involved in the process.
Record of Trainer/Assessor Validation & Moderation• Validation and Moderation activities with other assessors
Provides documented evidence of validation and moderation activities with assessors not involved in the development of the assessment tools.
Record of Internal Moderation• Internal moderation of judgements
Provides documented evidence of internal moderation where more than one assessor is involved in making judgements for students enrolled in the qualification.
Internal Audit ReportIncludes review of the validation processes and the application of these with documented evidence and the Internal Review Report.
The Review & Audit process lead into continuous improvement in relation to the TAS and other documentation and processes. All documentation generated from the Validation & Moderation process leads into the Internal Audit Process.
Internal Review ReportIncludes review of the validation processes and the application of these with the documented evidence provided (described below).
Industry Validation of TAS
Who are my students?
What does competent look like in the current workplace?
How will I know someone is competent?
What evidence do I need to collect to confidently judge someone as competent?
What methods will I use to collect that evidence?
What assessment tools will I need to collect that evidence?
Will I have enough evidence?
Competency Mapping
Step 1: Unpack the unit of competency to identify its critical components.
Step 2: For each assessment method, list the tasks to be performed by the candidate.
Step 3: For each assessment method, map the critical components of the unit to each assessment task.
Level of specificity in mapping – Risk Assessment
Risk can be determined by consideration of:• safety (e.g. potential danger to clients from an incorrect
judgement)• purpose and use of the outcomes (e.g. selection purposes)• human capacity (e.g. level of expertise and experience of the
assessors)• contextual (e.g. changes in technology, workplace processes,
legislation, licensing requirements and/or training packages)
Decision Making Rules
The rules to be used to:• check the quality of the evidence (i.e. the rules of
evidence)• judge how well the candidate performed on the task
according to the standard expected• interpret evidence from multiple sources to make an
overall judgement
Reasonable Adjustments• this section of the assessment tool should describe the
guidelines for making reasonable adjustments to the way in which evidence of performance is gathered without altering the expected performance standards
• in some cases the assessment tasks will need to be adjusted or alternative tasks developed. A simple example would be responding to questions orally rather than written.
• however, remember that the adjustments must not alter the expected standard of performance specified within the unit/s of competency.
Simulated assessment
• for the purposes of assessment, a simulated workplace is one in which all of the required skills are performed with respect to the provision of paid services to an employer or the public can be demonstrated as though the business was actually operating.
• in order to be valid and reliable, the simulation must closely resemble what occurs in a real work environment.
• the simulated workplace should involve a range of activities that reflect real work experience. it should allow the performance of all of the required skills and demonstration of the required knowledge.
Activities• In your groups discuss what input employers could provide to
develop valid assessment tools and processes to employ a person with a Cert IV in Business .
• Note down 2/3 questions you could ask employers and how the responses will inform the development or review of assessment tools and/or processes.
Questions
References• MAXWELL, G S 2001, Moderation of Assessment in Vocational Education and Training, University of Queensland, Australia.• (Wheeler, 1993). • www.nqc.tvetaustralia.com.au • http://policy.ballarat.edu.au/tafe/validation/• http://www.nssc.natese.gov.au/nqc_archive/nqc_publications/publications/assessment• http://www.velgtraining.com/
Thank you
Cecilia SorensenProject Manager | Sarina Russo InstituteP: 07 3308 2200
F: 07 3221 5304E: [email protected]
W: www.sri.edu.au