develop competency-based assessment plans · 1. multiple-choice questions (mcq) multiple-choice...

- 1 -

Singapore Workforce Development Agency – Quality Assurance Division Develop Competency-Based Assessment Plans Version 1.1 (14 October 2012)

Develop Competency-Based Assessment Plans

- 2 -


Table of Contents

Message from the Director of Quality Assurance Division ............................................................. 5

Learner Profile ............................................................................................................................. 6

Reflection on Learner Profiles....................................................................................................... 7

Will Technology-assisted, Work-based, Continuous Assessments Work? ......................................... 8

Using Social Media-based E-Portfolios in Competency-based Assessment ....................................... 9

What is Competency? ................................................................................................................ 11

Packaging of Competency Standard (CS) and Curriculum, Training and Assessment Guide (CTAG) 11

Competency Standard Template ................................................................................................. 12

Key Changes to Competency Standard Template ........................................................................ 13

Revised Competency Standard Template .................................................................................... 14

Curriculum, Training and Assessment Guide (CTAG) .................................................................... 14

Curriculum, Training and Assessment Guide (CTAG) Components ................................................ 15

Key Changes to CTAG Template .................................................................................................. 15

Competency-based Assessment ................................................................................................. 17

Requirements of Competency-based Assessment ....................................................................... 17

Types of Assessment .................................................................................................................. 18

Commonly Used Assessment Methods ....................................................................................... 19

Knowledge-based Assessment Methods .......................................................................................... 19

1. Multiple-Choice Questions (MCQ) ........................................................................................ 19

2. Written Assessment (Short Answers) ................................................................................... 19

3. Projects ................................................................................................................................. 20

4. Portfolio ................................................................................................................................ 20

5. Written Assessment (Reports) .............................................................................................. 21

6. Oral Questions ...................................................................................................................... 21

Performance-based Methods ........................................................................................................... 22

7. Workplace Performance ....................................................................................................... 22

8. Role-Play / Simulation ........................................................................................................... 22

Attitudinal Evaluation Methods ........................................................................................................ 23

9. Observations ......................................................................................................................... 23

- 3 -


Principles of Assessment (POA) .................................................................................................. 24

Rules of Evidence (ROE) ............................................................................................................. 24

Adult Educator (AE) Qualifications Requirements and Timelines for WSQ Approved Training Organisations (ATO) ................................................................................................................... 25

How do These Changes to Competency Standard (CS) and Curriculum, Training and Assessment Guide (CTAG) Affect my Assessment Plan Design? ...................................................................... 26

What is an Assessment Plan? ..................................................................................................... 27

Components of Assessment Plans .............................................................................................. 27

WSQ Requirements in Relation to Assessment Plans ................................................................... 28

Common Shortcomings in Assessment Plan Design ..................................................................... 29

Example 1: Inappropriate Development of Assessment Tools ......................................................... 29

Example 2: Assessment Criteria do not Address the Performance Statement / Performance Criteria

Sufficiently ........................................................................................................................................ 30

Example 3: Incomplete Instructions in Assessment Specifications .................................................. 31

Example 4: Conflicting Instructions in Assessment Plan ................................................................... 33

Example 5: Difficulty in Gathering Evidence ..................................................................................... 34

Example 6: MCQ Questions with “Give-away” Answers .................................................................. 35

Other Shortcomings in Assessment Plans ......................................................................................... 36

Selection of Assessment Methods .............................................................................................. 37

General Good Practices in Assessment Plan Development ........................................................... 37

Assessment Plan Validation ........................................................................................................ 38

Assessment Plan Validation Checklist ......................................................................................... 39

POA / ROE Validation Checklist .................................................................................................. 40

WSQ Curriculum Development and Pre-Accreditation Checklist .................................................. 45

References ................................................................................................................................. 50

Useful Websites ......................................................................................................................... 51

Version Control Record .............................................................................................................. 52

- 4 -


BLANK PAGE

- 5 -


Message from the Director of Quality Assurance Division

Dear WSQ Developers,

The Quality Assurance Division (QAD) conducted

a survey of the WSQ Approved Training

Organisations (ATO) in March 2012. It was

found that most ATOs indicated a need to have

a better understanding of how to develop

assessment plans. This further supports QAD’s

audit observations that ATOs need to improve

the design of their assessment plans.

These notes are purposefully designed for

developers who are involved in the design and

development of competency-based assessment

(CBA) for the Singapore Workforce Skills

Qualifications (WSQ) System. WSQ is a national credentialing system that

trains, develops, assesses and recognises adult workers for competencies they

need to stay employable. Thus, assessment is the integral part of WSQ as it is

used to assess the learning gain and competencies of the workers.

I hope these notes will be beneficial to your future development of

competency-based assessment plans.

All the best in your future developments!

Sharon Tan (Ms)

Director

Quality Assurance Division

- 6 -


Learner Profile

Learner profile is a set of information collected on how learner learns most

effectively. This information may be collected through observations, surveys,

questionnaires, application forms, interviews, etc. It may include information

like:

Age

Barriers to learning (e.g. institutional, situational, psychosocial, etc)

Educational level

Employability Skills (ES) Level

Experience (e.g. work experience, life experience, etc)

Motivations to learn (e.g. job, self-actualisation, etc)

Orientation to learning (i.e. immediacy of application for problem solving,

response to societal expectations, etc)

Readiness to learn

Self concept (i.e. self-directed, dependent, etc)

______________________________________________

______________________________________________

______________________________________________

______________________________________________

______________________________________________

- 7 -


Reflection on Learner Profiles

Draw up the profile of learners who attend your programmes. Pick the

appropriate characteristics.

Given your learner profile, how do we design assessments to match learner needs and strengths?

Low-Skilled Elderly

Non-English Speaking

Physically challenged

Less confident

Fearful of training &

assessments

Tech-Savvy Gen Y

24/7 connected

Multi-tasking

Easily bored

Articulate and confident

PMEs

Wealth of experience

High expectations

Questioning

Intellectual

Learner Profile

- 8 -


Will Technology-assisted, Work-based, Continuous Assessments Work?

Discuss with your partner on the appropriateness of the three assessment

approaches for your learners. Tick the relevant boxes if the approaches provide

valid, sufficient, current and authentic evidence.

RULES OF EVIDENCE (ROE)

Evidence

Criteria

Approach 1

(ACTA V5)

Approach 2

(WTP)

Approach 3

(DACE) Comments

Validity of

evidence

Sufficiency of

evidence

Currency of

evidence

Authenticity of

evidence

Characteristics

of Each

Approach

Continuous

assessment

Blogs

In-class

performance

Workplace

evidence

Live

observation

Video clip of

presentation

Skype to

cover gaps

Possible Takeaway

I can apply the following ideas to modify the practice in my organisation:

- 9 -


Using Social Media-based E-Portfolios in Competency-based Assessment

A pilot project by Institute for Adult Learning (Lee Wee Chee)

Background

IAL recently conducted a pilot run of a new design for four modules of

Advanced Certificate in Training and Assessment (ACTA). One of the new

methods that were tested was the use of social-media based e-portfolios for

continuous, formative and summative assessment.

Assessment Process

Learners started collecting evidence (artefacts) of learning and competency

from the start of the course. These took the form of pictures, reflective blog

entries capturing their thoughts on the application of learning, and videos of

them practising their delivery of training and assessment. Using a blog host

(Wordpress) and video host (YouTube), learners published their entries

frequently, allowing assessors, trainers and fellow learners to give feedback on

performance, comment on entries, and identify gaps to cover.

An Alternative Expression of Competency-based Assessment

This project built social constructivism, reflection and personalised

contextualisation into competency-based assessment, allowing assessors to

have a holistic, developmental view of the learner’s journey towards

competence.

Assessment Framework

Based on a portfolio assessment framework developed by Cleveland Clinic

Lerner College of Medicine for medical students1, IAL adapted it to create a

similar framework that includes e-portfolio and social media elements.

- 10 -


Evidence Gathering Tools

Smartphones, tablets

With photo-taking and

video-capturing

capability

Evidence Portfolio

e-Portfolio using social

media like blog hosts

(e.g. Wordpress) and

video hosts (e.g.

YouTube)

1Dannefer, E. F., and Henson, L. C. (2007). The portfolio approach to competency-based assessment at the Cleveland Clinic

Lerner College of Medicine. Academic medicine, 82(5), 493-501.

Summative Assessment

Learners meet with assessor to review all evidence for decision

on competence

Formative Assessment

Learners meet with assessor to review evidence presented

so far to determine performance gaps and

actions to take to close the gaps

Informal Feedback

Trainers, assessors and fellow learners give feedback on blog entries for

developmental purposes

Evidence Database

Application to own workplace practice: Reflective entries on applicability of learning to workplace

Teaching and assessment skills: Videos of skills practices in class or from the workplace + reflection

Construction of knowledge: Photographs and entries of own learning + reflection

Professional development: Reflective entries on professional practice based on self and peer feedback

- 11 -


What is Competency?

A competency is a measurable set of knowledge, skills, and attitudes that a

person needs to perform a task effectively

A competency is not an entire job

In most cases, several competencies are required for a job

As competency is task-based, the person may need to transfer the

competencies to new situations and environment

Packaging of Competency Standard (CS) and Curriculum, Training and

Assessment Guide (CTAG)

1st Generation CS

2nd Generation CS

3rd Generation CS

- 12 -


Competency Standard Template

- 13 -


Key Changes to Competency Standard Template

Key CS Components Status of Revision

Cover page Retain

Relevant Job role(s)/occupation(s)

Retain

Performance Statements / Competency Element / Performance Criteria

Retain and Enhance

Assumed Skills and Knowledge Retain and Enhance

Underpinning Knowledge Retain and Enhance

Range and Context Retain

Evidence Sources Retain

- 14 -


Revised Competency Standard Template

Curriculum, Training and Assessment Guide (CTAG)

- 15 -


Curriculum, Training and Assessment Guide (CTAG) Components

Key Changes to CTAG Template

- 16 -


Key Changes to CTAG Template

- 17 -


Competency-based Assessment

A collection of evidence to demonstrate that a learner can perform or

behave according to specific standards

A form of assessment that is derived from the specification of a set of

outcomes articulated in the competency standard(s)

The assessment outcome that will allow the assessor to make judgment

with respect to the achievement or non-achievement of these outcomes

(i.e. competent / not yet competent)

Requirements of Competency-based Assessment

Focus on ‘outcomes’ – The assessment must allow for clear demonstration

of outcomes of learner’s performance. Evidence must be collected to show

that a candidate has met every single performance criterion

Competent/Not yet competent judgement only – Only two judgments can

be made

Assessment in workplace context – Performance must be demonstrated

and assessed under conditions as close as possible to those under which it

would normally be practiced.

- 18 -


Types of Assessment

Diagnostic

o Prior to learning, e.g. training needs assessment

Formative

o During learning, to gauge the learning progress and achievement of

learners for learning (Assessment for learning) – does not contribute to

final assessment result

Continuous

o During learning, to gauge the learning acquired for competence/ grading

(Assessment of learning) – contributes to final assessment result

Summative

o End of learning, to determine competence

Skills Recognition

o Recognition of Prior Learning (RPL)

- 19 -


Commonly Used Assessment Methods

Frequently used competency-based assessment methods may be categorised

into four broad groupings:

Knowledge-based Assessment Methods

1. Multiple-Choice Questions (MCQ)

Multiple-choice questions consist of a question, followed by a number of

options. One of the options is the correct answer. The other options, which

are incorrect, are called distracters.

Advantages Disadvantages

As there is minimal writing, a

substantial amount of content can be

assessed in a relatively short time

Can be administered to a group of

learners concurrently

Scoring is highly objective (i.e.

number of correct answers only)

Validity of evidence blurred by the 25%

(assuming 4-option MCQ) chance of

random success.

Relatively difficult to design good MCQ

questions - requires skilled designers

and pilot- testing.

Method is usually constrained to being a

supplementary method to test

knowledge and comprehension.

Validity decreases with frequent use

and exposure of MCQ to learners

2. Written Assessment (Short Answers)

Short answers require candidates to articulate understanding in written

form but the answers are relatively short as compared to essays and hence

may be appropriate for candidates with literacy issues.

Advantages Disadvantage

Easy to mark/grade as the answers are predictable.

Answers are limited hence writing ability is not unduly stressed.

Suitable for learners who do not have high literacy levels.

Cannot be used alone and has to be supplemented with other methods.

- 20 -


3. Projects

Projects involve an assessment of a combination of subject knowledge,

process skills and transferable skills. Projects require learners to apply their

learning, encourage pro-activity and need an extended duration for

completion. The outcomes of a project may be presented through different

forms such as a report, portfolio, and/or presentation.


More flexible (in terms of scope, medium for presentation) than other assessment methods

High validity as it requires the application of skills and knowledge in a given / real context

Time consuming to mark

Challenging to determine learner’s true contributions and correspondingly his level of competencies

Should be complemented with other supplementary methods to ensure authenticity of evidence

4. Portfolio

A portfolio refers to a collection of evidence assembled by the learner to

demonstrate competence. The main purpose of using a portfolio would be

the collection of evidence to establish that the learner has demonstrated a

set of prescribed performance criteria. A key point to note is that evidence

that was accumulated many years ago may not be current and has to be in

line with current practices and standards.


Flexible as learners take the initiative to decide on how the portfolio is to be presented and organised.

Valid form of assessment as it is based on real task rather than a series of educational tasks.

Challenging to some learners due to the flexibility

May be difficult for assessors to map the portfolio to assessment criteria and require a face-to-face interview to ascertain competence

- 21 -


5. Written Assessment (Reports)

A report is a form of extended written assignment whereby information and

often recommendations for action are presented.

Advantage Disadvantage

Relevance to realistic context hence more practical than an academic essay

Quite demanding as learners may be required to analyse on what they have done and draw conclusions

6. Oral Questions

Oral Questioning require learners to articulate their understanding through

a verbal face-to-face question and answer (Q&A) process. This method is

useful when addressing a diverse group of learners with varying literacy

levels as the assessor is able to paraphrase questions to help the candidates

in understanding.

Advantage Disadvantage

Able to address diverse needs of candidates

Challenging for inexperience assessors as they will have to be able to think on their feet and paraphrase the questions without providing undue stress to the candidate

- 22 -


Performance-based Methods

7. Workplace Performance

This is arguably the best method to collect reliable and authentic evidence of a learner’s actual competencies. This is because the assessment context is set under actual work conditions that require demonstration of actual occupational competencies and maximises the degree of realism in the assessment process.


Good indicator of a learner’s actual work performance

High authenticity and reliability of evidence

Constrained by opportunities available to observe performance for all range of activities specified under standards

Requires considerable time and resources to assess all the performance criteria

8. Role-Play / Simulation

Role-playing or simulation provides an alternative for workplace

performance. It can be seen as an orchestrated scenario that seeks to

gather evidence on a learner’s competencies.


Allows control of different settings required to assess a full range of activities stated under standards

Flexibility in time allocation

Allows standardization of activities to enhance fairness of assessment

Detached from realistic workplace conditions

Requires considerable investment in facilities and equipment to simulate actual work setting

- 23 -


Attitudinal Evaluation Methods

9. Observations

This method involves observing the behaviour of learners, recording as

much as possible, or whatever appears to be useful, important, or unusual.

Once the data are recorded, the observers’ task is to make sense and draw

conclusions from it. Observation requires preparation time. It is necessary

to determine why you are observing, what you expect to see or think you

might see and how you will record what you see.

Attitudinal evaluation requires determination of a proxy behaviour that

best presents the attitude item / component.

The selection and determination of the proxy behaviour requires research

backing and testing. And there has to be a relationship between the proxy

behaviour and attitude, before it can be safely used for attitudinal

evaluation.


Structured observation allows the observer to focus on specific behaviour that indicate positive and negative feelings or attitudes

Structured observation allows for recording of reliable and relevant data

Structured observation incurs the risk of the observer overlooking concurrent relevant events

In the case of third party observations where inputs from direct supervisors are sought, key issue is the degree of objectivity and reliability of the evaluation

- 24 -


Principles of Assessment (POA)

Validity refers to the strength of assessment tasks and criteria to accurately

measure learners’ attainment of the intended competencies level (i.e. how

well the test really reflects the knowledge, skills and attitudes of the

Competency Standard). It includes content validity, construct validity, face

validity and predictive validity

Reliability refers to the consistency of measured outcomes via clear and

consistent processes for setting, marking, grading etc when a test (all

factors being equal) is repeated on a population of individuals or groups.

Flexibility refers to the consideration of the various needs of learners and

allows assessments to be conducted either on or off the job at mutually

convenient times and situations.

Fairness refers to the equality in the conduct of assessment and does not

advantage or disadvantage particular learners (e.g. amount of assessed

work should be manageable).

Rules of Evidence (ROE)

Valid Evidence

o Meets the Competency Standards and assessment criteria

Sufficient Evidence

o Sufficient evidence to show competency for all Performance

Statements / Performance Criteria and Underpinning Knowledge

Current Evidence

o Evidence collected is based on current situation and not outdated

Authentic Evidence

o Learner’s own work

- 25 -


Adult Educator (AE) Qualifications Requirements and Timelines for

WSQ Approved Training Organisations (ATO)

In-House WSQ ATOs

Role Phase I –

By 1 Oct 2012 Phase II –

By 1 Oct 2013 Phase III –

By 1 Oct 2014

Trainer and/or assessor At least 50% with Workplace Trainer Programme (WTP) or equivalent

At least 65% with Workplace Trainer Programme (WTP) or equivalent

At least 80% with Workplace Trainer Programme (WTP) or equivalent

Master Trainer At least 1 master trainer* must attain full ACTA or equivalent by 1 Oct 2013

Curriculum Developer At least 1 developer* must attain full ACTA or equivalent by 1 Oct 2013

Note:

New ATOs or existing ATOs who are moving into new areas of training are required to meet requirements in Phase I with immediate effect.

WSQ In-house ATOs are companies/organisations that train and/or assess their own staff internally.

Figures are based on declaration during accreditation and deployment * Can be the same person

Public WSQ ATOs

Role Phase I – By 1 Oct 2012

Phase II – By 1 Oct 2013

Phase III – By 1 Oct 2014

Trainer and/or assessor At least 50% with full Advanced Certificate in Training and Assessment (ACTA) or equivalent

At least 65% with Advanced Certificate in Training and Assessment (ACTA) or equivalent

At least 80% with Advanced Certificate in Training and Assessment (ACTA) or equivalent

Curriculum Developer 100% with Diploma in Adult and Continuing Education (DACE) or equivalent by 1 Oct 2015

Note:

New ATOs or existing ATOs who are moving into new areas of training are required to meet requirements in Phase I with immediate effect.

WSQ ATOs are external training providers that are in the business of conducting training and/or assessment

Today, curriculum developers of ATOs are already required to meet ACTA or its equivalent. They will also be required to meet the new DACE requirement by 1 Oct 2015.

Figures are based on declaration during accreditation and deployment. ATOs are required to ensure that its actual deployed AEs for each class delivery under each approved framework meet the qualifications requirements

Partial Completion of DACE: Individuals who have already attained DACE competency units (CU) C2, C3, C5, C6 and C7, may function as WSQ Trainers and Assessors at WSQ ATOs in lieu of completing the ACTA Qualification. Such individuals are encouraged to complete the remaining units of the DACE qualification as the full DACE qualification will enable them to practice as a WSQ Curriculum Developer at WSQ ATOs.

- 26 -


How do These Changes to Competency Standard (CS) and Curriculum,

Training and Assessment Guide (CTAG) Affect my Assessment Plan

Design?

For Competency Standards (CS)

Examine the assumed knowledge, skills and attitudes (KSA)

Examine the level of Bloom’s taxonomy of cognitive, affective and

psychomotor domains used in the Performance Statements / Performance

Criteria and Underpinning Knowledge

Examine the evidence source requirements

For Curriculum, Training and Assessment Guide (CTAG)

Review the industry-adopted strategies and hours

Mandatory / legal requirements and relevant training and assessment

hours

Other recommended strategies

Domain knowledge (technical / vocational qualifications)

Domain industry experience

Adult educator qualifications and experience

- 27 -


What is an Assessment Plan?

An Assessment Plan is a document that provides information on how the assessment will be structured, what is involved and the assessment criteria that candidates will be assessed against.

Components of Assessment Plans

- 28 -


WSQ Requirements in Relation to Assessment Plans

Principles of Assessment (POA) are complied with

Rules of Evidence (ROE) are complied with

Mandatory assessment strategies and requirements (if any is specified in

the CTAG)

All Performance Criteria / Performance Statements and Underpinning

Knowledge are assessed

Proposed Assessor-Learner ratio

Assessor materials are complete, meet needs of assessor

Assessment Plan is developed with:

o Clear instructions for preparations for assessment and documentation of

assessment e.g. set up of equipment, conduct of specific methods, time

for responses, appeal process, checklists

o Assessment instructions to guide assessor

o Assessment tools are provided e.g. role play scripts, answers to written

tests

o Assessment records with clear assessment criteria indicated

o Resources / References (library, web)

o Glossary provided

o Version control

- 29 -


Common Shortcomings in Assessment Plan Design

Example 1: Inappropriate Development of Assessment Tools

Description: A role play script was provided. It was crafted in a dialogue as follows:

Problem: This role play script does not provide valid evidence of competence as the learners only needs to follow the script and the flow of the script. In a real workplace situation, the learners will not be given such a script when providing customer service. Learners should not be in the know of exactly what to expect during each customer service provided. General Advice: Role play scripts should be developed appropriately to guide and ensure that the learners are adequately assessed through a role play based on a real workplace situation. Detailed scripts should only be provided to the actor (i.e. person not being assessed) to ensure that the role play does not deviate from the intended flow of the role play. A scenario may be provided to the learner to explain the context of the role play and the instructions to the learner should generally cover what he/she will be assessed on during role play. A sample role play script developed to assess learner based on a real work place situation is as follows:

Role Play Script (Candidate Copy)

(Customer walks into the store)

Customer: Hello.

Candidate (retailer): Good Afternoon, Mr XXXX. Welcome to OIO Department Store.

Customer: I am looking for a LCD TV.

Candidate (retailer): Can you please share with me what kind of TV you are looking for?

Customer: HD, Cheap, slim, power saving, and good resolution.

Candidate (retailer): Sure Sir! We have a great range of TV sets that you may like

(Candidate brings the customer to the TV)

Candidate (retailer): This is one of the latest models of HD TVs that comes with..... .

Role Play Scenario It is a Sunday morning and your department store is going to launch a new product called the iRobot. Your organisation anticipates a huge demand for this new electronic item as shown by the large sales figures in other countries.

Role Play Script (Candidate’s Copy) You are the retail assistant at XYZ Company. You turn up for work on time this morning and get ready for work. You know that there is a new ‘HOT’ item to be launched today. You are to deliver service excellence to your customer according to your organisational policies and procedures. You will be assessed on the competencies in Deliver Service Excellence, i.e.

Use organisational service culture and values to guide service delivery

- 30 -



Example 2: Assessment Criteria do not Address the Performance Statement / Performance Criteria Sufficiently Description: Assessment criteria were set differently as compared to the Performance Criteria in the Competency Standard Problem: The Performance Criteria state that it is to ascertain the competency of the learner to review team and organisational business plans with the objective of identifying new and emerging skill requirements. The question set only assesses the individual knowledge on how to evaluate business plan and how to identify skill requirements. It does not assess how the individual reviews team and organisational business plans with the objective of identifying new and emerging skill requirements General Advice: Be very clear about what the Performance Criteria expect as Performance Criteria and Performance Statements require the learner to demonstrate competence holistically, rather than in segments.

Example:

Competency Standard: LPM-DEV-301C-0 Encourage People

Performance Criteria: Review team and organisational business plans to identify new and emerging skill requirements

Assessment Method chosen: Written Assignment

Question set for this PC: Complete a 100-word essay on evaluating business plans and how to identify emerging skill requirements

- 31 -



Example 3: Incomplete Instructions in Assessment Specifications Description: The ATO only submitted the following table as part of the assessment specifications. The rest of the assessment plan consists of the matrix of assessment methods, observation checklist and a role play script. S/N Specifications Guidelines

1 Assessment Method Role Play

2 Performance Criteria PC 1.1, 1.2, 2.3, 2.4, 2.5, 2.6, 3.1, 3.2, 3.3

3 Underpinning Knowledge All stated UK

4 Duration 20 minutes

5 Assessment Venue Training Centre or Office

6 Assessment Resources Assessment Checklist

Assessment Questions and Answers

7 Assessment Set-up Requirements To be conducted in quiet room with proper tables and chairs

8 Conduct of Assessment Conduct Interview

9 Recording the Assessment Result Record findings and conclusions in the Individual Assessment Record.

Record reasons for “NYC” in remarks column.

10 Candidate Summary and Feedback Record feedback and findings into the Summary Assessment Record.

11 Required Assessment Tools Course Assessment Booklet (refer to Annex A)

Problem: The specifications do not provide complete information on the following:

Assessment process

Decision making on assessment outcome for the method and for the whole assessment

Preparation for assessment (by the assessor and / or assessment centre) General Advice: Developer needs to ensure that the instructions and specifications are detailed enough to clearly cover the following (not limited to):

Assessment duration

Assessment venue

Assessment set up requirements

Assessment tools

- 32 -


Assessor to candidate ratio

Conduct / provision of feedback

Conduct of assessment

Decision making on assessment outcome for the method and for the whole assessment

Preparation for assessment (by the assessor and / or assessment centre)

Performance Statements / Performance Criteria and Underpinning Knowledge covered

Special instructions (e.g. safety precautions, etc)

- 33 -



Example 4: Conflicting Instructions in Assessment Plan Description: Assessment criteria and assessment specifications provided by the ATO were not reflective of information (e.g. assessment methods selected for each Performance Criteria / Performance Statement) provided in the Evidence Gathering Plan. Refer to the illustration below. Problem: Users of the assessment plan may get confused as to what is the correct assessment method/s and whether the assessment plan is of the correct version. This may reduce the confidence of the user of the assessment tools and create a sense of doubt in the assessment plan being used. General Advice: Ensure that all sections in the assessment plan are consistent and do not conflict with each other. Always do a check on whether there is any information that may confuse the user.

EVIDENCE GATHERING PLAN

ASSESSMENT CHECKLIST

Performance Criteria and Underpinning Knowledge

Assessment Criteria

Please Tick

Remarks

C NYC

1.2 Communicate the performance management system to employees and their roles and responsibilities in performance management.

OQ13: Explain how to communicate the performance management

system to employees and their roles and responsibilities in performance management. Suggested Answers: Candidate is able to explain how to communicate the performance management system to employees and their roles and responsibilities in performance management, which may include:

As part of the performance appraisal

............................………….

............................………….

Performance Criteria Evidence Gathering Plan Method

Assessment Tools

Asse

ssm

en

t

ch

eck

lis

t

Ins

tru

cti

on

s t

o

Can

did

ate

s

Ins

tru

cti

on

s t

o

Asse

ss

ors

Le

arn

ing

jo

urn

al

Ro

le P

lay s

cri

pts

1.2 Communicate the performance management system to employees and their roles and responsibilities in performance management. Assessment Criteria: Ability to communicate the importance of performance management and the roles and responsibilities to stakeholders

Work Activity: Able to verbally communicate and adapt communication methods of the roles and responsibilities of performance management to line managers and employees

Role Play AA1

The assessment method chosen here is role play

The assessment checklist states that it is an oral question

- 34 -



Example 5: Difficulty in Gathering Evidence Description: Assessment specifications require the assessor to observe more than two concurrent practical performances for assessment. Problem: In order for the assessor to observe any forms of practical performance reliably, the assessor needs to be very clear of the context and the required role of the candidate. Having to observe more than two different practical performances is even more challenging, more so for an assessor that is not very familiar with the process. General Advice: Unless there are strict criteria for assessors, and specific consequential actions (i.e. achievement of specific outcome from only one specific process) being observed, it is advisable for assessment of practical performance to be conducted on a one-to-one basis.

6 Assessment Set-up Requirements

A fully set-up restaurant. This includes providing all the relevant items found in a typical restaurant.

7 Conducting the Role Play

1. The assessment is to be conducted at the restaurant (actual workplace)

2. Three candidates will take on the role of waiter and serve customers according to the organisational policies and procedures.

3. Three other people will take on the role of an angry customer, an overly excited guest and an indecisive guest.

4. The role play begins with a guest entering and randomly selecting a waiter to serve him / her.

5. It will end when all the guests have paid the bill and left the restaurant

6. In the event that competency is not being able to be observed during the role play, the assessor could use oral questioning to ascertain the competency.

7. The candidate is required to complete the entire role play even if the assessor had rated the candidate as ‘Not Yet Competent’. The result of the assessment should only be made known to the candidate at the end of the assessment.

- 35 -



Example 6: MCQ Questions with “Give-away” Answers Description: Although the multiple-choice question may have addressed the UK / PC, it has been set such that the answer is very obvious. (See below for example) Problem: If the correct answer is too obvious, it may result in an unreliable result as it is unclear if the learner can actually answer the question or if the other choices (i.e. distracters) were plausible. General Advice: Ensure that the developer of the assessment plan is clear and well-qualified in developing a reliable MCQ Questionnaire. This will help ensure that the assessment is in line with the rules of evidence. Some general tips for designing MCQs include:

Measure important outcomes only

Write a clear question

Ensure that the grammar of the term matches that of the options

Avoid giving verbal clues in the question

Provide plausible distracters

Avoid ‘all of the above’ and ‘none of the above’

Vary the length of the correct answer

Vary the position of the correct answer

Avoid making the correct answer less definite

Example:

1. What is an attribute of a manager? (choose the one correct answer)

a. Able to communicate ideas to supervisors for implementation

b. Able to escape from all responsibilities

c. Able to change the world

d. Able to punish staff as and when he likes

- 36 -



Other Shortcomings in Assessment Plans

Assessment criteria are not specific (e.g. copied directly from Performance Criteria / Performance Statements without elaboration)

Did no address all the Performance Criteria / Performance Statements or Underpinning Knowledge

Did not indicate the assessment coverage in assessment plan

Inclusion of other instructions that do not relate to the assessment (e.g. payment policy, refund policy, etc)

Lack of assessment tools (e.g. oral questions and suggested answers, etc)

Lack of instructions to assessors on

o conduct of assessments

o usage of assessment tools

Lack of instructions to learners

______________________________________________________________

______________________________________________________________

- 37 -


Selection of Assessment Methods

General Good Practices in Assessment Plan Development

Maintaining an assessment tools bank (i.e. different assessment methods selected systematically to ensure reliability of assessments conducted)

Conducting regular validation sessions to ensure currency of assessment plans

Collecting feedback from various parties (e.g. assessors, trainers, developers, participants, observers, etc) on the usefulness of the assessment

Conducting pilot tests for assessment plans before making adjustments

______________________________________________________________

______________________________________________________________

- 38 -


Assessment Plan Validation

Purpose of validation

o To determine if assessment plan meets the Principles of Assessment and Rules of Evidence, learners’ profile, and contextualisation (organisation, industry and legal) and learners’ requirements

Who are the stakeholders

o Line managers/supervisors, training managers, subject matter experts (SME), trainers, assessors

What to focus for validation

o Assessment criteria and evidence requirements

o Assessment environment

o Assessment methods and tools

o Principles of Assessment and Rules of Evidence

o Practicality

o Clarity of instructions

o Cost effectiveness

o Contextualisation (e.g. organisation, industry and legal) and learner requirements

When should validation take place

o Before assessment

o During assessment

o After assessment

Follow up to validation

- 39 -


Assessment Plan Validation Checklist

Purpose of Validation:

Stakeholders involved: Areas of Focus:

Assessment criteria and evidence requirements

Assessment environment

Assessment methods and tools

Principles of Assessment / Rules of Evidence*

Practicality

Clarity of instructions

Cost effectiveness

Contextualisation (organisation, industry and legal) and learners’ requirements

______________________________________________________

______________________________________________________

Assessment Venue: Follow up: *Refer to POA / ROE Validation Checklist

- 40 -


POA / ROE Validation Checklist

Underlying Principle Suggested areas to look into

(not limited to)

Compliance Mitigating Factor (if non-

compliant) Yes No

Principle of Assessment - Validity

Assessment activities are based on actual workplace practices and contexts

Assessment criteria

Assessment specifications

Other assessment tools (where applicable): case study scenarios, role play scripts, scenarios, assessment checklists, observation checklists, etc

Assessment activities relates directly address the PS / PERFORMANCE CRITERIA and UKs

Assessment tools: case study scenarios, role play scripts, scenarios, assessment checklists, observation checklists, etc

Assessment tools address the candidate’s ability to meet the level of performance required by the Competency Standard


Assessment activities planned are holistic and integrated

Assessment methods

Assessment criteria


The assessment method and tools developed are appropriate for the assessment system specified by industry (where relevant)

Assessment methods

Assessment criteria


All information in the assessment Plan has been validated by an appropriate person (e.g. master trainer, subject matter expert, etc) with expertise in the competency/is being assessed

Developer / co-developer

Validation panel

Subject matter expert

- 41 -



(not limited to)


compliant) Yes No

Principle of Assessment - Reliability

Critical elements have been identified and sampling used to ensure that the most important aspects are assessed.



Clear assessment criteria have been provided for the assessor in the assessment / observation checklists (e.g. clear guide to assess and individual as ‘C’ or ‘NYC’)

Assessment / Observation Checklist/s

Clear instructions have been provided to ensure that assessors conduct the assessment activity consistently and make consistent decisions over time and with different candidates

Instructions to assessors



Where multiple assessors are involved in conducting parallel assessment events, assessment strategies and approaches are consistent

Assessment methods

Assessment criteria


Consistent instructions are provided to learner/s and procedures for undertaking the assessment

Instructions to candidates

Where work samples are used, learners are provided with clear guidelines on requirements relating to authenticity and currency of submissions

Instructions to assessors / candidates



Where applicable, assessment specifications and tools are able to produce consistent assessment outcomes conducted in different contexts

Assessment methods


Assessment criteria

Other assessment tools (where applicable): case study

- 42 -



(not limited to)


compliant) Yes No

scenarios, role play scripts, scenarios, assessment checklists, observation checklists, etc

Principle of Assessment - Flexibility

The assessment approach can be adapted to meet the needs of all candidates, workplaces and/or stakeholders.

Assessment methods


Assessment criteria


Wherever practical and appropriate, assessment can be negotiated and agreed between the assessor and the candidate



The assessment strategy adequately covers both on and off the job components of the training


Assessment criteria


Policies, procedures, guidelines and standards are put in place to support assessors in the consistent application of reasonable adjustment/s where required



Principle of Assessment - Fairness

Candidates are given clear and timely information on assessment and what is required.



Information provided to candidate/s covers:

the assessment method/s

assessment process

Instructions to candidates

- 43 -



(not limited to)


compliant) Yes No

the criteria against which they are being assessed

when and how they will receive feedback

appeal process

Assessment strategy caters for and supports the language, literacy and numeracy needs of all candidates in accordance with the assumed skills and knowledge provided in the CTAG (where provided)

Assessment methods


Assessment criteria


Special needs (e.g.; geographical, financial, social) of the candidate/s have been considered in the development and conduct of the assessment strategy.



Reasonable adjustment can be made to the assessment strategy to ensure equity for the candidate/s, while maintaining the integrity of the outcomes.



Opportunities for feedback and review of all aspects will be provided to candidates.



Rule of Evidence – Validity

Assessment strategy and tools accurately address all PS / PERFORMANCE CRITERIA and UK to prove competence of the learners


Assessment criteria


Rule of Evidence – Sufficiency

Enough / More than enough* evidence is provided to confirm that the learner is competent in meeting all PS /


Assessment criteria

Other assessment tools (where applicable): case study scenarios, role play scripts,

- 44 -



(not limited to)


compliant) Yes No

PERFORMANCE CRITERIA and UK * As a general guide, there should be multiple sources of evidence (e.g. at least two methods per assessment plan)

scenarios, assessment checklists, observation checklists, etc

Rule of Evidence – Currency

Evidence collected is based on current situation and is not outdated.


Assessment criteria


Evidence collected is not gathered across a period that is too long (e.g. six months)



Rule of Evidence – Authenticity

Evidence of competence provided by learners are produced by the learner himself / herself alone



Assessment criteria


- 45 -


WSQ Curriculum Development and Pre-Accreditation Checklist

This Checklist serves as a self-assessment tool for the ATO to ascertain if their proposed

course is likely to meet WSQ course accreditation (CA) requirements.

It is based on existing CA audit criteria of ‘Course Design and Development’, ‘Assessment

Design’, ‘Adult Educator Requirements’, and ‘Facilities and Equipment’.

Detailed descriptions of the criteria are provided to enable Curriculum Developers and ATOs

to determine if their proposed course meets the expectations of a WSQ course and if not, to

do the necessary revisions and enhancements before submitting the course to WDA for

accreditation.

After a course is submitted for accreditation, the ATO will be given up to 10 working days to

do any necessary revisions and enhancements. Thereafter, the application may be audited

based on the latest set of documents available to WDA.

It is therefore important that the ATO conducts a self-check before submitting the course for

accreditation.

The Checklist is a reference guide and added resource for Curriculum Developers and

ATOs. The Checklist does not need to be submitted with the CA application.

WSQ CA requirements ATO’s self-check

1.1 Course Design and Development

1.1.1 Meet CEs, PCs/PSs and UK Yes No

Do the learning outcomes and coverage of content meet all Competency Elements, Performance Criteria / Performance Statements and Underpinning Knowledge?

Are the Range and Context / Range of Application requirements complied with (as specified in the CS and CTAG)?

1.1.2 Mandatory learning strategies and requirements (if specified in CTAG) Yes No

Are mandatory learning strategies and requirements, including duration, complied with (if any is specified in the CTAG)?

1.1.3 Courseware is pedagogically sound Yes No

Learners Guide and/or supplementary materials for learners are available and sufficient

Appropriate learning strategies and instructional methods are used:

o Is the course pitched at the correct competency level as indicated in the Competency Standard (Bloom’s model is recommended)?

o Is the choice of learning strategy and instructional methods appropriate and aligned to programme outcomes and objectives?

Aligned to adult learning principles:

o Are active and learner-centric methods adopted?

- 46 -


o Are activities planned to have variety to sustain the learner’s interest, engage and motivate them?

o Are activities planned to incorporate checkpoints for summary, recap of learning etc to maximise learning retention?

Tailored to needs of targeted adult learners:

o Does the course incorporate ‘bite-size learning’?

o Does the course address needs of learners e.g. cultural background, language proficiency?

o Does the course cater to various learning styles of learners?

Is the course contextualised to organisation / industrial / legal requirements?

Is appropriate media e.g. social media, audio/video used to deliver the training?

Is the time allocated for learning and activities appropriate and sufficient?

1.1.4 Proposed Trainer-Learner ratio Yes No

Does the proposed Trainer-Learner ratio meet ratio specified / recommended in CTAG?

In the absence of a specified / recommended ratio, is the proposed ratio appropriate?

Note:

If the proposed ratio exceeds or do not meet specified recommendations, ATO may be asked to provide justification

1.1.5 Trainer Materials are complete, meet needs of trainer and comprise these documents :

Yes No

Lesson Plan (with clear indication of topic, duration and equipment)

Clear instructions for preparations required, timing to pace lessons, conduct of learning activities

Key presentation points to guide facilitator

Resources / References (library, web)

Intellectual Property rights observed e.g. acknowledge sources of information

Glossary provided

Version control

1.2 Assessment Design

1.2.1 Principles of assessment are observed Yes No

Is the assessment Valid?

- 47 -


o assesses what it claims to assess through collection of evidence that is relevant to the activity and demonstrates that the PC/PS/UK have been met

Is the assessment Flexible?

o allows for either on or off-the-job, at mutually convenient times and situations, and appropriate to the range of contexts / application

Is the assessment Fair?

o does not disadvantage candidates and takes into account the characteristics of the candidate being assessed? Equitable to all groups being assessed

Is the assessment Reliable?

o consistency of interpretation of evidence and assessment outcomes

1.2.2 Rules of evidence are met Yes No

Is Validity ensured?

o evidence meets the specified criteria of the competency standards

Is Authenticity ensured?

o evidence produced is/can be/will be verified to be the candidate’s own work

Is Sufficiency ensured?

o enough evidence is gathered to show competency against all specified criteria of the standard/s

Is Currency ensured?

o Evidence to be gathered are current, relevant and applicable in today’s context

1.2.3 Mandatory assessment strategies and requirements (if specified in CTAG)

Yes No

Are mandatory assessment strategies and requirements, including duration, complied with (if any is specified in the CTAG)?

1.2.4 All PCs/PSs and UK are assessed through : Yes No

A range of assessment methods

Assessment method is clearly indicated for each item

Note:

ATO may be asked to provide explanations and justifications of the assessment design, methods and tools employed

1.2.5 Proposed Assessor-Learner ratio Yes No

Does the proposed Assessor-Learner ratio meet ratio specified / recommended in the CTAG?

- 48 -


In the absence of a specified / recommended ration, is the proposed ratio appropriate?

Note:

ATO may be asked to justify and provide explanations if the proposed ratio exceeds or do not meet specified / recommendations

1.2.6 Assessor Materials are complete, meets needs of assessor, and comprises these documents :

Yes No

Clear instructions for preparations for assessment and documentation of assessment e.g. set up of equipment, conduct of specific methods, time for responses, appeal process, checklists

Key assessment pointers to guide assessor

Assessment tools are provided e.g. role play scripts, answers to written tests

Assessment records with clear assessment criteria/grading indicated

Resources / References (library, web)

Glossary provided

Version control

1.3 Adult Educator (AE) requirements

1.3.1 Proposed AEs meet the following requirements as specified / recommended in the CTAG

Yes No

All proposed developers, trainers, assessors possess adequate:

o Domain industry experience

o Domain qualification

o AE qualifications

o AE experience

Note:

CVs of all AEs to be submitted for reference and verification. ATO may be further asked to provide copies of relevant certificates/scrolls/transcripts.

If subject matter experts (SMEs) are used, the SME/s possess adequate:

o Domain industry experience

o Domain industry qualification

Note:

CVs of all SMEs to be submitted for reference and verification. ATO may be further asked to provide copies of relevant certificates/scrolls/transcripts.

ATO is to also note that submission of profiles for SMEs is not sufficient.

1.4 Facilities and Equipment

- 49 -


1.4.1 Proposed facilities, equipment and other resources Yes No

Are proposed facilities, equipment and other resources adequate to support the effective delivery of training and assessment in accordance to the requirements as specified in the CTAG?

1.4.2 Training and assessment site/s comply with all fire, health and safety requirements

Yes No

Do training and assessment site/s, including its facilities and equipment contained within, comply with all legislations and regulations governing their use?

Have all necessary and valid registrations, permits and licenses for the operation and use of the training and assessment site/s, facilities and equipment been obtained from the proper authorities?

- 50 -


References

Dannefer, E. F., and Henson, L. C. (2007). The portfolio approach to

competency-based assessment at the Cleveland Clinic Lerner College of

Medicine. Academic medicine, 82(5), 493-501.

Singapore Workforce Development Agency (2007). WSQ Assessment

Design Guide. Assessment methods 13–22.

Institute for Adult Learning (2011), DACE Develop Assessment Tools,

Learner Guide from IAL

Institute for Adult Learning (2008), CU5 Develop a Competency-Based

Assessment, Learner Guide from IAL

- 51 -


Useful Websites

CA self-assessment checklist -

http://app2.wda.gov.sg/data/ImgCont/948/WDA-WSQ-

Curriculum%20Development%20&%20Pre-

Accreditation%20Checklist(1Aug11)v1.pdf

WSQ terms and conditions -

https://www.skillsconnect.gov.sg/web/guest/termsAndConditions#

External Guide for Continuous Improvement Review -

http://app2.wda.gov.sg/data/ImgCont/948/CIR%20External%20Guide(v1.8

)4Nov11(A).pdf

Guidelines for submitting results and printing SOAs -

http://app2.wda.gov.sg/data/ImgCont/487/OnlineSubmissionForm.zip

Practical Guide on Managing For Training and Assessment Excellence -

http://app2.wda.gov.sg/data/ImgCont/487/WSQPracticalGuideforCDForu

mheldon24May07.pdf

Guide on updating WDA on changes in Course Information –

http://app2.wda.gov.sg/data/ImgCont/487/ViewandUpdateCourseProfile.

pdf

Guide on updating WDA on changes in Organisation Information -

http://app2.wda.gov.sg/data/ImgCont/487/ViewandUpdateOrganizationP

rofile.pdf

Credit exemption guide - http://www.ial.edu.sg/index.aspx?id=81

http://app2.wda.gov.sg/data/ImgCont/948/WDA-WSQ-Curriculum%20Development%20&%20Pre-Accreditation%20Checklist(1Aug11)v1.pdf



https://www.skillsconnect.gov.sg/web/guest/termsAndConditions

http://app2.wda.gov.sg/data/ImgCont/948/CIR%20External%20Guide(v1.8)4Nov11(A).pdf

http://app2.wda.gov.sg/data/ImgCont/948/CIR%20External%20Guide(v1.8)4Nov11(A).pdf

http://app2.wda.gov.sg/data/ImgCont/487/OnlineSubmissionForm.zip

http://app2.wda.gov.sg/data/ImgCont/487/WSQPracticalGuideforCDForumheldon24May07.pdf

http://app2.wda.gov.sg/data/ImgCont/487/WSQPracticalGuideforCDForumheldon24May07.pdf

http://app2.wda.gov.sg/data/ImgCont/487/ViewandUpdateCourseProfile.pdf

http://app2.wda.gov.sg/data/ImgCont/487/ViewandUpdateCourseProfile.pdf

http://app2.wda.gov.sg/data/ImgCont/487/ViewandUpdateOrganizationProfile.pdf

http://app2.wda.gov.sg/data/ImgCont/487/ViewandUpdateOrganizationProfile.pdf

http://www.ial.edu.sg/index.aspx?id=81

- 52 -


Version Control Record

Version Effective Date Changes Author/s

1.0 27th August 2012 Initial version QAD

1.1 14th October 2012

Inclusion of additional notes for CD workshop for Public ATO

QAD / IAL

develop competency-based assessment plans · 1. multiple-choice questions (mcq) multiple-choice...

Documents