assignment 1 – designing quality assessment - web viewassignment 1– designing quality...

1 Assignment 1 – Designing Quality Assessment

ASSIGNMENT 1– DESIGNING QUALITY ASSESSMENT

3143VTA - VOCATIONAL ASSESSMENT

COURSE CONVENER: DR. IAN JAMES

SUBMITTED BY: DAVID MARTIN

STUDENT #2636349

[email protected]

DATE SUBMITTED: 10TH APRIL 2008

David Martin 3143VTAStudent #2636349 Vocational Assessment


CONTENTS

Overview........................................................................................................................................3

Course Objectives......................................................................................................................4

Assessment Items......................................................................................................................5

Rationale........................................................................................................................................6

Validity........................................................................................................................................8

Reliability....................................................................................................................................8

Fairness.....................................................................................................................................9

Learners Needs..........................................................................................................................9

Appropriateness for the Wider Community................................................................................9

Conclusion...................................................................................................................................10

List of References........................................................................................................................11

Appendix 1: Assessment Matrix..................................................................................................12

Appendix 2: Marking Criteria Checklists......................................................................................14

Examination Marking Criteria Checklist...................................................................................14

Role Play Assessment Marking Criteria Checklist...................................................................15

Essay Questions Assessment Marking Criteria Checklist.......................................................17

Appendix 3: Candidate Instructions and Sample Assessments.................................................19

Assessment 1: Multiple Choice and Short Answer Examination.............................................19

Assessment 2: Role Plays.......................................................................................................21

Assessment 3: Essay Questions.............................................................................................22



OVERVIEW

This project will present and justify assessment for a group of teachers who are required to

teach employability and technical skills to unemployed youths. Working effectively with groups

of youths to enhance their employment prospects requires some special skills and knowledge

above the technical content normally being taught and will be part of a professional

development activity. The activity will be run as a two day workshop, involving delivery and

assessment of factual, procedural and meta-cognitive knowledge.

This project will detail the learning outcomes and the assessment items and will use a rationale

to describe the processes and techniques used. Assessment was planned at the same time as

the learning outcomes were specified and were reviewed several times. While formative and

summative assessment will be used as part of the professional development activity, the focus

of this project is the summative assessment tasks that were designed to measure factual and

procedural knowledge as well as problem solving and goal orientated situations. The validity,

reliability and fairness of how the assessment applies to the wider community will also be

examined. Included in the appendix is the assessment matrix, samples of the assessment and

complete candidate and marking instructions.



COURSE OBJECTIVES

Teaching technical content to a group of unemployed youth in order to enhance their job

prospects requires more than just technical knowledge and general teaching skills. Teachers

working with this sort of target group need to be equipped with a number of special skills and

knowledge sets; and a range of professional development activities could be used. In this

project one professional development activity has been focused on, with the overall objective:

teachers will be able to administer and assess a literacy and numeracy test to a group of

unemployed youth.

This course objective has been broken into eight learning goals. The learning goals have been

described as instructional objectives and sequenced in the most likely order of use when

applied to the real life situation of supporting a youthful job seeker.

The teachers will, without aid, be able to:

identify common uses of literacy and numeracy testing that apply to unemployed youth

describe and demonstrate specialised teaching strategies to engage youth students and

to prepare them for taking a literacy and numeracy test

identify correct procedures that are used when supervising a literacy and numeracy test

describe and demonstrate the methods used to grade a literacy and numeracy test using

an answer sheet

describe and demonstrate methods of using national literacy benchmarks to rate

students literacy and numeracy level

describe and demonstrate methods of giving feedback to students based on results of a

literacy and numeracy test

identify organisations that offer support to students at risk



identify methods of following up students at risk

ASSESSMENT ITEMS

Assessment will be performed throughout the professional development activity in an informal

way. Questions will be asked to gauge prior knowledge and what is being covered during the

activity and to provide short term motivation to those involved.

There will also be summative assessment; the success of the course objective will be measured

using three types of assessment against the eight learning goals. Appendix 2 has complete

marking criteria and checklists and appendix 3 has candidate instructions and sample test items

used to administer the assessments described below.

Firstly an exam will be given to assess factual knowledge and an understanding of basic

concepts and procedures that are relevant to administering and assessing a literacy and

numeracy test. In this exam both multiple choice and short answer questions are used.

Secondly role play activities will be used to assess application of processes and the use of

underlying concepts needed to analyse and evaluate literacy. Five scenarios will be used in the

role play assessment.

Finally an essay questions will be used to assess the integration of knowledge, understanding of

procedures, processes and concepts and how they are evaluated and analysed when used to

administer and assess a literacy and numeracy test.



RATIONALE

As was done with this project, assessment needs to be planned at the same time as

instructional planning and should be guided by what is specified as instructional objectives

(Gronlund, 2006, pp. 51-52). When defining the instructional objectives, different methods of

assessment were reviewed and analysed for their suitability.

Informal assessment would be undertaken during the delivery of this professional development

activity to guide student learning, check for any need to review material and indicate if the pace

of the session is too slow or fast (Gronlund, 2006, p. 50). Summative assessment will be

undertaken at the end of delivery to assess the learning objectives.

Three assessments were planned, an exam that covers basic concepts and procedures, a set of

role play activities and an assignment that involves a number of essay questions. The cognitive

aspects of learning need attention during the assessment process; it is not enough to measure

simple repetition of behaviour patterns. Expertise involves accessing complex connections that

allow tasks not only to be performed well, but the ability to solve new problems (Stevenson,

2003, p. 5) and the assessment for this project was designed to take this into account by using

knowledge dimensions and cognitive taxonomies to develop an assessment matrix.

The results of all the assessment will be criterion-referenced, giving evidence of the specific

knowledge and skills that the student can demonstrate (Gronlund, 2006, p. 26). While the use of

competency based assessment, such as this project is widely accepted in vocational education,

there have been notable objections. Firstly that the specific objectives found in competency

based training (CBT) is restricted in learning situations requiring complex cognition (Moss, 1981,

p. 16). Secondly, as a system CBT “ignores for integration and the transferability of skills”

(Kinsman, 1992, p. 56)



To measure factual and basic procedural knowledge, objective scoring will primarily be used in

the exam. Objective scoring means that there is only one correct answer, it is easier to score

and is useful for both a small number and large groups (Athanasou & Lampraianou, 2002, pp.

92-93). Short answer or essay questions will also be used in the exam, which uses more

subjective scoring which allows complex cognitive processes, analysis and judgements to be

assessed (Athanasou & Lampraianou, 2002, p. 93). Questions are written so that there is only

one correct answer and that specific information was asked for (Athanasou & Lampraianou,

2002, p. 111) and both identification and completion questions would be used. As competence

is attained knowledge becomes structured and interconnected (Glaser, 1991, pp. 26-27) and the

variety and increasing complexity of the exam questions will assess this.

The role play and essay question assessments are closely linked in that they are designed to

assess not only the recall of knowledge, but as a way of assessing expertise; the way

knowledge is related to problem solving and goal-orientated situations (Glaser, 1991, p. 28).

The essay questions were written as clear tasks and designed to relate to the learning

outcomes. This type of assessment tests how well facts can be applied rather then remembered

(Athanasou & Lampraianou, 2002, p. 97).

All assessment items should be written in a way that avoid giving clues to the answer, clearly

show what marks are assigned for the item and need to be worded precisely and concisely

(Athanasou & Lampraianou, 2002, p. 111).



VALIDITY

Assessment cannot measure everything that a student has learnt during a course. It is a

sample, and a key measure of validity is that an adequate sample has been assessed

(Gronlund, 2006, p. 204). Another important indication for validity is that the assessment

matches the learning outcomes of the subject (Athanasou & Lampraianou, 2002, p. 168), and

the assessment matrix is used to show that this is the case. As part of a review process the

assessment was checked to ensure that every learning outcome is covered by a question, that

the format and reading level is appropriate and the directions and questions clear and

unambiguous (Athanasou & Lampraianou, 2002, pp. 172-173). For the time and resources

available for this activity the assessment matrix shows that content can be measured

sufficiently.

RELIABILITY

Reliability is an important aspect of assessment and for any valid inferences to be made by the

results of assessment and must provide consistent results (Gronlund, 2006, p. 25) across

students, teachers, time and contexts. Further, “reliability is largely a statistical concept”

(Athanasou & Lampraianou, 2002, p. 176) and because this is a new set of assessment has not

yet been measured. However, consistency can be achieved by using well designed marking

criteria and answer sheets. Further, using multiple assessment items rather than a single

assessment item makes the overall result more reliable (Brady & Kennedy, 2005, p. 33). Short

answer questions have a high reliability of scoring, and provide better coverage of topics than

essays (Athanasou & Lampraianou, 2002, p. 111). By using clearly defined assessment items

with comprehensive checklists reliability has also been increased.



FAIRNESS

One method of ensuring that assessment items are fair is to improve the functioning content of

the items. This would include eliminating barriers that prevent knowledgeable students from

answering and removing clues that could lead the uninformed to the correct answer (Gronlund,

2006, p. 64). Another way of improving fairness is to ensure that “the instructions for

assessment tasks are clear, explicit and ordered” (Stevenson, 2003, p. 75) and this was also

done during the designing of the assessment items of this project. By creating fair assessment

in this project, all those being assessed will have an equal chance of success.

LEARNERS NEEDS

Good assessment helps those involved to learn, and shows where problems lie and helps

rectify mistakes (Race, Brown, & Smith, 2005, p. 5). Learners also need to know that they have

acquired the skills and knowledge necessary to complete the tasks set in the workplace or

educational setting. The assessment in this project does this by focussing on important, valid

and reliable samples of factual, conceptual and procedural knowledge.

APPROPRIATENESS FOR THE WIDER COMMUNITY

While this professional development activity has a focus on unemployed youth, and the

teachers that deal with them, the learning outcomes and assessment could easily be modified to

include other scenarios. Literacy and numeracy tests are applied in a wide range of

environments, schools, community learning and businesses and therefore the outcomes and

assessments are appropriate for the wider community. By using a variety of assessment

methods, and being able to adapt the assessment if necessary, for example using more verbal



questioning if reading and writing is an issue, the assessment can be considered more inclusive

and flexible, therefore appropriate for the wider community.

CONCLUSION

The ability to design and develop appropriate assessment instruments is a vital part of

vocational education and goes hand in hand with quality learning outcomes. In this assignment

valid, fair and reliable assessment tools have been designed for an imaginary scenario using a

range of theories and approaches. These same theories and approaches can be used in real

vocational situations to determine the worth of curricular activities and how to inform students of

their progress and reporting of this progress to others.



LIST OF REFERENCES

Athanasou, J., & Lampraianou, I. (2002). A Teacher's Guide to Assessment. Tuggerah: Social

Science Press.

Brady, L., & Kennedy, K. (2005). Celebrating Student Achievement: Assessment and reporting

(2nd Edition ed.). Frenchs Forest: Pearson Education.

Glaser, R. (1991). Experitise and assessment. In M. C. Wittrock, & E. L. Baker, Testing and

Cognition (pp. 17-30). Pretnice-Hall.

Gronlund, N. E. (2006). Assessment of Student Achievement (8th Edition ed.). Boston: Pearson

Education.

Kinsman, M. (1992). Competency based training in TAFE. The Australian TAFE Teacher ,

Fourth Quarter, 49-61.

Moss, J. (1981). LCBE: Limiting compentency-based education (or) let's curb behavioral

excesses. Journal of Industrial Teacher Education , 19 (1), 14-18.


12Assignment 1 – Designing Quality Assessment

APPENDIX 1: ASSESSMENT MATRIX

Session Objective: To analyse special needs of students using literacy and numeracy testing and to mark, give feedback and offer specialist

support where necessary.

Learning Goals:The teacher, unaided, will be able to:

Assessment Targets Assessment Methods

Type of Knowledge Cognitive Processes Item 1:Exam

Item 2: Role Play

Item 3: Essay Questions

Identify common uses of literacy and numeracy testing that apply to unemployed youth

Factual knowledge on what the common

uses areConceptual knowledge of how common uses

apply to youth

Remember common uses, and understand

how they apply to unemployed youths

Questions asking to identify

common uses

Verbal questions to test recall

Describe common uses and the relationship to

youths

Describe and demonstrate specialised teaching strategies to engage youth students to prepare them for taking a literacy and numeracy test

Factual knowledge of different strategies

Procedural knowledge of how to apply

strategiesConceptual knowledge of how strategies can

work together with youth issues

Remember strategies and apply them after

analysing the situation, and planning how

different strategies.

Questions on how students

should be prepped for

test

Demonstrate technique likely to be

used to engage youth

students.

Describe methods that might be

used to overcome hostility or inhibition.

Identify correct procedures that are used when supervising a literacy and numeracy test

Factual knowledge of testing procedures,

rules Procedural knowledge of how to apply testing

procedures

Remember, understand the procedures and

apply them.

Questions on common rules of testing

Demonstrate how test rules

are given verbally to

youths

Describe how to deal with a

student cheating on a test

Describe and demonstrate the methods used to grade a literacy and numeracy test using an answer sheet

Factual knowledge of marking rules

Procedural knowledge of marking

Remember, understand and apply preset rules of test. Evaluate situation

and use relevant strategy. Compare test

and answer sheet

Questions on how tests are marked and

graded

Describe the procedure

used to grade a test

Given a sample test, grade the

results



Learning Goal

Assessment Targets Assessment Methods

Type of Knowledge Cognitive Processes Item 1:Exam

Item 2: Role Play

Item 3: Essay Questions

Describe and demonstrate method of using national literacy benchmarks to rate students literacy and numeracy level

Factual knowledge of benchmarks

Procedural knowledge of using benchmarks

Conceptual knowledge of benchmarks

Remember how to access benchmarks, explain how

to rate students and demonstrate marking

system

Questions on accessing national

standards

Describe method of comparing test and

benchmark

Demonstrate comparison

procedure of test and benchmark

Describe and demonstrate methods of giving feedback to students based on the results of a literacy and numeracy test

Conceptual knowledge on youth behaviour

Procedural knowledge of how to give positive

feedback Meta-cognitive

knowledge of test requirements vs. youth

ability

Analyse results of test and evaluate feedback

required. Explain procedures used

to give feedback

Demonstrate methods of analysing

results

Demonstrate how to give

student results in a positive and motivating

way

Describe methods of feedback and

demonstrate written example

Identify organisations that offer support to students at risk

Factual knowledge on what organisations are

available on what services they offer

Conceptual knowledge of what organisations

can offer youths

Remember a list of organisations and

evaluate which offers best choice for youth

List different support services available

Demonstrate methods of

offering support services

Given an example result and student

background, describe which

service offers the best support

Identify methods of following up students at risk

Factual knowledge of resources used to

contact youthProcedural knowledge of how to contact and

motivate youthsConceptual knowledge

of how to motivate youths.

Remember procedures and evaluate which offers

best chance of working with youth

Questions on suitable follow up timeframe

Simulate a conversation with student

about progress and

goals.

Given an example situation, explain

follow up procedures



APPENDIX 2: MARKING CRITERIA CHECKLISTS

EXAMINATION MARKING CRITERIA CHECKLIST

Candidate name:

Assessor name:

Course Name: Administer and assess literacy and numeracy tests

Date of assessment:

Brief description of task and assessor instructions:

Correctly answer over 80% of questions from each section on procedures and techniques used to administer and asses a literacy and numeracy test. The assessor will use the provided answer sheet to mark the test items.

Did the candidate meet the following criteria : Yes No Comments80% of Section 1: Answer multiple choice questions, demonstrating factual knowledge of terms and procedures relating to literacy and numeracy testing

80% of Section 2: Answer short answer questions testing knowledge of facts and procedures

The candidate’s performance was: Satisfactory Unsatisfactory Feedback to candidate

This signature confirms candidate agreement that the above record is a true reflection of the task performed. Candidate signature: Date:

This signature confirms that the candidate has demonstrated competence in the practical performance and theoretical understanding of the observed task.

Assessor signature: Date:



ROLE PLAY ASSESSMENT MARKING CRITERIA CHECKLIST

Candidate name:

Assessor name:

Course name: Administer and assess literacy and numeracy tests

Date of assessment:


Respond to the situations presented in the role play, applying knowledge and processes to successfully administer or assess a literacy and numeracy test. The assessor will use this marking criteria checklist to ensure adequate knowledge is demonstrated.

Did the candidate meet the following criteria : Yes No Comments

Identify at least 3 common uses of literacy and numeracy tests Demonstrate at least 2 strategies that can be used to set youth at ease about testing

Describe all rules used to supervise tests Describe the procedure used to mark and grade tests Describe the process used to compare test results against national benchmarks

Demonstrate a method of giving feedback on results in a positive way

Demonstrate a method of offering support services Describe at least 3 support services available to unemployed youth

Describe and demonstrate a method of following up a student about results and access of support services

The candidate’s performance was: Satisfactory Unsatisfactory Feedback to candidate




This signature confirms that the candidate has demonstrated competence in the practical performance and theoretical understanding of the observed task.




ESSAY QUESTIONS ASSESSMENT MARKING CRITERIA CHECKLIST

Candidate name:

Assessor name:

Course name: Administer and assess literacy and numeracy tests

Date of assessment:


Demonstrate an integrated understanding of the knowledge, procedures and techniques used to administer and assess a literacy and numeracy test. By using several Essay Questions, write 300 words detailing how you would deal with the situations presented. The assessor will use this marking criteria checklist to ensure that adequate knowledge has been demonstrated.

Did the candidate meet the following criteria : Yes No CommentsApplies common uses of literacy tests to youth students

Interprets the relationship between the use of tests and unemployed youth.

Describe the application of at least one method or strategy that can be used to prepare youth students to take a literacy and numeracy test

Describe the application of at least one procedure used when supervising a literacy and numeracy test

Demonstrate the correct method of grading a literacy and numeracy test.

Demonstrate the correct method of using literacy benchmarks Demonstrate the use of written feedback procedures Evaluates the agencies that offer services to youth and selects an appropriate option

Describe at least two methods of the use of follow up procedures for youths at risk.

The candidate’s performance was: Satisfactory Unsatisfactory



Feedback to candidate


This signature confirms that the candidate has demonstrated competence in the practical performance and

theoretical understanding of the observed task.




APPENDIX 3:

CANDIDATE INSTRUCTIONS AND SAMPLE ASSESSMENTS

ASSESSMENT 1: MULTIPLE CHOICE AND SHORT ANSWER EXAMINATION

Candidate Instructions:

To successfully administer and assess a literacy and numeracy test, a range of factual and

procedural knowledge is required. The following 20 questions will test your knowledge in this

area.

The total examination time is 45 minutes. You will have 15 minutes to view the questions and

ask questions of the examiner that are non-content related. You will then have 30 minutes to

complete the questions. Your score will be the number of correctly answered questions, so

answer every item.

To obtain a competent result for this assessment you will need to answer 80% of the questions

from each section correctly.

Example Multiple Choice Questions

1. The most common use of literacy and numeracy testing for unemployed youths is to identify youths with low literacy and numeracy levels:

a. with a view to offering support servicesb. that want to attend universityc. who have applied for jobs that require high literacy levelsd. so that they can be taken off unemployment benefits

2. When marking a literacy and numeracy test using a marking sheet, answers can be marked correct if the answer given is:

a. at least an 80% match b. exactly the same c. at least a 50% match



Example Short Answer Questions

1. List at least 3 organisations that can offer support to youths for literacy and numeracy

issues:

___________________________________________________________________________

2. What is a suitable timeframe to follow up students after they have been referred to a

support agency?

______________________________

Example Multiple Choice Answer Sheet

Question No. Correct Answer

1 A

2 B

Example Short Answer Sheet

1. Community based providers

Registered Training Organisations

Government based providers such as TAFE

Private organisations

2. Students should be followed up prior to their appointment with the agency and within 2

weeks of the completion date of any course that they have been sent to.



ASSESSMENT 2: ROLE PLAYS

Candidate Instructions:

To successfully administer and assess a literacy and numeracy exam, and when dealing with

unemployed youth there is knowledge, procedures and strategies that are required. Your

assessor will present you with a brief introduction to the roles being played and you will need to

respond to questions and situations as they arise.

There is no time limit but there are 5 role plays that will run for around 3-5 minutes each.

Example Role Play 1:

You have arranged an interview with an unemployed youth. She is required to sit a literacy and

numeracy test as part of her unemployment agreement. The youth is anxious about tests and

can’t understand why she has to do it or get her payment cut off. Explain to the youth the uses

of literacy and numeracy tests and encourage her to attend an upcoming test session.

Example Role Play 2:

You have assessed the literacy and numeracy test of a student, his score was 207. You have

arranged an interview with the unemployed youth that sat the test. Using the table below as a

guide to his literacy level, provide feedback on his score and offer at least 3 organisations that

could offer support.

Educational Functioning Level



Beginning ABE/ESL Literacy Beginning ABE/ESL Basic Education Low Intermediate ABE/ESL Education High Intermediate ABE/ESL Education Low Adult Secondary Education/Advanced ESL High Adult Secondary Education/Advanced ESL

200 and below 201 – 210 211 – 220 221 – 235 236 – 245 246 and above

ASSESSMENT 3: ESSAY QUESTIONS

Candidate Instructions

To successfully administer and assess a literacy and numeracy exam, and when dealing with

unemployed youth there is knowledge, procedures and strategies that are required. The

following questions require you to demonstrate depth of knowledge and describe how

knowledge is applied in various situations. You will have 60 minutes to answer the 6 questions,

and each of the answers should be no more than 150 words in length.

Example Question 1

Compare at least two common uses of literacy and numeracy tests and the way that they relate

to unemployed youth.

Example Question 2

Describe at least two methods you could use to overcome hostility and inhibition in a youth that

is required to take a literacy and numeracy test.


assignment 1 – designing quality assessment - web viewassignment 1– designing quality...

Documents