6 TIPS FOR BETTER ASSESSMENTS IN THE MARITIME INDUSTRY

Goal:Achieve Comprehensive, Valid and Reliable Assessment

Comprehensive, Valid and Reliable Assessment

VALIDif it correctly

measures what it is supposed to measure.

An assessment technique is:

RELIABLEif it yields the same

results consistently given

the same input.

Comprehensive, Valid and Reliable Assessment

HOW DO WE ACHIEVE OUR GOAL?

Comprehensive, Valid and Reliable Assessment

TIP 1USE A COMBINATION

OF ASSESSMENT TECHNIQUES

No single assessment technique is perfect.

Achieve a better overall assessment

by combining techniques.

TIP 1: USE COMBINATION OF TECHNIQUES

Example:Combine a demonstrative exam with a multiple choice exam to improve objectivity and coverage.

TIP 1: USE COMBINATION OF TECHNIQUES

Highly subjectiveEach assessors can grade differently.

Demonstrative exams (“Please don a firefighting suit”) are good at testing

skills, but are:Not comprehensiveDoesn’t fully test underlying knowledge.

TIP 1: USE COMBINATION OF TECHNIQUES

Multiple choice exams are weaker in testing skills but:

● Are highly objective

● Give additional insight into depth of knowledge

TIP 1: USE COMBINATION OF TECHNIQUES

Using multiple techniques creates a larger, basket of data for final assessment.

TIP 1: USE COMBINATION OF TECHNIQUES

TIP 2 SEPARATE THE

TRAINER FROM THE ASSESSOR

TRAINER’S JOB:

● Support, instruct and provide resources

● Ensure candidate has required skills

TIP 2: SEPARATE TRAINER FROM ASSESSOR

ASSESSOR’S JOB:

● Ensure no candidate is assigned to a job they are not prepared for

If the trainer and the assessor is the same person:

TIP 2: SEPARATE TRAINER FROM ASSESSOR

Objectivity when assessing is affected:• Candidate result

reflects training ability

• There is an existing trainee-trainer relationship

TIP 2: SEPARATE TRAINER FROM ASSESSOR

TIP 3 USE RUBRICS IN

SKILL ASSESSMENTS

RUBRIC: a written guide for

assigning grades on an assessment.

TIP 3: USE RUBRICS

Example Rubric forProficiency in donning firefighting

gear:Smoothly, without hesitation 5 pointsProficiently, hesitates occasionally 4 points

Makes occasional mistakes which they correct themselves

3 points

Makes occasional mistakes with which they require assistance

2 points

Makes multiple mistakes with which they require assistance

1 point

Unable to without significant assistance 0 points

TIP 3: USE RUBRICS

The candidate demonstrates the ability to don the firefighting suit and related gear.

Yes/No

Compare the previous rubric to:

TIP 3: USE RUBRICS

Rubrics add objectivity and

reliability, providing better data for a

final decision.

TIP 3: USE RUBRICS

TIP 4 TRAIN AND

SUPPORT ASSESSORS

TIP 4: TRAIN AND SUPPORT ASSESSORS

Skilled assessors are: ● Partly made by

experience● Partly made by

training

1.Train your assessors○Choose expert mariners with interest

in training and assessment○Conduct a 1 or 2 day training

session on best practices and guidelines

TIP 4: TRAIN AND SUPPORT ASSESSORS

2.Support your assessors○Create documents on

assessment practices, philosophy, etc.

○Provide rubrics

○Create a place for assessors to exchange information anywhere, anytime

TIP 4: TRAIN AND SUPPORT ASSESSORS

TIP 5 VARY YOUR

ASSESSMENTS

��

Using the same questions on every test leads to the contents becoming common

knowledge.

��TIP 5: VARY YOUR ASSESSMENTS

Questions should be varied for each

assessment.

This prevents candidates from “studying to the test”.

��

TIP 5: VARY YOUR ASSESSMENTS

One way to vary assessments:

List categories of skills and knowledge you would like to cover

Create questions with equal difficulty and importance

for each category

Randomly select

questions from each category

for the test

��

TIP 5: VARY YOUR ASSESSMENTS

This technique works for almost any type of assessment: written, oral, demonstrative, etc.There should be at least 10 questions available to choose from for each question selected.

👷

��

TIP 5: VARY YOUR ASSESSMENTS

TIP 6 IMPLEMENT

CONTINUOUS IMPROVEMENT

CONTINUOUS IMPROVEMENT:

A process by which we periodically evaluate the

performance of a task with a view toward improving it.

TIP 6: IMPLEMENT CONTINUOUS IMPROVEMENT

Evaluate current performance quality with Key Performance Indicators (KPIs).

Apply changes that you hope will improve the outcomes.

After a period of time, re-assess the KPIs.

During each continuous improvement cycle:

TIP 6: IMPLEMENT CONTINUOUS IMPROVEMENT

If the effect is positive, keep the changes.

TIP 6: IMPLEMENT CONTINUOUS IMPROVEMENT

If not, undo the changes.

1.Ask supervisors how successful they believe assessments are in ensuring only qualified candidates are put into service.

Some assessment-related KPIs:

TIP 6: IMPLEMENT CONTINUOUS IMPROVEMENT

2.Ask candidates whether the assessment accurately reflected the knowledge and skills taught in training.

TIP 6: IMPLEMENT CONTINUOUS IMPROVEMENT

3.Monitor exam outcomes for trends. Are grades generally increasing or decreasing?

TIP 6: IMPLEMENT CONTINUOUS IMPROVEMENT

The bottom line for continuous improvement:If you are not actively engaged

in measuring and improving your performance, then you are passively engaged in performance deterioration.

TIP 6: IMPLEMENT CONTINUOUS IMPROVEMENT

Assessment is about giving us the data we need in order to make an informed, professional decision about individuals.

Comprehensive, Valid and Reliable Assessment

If assessment techniques are flawed, then the data is incorrect or incomplete - an informed decision cannot be made.

Comprehensive, Valid and Reliable Assessment

It is not difficult to take steps toward more reliable and valid assessments - steps which can improve organizational performance, safety and pride as a whole.

Comprehensive, Valid and Reliable Assessment

Download the in-depth guide on improving your assessments here.

FOLLOW USTwitter Linkedin BlogFacebook

CREDITS∎ Presentation template by SlidesCarnival∎ Photographs by Unsplash, Pixabay, Pexels Liz Novak, US Navy Mass

Communication Specialist 3rd Class Anna Van Nuys, Maine Maritime Academy, Peter Griffin, NATO Training Mission-Afghanistan, U.S. Navy photo by Mass Communication Specialist 3rd Class Liz Dunagan, U.S. Navy Photo by Mass Communication Specialist Seaman Christopher Farrington, SuperJet International, Kevin Collins, U.S. Navy photo by Photographer's Mate 3rd Class Gary B. Granger, Mercator Media, U.S. Coast Guard,