blending workforce readiness curriculum, assessments ... · #eatpconf blending workforce readiness...

#EATPconf

Blending Workforce Readiness Curriculum, Assessments, &

Credentials

Peter Shiras (International Youth Foundation)

Richard D. Roberts, Ph.D. (Center for Innovative Assessments, ProExam)

Overview

– Presenter Introductions

– Workforce Readiness: Background

– Impact from Policy

– Key Research Findings

– Workforce Readiness Assessments: Constructs and Methods

– Workforce Readiness Curriculum: An Overview of Passport to Success

– Workforce Readiness Assessment in an International Context: Challenges and Strategies

– Workforce Readiness Assessments in an International Context: Preliminary Findings

– Summary

About the Presenters: Peter Shiras

– International Youth Foundation

– Executive Vice President for Business Development

– Work in 70 countries

– Partnerships with major companies including Hilton Worldwide, PepsiCo, Caterpillar, Microsoft, Chevron, Walmart, and Cummins

– Passport to Success

– Signature life skills and work readiness training program

– Work with universities, TVET systems and companies

– Developed with support from GE Foundation

– Significant impact on work readiness, job placement, retention and advancement

About the Presenters: Richard D. Roberts

– NRC Fellow (USAF)

– University of Sydney

– ETS

– Background variables: PISA, PIAAC, OECD

– Cross-cultural competence (3C)

– Fake resistant assessments of noncognitive skills

– ProExam

– Assessment of Biases in Cognition (IARPA)

– Assessment of non-academic skills (Tessera)

– Develop innovative assessments, feedback, & action plans

– Books, published articles on workforce readiness, noncognitive (social and emotional) skills

#EATPconf

Workforce Readiness: Background

The Changing Nature of Work (Burrus et al., 2016)

6

Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.

Technological Innovation

The rate of technological innovation is increasing

• Internet subscriptions nearly tripled in OECD countries from 1999-2009 (OECD, 2013)

• Mobile phone subscriptions more than tripled OECD countries from 1999-2009 (OECD, 2013)

Work Activity ShiftTechnological innovation leads to increased technology use in day-to-day work activities

• Over 95%, 85%, and 65% of large, medium, and small business, respectively, in OECD countries now use the internet in their jobs (OECD, 2013)

• Increase in non-routine analytic and interpersonal job tasks performed by workers since 1960 (Autor & Price, 2013)

Job Market ShiftShift in work activities leads to a shift in the job market

• Growth in occupations needing highly educated workers and decline in occupations needing medium- and low-educated workers from 1998-2008 (OECD, 2013)

• 47% of U.S. jobs at high risk due to automation over the coming decades, with low skill and low paying jobs most likely to be automated (Frey & Osborne, 2015)

Skill ShiftShift in the job market requires a shift in worker skills emphasized

• Most job growth since 1980 has been in occupations that require strong social (Deming, 2015).

• Jobs that require high levels of analytical and mathematical reasoning have had poor job growth since 1980 (Deming, 2015)

Valued by The Workforce (Conference Board, 2005)

7


50

60

70

80

90

100

Req

uir

e Th

is S

kill

(%

of

Emp

loye

rs)

Mismatch Between Job Candidates and Employers

8


(Source: Hart Research Associates, 2015)

How Well-Prepared is Current Workforce on this Skill? Employers Students

Working in teams 37% 64%

Staying current on technologies 37% 46%

Ethical judgment and decision making 30% 62%

Locating, organizing, and evaluating information 29% 64%

Oral communication 28% 62%

Working with numbers/statistics 28% 55%

Written communication 27% 65%

Critical/analytical thinking 26% 66%

Being innovative/creative 25% 57%

Analyzing/solving complex problems 24% 59%

Applying knowledge/skills to the real world 23% 59%

Awareness/experience of diverse cultures 21% 48%

Valued by The Workforce (Sackett & Walmsley, 2015)

9


O*NET Worker Style CharacteristicJob Family Top Ranked Second Ranked Third RankedArchitecture and Engineering Analytical Thinking (O)/ Dependability (C) Integrity (C) Initiative (C)

Arts, Design, Entertainment, Sports, and Media

Dependability (C) Adaptability and Flexibility (N) Initiative (C)/ Stress Tolerance (N)

Building and Grounds Cleaning and Maintenance

Dependability (C) Cooperation (A) Self-Control (N)

Business and Financial Operations Integrity (C) Dependability (C) Analytical Thinking (O)/ Cooperation (A)

Computer and Mathematical Analytical Thinking (O) Dependability (C) Cooperation (A)

Construction and Extraction Dependability (C) Cooperation (A) Self-Control (N)

Education Dependability (C) Integrity (C) Self-Control (N)

Farming, Fishing, and Forestry Dependability (C) Self-Control (N) Independence (C)

Food, Preparation, and Serving Related Cooperation (A) Dependability (C) Self-Control (N)

Health Care Support Dependability (C) Concern for Others (A) Integrity (C)

Health Care Practitioners and Technical Integrity (C) Dependability (C) Concern for Others (A)

Installation, Maintenance, and Repair Dependability (C) Integrity (C) Cooperation (A)

Legal Integrity (C) Dependability (C) Analytical Thinking (O)

Life, Physical, and Social Science Integrity (C) Analytical Thinking (O) Dependability (C)

Management Dependability (C) Integrity (C) Leadership (E, A)Office and Administrative Support Dependability (C) Integrity (C) Cooperation (A)

Personal Care and Service Dependability (C) Self-Control (N) Integrity (C)

Production Dependability (C) Cooperation (A) Integrity (C)Sales and Related Dependability (C) Integrity (C) Self-Control (N)Transportation and Material Moving Dependability (C) Self-Control (N) Integrity (C)

Predictive of Workplace Performance

10


Job

Performance

Task

Performance

Organizational

Citizenship

Behavior

Counter-Productive

Work Behavior

Conscientiousness .33 .31 .40 -.40

Agreeableness .22 .13 .23 -.51

Emotional Stability .13 .11 .21 -.31

Openness .10 .14 .04 -.08

Extraversion .26 .15 .28 -.04

(Source: Roberts et al., 2015)

Noncognitive Skills Impact Long-Term Workforce Status

• Swedish enlistment data: Measured at age 18 and predict labor market outcomes for 30+ year olds

• Noncognitive skills measured

– Willingness to assume responsibility, independence, outgoing character, persistence, emotional stability, initiative, social skills

• Poor labor market outcomes (chronic unemployment, low earnings)

– Predicted by lack of noncognitive skills rather than cognitive ability

– Noncognitive skills determine productivity irrespective of occupation or ability level. Cognitive ability of value for those in highly skilled occupations (but this may be shifting)

(Source: Lindqvist & Vestman, 2011)


Noncognitive Skills And Change

12


Effe

ct S

ize

Roberts, B. W., Walton, K. E. & Viechtbauer, W. (2006a). Patterns of mean-level change in personality traits across the life course: A meta-analysis of longitudinal studies. Psychological Bulletin, 132, 1–25.

Leadership

0

0.3

0.6

0.9

1.2

10 to 18 18 to 22 22 to 30 30 to 40 40 to 50 50+ 60-70 70+

Social dominance Agreeableness Conscientiousness

Emotional Stability Openness

Leadership

Key Take Away Messages

• Emerging consensus from the world of policy suggests:

– Modern workforce may require noncognitive skills in equal measure to cognitive skills

– But generally not the focus of education

– There is a mismatch in perceptions of competence: Students => Employers

• Emerging research suggests:

– Equally predictive as cognitive tests for certain outcomes (e.g., job ratings, training time), better for others (e.g., attrition)

– Can actually be measured with precision (e.g., reliability, validity)

– Noncognitive skills are malleable (cf. cognitive constructs)


#EATPconf

Workforce Readiness Assessment: Constructs and

Methods

Competency Crosswalks I

15


Competency Big FiveIYF (PTS Lessons) O*NET

Child Trends

Dependability/ Responsibility

Facet of Conscientiousness

Responsibility

Workplace Protocol

Dependability (Conscientiousness, Work Style)

Self-Control, Higher-Order Thinking Skills

Work Ethic Facets of Conscientiousness and EmotionalStability

Goal Setting

Stress Management

Managing Emotions

Achievement/Effort (Achievement Orientation & Stress ToleranceWork Styles)

Positive Self-Concept, Self-Control, Higher-Order Thinking Skills

Competency Crosswalks II

16


Competency Big FiveIYF (PTS Lessons) O*NET

Child Trends

Time Management

Facet of Conscientiousness

Time Management

Time Management (Resource Management Skills, Conscientiousness, Work Style)

Self-Control

Integrity Facets of multiple Big Five constructs – especially Agreeableness (see also HEXACO model)

Keys to Being a Good Employee

Integrity (Conscientiousness Work Style)

Self-Control

Competency Crosswalks III

17


Competency Big Five IYF (PTS Lessons) O*NETChild Trends

Cooperation/ Team Work

Facet of Agreeableness

Team Player

Power of Positive Attitude

Managing Conflict

Respecting Authority

Respecting Diversity

Cooperation (facet of Interpersonal Orientation Work Style )

Social Skills, Communication Skills

Critical Thinking/Problem-Solving

Facet of Openness

Solving Problems

Creative Thinking

Analytical Thinking (facet of Practical Intelligence Work Style)

Higher-Order Thinking Skills

Traditional Likert Scale Items

18

Developed and © Copyright 2016 by Professional Examination Service. All rights reserved. (Source: PISA Background Variable Questionnaire 2012)

ST46

Thinking about the mathematics you do for school: to what extent do you agree with the following statements?

Strongly agree Agree Disagree

Strongly disagree

I finish my homework in time for mathematics class. 1 2 3 4

I work hard on my mathematics homework. 1 2 3 4

I am prepared for my mathematics exams. 1 2 3 4

I study hard for mathematics quizzes. 1 2 3 4

I keep studying until I understand mathematics material. 1 2 3 4

I pay attention in mathematics class. 1 2 3 4

Representative Finding

• Time Management: Determining one’s needs, setting goals to meet needs, and prioritizing and planning to meet goals (Lakein, 1973)

• Time Management typically relates to GPA around the same magnitude as cognitive test scores

Planning Organization

Grades .21** .38**

Grades (6 mo later) .25** .42**

(Source: Liu, Rijmen, MacCann, & Roberts, 2009)

• And evidence-based interventions of time management exist, including elements that have been infused into school curricula (Burrus, Jackson, Holtzman, & Roberts, 2013)


The Little Problems of Likert Scales

20


– Faking

– Self-insight

– Above Average Effects

– Outcome evaluation

– Reference Bias

– Cross-cultural comparison ST84

Q Below you will find descriptions of three mathematics teachers. Read each of the descriptions of these teachers, then let us know to what extent you agree with the final statement.

(Please tick only one box in each row.)

Strongly

agree Agree Disagree Strongly

disagree

a) The students’ in Ms. <name’s> class

frequently interrupt her lessons. She

always arrives five minutes early to

class. Ms. <name> is in control of her

classroom.

1 2 3 4

b) The students’ in Ms. <name’s> class

are calm and orderly. She always

arrives on time to class. Ms. <name> is

in control of her classroom. 1 2 3 4

c) The students’ in Mr. <name’s> class

frequently interrupt his lessons. As a

result, he often arrives five minutes

late to class. Mr. <name> is in control

of his classroom.

1 2 3 4

ST85

Q Thinking about the mathematics teacher who taught your last mathematics class: to what extent do you agree with the following statements?

(Please tick only one box in each row.)

Strongly

agree Agree DisagreeStrongly disagree

a) My teacher gets students to listen to him

or her. 1 2 3 4

b) My teacher keeps the class orderly. 1 2 3 4

c) My teacher starts lessons on time. 1 2 3 4

d) The teacher has to wait a long time for

students to <quiet down>. 1 2 3 4

Vignettes Self-report

Forced-Choice Assessment

For each pair of statements, please click on the one that is most like you.

1. I respect my boss

2. I am always prepared

• New development: Can use psychometric modeling to recover both ipsative and normative scores

• Representative finding: Forced-choice had similar validity coefficients as Likert-scale for predicting valued educational outcomes (Anguiano-Carrasco, MacCann, Geiger, Seybert, & Roberts, in press)

• Advantage: This method appears less fakable, even in high stakes situations (Zigler, MacCann, & Roberts, 2011)


Situational Judgment Test

DEPENDABILITYYour job involves work for multiple different clients. You have a deliverable due in 6 weeks, and your manager has estimated that this will probably take 2 weeks of your time to complete. You do not have anything urgent due this week.

1. Start preparing the client deliverable right away.2. Start preparing the client deliverable 3 weeks before the due date.3. Start preparing the client deliverable 2 weeks before the due date.4. Start preparing the client deliverable 1 week before the due date.

• Suggestive finding: Correlates over .40 with similar self-report constructs, and predicts valued outcomes (e.g., sales performance, GPA) often more highly than “flat” Likert scales (Roberts et al., 2016a, 2016b)

• Can be used as both a summative and formative assessment approach


Situational Judgment Test (Continued)

Managing EmotionsAnna has been overseas for a long time and returns to visit her family. So much has changed that Anna feels left out. What action would be the most effective for Anna?

a. Nothing, it will sort itself out soon enoughb. Tell her family she feels left outc. Spend time listening and getting involved againd. Reflect that relationships can change over time

Construct Correlation (raw) with STEM

TAS-20 (Externally-Oriented Thinking) -.36** (-.32**)

Psychology Grade (Actual) .26** (.18)

(MacCann & Roberts, 2008)


Sample Individual Reports

24Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.

Sample Institutional Reports

25Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.

#EATPconf

Workforce Readiness Curriculum: An Overview Of

Passport to Success

#EATPconf

Best practice curriculum

©2016. All Rights Reserved. International Youth Foundation.

Original Development Partner: General Electric (GE) Foundation

Goal: To equip young people, ages 14-29, with key skills to help them stay in school, find employment, and succeed in life

Key features:

• Implemented in 40+ countries and

utilized in 19 languages

• Implemented in different settings: on the

job training, secondary schools, post-secondary schools,

vocational/technical colleges, and teacher training programs

• More than 3,300 IYF-trained trainers

• More than 113,000 young people trained

#EATPconf

Key course elements

■ A 78-module curriculum with success at contextualizing content to specific contexts (i.e. cultural setting; youth demographic; employment sectors; etc.).

■ Trainer’s Guide that supports trainers to deliver PTS lessons with consistency and quality.

■ Trainee Handbook that supports students as they progress through the course.

■ Comprehensive Training of Trainers and Master Trainers program that equips trainers through multi-day, interactive workshops with the skills and confidence necessary to successfully deliver experiential, student-centered methodology.

■ Coaching resources to support ongoing guidance and support to trainers.

■ Certification of trainers and coaches to implement PTS program in their institutions based on IYF certification standards and protocol.

■ Tracking and Performance Measurement tools including pre- and post- tests measuring skills and behavior change, teacher and student focus groups and employment and educational outcomes tracking mechanisms.

■ Quality assurance standards to support effective and impactful programming.


#EATPconf

COMPREHENSIVE CURRICULUM


Core Life Skills

• Self-confidence

• Managing Emotions

• Personal responsibility

• Respecting self and others

• Cooperation/Team work

• Communication/Inter-personal skills

• Creative Thinking

• Critical Thinking/Problem solving

• Decision making

• Conflict management

Curriculum Structure

80 Lessons / 6 units available:

• Personal Competencies

• Problem Solving & Managing Conflict

• Healthy Behaviors

• Job Search & Effective Work Habits

• Skills for Professional Growth/ Entrepreneurship

• Service Learning

#EATPconf

PTS speaks to a youth audience, while enhancing teachers’ training styles

PTS provides different learning experiences and suits different learning styles (auditory, visual, kinesthetic)

Generating Interest in Topic

Information to Share

Small and/or Large Group Activity

Youth learn to apply lessons outside the classroom

Personal Application

Pedagogy & methodology


#EATPconf

Select global results

■ Jordan: 90% of employers say they would hire more graduates in the future 80% of employers surveyed report that PTS graduates meet their company’s needs.

■ Egypt: 75% of employers stated there is a remarkable and noticeable difference between PTS graduates and other young employees.

■ South Africa: 91% of young Hilton Team members who received PTS in South Africa reported being more thoughtful listeners; 75%indicated improvement in conflict management and cooperation skills.

■ Caribbean: 90% of employers in the Caribbean hiring PTS graduates said they would do so again.


#EATPconf

Workforce Readiness in an International Context:

Challenges and Strategies

Assessment Uses

Possible uses include:

– Training

– Outcomes assessment (evaluating efficacy of Passport to Success program)

– Self-driven professional development and enhancement

– International comparison (cf. OECD)

– Issuing a micro-credential (badge)


Legal Issues Related to Assessment Vary by Country

Country

Evidence Needed to

Refute a Claim

Consequences of

Violation

Permissibility of Preferential

TreatmentChile Unclear, unless for sexual

harassment or unionization

suits. Empirical evidence not

required

Unknown. Currently, sexual

harassment suits may result in

monetary compensation and up

to 3 years imprisonment.

Government has enacted an informal quota

for women in minister positions; however,

this has not crossed over into the private

sector.

Kenya Must show that decisions

were based on applicant

aptitudes and abilities.

Empirical validity evidence

not required.

Remedy by following

recommendations of Ministry of

Health, Labor, and Welfare.

Possible public announcement of

violation. Civil fine of maximum

¥200,000

Different cutoff scores are set for members

from different ethnic groups to ensure that

some members from each group will be

selected. There are required quotas of 5% in

both the private and the public sector for

disabled individuals.

South Africa Both qualitative and

empirical data can be brought

to bear to support the

validity.

Fines. Possible cancellation of

government contracts.

Preferential treatment is permitted and

applied. Racial quotas are legal and

practiced by many large employers. The

practical implication for this is that it is legal

in South African context to use race

norming, or within-group top-down

selection strategies, in order to address

affirmative-action needs of organizations.

(Source: Adapted from Myors et al., 2008)Developed and © Copyright 2016 by Professional Examination Service. All rights reserved.

Translation Issues & Solutions

– Social and cultural mores vary, so writing SJTs is a complex undertaking (e.g., party scenario inappropriate in Saudi Arabia)

– Key guidelines for writing noncognitive items follow emerging guidelines from various sources (e.g., OECD): Avoid negations, idioms, use <“radicals”> to replace names, monetary units etc in different countries

– What is now standard fare:

– Forward and backward translations

– Local cognitive labs

– DIF analysis


Validation Framework

#EATPconf

Workforce Readiness in an International Context:

Preliminary Findings

Pilot Study: Item Try-Out

– Small bilingual sample from South Africa (N =20, F = 11, Age = 21 to 31)

– Situational judgment (8 items)

– 6 items had acceptable properties, scores ranged between .55 and .85 (Mean = .69)

– Self-report (24 items)

– Mean 4.3 (mean SD = .75) on a 1 to 5 scale

– This suggests high degree of social desireablity in these items


Pilot Study: Usability

Test-Specific Feedback Results from South Africa Cognitive Laboratory:

Percentage of Participants Who Responded “Agree“ or “Strongly Agree”

General Test-Specific Feedback Questions

Situational

Judgment

Test

Self-Report

Personality

I enjoyed responding to these items more than responding to

the questions on most tests I have taken. 70 85

These items seemed more relevant to my day-to-day life than

the questions on most tests I have taken. 70 85

In general, I enjoy taking tests that help me to learn more

about my personal qualities. 100 100

I would like to take another test like this in the future. 90 100

I would gladly take another test like this if doing it meant I got

feedback about my personal qualities. 90 100


#EATPconf

Summary

#EATPconf

Next Steps

– Item pool being expanded

– Translation protocols to be followed (note, extra burden here regarding score reports, which also must be translated)

– Pilot studies in Mexico, Indonesia, and South Africa

– Studies to establish cut-scores

– Possible use in evaluation studies

– Local validation studies

– Delivery: IT in each country

– Marketing and branding


blending workforce readiness curriculum, assessments ... · #eatpconf blending workforce readiness...

Documents