epso prometric nicoletta vullo - etouches · pdf file#eatpconf presentation outline 1....

#EATPconf

EPSO – Gilles Guillard

Hudson – Nikola Trbovic

Prometric – Nicoletta Vullo

#EATPconf

Presentation outline

1. Starting situation- About EPSO, Prometric & Hudson

- Two single-provider collaborations

- Original candidate trajectory

2. Need for a new candidate trajectory- Insertion of intermediate eliminatory test

- New trajectory

- Different providers, different solutions

– Hudson’s E-tray

– Prometric

3. In practice- Multi-provider collaboration process

- Reliability

- Equal opportunities

- What worked well

- Lessons learned

28-30 September 2016 2

#EATPconf28-30 September 2016 3


European Personnel Selection OfficeEPSO selects highly qualified staff for


European Personnel Selection OfficeThe EU Institutions

#EATPconf

European Personnel Selection OfficeA few figures


Since the creation of EPSO in 2003:

Over 600,000 candidates have participated in EPSO

selection processes

Over 510,000 EPSO candidates have taken a

computer-based test delivered via Prometric

Over 70,000 CBT exams were delivered in 2015

Between 1 % and 2 % of all candidates in EU open

competitions are successful and can be recruited

Some competitions have attracted up to 52,000

applicants for the approx. 300 positions offered

#EATPconf

Present Prometric


About Prometric

Wholly owned subsidiary of ETS

A trusted market leader in technology enabled testing and assessments

Comprehensive suite of services, including test development, test delivery and data management capabilities

350+ clients in every major

industry segment

Over 9 million exams

delivered annually

160 countries

26 languages

#EATPconf

Starting situation:

Two single-provider collaborations


Prometric has been providing Computer-Based Admission

Testing to EPSO since 2005

Over the years, Prometric has implemented innovative

and tailored solutions to meet EPSO’s specific testing

needs

Prometric exam delivery for the EPSO includes:

Unproctored internet-based testing in candidates’ locations;

Proctored Multiple Choice exams of various types, Case

Studies, Audio-Video exams in the Prometric testing network;

In all 24 official EU languages and across all 28 EU Member States and worldwide


About Hudson


About Hudson Belgium

Since 1982 (°De Witte & Morel): more than 30 years experience

Market leader with a strong reputation

6 regional offices: Brussels, Gent, Antwerp, Kortrijk, Hasselt & Louvain-La-Neuve

Over 200 people including over 100 consultants

More than 85% loyal customers

Worldwide Research & Development Center located in Belgium

Partnerships with universities & top business schools e.g. Vlerick Business School, Université

catholique de Louvain, Louvain School of Management, UGent, …

REWARD

MANAGEMENT,

COMPENSATION &

BENEFITS

ASSESSMENT &

DEVELOPMENT

CENTERS

CAREER GUIDANCE

INTERIM

MANAGEMENT &

CONTRACTING

RECRUITMENT &

SELECTION

COMPETENCY &

PERFORMANCE

MANAGEMENT

LEADERSHIP

DEVELOPMENTJOB GRADING

http://be.hudson.com/

#EATPconf

Starting situation:

Two single-provider collaborations

Hudson as trusted advisor for EPSO — Assessment Centres

Develops new test material (and test variations) and

accompanying documentation (marking grids, manuals,

guidelines)

Adapts pre-existing tests and provides off-the-shelf exercises

Trains permanent and non-permanent Selection Board

members, markers and EPSO support staff

Provides advice to EPSO on the further development of its

Assessment Centres

Provides IT tools, support and services relating to the

Informatics tools used by EPSO

28-30 September 2016 11

#EATPconf

Starting situation:

Original candidate trajectory (AD-cycle)

28-30 September 2016 12

MONTHS1 9

1 month 4 to 6 weeks 2 to 3 months

33,837

candidates25,544

candidates

504

candidates

185

laureates

#EATPconf

2. Need for a new candidate trajectory

28-30 September 2016 13

#EATPconf

New candidate trajectory

Intermediate test: E-tray

28-30 September 2016 14

Decision to insert an intermediate eliminatory test between pre-selection tests (CBT) and Assessment Centre: invite 10 x the number of candidates sought to take the e-tray in order to reduce invitees to AC to 2 x the number of candidates sought

Invite only the very best candidates to AC Cost effective (financial & human resources) Improved gender balance Assesses generic competencies at an early stage Automated scoring Limited scope for training

#EATPconf

New candidate trajectory:

Need for a multi-provider collaboration

28-30 September 2016 15

#EATPconf


28-30 September 2016 16

40,666

candidates

27,388

candidates

2,190

candidates541

candidates235

laureates

MONTHS1 9

#EATPconf

New candidate trajectory:

Different providers; different solutions

28-30 September 2016 17

Implementing the new trajectory was not without challenges:

EPSO’s needs Hudson Prometric

Testing facilities and administrators in 28 MS

E-tray to measure competencies

IT platform for test administration

(need for alignment)

Equal opportunities for all candidates in all MS

Parallel tests and randomisation of

test material

Standardised testing conditions

in all MS

#EATPconf


Description of Hudson E-tray

28-30 September 2016 18

Computer-based test using documentation provided in the form of an email inbox

Duration: 50 minutes

Assesses 4 general competencies:

Analysing & Problem-solving;

Delivering Quality & Results;

Prioritising & Organising;

Working with Others

#EATPconf


Hudson E-tray

9/20/2016 19

Step 0:

EPSO request

Step 1:

Development planning & check

NoC

Step 2:

Contact Subject Matter Expert

Step 3:

Meet Subject Matter Expert

Step 4:

Develop exercise

Step 5:

Selection Board validation & adaptations

Step 6:

Linguistic revision and translation

Step 7:

Trial session

Step 8: Psychometrics

Step 9:

IT programming

#EATPconf


Hudson E-tray

28-30 September 2016 20

#EATPconf


Prometric solution

9/20/2016 21

Prometric’s Web-launch technology supports clients who have their own browser-based test driver but who require a secure, proctored environment for test delivery

Web-launch acted as a proxy for Hudson’s website, providing an interface with Prometric’s administrative system

Hudson’s website had to operate using the infrastructure in place at the Prometric testing centres

In summary, using Web-launch required integration between Prometric’s and Hudson’s systems

#EATPconf

Prometric solution

Web-launch functionality

Start the browser and navigate to Hudson’s start URL

using agreed parameters passed in the URL

Interface to the administrative system to set the status

of test delivery

Maintain a test seat session timer

Ensure that candidates navigate only to known

internet domains

9/20/2016 22

#EATPconf

Prometric solution

Web-launch functionality

Monitor for pages that indicate conditions such as

end-of-test or error

Monitor for a loss of internet connectivity

Support restart of a candidate’s sessions

Close the browser and return control back to the

administrative system

9/20/2016 23

#EATPconf

Prometric solution

Hudson’s browser

Supported the Start and End control pages

Supported checkpoint and recovery of testing sessions

Supported test-specific timing (for ex-time extension

for Testing Accommodations)

9/20/2016 24

#EATPconf

3. In practice

28-30 September 2016 25

#EATPconf

In practice:

Multi-provider collaboration process

28-30 September 2016 26

Explore the capabilities of Hudson and Prometric

Define the requirements

for information flow between

the three parties

IT implementation

(platform integration)

Training of test administrators by Prometric

Trials on dummy candidates in

Brussels

First roll-out:

1,500 candidates

6 days

28 MS

#EATPconf

Hudson and Prometric readiness for Web-

launch implementation

Prometric and Hudson determined parameters to be

passed from Web-launch to Hudson’s website

Prometric and Hudson determined Start and End URLs

for QC and Production

Prometric published service scripts for Hudson tests,

representing parameters, URLs, and other

configuration decisions

9/20/2016 27

#EATPconf

Hudson and Prometric readiness for Web-

launch implementation

A test harness was created and Prometric and Hudson

performed integration testing in the QC environment

Four dry-runs were held in Brussels, one of which used

real candidates’ eligibilities

An escalation matrix was established between EPSO,

Prometric and Hudson prior to going live.

9/20/2016 28

#EATPconf

Prometric readiness for test delivery

in the Prometric network

Information was gathered on the ISP bandwidths of all EU28 testing centres

Hudson’s exams were downloaded in Prometric technical support labs in order to issue knowledge-based documentation

Beta testing was conducted in all participating EU28 MS testing centres (38 locations in total)

Hudson verified all results

A Command Centre was put in place during go-live, and bi-hourly reports were issued during testing

9/20/2016 29

#EATPconf

In practice:

Challenges

28-30 September 2016 30

The process was not without challenges:

A few limits to the integration of both IT platforms (example: abort functionality not possible)

Candidates had some questions on the scoring (need to publish more information on scoring methods on EPSO website/ombudsman enquiry)

Isolated technical issues encountered during testing (restarts, loss of connectivity, etc.)

#EATPconf

In practice:

Reliability

28-30 September 2016 31

E-tray test is stable and reliable.

Analysing &Problem Solving

Delivering Quality &

Results

Prioritising &

Organising

Working with

others

Cronbach's

Alpha.528 .567 .504 .575

#EATPconf

In practice:

Equal opportunities

28-30 September 2016 32

In terms of equal opportunities, no adverse impact for gender

ANOVA

Sum of Squares

dfMean

SquareF Sig.

APS_C Between Groups 1.220 1 1.220 .508 .476

Within Groups 1867.294 777 2.403

Total 1868.514 778

DQR_C Between Groups .810 1 .810 .223 .637

Within Groups 2818.667 777 3.628

Total 2819.477 778

POR_C Between Groups 1.758 1 1.758 .794 .373

Within Groups 1719.903 777 2.214

Total 1721.661 778

WWO_C Between Groups 5.989 1 5.989 2.452 .118

Within Groups 1898.246 777 2.443

Total 1904.236 778

Count

Gender_Female 341

Gender_Male 438

Total 779

#EATPconf

In practice:

Equal opportunities

28-30 September 2016 33

WITH E-TRAY INTERMEDIATE STAGE

Increased gender balanced achieved:

WITHOUT E-TRAY INTERMEDIATE STAGE

#EATPconf

In practice:

Equal opportunities

28-30 September 2016 34

In terms of equal opportunities, no adverse impact for test language

Count

language DE_BE 18

EN_INT 702

FR_BE 59

Total 779

ANOVA

Sum of Squares

df Mean Square F Sig.

APS_C Between Groups11.687 2 5.843 2.442 .088

Within Groups 1856.827 776 2.393

Total 1868.514 778

DQR_C Between Groups19.409 2 9.705 2.689 .069

Within Groups 2800.068 776 3.608

Total 2819.477 778

POR_C Between Groups1.483 2 .742 .335 .716

Within Groups 1720.178 776 2.217

Total 1721.661 778

WWO_C Between Groups5.019 2 2.509 1.025 .359

Within Groups 1899.217 776 2.447

Total 1904.236 778

#EATPconf

In practice:

Equal opportunities

28-30 September 2016 35

In terms of equal opportunities, no adverse impact for test day

Count

Testingday 09/21/2015 212

09/22/2015 270

09/23/2015 297

Total 779

ANOVA

Sum of Squares

df Mean Square F Sig.

APS_C Between Groups11.225 2 5.612 2.345 .097

Within Groups 1857.289 776 2.393

Total 1868.514 778

DQR_C Between Groups2.711 2 1.355 .373 .689

Within Groups 2816.766 776 3.630

Total 2819.477 778

POR_C Between Groups1.112 2 .556 .251 .778

Within Groups 1720.549 776 2.217

Total 1721.661 778

WWO_C Between Groups13.750 2 6.875 2.822 .060

Within Groups 1890.486 776 2.436

Total 1904.236 778

#EATPconf

EPSO’s idea in practice

What worked well

28-30 September 2016 36

Interaction between the three parties was very positive,

prompt and transparent

Dry-runs were effectively used to test and validate the exam

prior to delivery and helped identify and solve a number of

issues

Command centre structure and daily testing reports were

effective

Operations worked well for a first-time delivery, with no

candidates displaced and five re-tests in total

Escalation path and communication with Hudson worked

very well

What worked well

#EATPconf

EPSO’s idea in practice:

Lessons learned

28-30 September 2016 37

The multi-provider collaboration was a positive experience

Will be retained for future competitions

Further improvements are being introduced to enhance the

process

A dashboard was created to facilitate follow-up in

real time by EPSO

28-30 September 2016 38

epso prometric nicoletta vullo - etouches · pdf file#eatpconf presentation outline 1....

Documents