Evidence based policy: Presenting statistical information for policy

makers14 March 2014

Sandra Pattison & Patrick Lim

What is the VET policy environment? Australian Constitution National, State, Regional, Industry needs Government funding 2012

► $8428.5 million – $4 333.4 million states and territories– $2 325.4 million Aust government

Private markets, market failure, substitution.

What are the key policy questions? Objective

► A productive and highly skilled workforce enabling all working age Australians to develop the skills and qualifications needed to participate effectively in the labour market and contribute to Australia's economic future; and supports the achievement of increased rates of workforce participation.

Outcomes► the skill levels of the working age population

are increased to meet the changing needs of the economy;

► all working age Australians have the opportunity to develop skills;

► training delivers the skills and capabilities needed for improved economic participation for working age Australians.

How do policy makers like their information presented to them?

How do policy makers like their information presented to them?

How do policy makers like their information presented to them?

Effect β SE df t Pr > |t|

Intercept 2.383 0.099 325 24.01 <0.0001

Student SES 0.082 0.070 4381 1.17 0.2418

Student academic achievement

1.220 0.077 4381 15.83 <0.0001

School quality 0.470 0.091 325 5.14 <0.0001

Student SES by student academic achievement

-0.045 0.056 4381 -0.81 0.4159

Student SES by school quality

-0.150 0.053 4381 -2.84 0.0046

Student academic achievement by school quality

-0.132 0.063 4381 -2.09 0.0366

vs

How do policy makers like their information presented to them?

vs

Characteristics

Courses/units

Intentions

Outcomes

Qualifications

Characteristics

Employer views

Activity

Funding

VET ProviderApprentice & TraineeStudent Outcomes SurveyEmployer SurveyVET Financial Data

Plus other relevant data (eg: ABS Survey of Education and Training, Census, HILDA)

Data Sources?

Data Quality Data quality is vital. Quality is an on-going and continuous

process1. Import and collection.

2. Robust quality assessment when reporting.

3. Internal and external review of reports and publications.

Data Quality on collection Administrative Data:

► Governed by data standards (AVETMISS) and data input tools that ensure consistency with the data standard.

► Quality checks undertaken by NCVER upon receiving of the data.

Survey Data:► Close cooperation with the market research companies► Define ranges of acceptable responses for quantitative

data.► Quality checks undertaken upon receiving of the data.

Data Quality for reporting Usual exploratory data analysis. Cross-checking of tables against known

bottom-lines. Research papers - traditional academic

referee process:► 2 referees (either internal or external).

Defining the question? Often NCVER is asked by Government to

assess a broad general statement:► eg: Has the Productivity Places Program

(PPP) worked? What does this actually mean?

We spend considerable effort in helping refine questions.

Data is collected with no specific policy questions in mind.

Defining the question? Often, there is no robust, reliable data to adequately

answer a question. It is “our” role as the key agency in VET to identify

weaknesses in data collections and to advise on how to collect the required data/information.

Policy questions can and should inform data development & collection. However, most data collections and surveys exist longer than policy.

Thus, changes to standards/questionnaires needs to be undertaken systematically.

Evaluation needs to be considered when setting up a program, not once it has been delivered.

Informing policy? Very difficult to measure the impact of

research, even more-so when trying to determine if it informs policy.

Difficult to attribute a research project to a particular policy development.

Academic journals struggle to inform policy. Often - unpublished reports have the biggest

impact.

Informing policy?

Informing policy?

Layering the dissemination

Journal Articles

Synthesised into a very short briefing notes for Ministers, Advisors or other parliament committee

Very complicated piece of research may often be reduced to two or three lines in a ministerial briefing

Robust and accurate statistical methodology using reliable data is vital!

Policy makers don’t care that you have fitted a particular type of model.

They need to have faith in the data and methodology.

It is up to us to highlight shortcomings, but this needs to be done succinctly and clearly.

Informing policy?

Apprentice and trainee incentive changes Completion rates Indigenous systematic review Part-time work and study School Effects Youth Allowance

Informing policy?

100

35

Completion rate = 35%

Problems

Estimating course completion rates

Commencing Continuing

Quit Completed

Commencing Continuing

Quit Completed

?

?? ? ?

?

11

Commencing Continuing Continuing Completed

Markov Chain model

0.20.2

0.3

0.60.1

0.4

0.4

Commencing Continuing

Quit Completed 11

Probability of eventually completing

31

commence continue

quit complete

commence continue

quit complete

or

p1

p3

q1

q3

3

131 1

)course completing eventuallyPr(q

qpp

Key Message:

Our research won’t influence anyone if policy makers can’t understand what it is we are presenting!

Informing policy?

Discussion Evidence base policy – does it exist? Presenting complicated statistical models (yes, the most basic

regression is complicated to others)? Preserving statistical rigour - how do we ensure that the information

presented remains accessible? How do we balance the need for providing timely and relevant

information/data when:► Data sources may be scarce or unreliable► We need to maintain independence

Data Quality Statements – Fit for purpose? Accepting the fact that your analysis or work may in fact be

misrepresented or misunderstood and be prepared for it?