aes, darwin 10 th september 2014 traversing the interplay of politics and evaluation: education...
TRANSCRIPT
AES, Darwin 10th September 2014
TRAVERSING THE INTERPLAY OF POLITICS AND
EVALUATION: EDUCATION REFORM IN AUSTRALIA
Assoc Prof Janet ClintonDr Amy GullicksonRuth Aston
Edmund MissonPauline Ho
Introduction
Paper 1: Collaborative Evaluation
Paper 2: Evaluation Methodology
Paper 3: Dissemination of Evaluation Findings
Paper 1: Collaborative Evaluation
Janet Clinton & Edmund MissonUniversity of Melbourne & AITSL
Evaluation context - 1
Overview of the Standards
Map progression: Graduate Proficient Highly Accomplished Lead
Evaluation context - 2
Dual purpose: Improvement
Performance and development Professional learning
Career progression Accreditation – Graduate Registration – Proficient Certification – Highly Accomplished and Lead
Support Materials
Why this evaluation? Evaluation embedded in policy implementation to inform
and guide implementation How?
Collaborative Formative Multi-method Multi-year Utilising existing infrastructures
Considerations Stakeholder communication and engagement Sharing information and findings Timing Alignment of requirements/needs Changing political context Evaluation is designed to add value
Paper 2: Evaluation Methodology
Amy Gullickson & Ruth AstonUniversity of Melbourne
Evaluation methodology
WE ARE HERE
Evaluation methodology
Mixed-Methods Informing policy development =
Evidence Based Policy Implementation
Evaluation phases
Phase 1• Build evaluation
foundation• Establish stakeholder
groups and team• Lit review & program
logic
Phase 2
• National Forum• Stakeholder interviews• Collect existing
documents
Phase 3
• National Survey #1• National online
depository
Phase 4
• Case studies• Data collection round
#2
Phase 5
• Stakeholder interviews• Final data collection
round• National Survey #2
Phase 6
• Triangulation of findings
• Revisit program logic • Draw overall
conclusion
Stage 1- Develop and refine design
Stage 2- Collect evidence
Stage 3- Finalise and make recommendations
Levels of influence ACCREDITATIO
N(GRADUATE)
REGISTRATION(PROFICIENT)
CERTIFICATION(HA AND
LEAD)
PERFORMANCE AND
DEVELOPMENT
PROFESSIONAL LEARNING
SUPPORT MATERIALS
AND RESOURCES
STUDENTS
TEACHERS AND SCHOOL LEADERS
SCHOOLS AND COMMUNITIES
SYSTEMS/SECTORS &
AUTHORITIES
STATES/TERRITORIES
NATIONAL
POLICY
National Forum Interviewees
= 82
Workshop participants
= 174
Government teacher employersCatholic teacher employersIndependent teacher employersRegulatory Authorities
Union
Principal AssociationNational
BodiesDeans of EducationAITSL Board MembersAITSL Board
Alumni
National Survey#2 (June 2015)
National Survey #1
Respondents TotalTeacher 4141School leader 1427Teacher educator 214Pre-service teacher 219Combined 6001
Case Studies
Paper 3: Dissemination of Preliminary Findings
Edmund Misson & Pauline HoAITSL
Driving a dynamic communications strategy to evidence change and impact for the Evaluation Edmund Misson, Pauline Ho
In collaboration with Sam Hussein, AITSL Communications, Online & Social Media
Introduction
The Evaluation is a complex and dynamic process of reform implementation
- involves diverse stakeholders across varying contexts, levels, interests and needs.
Engaging the
Profession
Practising Teachers
School Leaders
Pre-service
Teachers
Teacher Educator
s
Members of key
orgs and agencies
System Leaders
AITSL’s Communications & Social Media Strategy Part of the whole of organisation Universal
Analytics Framework (UAF) tracks the promulgation and engagement of AITSL’s tools and resources
Multiple levels of analytics: CEO & Board Teams’ analytics to track engagement with tools and
resources ‘Deep dives’ to explore engagement of specific
requests ‘Personas’ to understand demographics, behaviours
and horizon scanning of AITSL’s audience
Aim of the Evaluation’s Communications & Social Media Strategy Methodological Rigour: Gain awareness and participation in the key data
collection activities of the Evaluation e.g. National Forum, National Surveys, Case Studies.
Engage the Profession: Encourage professional conversations on the Evaluation’s findings, analysed and reported in a variety of data visualisation ways.
Promulgate Findings: Share findings through various communication channels cross stakeholders to value add to the policy implementation.
Ultimately, the key goal is to increase the effectiveness of the implementation of Australian Professional Standards for Teachers in schools and organisations across jurisdictions and sectors.
Acquisition
Behaviour
Outcomes
Digital Marketing & Measurement Model by Avinash Kaushik
What are we doing to gain stakeholders’
attention to the Evaluation?
What are they doing and how are they
accessing the Evaluation’s activities?
What is the impact and outcomes of
their participation?
Objective: Engage stakeholders of the education profession to participate in the Evaluation and to drive implementation of the
StandardsMeasuring Impact
Engage
Agents
Participate
Reimaging Data
Web analyticsSocial media
analyticsCampaigns
Measu
re
Knowledge translation
Acquisition - Using diversity of acquisition channels
Social Media
FacebookTwitter
Print Collaterals
ReportsJournal
publicationsNewslettersInfographicsFact Sheets
Website & Online
MarketingAITSL E-News
Evaluation webpageInteractive
Infographics
Knowledge translation
6233New visi-tors
2140Return-ing Visi-
tors
desktop tablet mobile0
1,000
2,000
3,000
4,000
5,000
6,000
7,000 6,425
1,133 585
Utility of Devices
8,143 unique visitors
E-ne
wsle
tter
Twitt
er
Face
book
Ning
Googl
e+
Wor
dPre
ss
Blogg
er
Pock
et
Slid
eSha
re
010002000
Social Networks
% of audience accessing the survey 70%
898
121
1,780
1,182
3,490
227
1st interaction
2nd interaction
Evaluation page
50.5% 18.7%
Survey page
62.3% 51.8%
Behaviour – Engaging with our content
Avg. Visit Duration: 07:23
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
682 763
912
7893
730 325 276 380
971
969
3721
2277
1543
2045
National Survey (Oct-Nov 2013)
Case Studies 2014 (Mar-May 2014)
Monthly Analytics ReportTracking unique visitors
Behaviour – Engaging with our content
Aug 2013
Sept 2013
Dec 2013
Mar 2014
Apr2014
May 2014
National Survey 1
Research
6,001 respondents
across jurisdictions
SR 1 LAUNCH
Disseminatefindings
Case Studies LAUNCH
Case Studies*closed*
140 submissions
across jurisdictions
June 2014
Stakeholder 1 Fact Sheet –Infographics
Disseminatefindings
Targeted Marketing
Ongoing promulgation
Targeted Campaign
Research Research
August
2014
SR 2LAUNCH
Outcomes – Impact of the campaigns
Short-term- Create awareness of the Survey and Case Studies
Mid-term- Participate in ongoing data collection activities- Inform findings and shape the Evaluation
Long-term- Analyse and reporting of results and findings
Investigate the Impact of the implementation of the Standards on improving teacher quality
Outcomes – Impact of the campaigns
Next Steps
Not purely an online/social media strategy Involves other forms of stakeholder engagement
including workshops, symposiums, forums and meetings
Rethinking about how we deepen our understandings of the Evaluation through other ways of working across jurisdictions, sectors, schools and organisations
Paper 4: Bringing it together
Janet Clinton & Edmund MissonUniversity of Melbourne & AITSL
Final Comments
Adding value
Defining boundaries of collaboration
Incorporating knowledge translation
Transparency
Demonstrating worth
Understanding Collaborative Evaluation
2013- defining the approach and how it was going work Difficult– the literature often promotes it but doesn't really tell
us how It about all about:
RELATIONSHIPSUNDERSTANDING
TRUSTRIGOUR
ROLE DEFINITION
Collaborative Evaluation
How do we truly be collaborative and maintain objectivity?
Relationship• How it evolves• Practically how it works• What does it mean for
methods
Understanding the influences
Rigorous methods that are open and transparent
Understanding the Influences for each Party
Collaboration
Historical
Economics
Social
PsychologicalPolitical
Cultural
Contextual
Have the conversations and act! Relationships tensions Strong personalities Different management
styles Variable flexibilities Extended partners
Conversation---• Open & frank• Lose the emotion• Process a solution
Collaboration about Methods
Forums
Case Studi
es
Survey
Conversations
Model of Objectivity
OBJECTIVITY
• Recognised Standard evaluation framework• Rigorous methodology• Mixed methods• Evidence of analysis at every level• Triangulation clarity• Sophisticated social science analysis• High level of dissemination• Evidence base content• Every step transparent & reproducible
DIAMOND STANDARD APPROACH
Re-analyse -- suggest—advise– reflect—promote - use
Analyse –demand—contact--micro-manage-- change content- little dissemination
High
Low
High-stakes EvaluationIf it doesn’t work
But….
HANDLE WITH CARE
Questions?
