1 the rrfss evaluation framework 2005 background information and progress to date presenters amira...

1

The RRFSS Evaluation The RRFSS Evaluation Framework 2005Framework 2005 Background Background

Information Information and Progress to Dateand Progress to Date

PresentersPresenters Amira AliAmira Ali Jane HohenadelJane Hohenadel Kate O’ConnorKate O’Connor Deborah Radcliffe Deborah Radcliffe

June 22, 2005June 22, 2005

2

Group MembersGroup Members

RRFSS Evaluation Group: RRFSS Evaluation Group: Amira Ali, Elaine Hector, Jane Amira Ali, Elaine Hector, Jane Hohenadel, Irene Koren, Kathy Hohenadel, Irene Koren, Kathy Moran, Karen Moynagh, Lynne Moran, Karen Moynagh, Lynne Russell, Louisa WongRussell, Louisa Wong

PHRED RRFSS Evaluation Group: PHRED RRFSS Evaluation Group: Jane Hohenadel, Adriana Jane Hohenadel, Adriana Newbury, Kate O’Connor, Louise Newbury, Kate O’Connor, Louise Picard, Deborah RadcliffePicard, Deborah Radcliffe

3

Outline of PresentationOutline of Presentation

Link to strategic planning sessionLink to strategic planning session Link to previous evaluation (2002)Link to previous evaluation (2002) Evaluation Framework – RRFSS Evaluation Framework – RRFSS

Evaluation 2005Evaluation 2005 Tasks and TimelinesTasks and Timelines

4

HistoryHistory Project was identified in March 2004Project was identified in March 2004 Proposal developed in May 2004 and an Proposal developed in May 2004 and an

Evaluation committee was formed with Evaluation committee was formed with representation from each of the RRFSS representation from each of the RRFSS regional working groupsregional working groups

The PHREDs were identified as partners in The PHREDs were identified as partners in the project – proposal submitted to them in the project – proposal submitted to them in October 2004October 2004

Terms of reference for the RRFSS group and Terms of reference for the RRFSS group and the PHRED group were finalized in December the PHRED group were finalized in December 20042004

5

Link to Strategic Planning Link to Strategic Planning SessionSession

6

Strategic Planning SessionStrategic Planning Session Strategic Planning Session in December Strategic Planning Session in December

20042004• To develop strategic statements To develop strategic statements

(vision, mission, values, (vision, mission, values, ggoals, oals, strategies) that will guide the collective strategies) that will guide the collective RRFSS work over the next 3 yearsRRFSS work over the next 3 years

• To determine the specific objectives To determine the specific objectives and activities for the next yearand activities for the next year

• To create an action document To create an action document summarizing the directions setsummarizing the directions set

7

Strategic Planning SessionStrategic Planning Session Vision:Vision: RRFSS envisions that all RRFSS envisions that all

decisions within the public health decisions within the public health system to promote and protect health system to promote and protect health and wellbeing and prevent adverse and wellbeing and prevent adverse health events are informed by valid, health events are informed by valid, timely and relevant health intelligencetimely and relevant health intelligence

Mission:Mission: Ontario’s RRFSS is a flexible, Ontario’s RRFSS is a flexible, timely and responsive surveillance timely and responsive surveillance system designed to meet local Public system designed to meet local Public Health intelligence needs and address Health intelligence needs and address information, geographic and time-information, geographic and time-related data gapsrelated data gaps

8

Strategic Planning SessionStrategic Planning Session Four strategies were identified:Four strategies were identified:

1.1. AdvocacyAdvocacy

2.2. CommunicationCommunication

3.3. GovernanceGovernance

4.4. Quality Improvement and Quality Improvement and DevelopmentDevelopment

9

Strategic Planning SessionStrategic Planning Session

Strategy: Quality Improvement and Strategy: Quality Improvement and DevelopmentDevelopment

Priority ActivityPriority Activity: “Conduct the RRFSS : “Conduct the RRFSS 2005 Evaluation to identify areas of the 2005 Evaluation to identify areas of the surveillance system in need of surveillance system in need of improvement”improvement”

The full report “Strategic Plan for The full report “Strategic Plan for Action” can be viewed at: Action” can be viewed at: www.rrfss.on.cawww.rrfss.on.ca

10

Link to RRFSS Evaluation Link to RRFSS Evaluation 2002 2002

11

RRFSS Evaluation 2002 RRFSS Evaluation 2002 Covered first year of operation (2001)Covered first year of operation (2001) Number of health units ranged from 6 to 12 Number of health units ranged from 6 to 12

during the evaluation periodduring the evaluation period Purpose:Purpose:

Inform change to system proceduresInform change to system procedures Contribute to continuous quality improvementContribute to continuous quality improvement Act as a baseline and template for future Act as a baseline and template for future

evaluationsevaluations

The full evaluation report can be viewed at: The full evaluation report can be viewed at: www.rrfss.on.cawww.rrfss.on.ca

12

RRFSS Evaluation 2002RRFSS Evaluation 2002

Evaluation Framework covered Evaluation Framework covered 4 areas:4 areas: Process EffectivenessProcess Effectiveness Collaboration EffectivenessCollaboration Effectiveness UsefulnessUsefulness Cost EffectivenessCost Effectiveness

13

RRFSS Evaluation 2002RRFSS Evaluation 2002

Performance criteria linked to Performance criteria linked to following outcomes expected from following outcomes expected from RRFSS:RRFSS: Increased local awareness of Increased local awareness of

surveillance informationsurveillance information Increased involvement in generating Increased involvement in generating

appropriate, accurate informationappropriate, accurate information Increased practice of evidence-Increased practice of evidence-

based program planningbased program planning Demonstrated or potential cost Demonstrated or potential cost

benefit due to RRFSSbenefit due to RRFSS

14

RRFSS Evaluation 2002 RRFSS Evaluation 2002 Fourteen Key Issues Fourteen Key Issues

IdentifiedIdentified(grouped as below for 2005 (grouped as below for 2005

evaluation)evaluation) Local Resources (2)Local Resources (2) Centralization IssuesCentralization Issues

• Provincial Implications (3)Provincial Implications (3)• Implications for ISR (4)Implications for ISR (4)

Process Considerations (5)Process Considerations (5)

15

Key Issues from 2002 Key Issues from 2002 EvaluationEvaluation

Local ResourcesLocal Resources Adequate staff/resources for data Adequate staff/resources for data

analysis and making data useableanalysis and making data useable

RRFSS coordinators overwhelmed RRFSS coordinators overwhelmed by work at the local levelby work at the local level

16


Centralization – Provincial Centralization – Provincial ImplicationsImplications Need for stability and support for the Need for stability and support for the

central work of the RRFSS central work of the RRFSS partnership partnership

Sustaining funding for existing and Sustaining funding for existing and new participants (smaller health new participants (smaller health units in particular); includes funding units in particular); includes funding for survey administration and for survey administration and analysis analysis

Provincial representation of dataProvincial representation of data

17

Key Issues from 2002 Key Issues from 2002 Evaluation Evaluation

Centralization – Implications Centralization – Implications for ISRfor ISR

Increasing the sample size for a Increasing the sample size for a health unithealth unit

Access to ISR, decision-making, Access to ISR, decision-making, workload and resourcesworkload and resources

Improved timeliness of data from ISRImproved timeliness of data from ISR ISR Ability to handle increase in HU ISR Ability to handle increase in HU

participationparticipation

18


Process Consideration IssuesProcess Consideration Issues Increasing use of data in program Increasing use of data in program

planning and evaluation in the health planning and evaluation in the health unitsunits

Limited new question/module capacity Limited new question/module capacity under current questionnaire under current questionnaire

Process for revising old modules and Process for revising old modules and developing new modulesdeveloping new modules

Upgrading RRFSS websiteUpgrading RRFSS website Sharing data among organizationsSharing data among organizations

19

RRFSS Evaluation 2005 RRFSS Evaluation 2005

20

2005 Evaluation 2005 Evaluation Framework ObjectivesFramework Objectives

1.1. To document progress made in the To document progress made in the key areas identified in the 2002 key areas identified in the 2002 EvaluationEvaluation

2.2. To document some of the “best To document some of the “best practices” used by Health Unitspractices” used by Health Units

3.3. To envisage what it would take to To envisage what it would take to get RRFSS on board for all Ontario get RRFSS on board for all Ontario Health Units and how it could work Health Units and how it could work as a provincial systemas a provincial system

21

2005 Evaluation 2005 Evaluation FrameworkFramework

Refinement of the Key Refinement of the Key IssuesIssues To address the evaluation objectives, we To address the evaluation objectives, we

began with what were identified as began with what were identified as Key Key Issues Issues in the 2002 Evaluationin the 2002 Evaluation

Evaluation group members were asked to Evaluation group members were asked to review each of those 14 Key issues, and:review each of those 14 Key issues, and:• report on progressreport on progress• whether it was still an issue nowwhether it was still an issue now• if so, how important it was for the 2005 if so, how important it was for the 2005

evaluationevaluation

22

2005 Evaluation 2005 Evaluation Framework Framework

Refinement of the Key Refinement of the Key IssuesIssues It was felt that seven issues had been It was felt that seven issues had been

largely addressed, and that the 2005 largely addressed, and that the 2005 evaluation needed only to document their evaluation needed only to document their evolution since 2002evolution since 2002

The issues that we felt were addressed: The issues that we felt were addressed: • increasing the sample size for a increasing the sample size for a

health unithealth unit• improved timeliness of data from ISRimproved timeliness of data from ISR• access to ISR, decision-making, access to ISR, decision-making,

workload and resources, andworkload and resources, and• limited new question/module capacitylimited new question/module capacity

23


Refinement of the Key Refinement of the Key IssuesIssues

The issues that we felt were addressed The issues that we felt were addressed (cont’d): (cont’d):

• limited new questionnaire/module limited new questionnaire/module capacity under current capacity under current questionnairequestionnaire

• process for revising old modules process for revising old modules and developing new modulesand developing new modules

• upgrading RRFSS websiteupgrading RRFSS website

24


Refinement of the Key Refinement of the Key IssuesIssues

The remainder were felt to still be The remainder were felt to still be issues for the RRFSS partnership and issues for the RRFSS partnership and should remain in the 2005 evaluationshould remain in the 2005 evaluation

The outstanding issues were refined, The outstanding issues were refined, defined and expressed as evaluation defined and expressed as evaluation questionsquestions

25

2005 Evaluation Framework 2005 Evaluation Framework Evaluation Questions for Evaluation Questions for

Outstanding IssuesOutstanding IssuesIssueIssue Local ResourcesLocal ResourcesQuestionQuestion Do HUs have adequate staff and Do HUs have adequate staff and

resources for regional and local resources for regional and local administrative tasks of RRFSS?administrative tasks of RRFSS?

Data Collection ToolData Collection ToolQuestionnaireQuestionnaire

Source of DataSource of DataRRFSS reps, MOH of participating HURRFSS reps, MOH of participating HU

26


Outstanding IssuesOutstanding IssuesIssueIssue

Local ResourcesLocal ResourcesQuestionQuestionDo HUs have adequate staff and Do HUs have adequate staff and

resources locally for data analysis and resources locally for data analysis and making data useable?making data useable?


Source of DataSource of DataRRFSS reps, MOH of participating HURRFSS reps, MOH of participating HU

27


Outstanding IssuesOutstanding IssuesIssueIssue Local ResourcesLocal ResourcesQuestionQuestionAre local RRFSS reps able to meet the Are local RRFSS reps able to meet the

local work demand for RRFSS module local work demand for RRFSS module development?development?


Source of DataSource of DataRRFSS reps, MOH of participating HU, RRFSS reps, MOH of participating HU, program and research staff, RRFSS program and research staff, RRFSS coordinator, RRFSS Evaluation Group, ISRcoordinator, RRFSS Evaluation Group, ISR

28


Outstanding IssuesOutstanding IssuesIssueIssue Local ResourcesLocal ResourcesQuestionQuestionAre local RRFSS reps able to meet the Are local RRFSS reps able to meet the

local work demand for RRFSS module local work demand for RRFSS module development?development?


Source of DataSource of DataRRFSS reps, MOH of participating HU, RRFSS reps, MOH of participating HU, program and research staff, RRFSS program and research staff, RRFSS coordinator, RRFSS Analysis Group, ISRcoordinator, RRFSS Analysis Group, ISR

29


Outstanding IssuesOutstanding IssuesIssue Issue Local ResourcesLocal ResourcesQuestionQuestionAre local RRFSS reps able to meet the Are local RRFSS reps able to meet the

local work demand for RRFSS module local work demand for RRFSS module review?review?


Source of DataSource of DataRRFSS reps, MOH of participating HU, RRFSS reps, MOH of participating HU, program and research staff, RRFSS program and research staff, RRFSS coordinator, RRFSS Analysis Group, ISRcoordinator, RRFSS Analysis Group, ISR

30


Outstanding IssuesOutstanding IssuesIssueIssue Centralization Issues: Provincial ImplicationsCentralization Issues: Provincial ImplicationsQuestionQuestionWhat resources are needed currently to What resources are needed currently to

support and to ensure the stability of the support and to ensure the stability of the provincial and administrative work of the provincial and administrative work of the RRFSS Partnership?RRFSS Partnership?


Source of DataSource of DataRRFSS reps, MOH of participating HU, ISR, RRFSS reps, MOH of participating HU, ISR, RRFSS CoordinatorRRFSS Coordinator

31


Outstanding IssuesOutstanding IssuesIssueIssueCentralization Issues: Provincial ImplicationsCentralization Issues: Provincial ImplicationsQuestionQuestion

What funds are required to enable What funds are required to enable continued participation of HUs continued participation of HUs currently participating in RRFSS and currently participating in RRFSS and allow non-participating HUs to join?allow non-participating HUs to join?

Data Collection ToolData Collection ToolKey Informant InterviewsKey Informant Interviews

Source of DataSource of DataRRFSS reps, MOH of all HUs, ISR, RRFSS reps, MOH of all HUs, ISR, MOHLTCMOHLTC

32



Centralization Issues: Provincial Centralization Issues: Provincial ImplicationsImplications

QuestionQuestionIs provincial representation of data a Is provincial representation of data a desired product of RRFSS?desired product of RRFSS?


Source of DataSource of DataRRFSS reps, MOH of all HUs, MOHLTC, RRFSS reps, MOH of all HUs, MOHLTC, PHRED, Chief MOH, external partnersPHRED, Chief MOH, external partners

33


Outstanding IssuesOutstanding IssuesIssueIssue Centralization Issues: Provincial ImplicationsCentralization Issues: Provincial ImplicationsQuestionQuestion

What would it take to extend RRFSS to What would it take to extend RRFSS to all Ontario HUs?all Ontario HUs?


Source of DataSource of DataRRFSS reps, epidemiologists in non-RRFSS reps, epidemiologists in non-participating HUs, MOHLTC, ISR, Chief participating HUs, MOHLTC, ISR, Chief MOH, MOH of all HUsMOH, MOH of all HUs

34



Process ConsiderationsProcess Considerations

QuestionQuestion

To what extent are RRFSS data being To what extent are RRFSS data being used in PP&E in the health units?used in PP&E in the health units?

Data Collection ToolData Collection Tool

Questionnaire, Case ExamplesQuestionnaire, Case Examples

Source of DataSource of Data

RRFSS reps, program and research staffRRFSS reps, program and research staff

35




QuestionQuestion

To what extent are provincial RRFSS To what extent are provincial RRFSS data being shared among agencies data being shared among agencies and organizations?and organizations?


Questionnaire, Key Informant InterviewQuestionnaire, Key Informant Interview


RRFSS Coordinator, ISR, RRFSS repsRRFSS Coordinator, ISR, RRFSS reps

36

Evaluation Framework for Evaluation Framework for RRFSSRRFSS

Evaluation Questions for Evaluation Questions for Outstanding IssuesOutstanding IssuesIssueIssue

Process ConsiderationsProcess ConsiderationsQuestionQuestion

To what extent are health unit To what extent are health unit specific RRFSS data being shared specific RRFSS data being shared among agencies and organizations?among agencies and organizations?


Source of DataSource of DataRRFSS repsRRFSS reps

37

2005 Evaluation Framework2005 Evaluation Framework Evaluation Question for Objective Evaluation Question for Objective

22IssueIssue


QuestionQuestion

What Success Stories can be What Success Stories can be identified for RRFSS? identified for RRFSS?


Workshop, QuestionnaireWorkshop, Questionnaire


RRFSS reps, program and research staffRRFSS reps, program and research staff

38

2005 Evaluation 2005 Evaluation Framework Evaluation Framework Evaluation

QuestionQuestionIssueIssue

Process Considerations (Parked Issue)Process Considerations (Parked Issue)

QuestionQuestion

What is the quality of RRFSS data? What is the quality of RRFSS data?


QuestionnaireQuestionnaire


RRFSS reps, ISR, partnersRRFSS reps, ISR, partners

39

Tasks and TimelinesTasks and Timelines

40

RRFSS Evaluation TasksRRFSS Evaluation Tasks Finalize development of research tools. Finalize development of research tools.

• QuestionnaireQuestionnaire• Focus groupsFocus groups• Key informant interviewsKey informant interviews• Document reviewsDocument reviews

Data collection:Data collection:• RRFSS Workshop-- “Success stories”RRFSS Workshop-- “Success stories”• Primarily August and SeptemberPrimarily August and September

Data AnalysisData Analysis Draft and finalize the reportDraft and finalize the report

41

RRFSS Evaluation RRFSS Evaluation TimelinesTimelines

RRFSS Evaluation Project Tasks and Timelines, 2005

Month

June

July

August

Sept

Oct

Nov

Dec

Task

Develop Research Tools

Data Collection

Data Analysis

Draft Report

Finalize Report

1 the rrfss evaluation framework 2005 background information and progress to date presenters amira...

Documents