internet and program evaluation the non existent relationship

Internet and Program EvaluationInternet and Program EvaluationThe Non Existent RelationshipThe Non Existent Relationship

Benoît Gauthier

Réseau Circum inc. / Circum Network Inc.September 26, 2000

Version 1

2© Benoît Gauthier & Réseau Circum inc. / Circum Network Inc.

OutlineOutline

What non existent relationship? Internet program delivery

• how does it work?

• how is it influenced by program evaluation?

• potential influence

Program evaluation• how does it work?

• how is it influenced by the Internet?

• potential influence

Why the gap?


Who's this guy?Who's this guy?

Political science, research methods, public administration

1983: social marketing research, Justice Canada 1984-1986: program evaluation, CMHC 1986-1989: program evaluation, DOC 1989-1996: mainly program evaluation, Ekos Research 1996-: social and organisational research as well as

program evaluation, Circum Network


What non existent relationship?What non existent relationship?

In ternetas a m eans o f

com m unication

P rogram evaluation

modifies / improves

In ternetprogram delivery

modifies / improves1

2


Internet program deliveryInternet program deliveryHow does it work?How does it work?

Inputs

ActivitiesDeveloping

Outputs (tangible)

Immediate impacts

Long term impacts

Web siteInterface with other

delivery m ech.Usage reports

H um an resources Techno logy Other resources

S erveM anage serversM anage telecomM anage loadsEnsure security

R eceive and dealwith user input

Com plete transactionsAnswer e-m ail

Generate statisticsS um m arise feedbackP roduce reports

Access to contentTarget population

reach

S eek user feedback

Expediency o ftransaction

P rogram -specificim pacts

Target populationwelfare

Increased info andknowledge

ActivitiesDelivering

P lan content anddesign

Review docum entationInterview, discussReview literatureDraw plans

B uild content anddesign

Review best practicesDevelopTest

Update content anddesign

Review feedbackReview best practicesDevelopTest

R eport

User satis faction

R educed costs


Influence of program evaluationInfluence of program evaluationon Internet program deliveryon Internet program delivery

Inputs


Outputs (tangible)

Immediate impacts

Long term impacts









reach

S eek user feedback












R eport

User satis faction

R educed costs

Usabilty

Quality of co

ntent

Currency of co

ntent

User satisfa

ction

measurement


Potential influence of PE on Potential influence of PE on Internet program deliveryInternet program delivery

Inputs


Outputs (tangible)

Immediate impacts

Long term impacts









reach

S eek user feedback












R eport

User satis faction

R educed costs

UsabiltyQuality

of content,

exhaustivity

Currency of co

ntent

User satisfa

ction

measurement

Technical

performanceTurn around tim

e,

quality Performance

indicators

Proportion

reached, prop.

beyond reachContent se

rved,

extent of

duplication, ease of

access

Integrity of

transaction,

comparative speed

Knowledge gains...

Costs in

comparison to

alternative delivery


Program evaluationProgram evaluationHow does it work?How does it work?

Inputs

ActivitiesPlanning

Outputs (tangible)

Immediateimpacts

Long term impacts

ActivitiesImplementing

P rogram s P rogram EvaluatorsM eans o f

com m unicationOther resources

Describe programlogic and context

Identify evaluationissues

P lan evaluationm ethodo logy

Review literatureInterview, discussAnalyse, strategize

Review literatureApply social sc ienceUse creativity

Collect in form ation Analyze inform ationCom m unicate

inform ation

W rite "reports"M ake presentationsD iscuss resultsP ublish findings

Assess optionsP lan

Review docum entationInterview, discussReview literatureAnalyse, m odel

Interview individualsInterview groupsS urveyObserveRecycle secondary data

P rocess qualitative infoP rocess quantitative infoM odel, test hypothesesReconcile

Data setsP rogram profile and

evaluation planR eports and

presentations

Linkages b/wknowledge m akers

and users

K nowledgedissem ination

Im proved decis ionsIm proved programdesign and delivery

B etter perform ingprogram s

A better soc iety


Influence of the Internet on Influence of the Internet on program evaluationprogram evaluation

Inputs

ActivitiesPlanning

Outputs (tangible)

Immediateimpacts

Long term impacts








Review literatureApply social scienceUse creativity


inform ation

Write "reports"M ake presentationsD iscuss resultsP ublish findings

Assess optionsP lan






presentations


and users




A better soc iety

Internet

Web surveys, on-

line focus groupsSpeedy, fr

ee

distribution

Increased speed

Increased speed

Increased speed


Potential influence of the Internet Potential influence of the Internet on program evaluationon program evaluation

Inputs

ActivitiesPlanning

Outputs (tangible)

Immediateimpacts

Long term impacts








Review literatureApply social scienceUse creativity


inform ation

Write "reports"M ake presentationsD iscuss resultsP ublish findings

Assess optionsP lan






presentations


and users




A better soc iety

Web surveys, on-

line focus groups,

real-time data

Increased speed,

two-way exchanges

Increased speed,

two-way exchanges

Increased speed, two-way

exchanges, more effective

&

efficient methods

Distributed

applications,

interactive multi-

party analysis

Speedy, free

distribution,

hyperdocument

model

Increased speed,

depth and reach

On-going dialog

Participation, buy-

in Participation, buy-

in


Why the gap?Why the gap?

Impact of the program evaluation on IPD• experimental character of IPD

• cost of implementing rigourous research methods

• "low" cost of IPD compared to alternatives

Impact of the Internet on program evaluation• resistance to true participation (political cost)

• lack of openness to evidence from others

• knowledge silo model vs. knowledge addition model


ReferencesReferences

Simon Roy, Evaluating Public Web Sites, http://www.ggi.ca/E/WhatsNew.html

This presentation at http://circum.com (original texts)


Where to find meWhere to find me

Benoît Gauthier

Réseau Circum inc. / Circum Network Inc.

819.770.2423, t/f 819.770.5196

[email protected]

http://circum.com

internet and program evaluation the non existent relationship

Documents