Measuring and reporting outcomes for BTOP grants:the UW iSchool approach

Samantha BeckerResearch Project ManagerU.S. IMPACT Study

1UW iSchool evaluation framework

Communities Connect NetworkStudy of Community Technology in Washington State

In 2007, UW conducted a phone survey across 211 agencies in the state identified as community technology providers. 47of these agencies completed the survey, and 7 sites were visited for more in-depth study.

This was combined with earlier data to provide a snapshot of CT in Washington State

2UW iSchool evaluation framework

Findings from the CCN survey indicated that CT was having an impact on communities.

• Three levels of benefits were identified– Individual– Family – Community

• Six domains were found to be important– Employment/economic– Academic skills and literacy– Social inclusion and personal growth– Independence– Access to information and resources– Communication

From 2007 CCN study

3UW iSchool evaluation framework

This lead to the development of the community technology impact analysis framework

Context Analysis

Situated Logic Model

Outcome Measuremen

t

Validation and

Reflective Practice

• Used to frame the Washington Community Technology Opportunity Program goals

• Six domains were found to be important– Employment/economic– Academic skills and literacy– Social inclusion and personal growth– Independence– Access to information and resources– Communication

– A similar process was used to frame the current WA BTOP evaluation framework

4UW iSchool evaluation framework

Identify major policy issues confronting the PCC’s community, such as:• Workforce development• Education• Poverty• Civic engagementAnd link them to activities used to address these issues, such as:• Access to technology and information• Skills building• Development of local content

Step 1: Policy issue mappingContext Analysis

5UW iSchool evaluation framework

Identify other stakeholders concerned with the policy goals

– individuals– groups– organizations– institutions

Collect data about what they do and how they experience PCC servicesInclude stakeholders who are working to achieve similar policy goals or who are affected by the PCC

Step 2: Stakeholder analysisContext Analysis

6UW iSchool evaluation framework

Step 3: Develop a policy or stakeholder logic model

Situated Logic Model

Inputs

• Facilities• Technology• Knowledge• Relationship

s

Activities

• Technology access

• Training and support

• Awareness building

Outcomes

• Effective and efficient service delivery

• Improved decision making

• Gain employment

Impacts

• Improved workforce

• Reduction of poverty

• Improved community health

Policy Issue: Workforce Development

7UW iSchool evaluation framework

Step 4: Develop a PCC Logic Model to link goals with measurable indicators

Situated Logic Model

Inputs

• Facilities• Technology• Knowledge• Relationships

Activities

• Technology access

• Training and support

• Awareness building

Outputs

• Hours of access to technology

• Number of clients participate in training

Impacts

• Clients get jobs

• Clients earn GEDs

• Clients are able to use technology independently

• Clients engage in their communities

8UW iSchool evaluation framework

Sidebar: what’s a measurable indicator?Indicators need to meet certain utilitarian standards. Beyond the actual content of the indicator, they should also be:• specific, unique and unambiguous;• observable, practical, cost effective to collect,

and measurable;• understandable and comprehensible;• relevant (measures important dimensions,

appropriate and related to the program, that are of significance, predictive and timely);

• time bound; and • valid, providing reliable, accurate, unbiased,

consistent, and verifiable data(Hatry, 2006)

UW iSchool evaluation framework 9

Step 4: Bridge the logic models to show how the CTC supports larger policy goals

Situated Logic Model

Inputs

• Facilities•Computers• Internet

connection• Software• Staff

Activities

•Open technology access

•Computer classes

•Tutoring

Outputs

•Hours of access to technology

•Number of clients participate in training

•Number of clients looking for jobs

Impacts

•Clients get jobs•Clients earn

GEDs•Clients are able

to use technology independently

•Clients engage in their communities

Inputs

• Facilities• Technology• Knowledge• Relationships

Activities

• Technology access• Training and

support• Awareness building

Outputs

• Citizen technology access

• Increased citizen knowledge

• Use of technology to support social services

Impacts

• Improved workforce

• Reduction of poverty

• Improved community health

Example: Workforce Development Community technology

1. Workforce development clients use community technology to look for and apply for jobs

2. Clients get jobs

3. Workforce is improved

1 2 3

Workforce Development

Community Technology

10UW iSchool evaluation framework

Identify the most important outcomes to measure:• Tie activities to immediate policy goals

in the PCC community• Link to the larger community policy

context to evaluate overall impact on stakeholders and the community

Step 6: Measure outcomesOutcome Measurement

Step 7: Report outcomesReport outcome measures in the context of the situated logic model to show contribution to community policy goals.

11UW iSchool evaluation framework

Re-examine relationship between outcomes and policy issues:• Validate measures• Challenge assumptions• Interview stakeholders

Step 8: Validate Outcome MeasuresValidation and Reflective Practice

Step 9: Reflect on performanceUse outcome/impact measures to inform your work:• Establish the value of your work• Improve effectiveness• Understand your organization’s contribution to

the community you service

12UW iSchool evaluation framework

Measuring and reporting outcomes for WA BTOP

1UW iSchool evaluation framework

WA BTOP’s reporting system relates to a community technology logic model

Inputs Activities Outputs Impacts

SRs are asked to keep track of and report inputs, activities, outputs, and impacts on a quarterly basis for the duration of the grant in order to:

• Show the value of the services offered• Show the changes in use as a result of the grant• Evaluate the impact of the grant on the outcomes of clients

2Measuring and Reporting OutcomesUW iSchool evaluation framework

Impact types for WA BTOP were informed by the CCN study, BTOP policy goals, and PCC self-identified anticipated outcomes

Computer skills enhancement• Help clients gain computer and

Internet skills to enable them to independently use technology

• Can be at any skills level and includes multimedia training

Employment skills and opportunities• Help clients learn how to use

computers and the Internet to strengthen their ability to become employed

• Provide access to employment opportunities

Education enhancement• Provide supplemental learning

support for students challenged by limited learning resources

Access to information and services• Enable clients to use digitally accessible

information for personal needs like managing health problems

• Enable clients to access government information, benefits, or services

Life skills and social inclusion• Includes special services for people with disabilities• Help clients learn to use technology for managing personal relations• Help clients learn about and use technology for money management,

finding support, interacting with the community, and daily living skills (e.g. getting bus schedules)

• Teach and empower clients to use digital technology to express themselves and participate in their communities

4UW iSchool evaluation framework

• SRs are not expected to have outcomes to report for every category or indicator.

• SRs were asked to review the list of possible outcomes and choose those they wish to track.

• No single client survey can accommodate all PCCs, but samples and advice are provided for SRs for designing instruments and tracking sheets.

Tracking outcomes

4UW iSchool evaluation framework