attc network evaluation 1 national evaluation of the addiction technology transfer center (attc)...

ATTC Network Evaluation

1

National Evaluation of the Addiction Technology Transfer Center (ATTC) Network:

ATTC Event & Activity Reporting Database Review

Roy M. Gabriel, Ph.D.Jeffrey R.W. Knudsen Margaret Gwaltney, M.B.A. Richard Finkbiner, Ph.D.RMC Research Corporation Abt Associates, Inc. MANILA Consulting GroupPortland, OR Bethesda, MD McLean, VA

Presentation to ATTC Directors

November 5, 2009

ATTC NetworkEvaluation

2

Agenda

The ATTC Event & Activity Reporting Database: Review of past action steps

The NO, NET, and the ATTC Network Implementation to date Examination of current data

All events vs. GPRA only

Role of these data in overall evaluation Next steps???


3

Review—Where We’ve Been


4

ATTC Event & Activity Reporting Database and the NET

ATTC Event & Activity Reporting Database was augmented to meet multiple reporting needs: National Office/CSAT, NIDA, and the national evaluation.

In coordination with ATTC National Office, the NET developed a web-based query system to extract and summarize information from the database. Examples: Query the entire Network, specific region(s), time periods,

event types, etc. Tally number of events, number of participants, etc., as

ATTC “outputs” (both GPRA and non-GPRA). Cross-tab tallies w/many other event characteristics

(e.g., funding source, materials source, GPRA/non-GPRA, collaborator type, delivery mode, etc.).


5

The Underlying Premise . . .

During the design phase of the ATTC Evaluation contract, ATTC Directors clearly indicated that, from a service provision perspective, GPRA data did not tell the entire story.

It was clear that other very important services were being provided by ATTC centers, but were not being counted by the GPRA system.


6

The Output is Only as Good as the Input . . . Data Quality

The NET presented a series of sample summary tables at the December 2008 ATTC Directors meeting.

It was determined that assuring useful (and accurate) data queries and reports is contingent on the NET and the Network working together: The ATTC NO and the NET hosted a 2-part webinar series in

March 2009 reviewing (a) the purpose of the Event & Activity Reporting Database (from the perspectives of CSAT, NIDA, and the NET) and (b) data entry procedures.

The ATTC Performance Monitoring Subcommittee was working to (a) re-disseminate activity/event definitions and to (b) establish rules of thumb regarding what non-GPRA events should be entered into the system. The NO provided draft guidelines for review (April 2009).


7

NET Reporting Plans—On Hold

The NET prepared templates for quarterly summaries of events/activities to each regional ATTC. Reports were to feature regional activity data along

with a Network-wide summary of the same information.

The NET had hoped such summary reports would increase data utilization within Network.

Production and dissemination were contingent upon more consistent implementation of data system. The NET does not want to issue reports if numbers

are misleading, skewed, or simply inaccurate.


8

Implementation of Event & Activity Reporting Database

January 1–June 30, 2009


9

GPRA vs. Non-GPRA Events

GPRA? # of Events % of Events Valid % of Events

(Not Specified)a 29 4%

Yes 443 59% 62%

No 273 37% 38%

Totalb 745

aNote: This variable is only required once an event has occurred. “Not specified” responses likely represent event records not updated after the event took place.

bNote: Unduplicated event count.


Illustrative NET Interpretation

In a 6-month period, the Network conducted nearly 750 events and activities. An average of about 50 events/activities per region.

Much variability in these numbers from region to region.

Nearly 40% of these are not captured by the GPRA reporting system.

“Natural” follow-up questions: What kind of events?

Provided to whom? With whom?

On what topics?

10


11

Variability in Entering Non-GPRA Events

3 Regional Centers account for 70% of all non-GPRA events entered into database. These 3 Regional Centers demonstrated a 25%

GPRA/75% non-GPRA balance.

Time Period 0 events 1- 5 events 6- 10 events 10- 15 events 16 + events

Yr2 Q2 3 regions 8 regions 0 regions 0 regions 3 regions

Yr2 Q3 4 regions 4 regions 3 regions 1 regions 2 regions

Yr2 Q2 max/min (non-GPRA Events entered): 54/ 0 Yr2 Q3 max/min (non-GPRA Events entered): 30/ 0


NET Observations/Questions

Is it accurate that: 3–4 regions conducted no non-GPRA

events in these quarters? The majority of regions conduct fewer than

5 non-GPRA events in a quarter? In contrast to the earlier slide that indicated,

“Network-wide,” nearly 40% of the events conducted fall in the non-GPRA category.

What is the balance between GPRA and non-GPRA events in the Network: 60%/40%? 25%/75%? Something in between?

12


13

A Closer Look at Data:Event TypeYR2, Q2–3


14

Event Type (n = 745)

16%

0%

60%

1%

2%

4%

10%

7%

Meeting

Assessment

Educational Event

Formal Presentation

Conference

Brief Tech Asst

Tech Asst

Other


15

Event Type—GPRA Only (n = 443)

88%

5%3%1%

3%

Meeting

Assessment

Educational Event

Formal Presentation

Conference

Brief Tech Asst

Tech Asst

Other


16

Event Type—Non GPRA Only (n = 273)

15%

0%

35%

13%

24%

2%1%10%

Meeting

Assessment

Educational Event

Formal Presentation

Conference

Brief Tech Asst

Tech Asst

Other


17

Event Types for 3 Centers Consistently Logging Both GPRA & Non GPRA Events (n = 256)

32%

0%

22%

11%

23%

1%2%9%

Meeting

Assessment

Educational Event

Formal Presentation

Conference

Brief Tech Asst

Tech Asst

Other


18

A Closer Look at Data:Collaborator Type

YR2, Q2–3


19

Collaborator Type (n = 745 events)

6%

5%

35%

5%4%1%

14%

1%

4%

12%

13%

Another ATTC

CTN Node

SSA

Community Tx Programs

Provider Associations

Recovery Org

State/Regional Org

National Org (non-Fed)

Federal Org

Other

None


20

Collaborator Type—GPRA Only (n = 443)

6%

7%

25%

7%

4%1%16%

2%

5%

14%

13%

Another ATTC

CTN Node

SSA



Recovery Org

State/Regional Org


Federal Org

Other

None


21

Collaborator Type—Non GPRA Only (n = 273)

5%3%

54%

2%

3%

0%

10%

0%

3%

6%

14%

Another ATTC

CTN Node

SSA



Recovery Org

State/Regional Org


Federal Org

Other

None


22

Collaborator Types for 3 Centers Consistently Logging Both GPRA & Non GPRA Events (n = 256 events)

5%2%

45%

5%2%

0%

25%

1%

2%

3%

10%

Another ATTC

CTN Node

SSA



Recovery Org

State/Regional Org


Federal Org

Other

None


23

Many More Examples Available . . .

Previous slides provided only as illustrative examples.

Many other queries could have also been used to demonstrate the impact of non-GPRA data, for example: Primary funding source Tech transfer objective Delivery mode Target audience State-level service saturation


24

What This Means . . .

The inclusion of non-GPRA data changes the overall service delivery “picture” for the Network. This is not a revelation, nor unexpected; but nonetheless,

from the NET’s perspective, very important to the Network.

The change in picture is even more apparent when looking at those few Centers who seem to be higher implementers (or reporters) of non-GPRA activities.

However, given the variability in reporting, the NET can only confidently put forth the service delivery picture drawn from GPRA. When 70% of non-GPRA entries are coming from 20% of the

Network, the climate for skew exists.


Importance of Event & Activity Reporting Database in National Evaluation

The NET has designed a comprehensive series of data collection activities to: Address and assess multiple objectives of tech

transfer in the Network.

Obtain a variety of perspectives on the effectiveness and value of the ATTC Network.

Across all of these data collection activities, the database is the only one that can capture the sheer volume of work produced by the Network (and volume is valued by many key stakeholders).

25


26

Next Steps

The NET will run all developed queries on GPRA events entered into ATTC Event & Activity Reporting Database to learn more about planning processes, partners, funding, materials, objectives, etc. of ATTC provided services.

If the NET can determine that entry of non-GPRA events/activities has improved, pool of events targeted for inquiry will be expanded.

attc network evaluation 1 national evaluation of the addiction technology transfer center (attc)...

Documents

attc network evaluation

attc network implementation

regional attc

attc outputs

attc centers

attc evaluation contract

attc national office

attc directors meeting