nFORM Training #4: Collecting High Quality Data throughout Your Organization:

Equipping Staff to Use nFORM

Grace Roemer MATHEMATICA POLICY RESEARCH

Mathew Stange MATHEMATICA POLICY RESEARCH

Purpose of Today’s Presentation

• To help grantees plan for collecting high quality performance measures data – Performance measurement requirements – Data collection planning and implementation – nFORM and data collection resources – Special data collection scenarios – Answer your questions about data collection

2

1. Performance Measurement Requirements

Performance Measurement Requirements

• Grantees use nFORM to collect performance measures data – Client characteristics and outcomes • Applicant Characteristics Survey • Entrance Survey • Exit Survey

– Enrollment and participation (service delivery) – Program operations • All performance measures posted on FaMLE Cross-Site website:

www.famlecross-site.info

4

Performance Measures Come from Multiple Sources

What is the source? Who enters data? What is the timing

of data collection? Where are

data stored? Applicant characteristics survey Client Intake

Entrance survey Client First workshop

Exit survey Client Last workshop

Enrollment and participation data Program staff Ongoing

Grantee program operations data Program staff Quarterly

nFORM

2. Data Collection Planning

Follow Best Practices in Data Collection

Privacy Consistency

Completeness Timeliness

7

Written Data Collection Plan

• Outline each step of the data collection process– Ideally, the data collection plan will track closely to the

nFORM user manual and provide grantees with specificinformation on using nFORM and collecting data fortheir program

• List and describe data collection steps in as muchdetail as possible

• Include optimal timing of each step, responsiblestaff person, and backup

• Summarize roles and responsibilities so staff knowwhich parts of the plan they are in charge of

8

Example Table of Contents for Data Collection Plan

• Overview of data collection

• Administrative functions and nFORM setup

• Enrolling clients • Setting up and

administering surveys – Applicant Characteristics – Entrance – Exit

• Recording services and workshop participation

• Using nFORM to monitordata collection

• Staff roles and responsibilities

• Data collection checklists • Staff training plan • Data collection process

updates

Include Procedures for Maintaining Client Privacy

• Throughout the plan, identify ways to reduce risk, and what to do ifclient privacy is compromised

• Refer to User Manual Module I and Performance Measures andData Collection Logistics document—Appendix A– Keep discussions about clients confidential– Secure computer accounts– Use password protected screen savers– Store paper materials with personally identifiable information

(PII) in locked cabinets and shred when done• Consult your Institutional Review Board (IRB) if applicable

Include a Roles and Responsibilities Table

• Identify roles of each staffmember– Creating nFORM client

records– Connecting devices to the

Internet– Generating passcodes for

surveys– Handing out devices and

passcodes– Confirming completion of

surveys

Demo Grantee Roles and Responsibilities (August 2017) Names Roles Responsibilities Staff 1 Project

Director Work with site to plan data

collection Staff 2 Survey

Administrator Test Internet and nFORM access Set up surveys

Staff 3 Facilitator Run workshop and record workshop attendance in nFORM

Staff 4 Data Manager Run the Survey Completion operational report to confirm

clients are enrolled

11

Document Data Collection Logistics

• Logistics include: – Obtaining enough tablets or computers for clients to complete

surveys – Arranging for separate computers for program staff to generate

survey passcodes – Providing clean earbuds for clients to listen to survey questions – Bookmarking the survey URL on devices clients will use

https://www.famlecross-site.com/survey

12

Document Data Collection Logistics

• Connecting to the Internet in locations where clients willcomplete surveys– Test Internet connections in advance– Make sure there is enough bandwidth to support survey

administration– Determine whether locations have firewalls that prevent

access• Consider using mobile jetpacks if Internet is not available

– Ensure that jetpacks are optimized for the number of devices accessing nFORM and the surveys

– Test the strength of the jetpack signal in the location inwhich they will be used

Wi-Fi

13

Include Checklists for Data Collection Days

• Create a checklist for each stage of the data collection process – Ideally, tailor this to

constraints of location where surveys will be administered

– Include set-up and administration steps

• Checklists help the processrun smoothly and prevent missed steps fromdisrupting data collection

Applicant Characteristics Survey Setup Up to one hour before the class

begins, generate the Applicant Characteristics Survey passcode foreach client

Write the clients’ nFORM client ID numbers and passcodes on indexcards

Turn on the tablets or computers that clients will use to complete the survey

Open the bookmarked survey URL on each device

Plug earbuds into each device

Train Staff on Data Collection Processes

• Training approach should consider size and experience of your team

• Ask staff to study the data collection plan • Talk through the plan as a group

– Answer staff questions – Discuss logistical concerns – Incorporate staff feedback as needed

• Practice the data collection process with staff • Repeat this process regularly to ensure that it meets current needs

and that staff remain familiar with it

15

3. Data Collection Implementation

Create Client Records in nFORM

• Describe what information is needed – Names – Dates of birth – Contact information

• Discuss confidentiality procedures and IRB approval (if applicable)

• Identify eligible clients – No one under age 13 – Client records cannot be deleted

• Once the information is obtained,complete an application form for each eligible client to create nFORM clientrecord and client ID

19

Administer Surveys (1)

• Generate survey passcodes for clients • Introduce the applicable survey to clients • Hand out tablets or computers • Give each client his or her client ID and passcode • Ask client to log into their survey and confirm their name • Answer any client questions during the survey • Ask client to submit their survey when they are finished • Ask clients to bring their device to program staff, or raise their hand

to signal they are done • Confirm that each client submitted the survey

18

Administer Surveys (2)

• Same basic administration steps for each survey• Entrance survey at beginning of first workshop session

occurrence– Plan short activity to occupy clients while others finish

• Exit survey at end of last workshop session– Must be at least 28 days after Entrance Survey– Follow up with clients who have dropped out of the

program

19

Record Service Delivery Information in nFORM

• Staff record information in nFORM on all services provided to clients – Workshop attendance – Individual service contacts

(such as one-on-one case management) – Referrals – Incentives

20

Use nFORM to Monitor Data Collection (1)

• Client profiles• Operational reports• Query tool• Data export• QPR/PPR• Your monitoring approach should consider pace of client

enrollment and level of services provided

21

Use nFORM to Monitor Data Collection (2)

• Client profile includes client-specific reminders• Client Status and Survey Completion operational reports

– Confirm that all clients have completed enrollment• Enrollment is completed by submitting the Applicant

Characteristics web survey• Service delivery information and workshop attendance

cannot be recorded before enrollment date– Identify clients who need to make up surveys

• Series Session Attendance operational report– Ensure that attendance is recorded for each session

• Additional analysis options: Query tool and data export

22

F #HMRFPavingTheWay

Charlie Brown (Client 10 40001033)

Profile WoOcshops I Sessooos

Progam Information rJD

Marybelh Case Manager; Mathew Case Manager; Mathew Site Administrator Enrollment Date 9120/2015

Client Status Active

Slatus Change 9/20/2015 Cltent &.veys Date

Type 5catus Action

Chen.1 lnfomrat1on rJD Appficant Characteristics Survey Complete~ +~foa M

Entrance Survey 1 ncOO'l>lete •it Exit Survey lncOOl>lete Mi·

.... Appllcadon Date 9120/2015

Population Youth

Grantee location Nofthside

SeJVice Enhanced Services Type #Provided Most Recent

Assignment Service Contacts 11/ 100015 Date of Birth 1W30/1973

A 0 Referrals Follow up nffded 11/100015 Not screened for mtirrote partner violence Incentives

~ Contact Information

230 Main St. Workshop Surrmary

Anytown NJ 08888 Session(s) Attended last Session Attended Next Meeting Date : (908) 555-1032 Name I

Healthy Marriage WOlbhop 1 11/04/2015 11/25/2015 Additional Contacts

Healthy Marriage WOlbhop 3 0 11/19/2015 0 No add1t1onal cootact(s) have been added

HMm

Use Client Profile as Case Management To-Do List

la

HMmF #HMRFPavingTheWay HM.nrYllMllA.:&~M--

Gnntee (iimo Grantee ::::J

I~ ~ LJof4 ~ ~I ~ I ! Fm< I Next Iii\ · ~

Client Status Report Detail as of 8/31/2016 Client Grantee I Chent IO Couple IO Client~ Name Chent First Name Enrollment Date I Chent Status Loe<1t1on

10001439 Bird Tweety Duplicate pending

10001701 22715 Mertz Fred Applicant pending enrollment

Center City 10006599 Allen 1im 8.11812016 Duplicate confirmed

Center City 40001036 Chipmunk Alvin the 8.19/2015 Active

Center City 40001019 CO'(ote Wile E Applicant pending enrollment

Center City 40001038 Cricket Jiminy 812212015 Ac tive

Center City 40001003 Duck Daffy Applicant pending enrollment

Center City 40001005 Gorilla Magi Ila Applicant pending enrollment

Center City 40001026 10092 Jetson George 8.17/2015 Active

Center City 40001027 10092 Jetson Jane 12/1312015 Active

Center City 10001866 24355 Johnson Bill Applicant pending enrollment

Center City 10001879 24355 Johnson Pat Applicant pending enrollment

Center City 40001013 Leghorn Foghorn Applicant pending enrollment

Center City 40001047 Martian Marvin the Duplicate pending

Center City 40001040 McGraw Ouickdraw 911512015 Duplicate confirmed

Center City 10001484 22715 Mertz Ethel 412912016 Active

Center City 40001015 Mertz Fred Applicant pending enrollment

Client Status Operational Report

24

St.rt u oit.. ~15t_61_2_0_16 ______ ~1 ii1!!!1 End Date I SJ27J2i0~.6 lii1!!!1

Gr~ntee I o .. mo Gr~nt:ee "' I

~<4 ~ LJ IOI l ~ ~~ ~ I to0% 'V I I I Find I f\=t ----------------------------~

Survey Completion Detail as of 5/6/2016 Survey Completion Date

"" -Adult couple 10001701 Matt Fred Applicant p:mdimg i emmllment

Adult indM:!wl 10001439 Bird Tw~ty Duplicate p:mdiing

C

C

G

C

ent'=ir Oty Adult couple 40001026 Jet=on G=arge· Att:i'.o,: 8/7/Z015 11/ 9/2015

ent=ir Oty Adult oouple 40001027 Jet=on Jane Acti\'<2 12/Ul:!Ol S L2/13/2015

ent=r Oty Adult oouple 10001484 IMelrtt Etlrsl Arti'li''2 4/29/2016 4/29/2016

ent=r City Adult oouple 40001029 Rubble Barney Ai:th·a 9/ t 4/20t 5 11/ 9/2015

HMIRF #HMRFPavingTheWay HM.nrYllMllA.:&~M--

Survey Completion Operational Report

25

Client Status Report Summary as of 8/4/2017 Grantee Total

(excl Duplicates)

Applicant Pending

Enrollment

Active Completed/Graduated

Consent Revoked

Drop out

Demo Grantee 75 10 62 2 0 0

The program enrolled its new cohort of clients. Does this report show any issues?

Ten applicants have not completed

enrollment. Why might this

happen?

Client ID Couple ID Client Last Name

Client First Name

Enrollment Date

Client Status

10011526 Krueger Freddy Applicant pending enrollment

10011924 31167 Lyman Josh Applicant pending enrollment

10001701 22715 Mertz Fred Applicant pending enrollment

40001019 Coyote Wile E Applicant pending enrollment

10013595 Davis Bette Applicant pending enrollment

40001003 Duck Daffy Applicant pending enrollment

40001005 Gorilla Magilla Applicant pending enrollment

Ten applicants have not completed enrollment. How do you find them?

Use Excel to sort the detail section of the

Client Status operational report.

Survey Completion Summary

Clients Enrolled 4/1/2014 - 8/4/2017

Number of Clients who Completed Grantee Population Applicant

Characteristics Survey

Entrance Survey Exit Survey

Demo Grantee All 33 0 0 Adult couple 0 0 0 Adult individual 33 29 0

Youth 0 0 0

Clients completedtheir entrance survey during thefirst workshopsession. Does this report show any issues?

Four clients did not complete their

entrance survey.

Population Client ID Client Last

Name

Client First

Name

CurrentClient Status

ApplicationDate

Applicant Characteristics Survey*

Entrance Survey

Exit Survey

Adult individual 10006599 Allen Tim Active 4/9/2017 4/9/2017

Adult individual 10011872 Bishop Winston Active 3/31/2017 3/31/2017

Adult individual 40001038 Cricket Jiminy Active 3/31/2017 3/31/2017

Adult individual 10008681 Day Jessica Active 3/31/2017 3/31/2017

Adult individual 40001035 Mouse Mickey Active 3/31/2017 3/31/2017 4/10/2017

Adult individual 10008827 Miller Nick Active 4/2/2017 4/2/2017 4/10/2017

Adult individual 40001017 Bird Tweety Active 4/2/2015 4/2/2015 4/10/2017

Four clients did not complete their survey. How do you identify them?

Sort the detail section of Survey Completion operational report to identify those clients.

Authorization(s) by Active User Date of

User Last Name

User First Name

User Type User Name (email) Account Creation

Last Login # Days since Last

Login Activity

User AccountStatus

Case Manager Marybeth General testuserMG@mpr.com 9/17/2015 4/6/2016 485 Disabled

Site Admin Mark Site Administrator testuserMS@mpr.com 9/17/2015 7/21/2017 14 Active

Site Administrator

Grace Site Administrator testuser127@mpr.com 9/17/2015 8/4/2017 0 Active

Site Admin Gene Site Administrator gsmith@mpr.com 3/29/2016 8/1/2017 3 Active

Site Administrator

Mathew Site Administrator testuser177@mpr.com 9/17/2015 8/4/2017 0 Active

General Mathew General testuser147@mpr.com 9/17/2015 687 Disabled

Site Administrator

Marybeth Site Administrator testuser108@mpr.com 9/17/2015 9/1/2016 337 Disabled

The grantee has four current staff who use nFORM. Does this report show any issues?

No issues! Site administrators

can disable accounts for staff who no longer work for the program.

Accounts will automatically disable

if inactive for 180 days.

Clients enrolled in the program this grant year. Does the query tool show any issues?

Clients have not engaged in services

quickly.

A case managerleaves nFORM’s All Clients screen openwhen he walks across the room to answer a client’s question about theentrance survey. Are there any data collection issues?

To protect client personally identifiable

information, never leave nFORM open

when not in use.

Caseload Summary as of: 8/4/2017 Staff with Case Management Capability

Staff User Account Type

User Account Active?

Number of Active clients

Total number of clients

Site Admin, Mark Site Administrator

Y 2 4

Site Administrator, Grace Site Administrator

Y 2 3

Site Administrator Gene Site Administrator

N 3 4

Unassigned N/A N/A 19 65 Site Administrator, Mathew Site

Administrator Y 7 9

Site Administrator, Marybeth Site Administrator

N 4 8

Average Caseload 3.7

The programrequires allclients be assigned to a case manager. Does this report show any issues?

19 active clients are not assigned to a case

manager.

Sort the All Clients Search to Find Clients Not Assigned to Case Managers

Client ID . ... Last Name~

First

Name. ...

Service

Assignment. ...

Grantee

Location . ... Case Manager(s)

Application

Date . ...

Client

Status . ...

Ell•M•IM•ll Bunny Bugs Enhanced

Services

Northside 8/412015 Active

E H ll·l·l·ll·! Cricket Jiminy Enhanced

Services

Center

City

812212015 Active

Ell •l•l•MHI Day Jessica Center

City

10114/2016 Active

El l l·l·l·ll·tl Fatale Natasha Control - No

Services

Northside 8/6/2015 Active

Ell•M•ll•til Jetson George Control - No

Services

Center

City

81712015 Active

E IM·l·ll·tl Jetson Jane Control - No

Services

Center

City

81712015 Active

EIM•l•Hill Malfoy Draco Enhanced

Services

Southside 31712016 Active

HMIRF #HMRFPavingTheWay HM.nrYllMllA.:&~M--

Survey Completion Summary

Clients Enrolled 4/1/2014 - 8/4/2017 Number of Clients who Completed

Grantee Population Applicant Characteristics Survey

Entrance Survey Exit Survey

Demo Grantee All 33 0 0 Adult couple 0 0 0

Adult individual 12 0 0

Youth 21 0 0

The grantee set enrollment targets only for Youth. Does this report show any issues?

12 clients were enrolled as Adult

Individuals. The client population can be edited until theclient completes their

entrance survey.

The program provides 3 curricula. What is wrong with how these workshops are set up?

One Curriculum should NOT be

Entered as Multiple Workshops

4. Special Data Collection Scenarios

Special Data Collection Scenarios

• Performance Measures and Data Collection Logistics Document providesguidance on each of these scenarios: – Youth in schools – Incarcerated individuals – Administering surveys without internet access – Languages other than English or Spanish

• Work with a point of contact ahead of data collection to determine how to administer client surveys in special settings – Incorporate into data collection plan – Plan for group administration (if applicable) – Plan to offer make-up surveys to clients if needed

• Never administer Entrance and Exit Surveys on the same day

38

Administering Surveys Without Internet Access

• Paper copies of surveys (English and Spanish) available for very rare instances when Internet access is not available – Requires FPS approval

• Data collection plan should include staff responsibilities and timelines for data collection, data entry, and secure destruction of paper surveys

• Clients can complete survey on their own, or staff can read questions to clients with low literacy and help navigate skip patterns

• Record client ID number and date of administration in the designated place on paper survey

• Staff should input responses into nFORM same day if possible 39

Data Collection for Other Languages

• Plan ahead for clients who speak other languages – Review the survey and translate it in advance, not on

the fly • Grantee staff (or interpreter) can read translated

surveys to clients who speak languages other than English or Spanish • Enter the client’s responses into nFORM same day if

possible

40

Recap: Key Considerations

• Determine data collection logistics wellbefore data collection begins– Make sure to have enough devices for clients and

reliable Internet access– Work through any special data collection scenarios

• Prepare a written data collection plan– Practice the plan and revise as needed– Train staff on the plan– Schedule enough staff to perform data collection

within the time frame available

41

Recap: Privacy

• Client privacy is paramount• Follow procedures to safeguard client

information– Module I of the nFORM User Manual – Data Collection and Logistics Document– Follow your IRB procedures, if applicable

42

5. nFORM and Data Collection Resources

43

Information About Data Collection Available on nFORM’s Help Tab

44

Data Collection Logistics Document Available on Help Tab

45

Data Collection Logistics Document

46

6. Answering Your Questions

47