can we leverage facebook’s social structures for survey recruitment? presented at aapor 2015...

Can We Leverage Facebook’s Social Structures

for Survey Recruitment?



Presented at

AAPOR 2015

Hollywood, FL

May 15, 2015

1

Center for Survey ResearchUniversity of VirginiaCenter for Survey ResearchUniversity of Virginia

Thomas M. GuterbockCenter for Survey Research

[email protected]

Rupa S. ValdezPublic Health Sciences

[email protected]

Deborah L. RexrodeCenter for Survey Research

[email protected]

Ishan C. WilliamsSchool of Nursing

[email protected]

University of Virginia

2

mailto:[email protected]





OverviewOverview• Purpose: Improve design of health IT• Challenge: Sample Facebook users who have diabetes• Study design: Screener, then 3 phases of data collection• Bhutta’s method and her recruitment success• How we implemented Bhutta’s method• Results: Screening survey• Results: 3 study phases• Cost, feasibility, ethical issues• Why did Bhutta’s method not work better for us?

3


The substantive problem:The substantive problem:

How can we design better tools that will help community members use information technology to

better manage their health?

How can we secure a representative

sample of Facebook users with type 2 diabetes?

4

The methodological problem:The methodological problem:


5

Health information communication with social network members

Health information communication with social network members

• Health information communication between patients and social network members is a burdensome task (Pratt et al., 2006)

• There are few health IT solutions to support this task, e.g., Epic MyChart, Microsoft® HealthVaultTM , Caringbridge.org

• There are few studies of the design space for the task of health information communication with members of the social network (Newman et al., 2011; Skeels et al., 2010; Weiss & Lorenzi, 2005)


Health information communication on Facebook?

Health information communication on Facebook?

• Developed to support communication with social network

• Most widely used form of social media (1.44 billion active monthly users at end of first quarter 2015)

• Diverse users across gender, race, ethnicity, SES, geographic location

• Already adopted by some patients for health management

• Q: How should features be replicated, modified, removed, or added for consumer health IT?

6


AHRQ Funding AnnouncementAHRQ Funding Announcement

7


Overview of Study DesignOverview of Study Design

Phase 1:

Qualitative exploration

Proposed N=36

revise

Phase 2:

Survey pilot focusgroups

Proposed N=24

General design guidance

Targeted design guidance

Develop questionnaire

Phase 3:

Large sample survey

Proposed N=600

8

Recruitmentvia Facebook

Screening questionnaire


The methodological problemThe methodological problem

• How do we draw an adequately representative national sample of people with type 2 diabetes– . . .who use Facebook?

• Our answer:– Follow the Facebook recruitment method described by

Bhutta (2012)

9


Bhutta’s Facebook studyBhutta’s Facebook study

10


Bhutta’s three easy stepsBhutta’s three easy steps

1. Created a “research” group on Facebook in December 2008

• “Please help me find baptized Catholics!”

2. Found relevant existing groups on Facebook and asked group managers to invite members to join the research group (January 2009)

3. Once participants joined the research group, invited them to do a survey by a direct message

11

Bhutta’s invitationBhutta’s invitation

12


Solicitation resultsSolicitation results

13


Recruitment successRecruitment success

Bhutta also solicited participation and further recruitment from her 200 Facebook friends

Research groups capped at < 5,000 members Had to start three groups to handle all Nearly 7,500 people joined one of three groups

over the course of one month(!) 4,016 people completed 12-minute surveys

70% completed in first 5 days

14


Representativeness?Representativeness? Bhutta’s sample seriously over-represents observant Catholics Demographic weighting did not correct this

15


Our implementationStep 1: Creating study groupOur implementation

Step 1: Creating study group

• Finding a name• Establishing transparency

– “About” section– Files (Who may participate, Study description, FAQs)

• Establishing privacy settings• Establishing ground rules

16


17


Step 2: Messaging administratorsStep 2: Messaging administrators

• Identified groups by search of keywords – Focused on health condition and targeted demographic

groups• We paid $0.19-$1.06 to Facebook to ensure

messages delivered to personal “Inbox”• Follow up messages sent to those not responding

within 5 days• Personalized replies to establish trust and

transparency

18

(Valdez et al 2014)19


Response rate from groups and pagesResponse rate from groups and pages

20


Messages and study group membership Messages and study group membership

21


Step 3: Managing the study groupStep 3: Managing the study group

• Overall goal was to promote engagement without overwhelming members

– Weekly posts to update on study progress• Establishing ground rules

– Use of group page for general questions only– No posting of medical advice and solicitations– Resolved issues by messaging members or blocking

specific profiles

22


Content of interaction with administrators and membersContent of interaction with administrators and members

• 92 messages from administrators and study group members during recruitment

• 26 posts and 14 comments on group’s page that did not violate guidelines

• Qualitative content analysis using NVivo to inductively derive themes from data

– Administrators: assistance; interest; encouragement; clarification; concern; health info; solicitations

– Members: appreciation; support; health info– Details: Valdez, Guterbock et al. 2015

24


Step 4: Recruitment for screening surveyStep 4: Recruitment for screening survey• Private messages to each member after reaching 100-member

threshold• Anonymous online survey programmed in Qualtrics (one link for all

participants)• Up to two reminders sent to those who did not complete the survey• Brief survey assessed:

– Eligibility (over 18, US citizen or residing in the US, diagnosed with T2D, Facebook user)

– Demographics– Facebook use– Interest in study phases– Preferred contact information

25


Screening survey responses receivedScreening survey responses received

27

June 2014Start of Phase 1

May 2015Start of Phase 3

N of group members ~110 182

Total responses 100 163

Unique completions 79 140

Eligible respondents 61 100


Eligibility of screening survey respondents

Eligibility of screening survey respondents

28

May 2015 N %

Eligibility missing 10 7.1

Under age 18 1 0.7

18+, not US resident 20 14.3

US citizen, no diabetes 8 5.7

Has diabetes, not on FB 1 0.7

Fully eligible 100 71.4

Totals 140 100%


General Characteristics Valid %

Gender(n = 96) Male 27.1%

Female 72.9%

Race/Ethnicity(n = 95) White 73.7%

Black or African American 10.5%

American Indian/Alaskan Native 2.1%

Asian 2.1%

Hispanic 11.6%

29

Is our sample representative?Is our sample representative?


Estimated Population DemographicsEstimated Population Demographics

• Population of interest: Facebook users with Type II Diabetes

• Prior research gives us data on US population:– Adults who use Facebook (Duggan et al., 2013)

– Adults with Type II Diabetes (CDC, 2014)

• We can estimate the demographic makeup of our population of interest, if we assume that these two distributions are independent of each other.


General Characteristics Valid % Expected %

Gender(n = 96)

Male 27.1% 49.0%

Female 72.9% 51.0%

Race/Ethnicity(n = 95)

White 73.7% 51.3%

Black or African American 10.5% 18.7%

American Indian/Alaskan Native 2.1% 2.1%

Asian 2.1% 5.0%

Hispanic 11.6% 23.0%

Women, whites are over-represented:

– May be explained by engagement of our contacted groups and pages, frequency of Facebook group use among these populations, and attitudes about scientific research

31

Women, whites are over-represented:

General Characteristics Valid %

Community(n = 96)

Urban 22.9%

Suburban 40.6%

Rural 30.2%

Other 6.3%

Region(n = 84)

Northeast 21.4%

Midwest 20.2%

South 41.7%

West 16.7%

Education(n = 99)

Less than high school 1.0%

High school diploma or equivalent 16.2%

Some college, but no degree 30.3%

Associate’s or Bachelor’s degree 35.4%

Graduate degree 17.2%

Household Income(n = 87)

Less than $30,000 26.4%

$30,000-$74,999 39.1%

$75,000-$149,999 31.0%

$150,000+ 3.4%

Age(n = 99)

18-29 2.0%

30-49 36.4%

50-64 47.5%

65+ 14.1%

32


Results Valid %

General Health(n = 99)

Excellent 2.0%

Very good 20.2%

Good 46.5%

Fair 22.2%

Poor 9.1%

Years Diagnosed(n = 84)

1 Year or less 15.5%

2-3 Years 15.5%

4-10 Years 36.9%

More than 10 years 32.1%

Facebook Use(n = 97)

More than once a day 76.3%

Once a day 17.5%

A few times a week 6.2%

Technologies Used for Facebook Access*(n = 100)

Desktop computer 48.5%

Laptop computer 54.5%

Tablet computer 42.4%

Smartphone 73.7%

Decline to answer 1.0%

*Respondent could choose more than one answer

33


Results Valid %

How often do you share health information on Facebook?(n = 99)

Always 5.1%

Frequently 29.3%

Occasionally 35.4%

Rarely 26.3%

Never 4.0%

How often do you use Facebook to talk about your health?(n = 98)

Always 4.1%

Frequently 25.5%

Occasionally 37.8%

Rarely 20.4%

Never 12.2%

How often do you use Facebook to talk about another person’s health?(n = 98)

Always 1.0%

Frequently 12.2%

Occasionally 33.7%

Rarely 30.6%

Never 22.4%

34


N Valid %

Interest in Study Phases*(n = 100)

Phase 1 70 72.9%Phase 2 66 68.8%Phase 3 75 78.1%Not sure 10 10.4%

*Respondent could choose more than one answer

35

Interest in Study PhasesInterest in Study Phases


Phase 1—Qualitative interviewsPhase 1—Qualitative interviews

• Started when group membership topped 100– June 2014

• Purposive sampling– Emphasizing recruitment of minorities, males

• $25 gift card for those who complete• Semi-structured interviews about:

– How R uses Facebook– How R communicates health information

• 25 completed interviews– Average length = 43 min

• Very rich data (see Guterbock & Valdez 2015, Menefee 2015)– Recorded, transcribed, analyzed with nVivo

36


Phase 2: Focus groups to pilot the main survey

Phase 2: Focus groups to pilot the main survey

• Information from qualitative Phase 1 interviews used to develop detailed, structured survey instrument– Includes randomized vignette experiment

• Instrument ready for pretest January 2015• Contacted 30 participants interested in Phase 2

– Sought out minority and male participants– Offered $25 gift card

• Conducted 5 “Webinar” virtual focus groups in February• Total of 13 participants

– Highly cooperative and informative

• Significant re-structuring of instrument as a result

37


Phase 3: Production surveyPhase 3: Production survey

• Only 34 respondents remained who were eligible and expressed interest in Phase 3

• No incentives offered– Each person sent individual link to Qualtrics survey

• Low rate of response to repeated invitations • 3 completes, 3 partials out of 34 invited.

– RR3= 9%; RR4 = 18%

38


Direct study expendituresDirect study expenditures

• Direct costs of paying FB to put messages in private mailboxes of group and page managers:– $1.94 per completed survey from eligible individual prior to Phase

1 launch

• No payment for completion of screening survey• Phase 1 (Qualitative): $25 gift card for each respondent• Phase 2 (Focus groups): $25 gift card for each focus group

participant• Phase 3 (Main survey): no payments

39


FeasibilityFeasibility• Did not yield sufficient participation for conducting large-sample

surveys, which contrasts with Bhutta’s experience• Would it have been more effective just to advertise on Facebook?

– Advertisements have yielded between 0 and 1548 completed surveys

– Our direct costs were substantially lower than recruitment through Facebook advertisements ($4.28-$32.26 per participant)

– Personnel time is probably greater for our strategy• Demonstrated potential for recruiting participants for qualitative

inquiry– Substantial effort required to build rapport, trust, group size– Not unlike building and maintaining any opt-in panel

40


Ethical ChallengesEthical Challenges• Balancing trust and relationship-building while maintaining

appropriate distance– Facebook’s policy of one account per user– Decision not to accept friend requests– Including more publicly available information on profiles

• Targeting demographic groups not focused on phenomenon of interest– That is: Contacting racial and ethnic minority communities

focused on topics unrelated to health– Possibility of alienating populations we are trying to

engage– Need to state more explicitly: our rationale for contacting

these groups

41


Why did this study not achieveBhutta’s rate of success?

Why did this study not achieveBhutta’s rate of success?

• Catholics ≠ People with diabetes– Diabetes is a hidden and significantly stigmatized disease

• Youthful Catholics ≠ older patient population• Appeal from young grad student ≠ appeal from profs• We offered contingent incentives (gift cards); Bhutta used purely

social appeals• We did not recruit through personal networks

– But most of Bhutta’s recruits came from her appeals to groups

• Facebook 2009 ≠ Facebook 2015 – {height of fad, youth movement} ≠ {older users, growing privacy concerns}

Whatever the reasons: Don’t expect to repeat Bhutta’s success on Facebook today!

42


So, how will we get 600 respondents?So, how will we get 600 respondents?

• Turning to: SSI on-line panel• Seems to offer adequate number of people with diabetes

– Potential to fill most of our racial/ethnic quota cells

• Able to support individual, personal links to our Qualtrics survey – Necessary to control randomization of multi-factor vignettes

• Affordable cost (funded outside AHRQ grant) • Quick turnaround expected

• Data collection now in progress . . .

43


AcknowledgementsAcknowledgements• We would like to thank the administrators of the

Facebook groups and pages that supported our study, and all of our members and participants.

• This research was sponsored by the Agency for Healthcare Research and Quality (R03 HS22930-01). This content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality.

44


ReferencesReferences• Bhutta, C. B. (2012). Not by the book: Facebook as a sampling frame. Sociological Methods & Research, 0049124112440795.

• Center for Disease Control and Prevention. (2014). National diabetes statistics report: estimates of diabetes and its burden in the United States. Retrieved from http://www.cdc.gov/diabetes/pubs/statsreport14/national-diabetes-report-web.pdf

• Duggan, M., Ellison, N., Lampe, C., Lenhart, A., & Madden, M. (2015). Social media update 2014. Retrieved from http://www.pewinternet.org/files/2015/01/PI_SocialMediaUpdate2014_pdf

• Menefee, H. (2015). Type 2 diabetes patients’ health information communication on facebook: A qualitative study of online privacy concerns and health-related social stigmas to inform consumer health IT design guidance. (Masters Thesis, University of Virginia).

• Newman, M. W., Lauterbach, D., Munson, S. A., Resnick, P., & Morris, M. E. (2011). It's not that i don't have problems, i'm just not putting them on facebook: Challenges and opportunities in using online social networks for health. Paper presented at the Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work, pp. 341-350.

• Pratt, W., Unruh, K., Civan, A., & Skeels, M. M. (2006). Personal health information management. Communications of the ACM, 49(1), 51-55.

• Skeels, M. M., Unruh, K. T., Powell, C., & Pratt, W. (2010). Catalyzing social support for breast cancer patients. Paper presented at the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 173-182.

• Valdez, R. S., Guterbock, T. M., Thompson, M. J., Reilly, J. D., Menefee, H. K., Bennici, M. S., et al. (2014). Beyond traditional advertisements: Leveraging facebook's social structures for research recruitment. Journal of Medical Internet Research, 16(10).

• Weiss, J. B., & Lorenzi, N. M. (2005). Online communication and support for cancer patients: A relationship-centric design framework. AMIA ...Annual Symposium Proceedings / AMIA Symposium.AMIA Symposium, 799-803.

45





Presented at

AAPOR 2015

Hollywood, FL

May 15, 2015

46

Thomas M. Guterbock Deborah L. [email protected] [email protected]

Rupa S. Valdez Ishan C. [email protected] [email protected]

University of Virginia





can we leverage facebook’s social structures for survey recruitment? presented at aapor 2015...

Documents

design of health itchallenge

valdezpublic health

health managementq

survey recruitment

design space

information technology

social networkmost

social structures