Sustainable Attitudes and Behaviors at OSU

Assessing Undergraduate Sustainability Knowledge Campus Wide:Adam Zwickle - OSUTomas Koontz - OSUAndrew Bodine - OSU

Mark Stewart UMDNicole Horvath - UMDEnvironmental and Social Sustainability Lab

From design to implementation and analysis

1OverviewHow we developed our Assessment of Sustainability Knowledge (ASK)

Why an ASK is important & how it can help

An aside on Knowledge and Literacy

Conducting an assessmentThinking long term

22Developing an assessmentBuilt upon the triple bottom line, the three legged stool, the 3 ps Environmental (planet)Economic (prosperity)Social (people)

3knowledge domains

3Developing an assessmentReplicated questions used in the pastCoyle, 2005. Environmental Literacy in America.Solicited topics and questions from expertsHeld expert focus groupsPilot tested among professors, graduate, and undergraduate studentsNarrowed down to 30 questions

4Developing an assessmentDistributed those 30 to OSU studentsUsed IRT to throw out 14Added UMDs 16Distributed those to OSU and UMD studentsUsed IRT to throw out 2Current ASK has 28 items:ess.osu.edu

5Conceptualizing sustainability knowledgeSocialEconomicEnvironmentalSustainability6Which of the following is the most commonly used definition of sustainable development?

Meeting the needs of the present without compromising the ability of future generations to meet their own needsWhat is the most common cause of pollution of streams and rivers?

Surface water running off yards, city streets, paved, lots, and farm fieldsMany economists argue that electricity prices in the U.S. are too low because

They do not reflect the costs of pollution from generating the electricityADD EXAMPLE QUESTIONS

6How is this helpful?7

7Need for measuring knowledgeUniversity goals-More along the lines of:Become carbon neutral by 2050

Less common:Create sustainably minded citizens of tomorrow

Can help track improvement over time

8

By creating a way to quantify an abstract educational goal such as this, we hope to encourage more institutions to make this a priority.8Need for measuring knowledgeServes as an evaluation for specific academic efforts:Interdisciplinary programsSustainability majors/minors

STARS Credit:ER 6 (STARS 2.0) 3 points available

9By creating a way to quantify an abstract educational goal such as this, we hope to encourage more institutions to make this a priority.9Knowledge vs. LiteracyKnowledgeCan be objectively measuredCan be used to evaluate academic programs

Literacy= KnowledgeKnowledge + values, attitudes, and behaviorsCan be used to evaluate outreach efforts, sustainability campaigns

10By creating a way to quantify an abstract educational goal such as this, we hope to encourage more institutions to make this a priority.10Measuring Behaviors, Values, and AttitudesMake sure you are asking the right questions

Collaborate with sustainability departments to target specific behaviors (e.g., leaving lights on)Include questions on behavioral barriersOther behaviors that may get at the same concept

11By creating a way to quantify an abstract educational goal such as this, we hope to encourage more institutions to make this a priority.11Measuring Behaviors, Values, and AttitudesCollaborate with academic departments to develop a good survey designNo need to reinvent the wheelEach survey could be a Masters thesis

12By creating a way to quantify an abstract educational goal such as this, we hope to encourage more institutions to make this a priority.12Conducting an Assessment

13Conducting an Assessment14Find your partners!

IRB approvalRequired for publicationExempt status

Registrar approval*Students emails, majors, and demographics

Survey software*We use Qualtrics, but there are others

*If a large scale assessment is plannedMaximizing Response RatesMaximizing response rates is important to reduce uncertainty about how well your completed sample matches the population of interest.

Dillman (2008) and others have long studied how to maximize response rates for surveys that were telephone, mailed, or in-person.

There is growing research on electronic survey response rates.

15Research MethodsWe compare response rates from five different survey implementations:1: 2012 spring OSU (n=10,000)2: 2013 spring OSU sample A (n=10,000)3: 2013 spring OSU sample B (n=10,000)4: 2013 spring OSU School of ENR (n=538)5: 2013 spring UMD (n=10,000)

We tried different treatments and tracked survey responses with survey software (Survey Monkey and Qualtrics)

16Key Variables Affecting Response Rates among College StudentsTiming When to send the invitation and reminders

Incentives

Email textWho it is from

Questionnaire formatLong list of questions vs. more page clicks

17Timing MattersTime of Semester:Last week of semester and into finals weekMiddle of semester

Time of Day:6:00 am6:00 pm

Reminders are criticalbig spike in completed surveys after each reminder, with a fast decay

18Timing MattersOverall, you want students sitting at their computers but wanting to be distracted

19

IncentivesSome disagreement in research on best way to provide incentivesahead of time (Dillman 2008)randomly select winner of 1 big prizegive more/all respondents smaller prizes

May impact the validity of the data20Survey Responses21

UMDn=1,556OSUn=2,621DayDayEmail text

How the invitation emails are phrased affects response rates.

Besides making the invitation personal, clear, and as short as possible, prior research has found that who the invitation comes from matters. An appeal from a trusted authority increases response rates.

22Email text: Appeal from Authority

We compared an appeal from authority versus an appeal from a peer (student).

Two surveys, A and B, had an appeal from a higher authority (University VP) for first 3 contacts.

For the 4th contact kept the higher authority for A, but switched B to have to an appeal from a grad student

23

Email text: Appeal from Authority

Results:

= additional 538

= additional 348

over the next 65 hours

24

Questionnaire FormatTrade-off:Long list of questions to scroll downShorter lists with a page click to get to the next page

We analyzed respondent drop-outs spotsHighest spots were just after clicking to the next page

We recommend finding a balance25Non-Respondent BiasEven after efforts to maximize response rates, are the non-respondents different from the respondents?

We conducted a non-respondent short survey (5 questions + some demographics).

Results indicate that non-respondents are slightly but significantly less knowledgeable about sustainability topics, but no difference in GPA or pro-environmental behaviors.

26Planning aheadFor longitudinal studies:Write a multiple year IRB Let registrar know this is a yearly surveyFinding partnersSustainability officeAcademic departments with survey expertise (interdisciplinary social science, communications, environmental studies, sociology, political science, psychology)

27AcknowledgementsFunded by:The Ohio State Office of Sustainabilityhttp://sustainability.osu.edu/

OSUs School of Environment & Natural Resourceshttp://senr.osu.edu/

28Thank You!Environment and Social Sustainability Labess.osu.eduContains:This presentation The 28 question ASKForthcoming article

EmailZwickle.1@osu.eduKoontz.31@osu.edu

Questions?2929