research workshop series session 2:...
TRANSCRIPT
![Page 1: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/1.jpg)
Research Workshop Series Session 2: SurveysJill Walston
10/24/2017
![Page 2: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/2.jpg)
Workshop Series 2017September 28 Data, Research, and EvidenceOctober 24 Survey Development and Administration November 15 Interviews and Focus Groups
2018January 17 Observations and Rubric Development March 8 Data AnalysisApril 25 Communicating Research Findings
![Page 3: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/3.jpg)
Today’s Goals
1. Discuss initial considerations for planning a survey project.
2. Learn how to write and test survey items.
3. Learn about techniques for maximizing response rates.
![Page 4: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/4.jpg)
![Page 5: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/5.jpg)
Planning for a survey
![Page 6: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/6.jpg)
Gain consensus on the goals of the survey.
![Page 7: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/7.jpg)
Create a timeline for survey development, administration, data analysis, and sharing results.
![Page 8: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/8.jpg)
Define survey constructs.
Family engagementPrincipal leadershipAcademic engagementSchool safetyDiscipline
![Page 9: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/9.jpg)
Align and limit content to the goals.
![Page 10: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/10.jpg)
Consider who will provide the most accurate data.
![Page 11: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/11.jpg)
Define the target population.
![Page 12: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/12.jpg)
Choose the most appropriate mode for administering the survey.
![Page 13: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/13.jpg)
Activity: Handout 1Work in your group to complete the survey planning form for your survey.
![Page 14: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/14.jpg)
Developing survey content
![Page 15: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/15.jpg)
Explore existing surveys.
![Page 16: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/16.jpg)
NCES surveys
![Page 17: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/17.jpg)
NCEE reports
![Page 18: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/18.jpg)
Other resources: School climate
![Page 19: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/19.jpg)
Guidelines for writing items
![Page 20: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/20.jpg)
Be specific.Do you work full time?
This is more specific.Currently, about how many total hours per week do you typically work for pay, counting all jobs?
![Page 21: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/21.jpg)
Use clear directions where necessary—for example:
• Select all that apply.
• Select only one response.
• Round to the nearest whole number.
• Do not include long-term substitute teachers in your total count.
![Page 22: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/22.jpg)
Avoid double-barreled items.
Each item should represent a unidimensional concept.
![Page 23: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/23.jpg)
Use appropriate reading level.
![Page 24: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/24.jpg)
Typically, four to seven response categories are used for rating scales.
My teacher grades fairly.Strongly disagreeDisagreeAgreeStrongly agree
![Page 25: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/25.jpg)
Using a neutral category
![Page 26: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/26.jpg)
Types of rating scales
• Agreement• Interest• Importance• Frequency• Degree or extent• Similarity (like me to not like me)
![Page 27: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/27.jpg)
For example:
![Page 28: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/28.jpg)
Make responses mutually exclusive.
What is wrong with this item?How many years have you been teaching?
• 0–5 years• 5–10 years• 10+ years
![Page 29: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/29.jpg)
Make responses collectively exhaustive.
What is wrong with this item?How often do you assign homework?
• Monthly• Weekly• Daily
![Page 30: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/30.jpg)
Open-ended items
![Page 31: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/31.jpg)
Methods to pretest survey items
• Expert reviews• Cognitive interviews• Focus groups• Field testing
![Page 32: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/32.jpg)
Cognitive interviews are a common pretesting strategy.
![Page 33: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/33.jpg)
Concurrent “think-aloud”: Ask participants to verbalize what they are thinking.
![Page 34: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/34.jpg)
Retrospective probing: Ask specific questions after responses.
What does the word participate
here mean to you? In your own
words, what is this question asking
you to do?
How do you remember. . .?
Was it easy or hard to answer this
question?Can you tell me
how sure you are that you. . .?
![Page 35: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/35.jpg)
Cognitive interviews: Analysis
Review notes and listen to audio recordings of interviews.
![Page 36: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/36.jpg)
Activity: Handout 3Work in your group to review survey items.
![Page 37: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/37.jpg)
Survey administration considerations
![Page 38: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/38.jpg)
Avoid convenience samples.
![Page 39: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/39.jpg)
Start with a sampling frame.
![Page 40: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/40.jpg)
It is not necessary to survey every member of the target population.
![Page 41: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/41.jpg)
Large random samples are more representative than smaller random samples.
![Page 42: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/42.jpg)
Consider subgroups of interest.
![Page 43: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/43.jpg)
What is a response rate?
![Page 44: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/44.jpg)
How do you maximize response rates?
![Page 45: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/45.jpg)
Contact participants.
![Page 46: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/46.jpg)
Online survey platforms at DPI
• Qualtrics• Google Forms• Verint
4
![Page 47: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/47.jpg)
Procedures for gaining consent
![Page 48: Research Workshop Series Session 2: Surveysies.ed.gov/ncee/edlabs/regions/midwest/pdf/eventslides/DPI-Research-Methods...Learn how to write and test survey items. 3. Learn about techniques](https://reader034.vdocuments.site/reader034/viewer/2022050407/5f84618717dd53020f19c4b4/html5/thumbnails/48.jpg)
Final thoughts: Survey projects will need time for:
• Planning• Survey content development• Survey administration• Data analysis• Sharing findings
See Handout 4 for an example Gantt chart.