1 cross-cultural surveys and translation esrc question bank conference survey measurement: assessing...
TRANSCRIPT
![Page 1: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/1.jpg)
1
Cross-cultural surveys and translation
ESRC Question Bank ConferenceSURVEY MEASUREMENT: ASSESSING THE
RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS
April 10Royal Statistical Society
Janet A Harkness
gesis-ZUMA
![Page 2: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/2.jpg)
gesis-ZUMA 2
Outline
I. FrameworkII. Progress in survey translation
production and assessmentIII. Persistent problems & consequencesIV. Towards solutions
![Page 3: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/3.jpg)
gesis-ZUMA 3
I. Framework
1. Survey translation uses2. Expectations3. Current practice
![Page 4: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/4.jpg)
gesis-ZUMA 4
1. Survey translations uses
Within country research Groups may partially share larger
context Shared contexts, degree of
interaction, immigrant acculturation affect translations required
Across country research Different contexts and languages
![Page 5: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/5.jpg)
gesis-ZUMA 5
2. Expectations for survey translations
Assumption is a good translation asks the same question
maintains semantic & pragmatic meaning maintains measurement properties
retains source design features satisfies multiple other requirements
(askable and answerable, burden, saliency, etc)
![Page 6: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/6.jpg)
gesis-ZUMA 6
3. Common current practice
Depending on discipline Do not assess translation process quality Assess translation quality through back
translation Pretest translated questions Assess translated instrument quality on
basis of statistical analysis (dif, IRT)
![Page 7: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/7.jpg)
gesis-ZUMA 7
II. Progress in survey translation production and assessment
![Page 8: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/8.jpg)
gesis-ZUMA 8
Progress in survey translation production and assessment
1. Guidelines and know-how2. Tools3. Basic research
Procedures, strategies, outcomes
![Page 9: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/9.jpg)
gesis-ZUMA 9
Progress in survey translation production and assessment
1. Guidelines and know-how2. Tools3. Basic research4. Procedures and strategies
![Page 10: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/10.jpg)
gesis-ZUMA 10
1. Guidelines and Know-How
![Page 11: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/11.jpg)
gesis-ZUMA 11
1. Guidelines and Know-How
ESS, USA Census Bureau, International Test Commission, QoL research and publications
Comprehensive sets of guidelines CSDI Workshop Guidelines Initiative (lead groups at ISR, Michigan, UNL and gesis-ZUMA)
![Page 12: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/12.jpg)
gesis-ZUMA 12
2. Tools
![Page 13: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/13.jpg)
gesis-ZUMA 13
2. Tools
Process documents Note-taking templates
Queries, decisions, rationales Harmonization templates
Decisions, rationales Version records
Decisions, rationales
![Page 14: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/14.jpg)
gesis-ZUMA 14
2. Tools Technological options
Memory files -- repeated questions, instructions, answer scales …
Translation software support Parallel presentation of source and target
versions Importing modifications to existing text
Potential do-it-all tools Combine questionnaire production-documentation
with translation version production-documentation
![Page 15: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/15.jpg)
gesis-ZUMA 15
3. Basic research
![Page 16: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/16.jpg)
gesis-ZUMA 16
3. Basic research
Survey translation research on Assessment procedures & outcomes Translation procedures & outcomes Answer scale translation Oral translation and interpreting Tool options Impact of source questions
![Page 17: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/17.jpg)
gesis-ZUMA 17
3. Basic research
Survey translation research on Assessment procedures & outcomes Translation procedures, strategies
& outcomes Answer scale translation Oral translation and interpreting Tool options Impact of source questions
![Page 18: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/18.jpg)
gesis-ZUMA 18
Procedures, strategies, outcomes
Team translation efforts Interdisciplinary expertise Translators, reviewers, adjudicators,
consultants Iterative process
![Page 19: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/19.jpg)
gesis-ZUMA 19
Team Translation
TRAPD model --an iterative cycleTranslationReviewAdjudicationPretesting and refinementDocumentation underpins all stages(cf. frameworks in ESS, SHARE, and US Bureau of Census, Westat, WMHI)
![Page 20: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/20.jpg)
gesis-ZUMA 20
Why iterative TRAPD procedures may need to be repeated at different stages. For example, pre-testing and debriefing sessions with fielding staff and respondents will lead to revisions; these call for further testing of revised translations.
![Page 21: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/21.jpg)
gesis-ZUMA 21
Translate and document
Review, adjudicateand document
SOURCEPretest and document,reiterate if necessary
![Page 22: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/22.jpg)
gesis-ZUMA 22
Basic Team Players
Translators: selected, competent, briefed
Reviewers: selected, competent, briefed
Adjudicator: (takes final decision) selected as possible on basis of skills as well as seniority.
May need to work with Consultant.
![Page 23: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/23.jpg)
gesis-ZUMA 23
Other Team Players Translators: Reviewers: Adjudicator-------------- (Co-ordinator) (Substantive experts) (External assessors)Copy-editors (Programmers)Back-ups (illness, vacation, leave) (Oral translation and interpreting extra)
![Page 24: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/24.jpg)
gesis-ZUMA 24
Basic Procedures
Translators translate Review session: reviewers discuss and review each question Adjudicator decides/signs off
consults with senior reviewer; if sensible/possible also joins in review session(s).
![Page 25: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/25.jpg)
gesis-ZUMA 25
Scenes from a review session …
![Page 26: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/26.jpg)
gesis-ZUMA 26
The team
Senior reviewer, co-adjudicator Translator 1 Translator 2 Project coordinator, survey researcher Pre-tester, survey researcher Survey researcher, lay translation
talent
![Page 27: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/27.jpg)
gesis-ZUMA 27
Clip 1: The meeting begins Four in room discussing the weather Paul and Margrit enter Greetings Seating Framing the session
Paul: I’ve got a parking space till 5.06 pm. Janet: …Yes, we need to finish by five.
![Page 28: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/28.jpg)
gesis-ZUMA 28
![Page 29: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/29.jpg)
gesis-ZUMA 29
Clip 2: reaching decisions
Team has been trying at length to find a phrase that includes atheist views on religion.Translations proposed so far imply people do have religious views, rather than just a view about religion (so the clip begins)
![Page 30: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/30.jpg)
gesis-ZUMA 30
Clip 2 events
Discussion ongoing Janet halts discussion (hand motion) and
identifies the continuing problem Peter makes a new proposal Team consider it and accept Adjustments to other text discussed,
approval re-confirmed The core group takes notes
![Page 31: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/31.jpg)
gesis-ZUMA 31
JH Let’s recapitulate. There’s the problem that some have no faith and no religion … our discussion … solves perhaps what we could do for Muslims and Protestants or among Protestant groups, but does not really solve the other problem.
PM Well, there’s an odd word in German “glaubensvorstellungen” that means other views from those you /they have
JH Ah.. And would that cover atheists too?
.
![Page 32: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/32.jpg)
gesis-ZUMA 32
PM Yes, of course, they have a view about faith, namely none (= without faith)
MR And the atheists are covered
JH Good. Then we’ll take that. Then we have everything
MR And Muslims are also covered
JH Good
(More discussion of how good the solution is andthat earlier text should be modified)
.
![Page 33: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/33.jpg)
gesis-ZUMA 33
![Page 34: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/34.jpg)
gesis-ZUMA 34
What we learn from teams
Language challenges Source question issues Strategies Task knowledge available and needed
![Page 35: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/35.jpg)
gesis-ZUMA 35
Quality improvements considerable, but…
![Page 36: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/36.jpg)
gesis-ZUMA 36
III. Persistent problems
![Page 37: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/37.jpg)
gesis-ZUMA 37
III. Persistent problems
1. Views on translation and what it can do
2. Established common practices3. Nature of questionnaires4. Researchers and good questions
![Page 38: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/38.jpg)
gesis-ZUMA 38
1. Views on translation & what it can do
i. Anyone can translateii. Focus on wordsiii. Neglect/avoidance of adaptation
![Page 39: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/39.jpg)
gesis-ZUMA 39
i) Anyone can translate
My secretary speaks Czech My son studies French Harry spent a year in Turkey
Language ability is not a guarantee for translation ability
![Page 40: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/40.jpg)
gesis-ZUMA 40
ii) Focus on words
Have you felt blue or down recently?
![Page 41: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/41.jpg)
gesis-ZUMA 41
ii) Focus on words
Have you felt blue or down recently? NOT a matter of colour or idiom Temporary state Degree of depressed state [Disclosure] [Symptom relevance]
![Page 42: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/42.jpg)
gesis-ZUMA 42
iii) Neglect/avoidance of Adaptation
![Page 43: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/43.jpg)
gesis-ZUMA 43
Adaptation
Deliberate modification of a question or questionnaire to meet new requirements
Frequently but not necessarily associated with translation
Undertaken in source questions and/or translated questions
Various forms (cf. Harkness 2008)
![Page 44: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/44.jpg)
gesis-ZUMA 44
Adaptation
Do you have difficulty walking several blocks?
Do you have difficulty walking 100 yards?
Do you have difficulty walking 100 metres?
Do you have difficulty walking 200 metres? (Sweden)
![Page 45: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/45.jpg)
gesis-ZUMA 45
Adaptation and Design are related
Measurement properties should remain Intended latent construct should remain Burden or difficulty should not change Relationship to other questionnaire
elements should not change Adaptation can be anticipated in source
instruments
![Page 46: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/46.jpg)
gesis-ZUMA 46
Adaptation and Translation are related
Translation involves adaptation (Adaptation need not involve translation) In the context of translation, some general
types of adaptation can be identified Blends/entanglement of different
types not unusual
![Page 47: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/47.jpg)
gesis-ZUMA 47
2. Established practices as persistent problems
![Page 48: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/48.jpg)
gesis-ZUMA 48
2. Established practices as persistent problems
Back translation False economy Horse-and-cart structures in survey
implementation Horse and Cart are essentially
different and distinct Perceived interdependence minimal
![Page 49: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/49.jpg)
gesis-ZUMA 49
3. Nature of questionnaires as persistent problem
![Page 50: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/50.jpg)
gesis-ZUMA 50
3. Nature of questionnaires
Complicated text type Leads a double life
Covert measurement properties
Surveyspeak and scalespeak
![Page 51: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/51.jpg)
gesis-ZUMA 51
3. Nature of questionnaires
Complicated text type Leads a double life
Covert measurement properties
Surveyspeak and scalespeak
![Page 52: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/52.jpg)
gesis-ZUMA 52
Surveyspeak
Dentist When did you last visit a dentist?
Patient Dentist follow-up Are you sure?
Uh-huh.
I see.
![Page 53: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/53.jpg)
gesis-ZUMA 53
Surveyspeak (2)Interviewer: When did you last visit a dentist?
Respondent: About two months ago.
Survey follow-upWould you say that you are very certain / somewhat certain / neither certain nor uncertain / somewhat uncertain or very uncertain about the date you just gave?
(or attempt to get date)
53
![Page 54: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/54.jpg)
gesis-ZUMA 54
Scalespeak in an “importance” scale
very importantsomewhat importantneither important nor unimportantsomewhat unimportantvery unimportant
This is somewhat unimportant
This is very unimportant
This is somewhat important
![Page 55: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/55.jpg)
gesis-ZUMA 55
4. Good questions (and researchers) as persistent problems
![Page 56: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/56.jpg)
gesis-ZUMA 56
4. Good questions (and researchers) as persistent problems
What makes a question good?
Q measures what it should
Q understood as intended
Q salient for respondents
Q answerable and answered
Demonstrated quality through testing and use
![Page 57: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/57.jpg)
gesis-ZUMA 57
Good questions "fit" their context
Validity and reliability are facilitated by common ground, shared speech community usage and social norms
Intended meaning of question and answer options = perceived meaning
![Page 58: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/58.jpg)
gesis-ZUMA 58
A visual example of a good instrument
![Page 59: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/59.jpg)
gesis-ZUMA 59
Chinese diagnostic doll
Cultural norms on disclosure observed to enable response
Patient remains clothed, doll is naked
Cultural norms on sensitive topics observed
Note: feet are covered
![Page 60: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/60.jpg)
gesis-ZUMA 60
The diagnostic doll reflects and
accommodates the cultural embedding of the
instrument (doll), the researcher (doctor) and
the respondent (patient)
![Page 61: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/61.jpg)
gesis-ZUMA 61
When "good" questions go travelling...
In different contexts, good questions may be poor cultural fits change in "meaning"
different conceptual coverage
socially difficult to ask or to answer
lose or gain saliency
![Page 62: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/62.jpg)
gesis-ZUMA 62
When "good" questions go travelling...
In different contexts, good questions may be poor cultural fits
Translation may then be an inappropriate means to "ask the same question"
![Page 63: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/63.jpg)
gesis-ZUMA 63
Consequences: what can go wrong ….and why
![Page 64: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/64.jpg)
gesis-ZUMA 64
TRANSLATION
Translation (Spanish)We do not talk much about my illness athome
SourceCommunication about my illness at home is poor
Why? Remedy?
![Page 65: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/65.jpg)
gesis-ZUMA 65
PERCEPTION
Translation (French)
Do you have difficulty standing for 2 hours?
SourceDo you have difficulty sitting for 2 hours?
Why? Remedy?
![Page 66: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/66.jpg)
gesis-ZUMA 66
CULTURE or INTENDED MEANING
Translation (German)
Would you take part in a demonstration thatblocks the traffic?
SourceWould you take part in a demonstration?
Why? Remedy?
![Page 67: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/67.jpg)
gesis-ZUMA 67
GLITCH
Translation (Turkish)
Have you ever felt like shooting someone
SourceHave you ever felt like hitting someone
Why? Remedy?
![Page 68: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/68.jpg)
gesis-ZUMA 68
Translation process highlights design issues
Do you provide financial support for grown-up children or grandchildren?
How many hours TV do you watch on an average weekday?
Please give me the initials of your first and last name
Do you prefer OTC or prescription medicines?
![Page 69: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/69.jpg)
gesis-ZUMA 69
Unavoidable design changesEnglish
neither satisfied nor dissatisfied
isiZulu and Hebrew
neither nor not ... not
dissatisfied not satisfied
![Page 70: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/70.jpg)
gesis-ZUMA 70
Unavoidable design changesEnglish
neither satisfied nor dissatisfied
isiZulu and Hebrew "not satisfied and not not satisfied"
cf. Henningsson et al, 1998, Harkness et 2005cf. Henningsson et al, 1998, Harkness et 2005
![Page 71: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/71.jpg)
gesis-ZUMA 71
IV Towards solutions:concluding remarks
![Page 72: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/72.jpg)
gesis-ZUMA 72
IV Towards solutions
See the source as the source Source of information Source of challenges
Source needs to be appraised and tested for suitability for new contexts and languages
![Page 73: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/73.jpg)
gesis-ZUMA 73
IV Towards solutions
See the target as the target Determine aims for target language
questions Appraise source question goals and
means for target realization Engage in deep processing
Target may need degrees of freedom
![Page 74: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/74.jpg)
gesis-ZUMA 74
IV Towards solutions: more progress
Progress will be a process Base any change on evidence
Research and documentation Create critical mass
Collaboration in initiatives Record and share lessons learned
![Page 75: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/75.jpg)
75
Thank you
![Page 76: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/76.jpg)
gesis-ZUMA 76
Harkness, J. (2008) “Comparative Survey Research: Goals and Challenges.” Foundation chapter in: Dillman, D., Hox., J. and de Leeuw, E. (eds.) International Handbook of Survey Methodology, Hyattsville, VA: Erlbaum. Harkness, J., Schoebi, N., Joye, D., Mohler, P., Faass, T. and Behr, D. (2007) “Oral Translation in Telephone Surveys”. In: J.M. Lepkowski, C. Tucker, J.M. Brick, E. de Leeuw, L. Japec, P.J. Lavrakas, M.W. Link and R.L. Sangster. Advances in Telephone Survey Methodology, John Wiley and Sons, Hoboken, New Jersey.Harkness, J.; Pennell, B.-E., Schoua-Glusberg, A. (2004): Survey Questionnaire Translation and Assessment. In: Presser, Stanley, Rothgeb, Jennifer, Couper, Michael, Lessler, Judith, Martin, Elizabeth, and Singer, Eleanor (Eds.): Questionnaire Development Evaluation and Testing Methods, Wiley Series in Survey Methodology. New Jersey: John Wiley & Sons Inc.
Related Literature
![Page 77: 1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649cff5503460f949d0a48/html5/thumbnails/77.jpg)
gesis-ZUMA 77
Harkness, J. (2003): Questionnaire Translation. In: Harkness, Janet A., Van de Vijver, Fons J.R., Mohler, Peter Ph.(Hrsg.): Cross-Cultural Survey Methods. Wiley Series in Survey Methodology. New Jersey: John Wiley and Sons Inc
Harkness, J.; Van de Vijver, F. J. R.; Johnson, T. P. (2003): Questionnaire Design in Comparative Research. In: Harkness, Janet A., Van de Vijver, Fons J.R., Mohler, Peter Ph.(Hrsg.): Cross-Cultural Survey Methods. Wiley Series in Survey Methodology. New Jersey: John Wiley and Sons Inc
Harkness, J.; Schoua-Glusberg, A. (1998): Questionnaires in Translation. In: Harkness, J. (Hrsg.): Cross-Cultural Survey Equivalence. ZUMA-Nachrichten Spezial Band 3. Mannheim: Zentrum für Umfragen, Methoden und Analysen 1998, S. 87-128