Gile SURVEYS 1
SURVEYS IN TSSOME PRINCIPLES AND A CASE STUDY
www.cirinandgile.com
Gile SURVEYS 2
WHY SURVEYS?
IN EMPIRICAL RESEARCH“OBJECTIVE” DESCRIPTION AND MEASUREMENT
CANNOT COVER EFFICIENTLY ALL RELEVANT PHENOMENA
IN PARTICULAR:THOUGHTS, MOTIVATIONS, OPINIONS, INTENTIONSCANNOT BE OBSERVED OR MEASURED DIRECTLY
INDIRECT EXPERIMENTAL PROCEDURES ON INDICATORS ARE COMPLEX AND SOMETIMES
TRICKY
ASKING PEOPLE IS ANOTHER OPTION
Gile SURVEYS 3
CAUTION
PEOPLE’S UTTERANCES ARE NOT ALWAYS
TRUTHFUL OR ACCURATE
THEY ARE NOT NECESSARILY EASY TO INTERPRET
BUT THEY ARE OFTEN A USEFUL COMPONENT OF RESEARCH FOR:
- PRELIMINARY EXPLORATION
- COMPLEMENTARY COLLECTION OF INFORMATION
- VALIDATION OF INFORMATION OBTAINED THROUGH OTHER METHODS
RELEVANCE TO RESEARCH INTO TRANSLATION IS OBVIOUS BECAUSE OF:
COMPLEX PROCESSING IS PART OF TRANSLATION, ONLY PRODUCT IS VISIBLE, CORPORA ARE DIFFICULT TO OBTAIN, INTERPRETATION OF INDICATORS IS DIFFICULT
Gile SURVEYS 4
THERE ARE MANY TYPES OF SURVEYS
LARGE SCALE/SMALL SCALE
SPOKEN (INTERVIEWS) VS. QUESTIONNAIRES
COLLECT OBJECTIVE INFO VS. SUBJECTIVE INFO
FOR QUALITATIVE PURPOSES / SIMPLE QUANTITATIVE PURPOSES / ADVANCED QUANTITATIVE
PROCESSING
INDEPENDENT / PART OF A LARGE PROJECT WITH OTHER COMPONENTS
Gile SURVEYS 5
(SOME) OBSTACLES TO COLLECTION OF TRUTHFUL, ACCURATE INFO IN SURVEYS
LACK OF WILLINGNESS OF PEOPLE TO INVEST EFFORTS IN SURVEYS → LOW RESPONSE RATES, INCOMPLETE ANSWERS
- RESPONSES VARY AS A FUNCTION OF SELF-IMAGE AND IMAGE SUPPOSEDLY GIVEN TO INTERVIEWER
(SO LESS THAN TRUTHFUL RESPONSE POSSIBLE)
- RESPONSES VARY AS A FUNCTION OF ULTERIOR INTERESTS
- QUESTIONS (AND ANSWERS) MAY BE INTERPRETED DIFFERENTLY BY AUTHORS AND READERS (RESPONDENTS
AND ANALYSTS)
Gile SURVEYS 6
IMPORTANT STRATEGIES
- MINIMIZE EFFORTS FOR RESPONDENTS(CLEAR QUESTIONS, EASY TO ANSWER, LENGTH OF QUESTIONNAIRES)
- PILOT THE QUESTIONNAIRES TO MAKE SURE PEOPLE UNDERSTAND
PILOTING IS ALSO IMPORTANT TO DETECT MISSING ITEMS, AMBIGUITY AND OTHER WEAKNESSES
- TRY TO ELIMINATE OR REDUCE EFFECT OF SELF-IMAGE(QUESTIONING STRATEGIES, TRAINING OF INTERVIEWERS)
IF NOT POSSIBLE, USE RESULTS CONSERVATIVELY
Gile SURVEYS 7
SAMPLING AND REPRESENTATIVITY
SIZE IS NOT THE ONLY CONCERN
REPRESENTATIVITY IS PARAMOUNT
“POPULATION”“SUB-POPULATION”
“BIAS”(SYSTEMATIC ERROR – versus SAMPLING ERROR)
AVOIDING BIAS THROUGH RANDOM SAMPLING
MAY WANT STRATIFIED RANDOM SAMPLING(RANDOM SAMPLING WITHIN SUB-POPULATIONS THAT YOU
WANT TO INCLUDE)
Gile SURVEYS 8
SAMPLING ISSUES
SMALL SAMPLE SIZE DOES NOT IMPLY BIASTHROUGH IT MAY NOT HAVE FEATURES CLOSE
ENOUGH TO THOSE OF THE POPULATION DUETO SAMPLING ERROR
SAMPLING ERROR DECREASES WITH SAMPLE SIZE(BUT PROPORTIONALLY TO SQUARE ROOT OF SAMPLE
SIZE – MUTLIPLY SAMPLE SIZE BY 4 NAND YOU WILL ONLY REDUCE SAMPLING ERROR BY 2)
SO INCREASING SAMPLE SIZE IS ONLY EFFICIENT TO A MODERATE EXTENT
ESPECIALLY AS PATTERNS STABILIZE
Gile SURVEYS 9
CASE STUDY: PUBLISHING IN TS
EST SEMINAR IN GHENT IN 2005
EDITORS ASKED TO TALK ABOUT: - CRITERIA FOR PUBLICATION
- DIFFICULTIES IN PUBLISHING ACTIVITY- REFEREEING
DURING SEMINAR:JOURNAL EDITORS DID NOT SEEM AWARE OF THE
FACT THAT THEY ARE IN COMPETITION
DID NOT MENTION PROBLEMS WITH REFEREEING
SOSURVEY TO CHECK BOTH
Gile SURVEYS 10
PURPOSE OF THE SURVEY
• IDENTIFY FACTORS WHICH MAY BE RELEVANT/IMPORTANT(NOT QUANTIFY THEM)
• DO SOME EXPLORATIONIN PREPARATION FOR FURTHER STUDIES
IF INTERESTING ISSUES COME UP
SURVEY DESIGN TAKES THESE OBJECTIVES ON BOARD(LESS STRINGENT REQUIREMENTS THAN IF HYPOTHESES
WERE TO BE TESTED OR IF INFLUENCES WERE TO BE QUANTIFIED)
Gile SURVEYS 11
QUESTIONNAIRE OR INTERVIEWS?QUESTIONNAIRES - INTERNET
WHAT QUESTIONS?
ONE SET ON CRITERIA FOR SELECTIONONE SET ON REFEREEING PROBLEMS
(Not standard procedure, but for efficiency and higher response rate)
- THINK OF CERTAIN CRITERIA FOR SELECTION- BUT LET RESPONDENTS ADD
- ON REFEREEING, ASK ABOUT POSITIVE EXPERIENCE AND NEGATIVE EXPERIENCE
- ASK ABOUT REFEREE’S EXPERIENCE
- PLUS QUESTIONS ON RESPONDENTS TO ANALYZE BY EXPERIENCE, COUNTRY, LANGUAGES IF POSSIBLE
(MINIMAX)
Gile SURVEYS 12
CLASSIFY RESPONDENTS AS BEGINNERS / EXPERIENCED / IN THE MIDDLE
BECAUSE THIS MAY BE RELEVANT
TENTATIVE DUAL CRITERION WITH ARBITRARY THRESHOLDS (TURNED OUT TO BE ALL RIGHT – PERHAPS BECAUSE DESIGNER OF
SURVEY “KNEW” THE FIELD WELL ENOUGH)
- How long have you been in the field of TS?- Less than 5 years
- 5 to 10 years- More than 10 years
- How many papers have you published so far in the field, either in journals or in collective volumes?
- Less than 5- 5 to 10
- More than 10
Gile SURVEYS 13
SELECTION CRITERIA WHEN CHOOSING JOURNALS
• Reputation/Rating - ATTENTION this is subjective• Speed of publication
• Choosing the language of publication
Why these criteria?
What scale?- Not at all important- Not very important
- Quite important- I don’t know
Why not numerical? Why not more?
Gile SURVEYS 14
REFEREEING
- How often have you felt that the referee’s comments were useful to you? - ATT
- How often have you felt that referees misunderstood your statements? - ATT- If you have acted as a referee yourself, how often have you felt that you had
to perform this function because nobody else with the right qualification was available or willing to do the job, but were not fully qualified because of language problems or because you were not familiar enough with the
field, topic or research methods concerned? – ATT
• Often/Sometimes / Rarely / I don’t know
NOTE:• BOTH POSITIVE AND NEGATIVE
Why?
• AUTHOR’S AND REFEREE’S VIEWPOINTWhy?
Gile SURVEYS 15
PILOTING
DISTRIBUTION MODE : INTERNETADVANTAGES AND DISADVANTAGES?
SENT TO COLLEAGUES EXPERIENCED IN SURVEYS FOR INPUT (CHIARO AND NOCELLA)
Why?
SOME CHANGES ON BASIS OF THEIR COMMENTS AND SUGGESTIONS
- PILOTING 1: SENT TO ABOUT 15 SCHOLARS(AFTERWARDS, THEY ARE NO LONGER AVAILABLE AS RESPONDENTS)
- PILOTING 2: INITIAL PRODUCTION: FIRST BATCH – POSSIBILITY OF FINDING FURTHER PROBLEMS AND CORRECTING
- THEN FINAL “PRODUCTION”
Gile SURVEYS 16
PREPARATION OF DATA FOR ANALYSIS
• EXCEL FOR SIMPLICITY, PORTABILITY(DATA BASE MORE POWERFUL, BUT MORE DIFFICULT TO EXPORT TO
OTHER COLLEAGUES/SYSTEMS)
• “NAMES”, MEANINGFUL(FOR EASIER READING AND INTERPRETATION THAN CODES)
• W/ DIFFERENT INITIALS(FOR AUTOMATIC COPYING BY EXCEL – SAVES TIME)
• COLUMN FOR FURTHER COMMENTS
• IDENTIFIERS (NOT NAMES OF RESPONDENTS, UNLESS EXPLICIT PERMISSION)
• INITIAL INFO ON RESPONDENTS TO HELP FIND OUT ABOUT INFLUENCE OF POTENTIALLY RELEVANT FACTORS (COUNTRY,
EXPERIENCE)
Gile SURVEYS 17
Ident Country Experience
Publics Reputation of journal
Speed of publication
Language
Useful Misunderstings
Not qualified
Other criteria
1 AUS B 5-10 C 10+ QUITE 3 NOT VY ZERO OFTEN SMTMS NEVER INTERACTN W/ REFEREES
2 BELG B C Q N Z O S RARE POLITICS
3 CAN N/A
4 CHINA
5 CZ
6 DK
7 FINL
Gile SURVEYS 18
COUNTRIESn = 85
AUSTR 2 OEST 3
BELG 4
GERM 4
GR 1
IRL 1
ITAL 4
NL 1
PORT 1
SPAIN 14
UK 4
FRANCE 1
CAN 11
US 1
CHINA 2
IL 3
JAP 1
KOR 6
MALAY 1
CZ 2
POL 3
SLOV 1
DK 5
FINL 3
SWED 1
SOUTHAF 1
N/A 1
2 38 12 13 6 9 1
Gile SURVEYS 19
WHY RELEVANT AND WHAT CAN WE SAY ABOUT THIS SAMPLE?
• FROM ALL OVER THE WORLD
• BUT STRONG WEST EUROPEAN BIAS
• POSSIBILITY OF CHECKING BY REGIONS
• POSSIBILITY OF CHECKING BY COUNTRIES FOR SOME FACTORS
(IF HYPOTHESES ON LANGUAGE, REFEREEING W/O FULL QUALIFICATIONS)
Gile SURVEYS 20
Exper Publs Reput Speed Lang Useful Misunderst
Not qual Other crit
A:15/18%
B:16/18%
C:54/64%
A:14/17%
B:13/15%
C:58/68%
Q:75/88%
NV:10/12%
Q:50/60%
NV:33/37%
Z:3/4%
Q:39/45%
NV:32/38%
Z:13/15%
NA:1
O:42/50%
S:34/40%
R:5/5%
NA:4
O:3/4%
S:35/42%
R:42/49%
NA:5
O:2/3%
S:25/37%
R:24/37%
Z:15/23%
NA:19
INTL 8
SPCLZD 4
READERSHIP 4
POLITICS, OPEN MIND, INTEGRITY OF PAPER/EDITORS
N = 85
A 0-5 yrs
B 5-9 yrs
C > 10 yrs
A > 5
B 5-9
C > 10
Q quite impt
NV not very impt
NA: not available
O often
S stimes
R rarely
Z never
Z: zero imptance
Gile SURVEYS 21
INTERPRETING THE RESULTS
CRITERIA FOR CHOOSING JOURNAL
• REPUTATION: Q 88% CLEARLY IMPORTANT• SPEED: Q 60% - NV 37% OFTEN IMPORTANT – When? • LANGUAGE: Q 45% - NV 38% SOMETIMES NOT CLEAR BY COUNTRY IN SPITE OF HYPOTHESIS – FURTHER
EXPLORATION POSSIBLE
REFEREEING• USEFUL: O 50% - S 40% - R 5% POSITIVE ON THE WHOLE – OVERASSESSED BECAUSE
POLITICALLY CORRECT?• MISUNDERSTANDINGS: O: 4% - S: 42% - R: 49% OK ON THE WHOLE• I NOT COMPETENT: O: 3% - S: 37% • NOT INSIGNIFICANT – UNDERASSESSED? – NOT CLEAR BY
COUNTRY DESPITE SPECIFIC SITUATIONS – FURTHER EXPLORATION
Gile SURVEYS 22
SOME LIMITATIONS AND SHORTCOMINGS OF THE STUDY
- NOT RANDOM SAMPLE, DIFFICULT TO ASSESS BIAS
- QUESTIONNAIRE SENT IN ENGLISH ONLYPOSSIBLE LANGUAGE BIAS,
PARTICULARLY IMPORTANT FOR QUESTION ON RELEVANCE ON LANGUAGE
- IN LAST QUESTION, POSSIBLE AMBIGUITY BETWEEN NON APPLICABLE AND “NEVER”. SHOULD HAVE
ADDED “NON APPLICABLE” CATEGORY
Gile SURVEYS 23
WAS THE EXERCISE USEFUL?
• QUANTITATIVE RESULTS CONFIRM INTUITIVE KNOWLEDGE,
• GIVE GENERAL ORDER OF MAGNITUDE• NO BIG SURPRISES
• PERHAPS USEFUL TO KNOW THAT MORE THAN A THIRD OF THE RESPONDENTS SAY THAT
SOMETIMES THEY ARE NOT FULLY QUALIFIED
• “QUALITATIVE” RESULT: MANY RESPONDENTS WANT TO INTERACT WITH REFEREES – PERHAPS
USEFUL FOR EDITORS’ POLICY DECISIONS