survey research for elt
DESCRIPTION
this slides are used in my research classes at UKITRANSCRIPT
ParlindunganParlindungan PardedePardedeChristian University of IndonesiaChristian University of Indonesia
Definition
• Survey research designs are procedures inquantitative research in which investigatorsadminister a survey to a sample or to theentire population of people to describe theattitudes, opinions, behaviors, orcharacteristics of the population.
• It is a useful design to use when researchersseek to collect data quickly and economically,study attitudes and opinions, and surveygeographically dispersed individuals.
Goals of Surveys• To describe trends, such as high school students’
preferences in having native speaker teachers or non-native ones.
• To determine individual opinions about policy issues,such as whether English should be taught in primaryschools.
• To identify important beliefs and attitudes ofindividuals, such as college students’ perceptions onthe use of blogs to develop writing skills.
• To get information necessary to evaluate programs inschools, such as the success of using multimedialaboratory in English teaching.
Key Characteristics
1. Sampling from a population2. Collecting data through questionnaires or
interviews3. Designing instruments for data collection4. Obtaining a high response rate
Population, Target Population, and Sample
Senior High SchoolStudents in Jakarta
Students of SMAN16 Jakarta
11th Graders ofSMAN 16 Jakarta
Types of Sampling MethodsSampling Techniques
Non-RandomSampling Techniques
RandomSampling Techniques
ConvenienceSampling
JudgmentalSampling
QuotaSampling
SnowballSampling
SystematicSampling
StratifiedSampling
ClusterSampling
Other SamplingTechniques
Simple RandomSampling
Convenience Sampling
Convenience sampling attempts to obtain a sample ofconvenient elements. Often, respondents are selectedbecause they happen to be in the right place at the right time.
– use of students, and members of social organizations– mall intercept interviews without qualifying the
respondents– department stores using charge account lists– “people on the street” interviews
Judgmental Sampling
Judgmental sampling is a form of convenience sampling inwhich the population elements are selected based on thejudgment of the researcher.
– test markets– purchase engineers selected in industrial marketing
research– bellwether precincts selected in voting behavior research– expert witnesses used in court
Quota SamplingQuota sampling may be viewed as two-stage restricted judgmental sampling.– The first stage consists of developing control categories, or quotas, of
population elements.– In the second stage, sample elements are selected based on convenience
or judgment.
Population Samplecomposition composition
ControlCharacteristic Percentage Percentage NumberSexMale 48 48 480Female 52 52 520
____ ____ ____100 100 1000
Snowball Sampling
In snowball sampling, an initial group of respondents isselected, usually at random.
– After being interviewed, these respondents are asked toidentify others who belong to the target population ofinterest.
– Subsequent respondents are selected based on thereferrals.
Simple Random Sampling
• Each element in the population has a known and equalprobability of selection.
• Each possible sample of a given size (n) has a known andequal probability of being the sample actually selected.
• This implies that every element is selected independently ofevery other element.
Systematic Sampling• The sample is chosen by selecting a random starting point and then
picking every ith element in succession from the sampling frame.• The sampling interval, i, is determined by dividing the population size N by
the sample size n and rounding to the nearest integer.• When the ordering of the elements is related to the characteristic of
interest, systematic sampling increases the representativeness of thesample.
• If the ordering of the elements produces a cyclical pattern, systematicsampling may decrease the representativeness of the sample.For example, there are 100,000 elements in the population and a sampleof 1,000 is desired. In this case the sampling interval, i, is 100. A randomnumber between 1 and 100 is selected. If, for example, this number is 23,the sample consists of elements 23, 123, 223, 323, 423, 523, and so on.
Stratified Sampling
• A two-step process in which the population is partitioned intosubpopulations, or strata.
• The strata should be mutually exclusive and collectivelyexhaustive in that every population element should beassigned to one and only one stratum and no populationelements should be omitted.
• Next, elements are selected from each stratum by a randomprocedure, usually SRS.
• A major objective of stratified sampling is to increaseprecision without increasing cost.
Stratified Sampling• The elements within a stratum should be as homogeneous as possible, but
the elements in different strata should be as heterogeneous as possible.• The stratification variables should also be closely related to the
characteristic of interest.• Finally, the variables should decrease the cost of the stratification process
by being easy to measure and apply.• In proportionate stratified sampling, the size of the sample drawn from
each stratum is proportionate to the relative size of that stratum in thetotal population.
• In disproportionate stratified sampling, the size of the sample from eachstratum is proportionate to the relative size of that stratum and to thestandard deviation of the distribution of the characteristic of interestamong all the elements in that stratum.
Cluster Sampling• The target population is first divided into mutually exclusive and
collectively exhaustive subpopulations, or clusters.• Then a random sample of clusters is selected, based on a probability
sampling technique such as SRS.• For each selected cluster, either all the elements are included in the
sample (one-stage) or a sample of elements is drawn probabilistically(two-stage).
• Elements within a cluster should be as heterogeneous as possible, butclusters themselves should be as homogeneous as possible. Ideally, eachcluster should be a small-scale representation of the population.
• In probability proportionate to size sampling, the clusters are sampledwith probability proportional to size. In the second stage, the probabilityof selecting a sampling unit in a selected cluster varies inversely with thesize of the cluster.
Types of Cluster Sampling
Cluster Sampling
One-StageSampling
MultistageSampling
Two-StageSampling
Simple ClusterSampling
ProbabilityProportionate
to Size Sampling
Three Types of Information Surveys Can Provide
• Factual information, like the characteristics of individualteachers and learners (e.g., students’ age, gender,ethnicity, language background, proficiency level, etc.).
• Behavioral information, i.e. the one that describe whatstudents or teachers have done or regularly do in terms oftheir language teaching and learning, such how oftenstudents look up unfamiliar words in a dictionary or makean outline before they write an essay.
• Attitudinal information, which depict the opinions, beliefs,or interests of teachers or learners. These questions areoften used in needs analysis when researchers want togather information on such topics as what learning goalsstudents have or what skill areas they are most interestedin.
Types of Surveys
Survey
Cross-Sectional
data are collected atone point in time
Longitudinal data are collectedover time
CROSS-SECTIONAL vs. LONGITUDINAL SURVEY
Types of Survey: CROSS-SECTIONAL
A cross-sectional study is one that produces a ‘snapshot’ ofa population at a particular point in time. The researchercollects data at one point in time in order to:(1) to examine current attitudes, beliefs, opinions, or
practices;(2) to compare two or more educational groups (students
with students, students with teachers, students withparents) in terms of attitudes, beliefs, opinions, orpractices;
(3) to measure community needs of educational services asthey relate to programs, courses, or school facilitiesprojects;
(4) to evaluate a program, such as a survey that providesuseful information to decision makers.
Types of Survey: LONGITUDINAL
• The researcher collects data to study individuals over time. Thisdesign is differentiated into:(1) Trend study, involves identifying a population and examining
changes within that population over time, e.g., Gallup Poll, whichis used during elections to monitor trends in the population ofvoters from the primary to the final election;
(2) cohort study, in which a researcher identifies a subpopulationbased on some specific characteristic and then studies thatsubpopulation over time. For instance, a group of 18-year-oldstudents is studied in the year 2001. Five years later (in 2006), agroup of 23-year-olds is studied. (They may or may not be thesame individuals studied in 2001.) Five years after that (in 2011),a group of 28-year-olds is studied.
Surveys
Questionnaire
Telephone
Web/Email
Face-to-FaceInterviews
Survey Procedure
Questionnaire DesignTips for designing questionnaire:1) Consider whether a survey instrument is available to
measure your variables2) Consider modifying an existing instrument3) design your own instrument by following 3 steps:
Write different types of questions, including personal,attitudinal, and behavioral questions; sensitivequestions; and closed-and open-ended questions.Use strategies for good question construction, i.e. usingclear language, making sure the answer options do notoverlap, and posing questions that are applicable to allparticipants.Perform a pilot test of the questions, and make revisionbased on obtained feedback.
Guidelines for Designing Questionnaire1. Keep the questionnaire sufficiently short (30 minutes maximum).2. Avoid jargon. Seek simplicity but avoid being condescending.3. Keep questions short, as long and complex questions are difficult to understand.4. Split double-barrelled questions, e.g. How long have you studied English and
been in receipt of Government support? into two questions.5. Avoid leading questions which encourage a particular answer, e.g. Do you agree
that your supervisor is supportive?6. Avoid negatively framed questions or statements which are difficult to
understand, particularly when asked to agree or disagree.7. Make sure respondents have the knowledge, otherwise you may get false
answers.8. Check terms are suitable for the context in which they are used, as meanings
may vary for different age groups, religions, cultures etc.9. Ensure the frame of reference for each question is clear e.g. when asking for the
frequency of an event, specify the time period.10. Avoid creating opinions. Respondents do not necessarily hold opinions on
topics. Allow a no opinion alternative.11. Use personal wording if you want the respondents to express their feelings.
A good questionnaire …• is complete, i.e. gets all the data you need;• is short, i.e. doesn't abuse the respondents’ time or concentration;• asks only relevant questions;• gives clear instructions;• has precise, unambiguous and understandable questions;• has objective questions, i.e. doesn't suggest answers;• starts with general questions;• has appropriate questions;• puts sensitive questions at the end; is complete, i.e. gets all the data you need;• is short, i.e. doesn't abuse the respondents7 time or concentration;• asks only relevant questions;• 0 gives clear instructions;• has precise, unambiguous and understandable questions;• has objective questions, i.e. doesn't suggest answers;• starts with general questions;• has appropriate questions;• puts sensitive questions at the end;
Data Analysis Techniques• DESCRIPTIVE. Deals with the question of ‘what’
things are like, not ‘why’ they are that way, andincludes means, standard deviations, frequencycounts, graphs, and charts.
• ANALYTICAL. Seeks to explain relationships,causes or consequences, and include bivariateand multivariate analyses such as correlations,cross-tabulations and regressions
• CONTEXTUAL. Narrows down the context byreinterpreting the data for subgroups. E.g. EFLstudents vs. ESL students, academic vs non-academic employees.
Instrument Validity
• Construct validity: Does the questionnaire really measure theconstruct being examined?
• Criterion-related validity. Does the instrument accuratelypredicts (predictive validity) or diagnoses (concurrent validity)some particular variable (criterion).
• Content validity. Does the contents of the questionnaire reallymeasure the variable being measured ? To achieve this,compare your questionnaire to existing related instruments. Ifnone exists, gather expert opinion on each question on theinstrument to determine whether or not it actually tests whatit is supposed to.
Measures for Assuring a Survey’s Reliability
• Giving the same survey on two occasions to thesame individuals and checking the consistency ofthe same response to the same item.
• Having the same individuals taking two forms of asurvey.
• check the internal consistency of responses in asurvey, i.e. seeing how consistently the samerespondents answer similar questions formulatedin different forms
References
McKay, S. L. (2006). Researching second language classrooms.New Jersey: Lawrence Erlbaum Associates, Inc., Publishers
Burns. A. (2010). Doing action research in english languageteaching: A guide for practitioners. New York: Routledge:
Creswell, J. W. 2008. Educational research: Planning,conducting, and evaluating quantitative and qualitativeresearch. New Jersey: Pearson
Denscombe, M. (2010). The good research guide for small-scalesocial research projects. New York: McGraw-Hill
Goddard, W & Melville, S. (2006). Research methodology: Anintroduction. Lansdowne: Juta & Co, Ltd.
Ross, Kenneth N. (ed.). (2005). Educational research: Somebasic concepts and terminology. Paris: International Institutefor Educational Planning/ UNESCO.
Examples• A Survey of English Language Teaching in Lithuania: 2003-2004 by
Ministry of Education and Science of the Republic of Lithuania.(Available online athttp://old.smm.lt/en/stofedu/docs/edu_reform/english%20language%20teaching.pdf)
• A Survey on the Iranian ELT Community's Attitudes to CriticalPedagogy by Hossein Davari, Abutaleb Iranmehr, Seyyed MahdiErfani. (Available online athttp://www.ccsenet.org/journal/index.php/elt/article/view/14567)
• Survey of The English Language Needs of the Senior Students ofPhysical and Biomedical Sciences by: Inga Rozgien and AlbinaTre iokait . (Available online athttp://www.uki.vu.lt/file/Verbum/2011/2/rozgiene_treciokaite.pdf)