1

RESEARCH METHODOLOGY

Indawan SyahriGraduate Program in Language Education

PGRI University

2

THE ‘NUTS AND BOLTS’ OF RESEARCH

The tools of Research1. Model, typology and Ideal

types2. Paradigm3. Concept

Key Terms

3

THE TOOLS OF RESEARCHA good way to introduce key concepts used in

research is to suggest a continuum of research tools that have as their intention ‘explaining’ social phenomena at one end ‘describing’ social phenomena at the other.Explaining Describing

Theory

Model

Typology

Ideal type

Paradigm

Concept

Notes: An ‘explaining’ is not to be understood as being better or more significant that a ‘description of an event

4

TheoryTheory is a scheme or system of ideas or

statements held as an explanation or account of a group of facts or phenomena.

Theory is a statement of what are held to be general laws, principles, or causes of something known or observed.

Theories are understood as abstract nations which assert specific relationships between concepts.

The abstract ideas and propositions contained in theory are either tested in the field by the collection of data or derived from the data itself.

Theory ought to be ‘falsifiable’; i.e., it must be possible to falsify, then it by definition not a theory.

A good theory will be generalizable – and if possible, predictive – and able to be employed in different contexts to the original.

5

ModelA model is an abstraction from reality that

serves the purpose of ordering and simplifying our view of reality while still presenting its essential characteristics.

A model is a representation of reality; it delineates certain aspects of the real world as being relevant to the problem under investigation.

A model makes explicit the significant relationships among the aspects.

A model enables the formulation of empirically testable propositions regarding the nature of these relationships.

6

TypologyTypologies (also called ‘taxonomies’) are

similar systems of classification to ideal types.

Typologies consist of a system of categories constructed to fit empirical observations so that relationships among categories can be describes.

These devices can be seen as loose framework within which to organize and systematize our observations.

Typologies do not provide us with explanations; rather they describe empirical phenomena by fitting them into a set of categories.

7

Ideal typeThe ideal type is a construct that represents

an intellectual description of a phenomenon in its abstract from and does not provide us with explanation.

An ideal type should not be understood as an ‘ideal’ standard in the sense of being perfects, but rather that it is ‘ideal’ in sense of an intellectual construct that may never exist in the real world.

An ideal type is a conceptualization, e.g., the ‘working class’, with which the researcher can compare reality on the ground.

An ideal type does not posit relationships among variables.

8

ParadigmA paradigm is an institutionalization of

intellectual activity which, in effect, socializes students into their respective scientific community.

A paradigm is used in research when students and researchers describe crude and broad grouping of certain approaches to the study of a specific topic; ‘top-down’ and ‘bottom-up’ research paradigm.

A paradigm is used for describing broad approaches to research; ‘positivist’ and ‘interpretivist’ paradigms.

9

Paradigm (continued)Paradigm: this should be reserved for broad

definitions, e.g., the positivist paradigm of research (a realist; patterns and regularities, causes and consequences, explanation based, objectivity, etc.)

Discipline: this usually applies to ‘traditional’ academic disciplines like economics, history, linguistics, etc.

Perspective: an academic perspective pertains to (a) certain approaches within a disciplines, e.g., communicative approach in language teaching, and (b) approaches that transcend narrow academic disciplines, e.g., a ‘interlanguage’ perspective in language teaching.

10

ConceptsConcepts are the building blocks of theory,

hypotheses, explanation and prediction.A concept can be seen as an idea or notions

expressed and compressed into one or more words.

A concept carries with it a certain perspective and certain built-in assumptions or ways of looking at empirical phenomena and can be seen as a agreed-upon term among scholars.

A concept is an abstraction of empirical phenomena based on certain assumptions and can be used as a type of shorthand in explanation.

11

KEY TERMS IN RESEARCHMethod

All the root of research lies what the ancient Greeks term methodos meaning ‘the path towards knowledge’ and ‘reflections on the quest for knowledge-gathering’.

Research method can be seen as the techniques or procedures used to collate and analyze data.

The method (M) chosen for a research project are inextricably linked to the research questions (QS) posed and to the sources (S) of data collected

(RQ M S)Methods: quantitative research (quantity

and quantifying) and qualitative research (interpreting the subjective experiences)

12

MethodologyMethodology is concerned with the logic of

scientific enquiry; in particular with investigating the potentialities and limitations of particular techniques or procedures.

It pertains to the science and study of methods and the assumptions about the ways in which knowledge is produced.

A Project’s methodology in concerned with the discussion of how a particular piece of research should be undertaken and so can best be understood as the critical study of research methods and their use.

13

The Interrelationship between the Building Blocks of Research

Ontology Epistemology Methodology Methods Sources

What’s out there to know?

What and how to know about it

How to go about acquiring that knowledge

Which precise procedures used to acquire it Which data

to collect

14

ASKING APPROPRIATE QUESTIONSOwn interest and curiosity. “I wonder if…”

or “I wonder why…”Sources of Research questions

1. Experiences2. Journals3. Articles4. “critical friends”

The more specific the area, the easier the question should be to formulate clearly.

Looking at previous research you will have better prepared to ask relevant questions.

15

DEFINING THE RESEARCH QUESTION (RQ)What is Research?Asking Appropriate QuestionsStating RQ and HypothesisThe Highlights of Formulating

Your ResearchWhat is “a systematic may”?Internal ValidityExternal ValidityFeasibility of Research

16

WHAT IS RESEARCH?

A systematic approach to searching for answers to questions.

Asking questions, using systematic approach and end with answers to the questions.

Asking appropriate questions, selecting the best and optimally (the shortest) ways to find answers, and to interpret the findings in a way that we can justify.

17

APPROPRIATE QUESTIONS (continued)Research questions should:1. interest us2. promise new information or confirm old

information in new ways3. have reasonable scope4. have key terms that are clearly defined

and operationalized

18

Stating Research questions and Hypotheses

You should be able to state a more precise research question and have possible answers to the questions as well (the possible answers = hypotheses)

A hypothesis is a tentative statement about the outcome of the research.

While the research problem in a question form, the hypotheses are generally made in form of statements.

The hypothesis may be stated as a null hypothesis (Ho) and as an alternative hypothesis (H1). E.g. Ho: There is no order of acquisition of English

spelling patterns.H1: There is an order of acquisition of English

spelling patterns.

19

THE HIGHLIGHTS OF FORMULATING YOUR RESEARCH:

1. IDENTIFY A RESEARCH PROBLEM.2. NARROW THE TOPIC DOWN AS MUCH

AS POSSIBLE.3. REVIEW THE LITERATURE ON THE

TOPIC AS COMPLETELY AS POSSIBLE.4. STATE THE PROBLEM IN A QUESTION

FORM.5. STATE HYPOTHESES ABOUT THE

EXPECTED OUTCOME OF THE RESEARCH.

20

WHAT IS “A SYSTEMATIC WAY”?

It simply means that the researcher should follow established principles.

This is a matter of avoiding ad hoc solution during the investigation.

By clearly outlining your procedure and maintaining consistency, you can reduce the effect of your personal preferences as well as other extraneous factors which might influence the outcome of the research.

The researcher should consider which approach, e.g., longitudinal or cross-sectional approach will be most efficient in giving answers to the research questions.

If the procedures are flawed, then neither the results nor the interpretation of the result can be valid.

21

INTERNAL VALIDITYThe extent to which the outcome is a

function of the factor you have selected rather than other factors you have not controlled.

There are many factors which can influence the internal validity of the research study; among them are maturation, test effect, subject selection, and the history factor.

Maturation refers to the general developmental changes in subjects during the course of the research.

Test effect is another influencing the internal validity of research, e.g., one may have better score in posttest due to learning something in pretest.

22

INTERNAL VALIDITY (continued)

Subject selection may influence the results of research studies. History factor may happen when something else was happening at the same time as the research study being conducted.

Many possibilities are available to researchers to help obtain internal validity; among them are using random procedures and statistical procedures.

23

EXTERNAL VALIDITY

The extent that the outcome of any research study would apply to other similar situations in the real world.

Selection of subjects is a factor influencing external validity.

A random sample can help researcher obtain external validity.

24

FEASIBILITY OF RESEARCH

1. HOW MUCH TIME YOU HAVE TO INVEST?

2. HOW CAN YOU GAIN ACCESS? (QUANTITY AND QUALITY OF ACCESS)

3. HOW MUCH DOES THE PROJECT COST?

25

DESCRIBING VARIABLES Research Variables

Variable vs. LevelMeasurement of Variables

Nominal Scale VariablesInterval Scale VariablesFrequency Data Vs. Score Data

Functions of VariablesDependent & Independent VariablesModerator & Control VariablesOther Intervening Independent Variables

26

Research VariablesA Variable can be defined as an attribute

of a person, a piece of text, or an object which varies from person to person, text to text, object to object, or from time to time.

Variables can be very broad or very narrow.

The broader the variable, the more difficult it may be to define, locate, and measure accurately.

Variables can be assigned to groups of people as well as individuals.

27

Variable vs. LevelWe may wish to look at levels within a

variable.E.G., ESL student as a variable may be

divided into two levels; foreign students and immigrant students.

We can limit the range or expand them, e.g., bilingual as a variable ; either is or is not bilingual (two levels) – yes/no levels.

Proficiency as a variable can be expanded into three levels; advanced, intermediate, and elementary

28

Measurement of VariablesWhen variables are of the all-or-nothing,

yes/no sort, we cannot measure how much of the variable to attribute to the person, text, or object. So how?

Variables will be quantified in different ways depending on whether we want to know how often an attribute is present or how much of the variables to attribute to the person, text, or text.

29

Nominal Scale VariablesA nominal variable names an attribute or

category and classifies the data according to presence or absence of the attribute.

The classification numbers have no arithmetic value, e.g., in yes/no notation, 1 represents yes and 2, no. and in south Sumatra dialects, 1=Palembang, 2=Lahat, 3=Komring, 4=others.

There is no reliable method of measuring the degrees, e.g., of how much happiness one possesses at any moment.

30

Ordinal Scale VariablesAn ordinal scale variable shows how much

the variable attributes a person, text, or object in term of degree.

E.g., the variable happy can be measured in term of degree of happiness;very unhappy—unhappy—happy– very happy

Most researchers prefer to use a 5-point, 7-point or 9-point scale

You may use a Likert-type scale to tap students attitudes:The lesson were boring 1 2 3 4 5 6 7

31

Interval Scale VariablesAn interval scale tell us how much of the

variable to attribute a person, text or object which more precise measurement.

The intervals of measurement can be described.

Each interval unit has the same value so that units can be added or subtracted.

E.g., the difference in intervals between 1 and 10 in a test of 100 items.

32

Frequency Data vs. Score DataAnother way to think about difference in

measurement is to consider whether the study measures how much on an interval or ordinal scale or whether it measures how often something occurs– the frequency of nominal measurement.

This distinction is important because it will determine the appropriate statistical analysis to use with the data.

33

Functions of VariablesTo understand how the variables in a study

relate to one another, we need to be able to identify their functions.

Functions grow out of the RQ.Functions depend on the connection we

believe exists between the variables we have chosen to study.

One variables are expected to be related to one another or one affects the others.

34

Dependent Variable (DV)The dependent variable is the major

variable that will be measured in the research.

E.g., the construct communicative competence (CC) of a group of students, the dependent variable is the construct and it might be operationalized as the students’ scores or ratings on some measure of CC.

We expect the performance on the dependent variable will be influenced by other variables.

35

Independent Variable (IV)An independent variable is a variable that

the researcher suspects may relate or influence the dependent variable.

E.g., if we want to know something about CC of students, the dependent variable is the score for CC. We might believe that the male and females students differ on this variable. So we can assign gender as the independent variable.

36

Moderator Variable (MV)Some researchers makes a distinction

between IV and MV, others call them both IVs since they influence variability in the DV.

Specifying variables as “independent” and “moderator” helps us study how moderating variables mediate or moderate the relationship between the IV and DV.

E.g., we believe that gender as IV is the most important variable to look at L2 compliments, but we observe that length of residence suspected to influence the DV can be considered as MV

37

Control Variable (CV)CV is a variable that not of central

concern in a particular research project but which might affect the outcome.

It is controlled by neutralizing its potential effect on the DV.

E.g., if you assume that handedness can affect the Ss’ respond in many task, you can institute a control by including only right-handed Ss in your study

Remember when you control in this way, you limit the generalizability of your study.

38

Other Intervening Independent Variables

Young adults

Additional Education Increased Income

Older adultsIntervening variable

39

CONSTRUCTING RESEARCH DESIGNSDetermining the design

The dependent vs. independent variablesRepeated-measures vs. Between-groups designsMixed designs (split-plot designs)

Practice in drawing design boxesClassification of designs

Studies with intact groupsOne-shot designOne-group pretest-posttest designIntact group—single controlTime-series designsExperimental studiesRandom assignment posttestControl group pretest-posttestEx post facto designs

40

Determining the DesignIn addition to

determining the scope of the RQ, stating the RQ as clearly as possible, giving operational definitions to key

terms, identifying variables, understanding the roles variables will

play in the researchhow those variables will be observed, we need to plan the overall design of the

project.This is important for it will help us to

determine how the data will be analyzed.

41

Dependent vs. independent variables Common questions asked by advisors or

consultants:Tell me about your research.What are you trying to find out?So X is your dependent variables and A, B, and

C are you independent variables, right?

If the RQ is not clearly stated, it may take time to determine this first crucial piece of information: which variables in the research are which.

42

Dependent vs. independent (cont.)The conditions that we ourselves vary are called independent

variables.Those whose response we are measuring are dependent

variables. Independent variable Dependent variable

Type of teaching method Score on language test

Sentence complexity No. of sentences recalled correctly by subjects

Social class Percentage of –ing endings pronounced [in] by subjects

Topic of newspaper article Sentence length

43

Repeated-measures vs. Between-groups designs

Repeated-measures, e.g.:A is asked to teach literature to the students of

English study program. He is not sure just how appropriate his selection of short stories might be. Five different themes are presented in the stories. Since he will teach the course again in the next term, he wants to be sure of his choices. He asks the Ss to rate the stories on a number of factors. Each S’s ratings of the stories within a theme is totaled as that S’s score for the theme

T1 T2 T3 T4 T5

44

Between-groupsYou are comparing native and nonnative

teachers of English teaching English in term of their success in teaching. There two groups of Ss you observe, i.e., those who are taught by NT and NNT.

NT NNT

45

Mixed designs (split-plot designs)

If we survey all the Ss in English Study program regarding the value they placed language vs. literature courses in their university training, then we have only one group of Ss and we draw comparisons for the courses by repeated measures of that one group.

Ratings

language Literature

46

Practice in drawing design boxes Decide whether the following studies

involve a repeated-measures design or a between-groups design, or both and draw a design box.

1. In a cross-cultural analysis, you ask Ss to watch a video and judge the appropriateness of small-talk behavior in each of five different episodes. The Ss are Indonesian, Chinese, and Indian.

2. Using a 30-item questionnaire, you ask native speakers of English and non-native Ss to decide whether a single verb or a two-word verb would be more appropriate in a given context. Each question contains a context and then a choice between a single verb (such as telephoned) and a two-word verb (such called up). Fifteen items appear in a formal context and 15 in an informal context. Does context type influence verb type choice?

47

Classification of Designs1. Studies with Intact Groups

• The subjects/samples have been assigned on the basis of some principles.

• It is impossible randomly to select Ss to begin with.

• Ss may self-select a section according to their timetables.

• In intact group studies, we are unable randomly to select or randomly to assign Ss for research purposes.

• Such designs will not allow us to make causal (cause-effect) statements about the findings.

48

2. One-Shot DesignsIn many teaching programs, teachers

want to know whether Ss meet the objective set for the course.

At the end of the courses (and whatever happened during the course) Ss take a test.

T - XThis is a very simple design but the

results must be interpreted with great care.

The results may not be valid and you cannot generalize from them with confidence.

49

3. One-Group Pretest-Posttest DesignYou give a pretest on the first day of class

and a posttest at the end of the course.X1 – T – X2

By giving the pretest, you can ensure yourself that Ss did not already know the material tested on the posttest.

The pretest-posttest design for one group has many drawbacks that you can consider in your study group.

50

4. Intact Group-Single Control If you are working with a class and there

are Ss in other sections of that same course, it is possible to establish a control group for the research.

Ss have not been randomly selected for the course nor randomly assigned to sections of the course.

However, you could randomly assign the special treatment to the sections by flip of a coin.

G1 (intact) – T – XG2 (intact) – O – X

This design in an improvement over the previous design.

51

5. Time-Series DesignsBecause of the problems involved with

random assignment and the difficulties in finding control groups, researchers often turn into time-series designs.

The Ss are usually not randomly selected for such studies (they can be to maximize generalizability).

In these designs, the class is its own control group.

The time series means that several pretests and posttests are administered.

X1 – X2 – X3 – T – X4 – X5 – X6

52

Experimental StudiesTrue experimental studies are rare in applied

linguistics.True experimental studies do use control groups.They also assess and/or control for differences between

groups prior to the start of the experiment.They require random selection of Ss and random

assignment of Ss to control and experimental groups.The assignment of control and experimental status is

also done randomly.Random Assignment Posttest

G1 (random) – T – XG2 (random) – 0 – X

Control Group Pretest-PosttestG1 (random) – X1 – T – X2G2 (random) – X1 – 0 – X2

53

Ex Post Facto DesignsWe will look at the type of connection without

considering what went before.No treatment is involved.Good design requires, however, that we consider

all the possible threats to the validity of the study and try to control for as many of them as possible.

E.g., Your curriculum is set up for a total skills approach. You notice that there seems to be a great range within your class and also across all the course levels in terms of how quickly students are able to complete tasks. You wonder if this is due to slow reading speed. The first step, then, is to discover the average reading speed of students across the levels. Are students in the beginning sections reading more slowly than those in intermediate sections, and are the students in the advanced sections reading most rapidly? Or, is it the case that there are such wide ranges in reading speed at each of these class levels that course levels are not really related to reading speed?

54

The dependent variable is reading speed.You want to discover whether class level

(the independent variable) can account for differences in reading speed (the dependent variable).

There are three class levels: beginning, intermediate, and advanced.

The class level is in a nominal scale variable.

Reading speed is an interval scale variable.

Reading Speed

Beg Interm Adv

55

Is the research trying to show that performance has been improved on the basis of instruction?No, it is not

Are causal relations to be established?No.

Are any of the variables being manipulated to cause a change?Again, no.

So, this study is an ex post facto design.

56

DESCRIBING INTERVAL AND ORDINAL VALUE

MEASURE OF CENTRAL TENDENCY

MODE(the most frequently

occurring scores)

MEDIAN(the middle

score)

MEAN(the average of

all scores)

57

Measures of Central Tendency

Central tendency is used to talk about the central point in the distribution of value in the data.

58

Measures of variabilityIn order to describe the distribution of

interval data, the measure of central tendency will not suffice.

To describe the data more accurately, we have to measure the degree of variability of the data of the data from the measure of central tendency.

There are 3 ways to show the data are spread out from the point, i.e. range, variance, and standard deviation.

59

MEASURESOF VARIABILITY

RANGE(the highest minus the

lowestscore)

VARIANCE(the square of

Standarddeviation

STANDARDDEVIATION

(the square root of variance

60

RangeRange = X highest – X lowestE.g. The youngest student is 17 and the oldest is

42,Range = 42 – 17 = 25The age range in this class is 25.

If range is so unstable, some researchers prefer to stabilize it by using the semi-interquartile range (SIQR)SIQR = Q3 – Q1 / 2Q3 is the score at the 75th percentile and Q1 is the score at

the 25th percentile.E.g., the score of the toefl score at the 75th

percentile is 560 and 470 is the score at the 25th percentile. SIQR is 560 – 470 / 2 = 45

61

Variance To see how close the scores are to the average

for the test. E.g., if the mean score on the exam was 93.5

and a student got 89, the deviation of the score from the mean is 4.5.

If we want a measure that takes the distribution of all scores into account, it is variance.

To compute variance, we begin with the deviation of the individual scores from the mean.Stages:1. Compute the mean: X2. Subtract the mean from each score to obtain

the individual deviation scores x = X – X.3. Square each individual deviation and add: ∑

x²4. Divide by N – 1: ∑ x²/ N - 1

62

Standard DeviationVariance = standard deviation are to give us

a measure that show how much variability there is in the scores.

They calculate the distance of every individual score from the mean.

Standard deviation goes one step further, to take the square root of the variance.

S =√ ∑ (X –X)²/ N – 1 or s = √ ∑x² / N - 1

63

CORRELATIONCorrelation is that area of statistics which

is concerned with the study of systematic relationships between two (or more) variables.

It attempts to answer questions such as: Do high values of variable X tend to go together

with high values of variable Y? (positive correlation) Do high values of X go with low values of Y?

(negative correlation) Is there some more complex relationship between X

and Y, or perhaps no relationship at all?

64

Visual representation of correlation: Scatter diagram

X

Y

X

Y

X

Y

Y

X

High positive r High negative r Lower positive r

Lower negative r No r

YY

X XNonlinear r

65

Correlation coefficients:To supplement the information given by a scatter

diagram a correlation coefficient is normally calculated.

The expressions for calculating such coefficients are so devised that a value of +1 is obtained for perfect positive correlation, a value of -1 for perfect negative correlation, a value of 0 for no correlation at all.

For interval variables, the appropriate measure is the so-called Pearson product-moment correlation coefficient.

For ordinal variables (scattergrams are not really appropriate), they use the Spearman rank correlation coefficient.

For nominal variables, they use the phi coefficient.

66

1 3 5 7 9 11 13 15 17 19 21 23 25 27 290

1

2

3

4

5

6

7

8

9

PretestPosttest

Students

Scor

es

67

3%10%

20%

20%

33%

13%

Excellent Very Good Good Enough Fair Fail