sfu siat comprehensive examination

Comprehensive ExaminationResearch Methods Question Presentation

Vitomir Kovanovic

School of Interactive Arts and Technology,Simon Fraser University

Surrey, BC Canadavitomir [email protected]

December 9, 2013

Research Methods Question

Learning Analytics is a field that draws on numerous data about learners and thecontext in which learning happens.

Studying learning in the quasi-experimental setting brings certain issues that haveto be considered. Identify those issues; show how those drive choice of dataanalytics and techniques, type of outcomes and issues related to validity offindings. Illustrate on the case study of your choice.

Vitomir Kovanovic (SIAT) Comprehensive Examination December 9, 2013 2 / 50

Agenda

1 Research PlanResearch Plan OverviewTheoretical FoundationProposed Approach

2 Quasi-Experimental Research in Learning AnalyticsOverview of Learning Analytics Research MethodsQuasi-Experimental ResearchAnalytical Approaches

3 Conclusions


Research Plan

Research Plan



3 Conclusions


Research Plan Research Plan Overview

High Level Overview

ProblemHow we can best use the vast amount of data to improve learning process incontext of the socially-enabled learning environments?

SolutionProvide instructors with information on student’s learning so that appropriateinstructional interventions can be planned and implemented.

Provide learners with the feedback on their learning progress so that they canreflect more objectively on their own learning.

Approach

Adopt techniques of Learning Analytics to provide relevant information aboutstudents’ learning.


Research Plan Theoretical Foundation

Theoretical Foundations: Community of Inquiry - CoI

Community of Inquiry

Community of Inquiry is a conceptual framework outlying important constructsthat define worthwhile educational experience in distance education setting.

Three presences:

• Social presence: relationships and socialclimate in a community.

• Cognitive presence: phases of cognitiveengagement and knowledge construction.

• Teaching presence: instructional roleduring social learning.

CoI model is:

• Extensively researched and validated

• Adopts Content Analysis for assessment ofpresences


Research Plan Theoretical Foundation

Theoretical Foundations: Community of Inquiry - CoI

Issues and challenges:

• Used for analysis of learning long after courses are over.

• Require substantial manual and time consuming work for content analysis ofdiscussion messages for assessment of the levels of three presences.

• Not explaining reasons behind observed levels of presences.

• Not providing suggestions or guidelines for instructors to direct theirpedagogical decisions.


Research Plan Proposed Approach

Learning Analytics

Learning Analytics

Measurement, collection, analysis and reporting of data about learners and theircontexts, for purposes of understanding and optimizing learning and theenvironments in which it occurs.

Process of Learning Analytics

Five approaches of Social Learning Analytics(Ferguson and Shum, 2012):

• Social Network Analytics

• Content Analytics

• Discourse Analytics

• Disposition Analytics

• Context Analytics


Research Plan Proposed Approach

Learning Analytics for Communities of Inquiry

Automatic Content Analysis

LMS Database

System-use Feature Extractor

Prediction Component

SNA Extractionand

Metrics Calculation

NER ofLearning Concepts

Interventions and Feedback

Student Profiling


Quasi-Experimental Research in Learning Analytics

Quasi-Experimental Research in Learning Analytics



3 Conclusions


Quasi-Experimental Research in Learning Analytics Overview of Learning Analytics Research Methods

High-level View of Educational Research Methods

Most common classification:• Quantitative research

• Scientific Method• Positivism• Systematic investigation• Use of statistics

• Qualitative research• Postpositivism• Social constructionism• Study of human behavior• Frequent use of narratives &

interviews

• Mixed research• Multiple perspectives• Best of both worlds

Based on purpose of research:

• Basic Research• Broadening the knowledge

• Applied Research• Solve Practical Problems

Additional types of research:

• Design-based research• Naturalistic study of interventions• Iterative• Real-world setting• Researchers & Practitioners

collaboration

• Action Research• Practical approach to inquiry• Real-world setting• Practitioners lead research• Improving practice

Choice of research method is NOT a personal preference choice!

It should depend on the study goals and research questions.



Quantitative Research Methods

Four basic types of quantitative research in education (Creswell, 2012):

• Descriptive (survey) research.• What are the characteristics of a population?• Frequencies, averages• Often based on surveys

• Correlational research• Synonymous with nonexperimental research• Simply observes the size and direction of a relationship among variables

• Experimental research• Intervention is deliberately introduced to observe its effects.• Random assignment to different conditions• “Gold standard” for research.

• Quasi-experimental research• Almost experimental• Assignment to conditions is not random



Experimental Research

”The only legitimate way to try to establish a causal connectionstatistically is through the use of randomized experiments.”

(Utts, 2005)

• Direct manipulation ofindependent variable

• Random assignment todifferent treatments

• Usually at laboratory

• Can be very expensive

• Not always possible or desirable

• Issues of external and constructvalidity



Why Random Assignment?


Quasi-Experimental Research in Learning Analytics Quasi-Experimental Research

Quasi-Experimental (QE) Research

“by definition, quasi-experiments lack random assignment.”(Shadish, Campbell, and Cook, 2010)

Two types of assignment:

• Self-assignment

• Administrative assignment

Two types of Quasi-Experiments:

• “Person-by-treatment” experiments• In laboratories• At least one variable measured,

one manipulated

• “Natural” experiments• No control over assignment

Types of QE (Fife-Schaw, 2006):

• One-group design

• Non-equivalent control groups design• Posttest only• Pretest and posttest

• Interrupted time-series designs• Single• Multiple



One-group Design

Most common:

• Pretest-posttest design: O X O

Variations:

• Posttest-only design: X O

• Double pretest design: O O X O

• Nonequivalent dependent variable design: [Oa Ob] X [Oa Ob]

• Removed-treatment design (ABA): O X O ¬X O

• Repeated-treatment design (ABAB): O X O ¬X O X O



Non-equivalent Control Groups Design

Most common:

• Pretest-posttest design: Intervention Group: O X OControl Group: O O

Variations:

• Posttest-only design: Intervention Group: X OControl Group: O

• Double pretest design: Intervention Group: O O X OControl Group: O O O

• Switching replications: Intervention Group: O X OControl Group: O O X O



Interrupted Time-Series Designs



Multiple Time-Series Designs



Advantages and Disadvantages of Quasi-experiments

Advantages over true experiments:• Often easier to set up

• Often higher external validity

• Less ethical considerations

• Can sometimes give even betterresults

Disadvantages of quasi-experiments:

• Confounding

• Issues with internal validity

• Some ethical issues still exist

• Less control

• Sometimes comparable groups donot exist

• Replicability problem



Internal Validity

DefinitionThe extent to which a research design allows us to infer that a relationshipbetween two variables is a causal one or that the absence of a relationshipindicates the lack of causal relationship (Cramer and Howitt, 2004)

True ExperimenalResearch

Quasi-ExperimentalResearch

Correlational Research

High Internal ValidityLow External Validity

Low Internal ValidityHigh External Validity

Quantitative research validity



Internal Validity Treats

• Confounding

• Selection bias

• Experimenter bias

• History

• Contamination (Diffusion)

• Maturation effects

• Testing effect

• Regression toward the mean

• Instrumentality

• Mortality

• Competition


Quasi-Experimental Research in Learning Analytics Analytical Approaches

Analytical Approaches for Resolving Validity Threats

• Double pretest

• Nonequivalent dependent variable

• Regression discontinuity

• Matching




• Double pretest



• Matching



Double Pretest

Main ideaUse the “dry run” in order to check for internal validity treats

• Maturation effect

• Regression toward the mean

• Testing effect




• Double pretest



• Matching



Nonequivalent Dependent Variable

Hypothetical example

CBC starts running a 26-week long TV program called “LearningAlphabet” where every week children learn one new letter.

How effective is it for helping children to learn alphabet?

Naive approach (Pretest-posttest design): O X O

1 Select a sample of children

2 Assess their knowledge of the alphabet

3 Run the TV show for 26 weeks

4 Assess their knowledge again• If the difference is positive, then this is the evidence

that “Learning Alphabet” helped children to learn alphabet







Typical approach (Posttest-only NEGD design): Intervention Group: X OControl Group: O


2 Assess their knowledge of the alphabet


4 Split the sample into two groupsbased on whether they watched TV show or not

5 Assess their knowledge again• If the difference is bigger in the group that watched the show,

then this is the evidence that “Learning Alphabet” helped childrento learn alphabet







Alternative approach: [Oa Ob] X [Oa Ob]:


2 Assess their knowledge of the first 13 letters of the alphabet

3 Assess their knowledge of the second 13 letters of the alphabet


5 Assess their knowledge again• If the difference is bigger for first 13 letters than for the second 13 letters,

then this is the evidence that “Learning Alphabet” helped children to learnalphabet




Main idea

Use of the 2nd variable which is very related, but should not beaffected by the intervention




• Double pretest



• Matching



Regression Discontinuity

Real-world Example

How is the Gates Millennium Scholars (GMS) Programrelated to college students time use and activities?

(DesJardins et al., 2010)

Administered by Bill & Melinda Gates Foundation, provides $1 billion inscholarships over 20 year period.

• Provide help to high achieving, low-income students

• Provides a scholarship that covers unmet financial need

• Selection criteria:• Cognitive: 3.3 high school GPA• Non-cognitive: must be over a given threshold for a his ethnic group




Regression Discontinuity idea

Students just above and below the cutoff: distributed in an approximately randomfashion, similar to randomized trial




Results of the DesJardins et al. (2010) study:

• Receiving a scholarship significantly reduces hours of work

• No significant effects on hours spent studying, relaxing, or in extracurricularactivities.

• Receiving a scholarship significantly lowers hours worked by AfricanAmericans and Asian Americans.

• Receiving a scholarship significantly does not lower hours worked by LatinoAmericans.

• Latino Americans report significantly higher levels of participation in culturalevents relative to other racial/ethnic groups.




• Double pretest



• Matching



Matching

Real-world example

What are the effects of coaching on SAT scores?(Domingue and Briggs, 2009; Rock and Powers, 1998)

Rock and Powers (1998) study:

• ’95 Survey data about whether students used coaching

• Large section of the report dedicated to “who seeks coaching” question



Matching

Overall Idea of Matching

Balance treated and control groups on observable characteristicsas much as possible.



Matching

Use matching when:

• Very few subjects from both groups are directly comparable(e.g., regression to the mean)

• Selection of control subjects is hard due to the high dimensionality

Typical procedure:

• Run logistic regression on all available variables to retroactively predictchance of receiving a treatment.

• This chance is called “propensity score”

• Create control group where instances have similar propensity scores.Two common approaches:

• “Optimal” matching• Subclassification

Matching vs OLS Regression:

• In matching only untreated units that are similar are used as control group!

• OLS Regression can overestimate the effect of treatment due to the largedifferences between groups (e.g., selection bias)



Matching



Matching

Domingue and Briggs (2009) study

• Used propensity score matching to answer similar question as the study byRock and Powers (1998)

• Data from 2002 Education Longitudinal Survey

• Contains many variables about high school students:• Their SAT scores• Many demographic variables• Data about whether they used coaching to prepare for SAT



Matching

Domingue and Briggs (2009) study results



Arnold and Pistilli (2012) Example

Study by Arnold and Pistilli (2012) presented at Learning Analytics andKnowledge ’12 conference.

Typical example of LA system evaluation studies.

Authors presented Course Signals (CS) system:

• Feedback to students on their progress

• Some of the courses use this system

• Effect of use of Course Signals on student retention

• Comparing students who used CS and who didn’t



Arnold and Pistilli (2012) Results



Arnold and Pistilli (2012) Example

Challenges:

• Weak proof of causality

• Selection bias

• History effect

• Students who used CS have lower SAT scores• “this aspect needs to be further investigated” (Arnold and Pistilli, 2012)

Ways to improve:

• Make treatment and control groups more comparable

• Matching


Conclusions

Conclusions



3 Conclusions


Conclusions

Conclusions

Current Trends:

• Many of shown methods are not commonly used

• Learning Analytics research is most often correlational

• Big promise of Learning Analytics

• Still to define its “standard of research”

• Bigger impact of LA on educational practice


Conclusions

Conclusions

How this relates to my own research?

• Learning Analytics data are mostly observational

• Mostly working with “medium” data and will probably analyze BIG data formy PhD research

• Currently working on quasi-experimental data sets

• Overall idea is to advance both theoretical knowledge in CoI area andpractical impact of it by development of LA systems

• Making stronger claims about causality


References I

Arnold, Kimberly E. and Matthew D. Pistilli (2012). “Course Signals atPurdue: Using Learning Analytics to Increase Student Success”. In:Proceedings of the 2Nd International Conference on Learning Analytics andKnowledge, 267270. doi: 10.1145/2330601.2330666.http://doi.acm.org/10.1145/2330601.2330666.

Cramer, Duncan and Dennis Howitt (2004). The SAGE Dictionary ofStatistics: A Practical Resource for Students in the Social Sciences. English.Sage Pub.Creswell, John W (2012). Educational research: planning, conducting, andevaluating quantitative and qualitative research. English. Pearson.

DesJardins, Stephen L. et al. (2010). “A Quasi-Experimental Investigation ofHow the Gates Millennium Scholars Program Is Related to College StudentsTime Use and Activities”. en. In: Educational Evaluation and PolicyAnalysis 32.4, pp. 456–475. doi: 10.3102/0162373710380739.http://epa.sagepub.com/content/32/4/456.

http://dx.doi.org/10.1145/2330601.2330666

http://doi.acm.org/10.1145/2330601.2330666

http://dx.doi.org/10.3102/0162373710380739

http://epa.sagepub.com/content/32/4/456

References II

Domingue, Ben and Derek C Briggs (2009). “Using linear regression andpropensity score matching to estimate the effect of coaching on the SAT”.In: Multiple Linear Regression Viewpoints 35.1, pp. 12–29.

Ferguson, Rebecca and Simon Buckingham Shum (2012). “Social learninganalytics: five approaches”. In: Proceedings of 2nd International Conferenceon Learning Analytics & Knowledge, p. 23.

Fife-Schaw, Chris (2006). “Quasi-experimental Designs”. en. In: ResearchMethods in Psychology.

Rock, Donald A. and Donald E. Powers (1998). Effects of Coaching on SATI: Reasoning Scores. Tech. rep. 98-6. The College Board.http://research.collegeboard.org/publications/content/2012/

05/effects-coaching-sat-i-reasoning-scores.

Shadish, William R, Donald Thomas Campbell, and Thomas D Cook(2010). Experimental and quasi-experimental designs for generalized causalinference. English. Wadsworth, Cengage Learning.Utts, Jessica M (2005). Seeing through statistics. English. Thomson,Brooks/Cole.

http://research.collegeboard.org/publications/content/2012/05/effects-coaching-sat-i-reasoning-scores

http://research.collegeboard.org/publications/content/2012/05/effects-coaching-sat-i-reasoning-scores

Thank you

sfu siat comprehensive examination

Science

analysis of learning

learning progress

enabled learning environments

learning process incontext

quasiexperimental setting

social presence

teaching presence

cognitive presence