the connection between self-regulated learning … · the connection between self-regulated...

The Connection Between Self-Regulated Learning and Student Success in a Hybrid Course

Kevin R. Guidry

Jorge Cubillos

Kathleen L. Pusecker

University of Delaware

Paper presented at the Association for Institutional Research Annual Forum

Long Beach, California

May 21, 2013

Author Note

Kevin R. Guidry, Center for Teaching & Assessment of Learning, University of

Delaware; Jorge Cubillos, Department of Foreign Languages and Literatures, University of

Delaware; Kathleen L. Pusecker, Center for Teaching & Assessment of Learning, University of

Delaware.

This research was supported in part by Pearson Education and a grant from the University

of Delaware Center for Teaching & Assessment of Learning. This paper was originally titled

Using Self-Regulated Learning to Predict Student Success in a Hybrid Course and appears under

that title in the AIR Forum program; helpful suggestions from reviewers resulted in this change.

Additionally, this version correct two errors made in the original conference paper: age was

miscoded in the original data set and the reference level for the dummy-coded prior Spanish

Self-Regulated Learning and Student Success in a Hybrid Course 2

learning experience was mistakenly included in the first multiple regression model. These errors

have been corrected and had little impact on the overall results of this study.

Correspondence concerning this article should be addressed to Kevin R. Guidry, Center

for Teaching & Assessment of Learning, 212 Gore Hall, University of Delaware, Newark, DE,

19716. E-mail: [email protected]


The Connection Between Self-Regulated Learning and Student Success in a Hybrid Course

This paper describes preliminary results from a study at the University of Delaware

examining the relationship between self-regulated learning and student success in a hybrid

course using a modified version of the Motivated Strategies for Learning Questionnaire (MSLQ)

to determine if self-regulated learning is important to success in a hybrid course. By comparing

students’ performance in five sections of introductory Spanish, three face-to-face and two hybrid,

we examine how self-regulated learning behaviors and beliefs affect learning and student

success.

At the University of Delaware, there are currently 129 majors that have a foreign

language graduation requirement. In the face of increasing financial pressures, some colleges and

universities are turning to technology – specifically, online instruction – to preserve their

language offerings (Arnone, 2002). Indeed, current technological advances may make it

financially and pedagogically viable to transition away from the traditional classroom towards a

virtual one (Hokanson, 2000; Kinney & Robertson, 2003; Tunison & Noonan, 2001). Although

we are not yet experiencing direct political or financial pressure to reduce or streamline our

foreign language coursework, we anticipate that we may experience those pressures and we want

to ensure that we are prepared to meet them. Specifically, we want to ensure that (a) students in

hybrid courses exhibit similar or better gains in learning and (b) we understand some of the

characteristics of successful performance in hybrid courses so we can adequately advise students

who may have to select between hybrid and traditional face-to-face courses.


Literature Review

Hybrid Courses

Although a move away from traditional forms of delivery of instruction is now possible

(and in fact, quite attractive from space requirement viewpoint), there is limited information

about the pedagogical impact of such a change for foreign language instruction. The body of

applicable research available has focused on the applications of technology-assisted instruction

to different aspects of language learning such as vocabulary, grammar, and reading

comprehension. Unfortunately, the information available on the comparative pedagogical

advantage of this online learning environment versus the traditional face-to-face model is very

limited (Young, 2002).

A slightly larger body of research exists that focuses on hybrid instruction, a combination

of traditional face-to-face with online teaching (Carroll, 2003; Hopper, 2003; Oblender, 2002;

Patterson, 2004). This option is being used in other disciplines such as history, economics, textile

science, nursing, and computer science with encouraging results (Brown & Liedholm, 2002;

Combs, 2004; Krawiec, Salter & Kay, 2005, Riffell & Sibley, 2005; Salamonson & Lantz,

2005). Despite the recent popularity of “flipped classes,” there is little high-quality research that

specifically addresses college-level language coursework taught in a hybrid format.

Our primary concern with the introduction of hybrid language learning courses is the

efficacy of the courses in meeting learning outcomes. The lack of research in this area forces us

to reluctantly turn to the larger body of literature addressing the efficacy of different media on

college-level coursework with much of it focusing specifically on distance education. The bulk

of this research can best be characterized as finding that the medium usually has no significant

difference on student learning (Clark, 2001; Russell, 1999). The most well-known recent meta-


study concluded that students in hybrid courses performed better than students in online or face-

to-face courses; this finding, however, is clouded by the observation that such courses tended to

require students to spend more time-on-task (Means, et al., 2010).

Self-Regulated Learning

One of our primary beliefs is that students who practice appropriate behaviors and

beliefs, including adaptation to different circumstances, self-monitoring, and self-reflection, are

more successful. These ideas are nicely captured in the idea of self-regulation (Bembenutty,

2011; Weinstein, Meyer, Van Mater Stone, & McKeachie, 2006; Zimmerman, 2000). Broadly

construed, this concept includes areas such as time management, motivation, self-monitoring,

and management of anxiety. Alternatively, these ideas can be classified into cognitive,

metacognitive, and motivational strategies. Importantly, these are learned skills that we can help

students identify and improve (e.g. Weinstein, Husman, & Dierking, 2000).

Luckily, there is a well-tested and freely-available instrument to measure self-regulation:

the Motivated Strategies for Learning Questionnaire (MSLQ). This instrument was developed in

the 1980s with funding from the National Science Foundation and the Department of Education

(Duncan & McKeachie, 2005) and its psychometric properties have been thoroughly vetted

(Pintrich, Smith, Garcia, & McKeachie, 1993; Taylor, 2012). This vetting has also included

recommendations to modify the instrument to improve it such as those made by Dunn, Lo,

Mulvenon, and Sutcliffe (2012).

Research Questions

The following research questions guided this study:

RQ 1: What is the difference in student learning in face-to-face and hybrid sections of

this course?


RQ 2: What is the relationship between self-regulated learning, as measured by a

modified version of the MSLQ, and student success when student success is defined as (a)

improved performance on a normed language proficiency test or (b) a higher average grade on

course assignments and tests?

Methodology

This study examines characteristics and measures of approximately 100 undergraduate

students enrolled in five sections of introductory Spanish in the spring of 2013. This is a four-

credit course that typically meets four times each week for one hour each day. Two sections,

however, were taught as hybrid courses where students met only twice a week and conducted

much of their learning online. Class time was designated to be used primarily to practice Spanish

with peers and the instructor.

We employed as many controls in this quasi-experiment as practicable. Students did not

know that two sections were going to be taught as hybrid courses when they signed up for the

course, reducing the impact of self-selection (although students were, of course, permitted to

change course sections or withdraw). Although the two sets of sections were taught by different

faculty, they were scheduled at similar days and times to minimize the impact of those variables.

Finally, the eight tests and major assessments – four tests, two in-class compositions, one oral

recording, and one oral exam – were identical across all five course sections.

Sources of Data

Data were collected from and about the students in four ways. First, some data were

extracted from the university’s student information system. This included demographic

information such as gender, race/ethnicity, and first generation status. Second, grades from

assignments and assessments were included in the study. These were common for all sections of


the course and include four written tests, two in-class written compositions, an oral recording,

and an oral test.

A third source of data is a language proficiency exam administered at the beginning and

end of the semester. The STAMP 4S (STAndards-based Measurement of Proficiency – 4 Skills)

test was administered during class. Originally developed by language educators at the University

of Oregon’s Center for Applied Second Language Studies with a Fund for Improvement of Post-

Secondary Education (FIPSE) grant, STAMP 4S is a computer-adaptive test with four sections:

interpretative listening, interpretative reading, presentational writing, and presentational speaking

(Avant Assessment, 2013). Sixty-six students took both the pre- and post-test and completed

enough of the test to register scores in the listening and reading sections. These students were

beginning language learners, however, and too few completed the writing and speaking sections

to contribute meaningful data for this study. Avant Assessment, the company that has licensed

and administers the STAMP 4S, reports both general proficiency levels and scaled scores; in this

study, we use the scaled scores.

The fourth source of data is a survey that combined some demographic and demographic-

like questions with a modified version of the MSLQ. The first fourteen questions focused on

student characteristics, experiences, and preferences including several questions focused on

students’ use and access to Internet-connected devices. One critical question asked students how

long they had spent studying one or more foreign language courses. These questions were tested

in cognitive interviews with students in a Spanish course prior to finalizing the survey

instrument.

The remaining sixty questions are a modified version of the MSLQ. The MSLQ was

modified in three ways. First, two of the original subscales – Metacognitive Self-Regulation and


Effort Regulation – were removed and replaced with General Strategies for Learning and

Clarification Strategies for Learning subscales as recommended by Dunn, Lo, Mulvenon, and

Sutcliffe (2012). In their study, the removed subscales did not exhibit satisfactory psychometric

properties but the new subscales emerged. Second, questions that were reverse-coded were

modified or deleted. Some of these questions were specifically deleted per the recommendations

of Dunn et al. Four questions not included in their study were also modified or deleted both to

keep the instrument consistent and because the arguments made by Dunn et al. against their use

are convincing. Third, questions were slightly modified to better fit the context of this study; one

question could not be modified and was deleted.

The modified MSLQ used in this study includes two scales: Motivation and Learning.

The motivation scale includes questions about control of learning beliefs, extrinsic goal

orientation, intrinsic goal orientation, self-efficacy for learning and performance, and task value.

The learning scale includes questions about clarification strategies for learning, elaboration,

general strategies for learning, help seeking, organization, peer learning, rehearsal, and time and

study environment.

Analysis Methods

This study focuses on the relationship between self-regulated learning, as measured by a

modified version of the MSLQ, and two measures of student success: Improved performance on

a normed language proficiency test and a higher average grade on course assignments and tests.

To answer our research questions, we created three multiple linear regression models. Model 1

uses the students’ average score on course tests and assessments as the dependent variable.

Models 2 and 3 uses improvement on the student scores on the language proficiency test as the

dependent variables; model 2 focuses on the improvement in reading score and model 3 focuses


on the improvement in the listening score. The independent variables are listed below in Table 1.

Note that the models exploring improved performance on the language proficiency test do not

include prior Spanish language learning experience as an independent variable; we expected that

prior experience would have influenced performance on the pre-test so this is already accounted

for in those models.

Table 1: Independent Variables in Multiple Regression Models

Variable Models Variable type Age 1, 2, 3 Continuous First-generation status 1, 2, 3 Dichotomous (Non-first-gen as reference) Gender 1, 2, 3 Dichotomous (Male as reference) Hybrid course status 1, 2, 3 Dichotomous MSLQ learning scale 1, 2, 3 Continuous MSLQ motivation scale 1, 2, 3 Continuous Race/ethnicity 1, 2, 3 Dichotomous (White as reference) Spanish language learning experience

1 Dummy coded in three levels: less than two years (reference), at least two years but less than four, four or more years

Results

The first research question asks if there is a difference in student learning in face-to-face

and hybrid sections of this course. As discussed above, we employed three measures of student

learning in this study; means and standard deviations of those measures are shown in Table 2.

The differences between the means of the two groups are not statistically significant for any of

the measures.


Table 2: Average Grades and Proficiency Scores

Mode of instruction

Number of students

Grades Average reading proficiency score

change

Average listening proficiency score

change Mean Standard

deviation Mean Standard

deviation Mean Standard

deviation Face-to-face

66 83.9 9.0 33.9 36.3 28.2 40.1

Hybrid 34 83.7 7.8 47.5 39.3 16.2 44.0

Moving beyond simple t-tests to allow us to employ controls, in model 1 we regressed

our selected independent variables against the students’ average test and assessment score. The

model is statistically significant (p = .027) and has modest predictive power (R2 = .287, F(9, 52)

= 2.329. As shown in Table 3, only two variables in the model (other than the constant) were

significant: the MSLQ motivation scale (B = .419, p =.001) and the students’ age (B = -.307,

p=.026).

Table 3: Multiple Regression Model 1

Variable B Standard Error of B p Constant 68.645 8.819 .000 Hybrid course status 1.983 2.260 .384 MSLQ learning scale -.167 .103 .109 MSLQ motivation scale .419 .113 .001 Race/ethnicity .415 2.699 .878 Age -.307 .134 .026 First-generation status -1.275 2.973 .670 Gender 3.488 1.912 .074 Previous Spanish experience: Some 0.219 2.393 .927 Previous Spanish experience: Most -0.232 2.847 .935


Model 2 also produced interesting results. In this model we regressed our selected

independent variables against students’ gains in language proficiency reading scores. Like Model

1, this model has modest predictive power and is statistically significant (R2 = .400, F(7, 43) =

4.087, p = .002). Moreover, as shown in Table 4 two of independent factors had significant

loadings: the MSLQ learning scale (B = 1.537, p=.003) and the students’ age (B = 2.269,

p=.001). Model 3 was not significant nor did it have even modest predictive power (R2 = .029,

F(7, 42) = .182, p = .988). All three models satisfactorily satisfied tests of the assumptions of

multiple linear regression.

Table 4: Multiple Regression Model 2

Variable B Standard Error of B p Constant -28.334 48.378 .561 Hybrid course status 11.096 10.938 .316 MSLQ learning scale 1.537 0.492 .003 MSLQ motivation scale -1.173 0.605 .059 Race/ethnicity 6.565 12.723 .609 Age 2.269 0.646 .001 First-generation status 21.622 14.492 .143 Gender -22.449 10.779 .043

Discussion

First, we note that the regression models show that the mode of instruction did not

significantly affect students’ grades or language proficiency. This is consistent with the wider

body of research that has shown that the mode of instruction rarely has no significant effect on

learning. This reassures us that if we choose or are forced to implement more hybrid courses or

course sections they appear to be at least as effective as face-to-face ones that require more

classroom space and other campus resources.


Second, in two of our three models we found significant independent variables. Age and

the MSLQ motivation scale were significant predictors of higher test and assessment grades.

Age and the MSLQ learning scale were significant predictors of improved proficiency in reading

Spanish. This is a muddled picture.

The role of age in predicting these two outcomes is unclear as age is a negative predictor

of higher test and assessment grades but a positive predictor of improved proficiency in reading

Spanish i.e., younger students earned higher grades but older students performed better on the

Spanish reading portion of the proficiency test. It is possible that older students have different

motivations with less focus on earning higher grades in the course and more focus on acquiring

skills and knowledge.

It is not clear why only the MSLQ motivation scale predicted higher test and assessment

grades, only the MSLQ learning scale predicted improved proficiency in reading Spanish, and

neither scale predicted improved proficiency in listening to Spanish. Our hypotheses were that

self-regulated learning – motivation and learning – would lead to higher grades and increased

learning. These muddled, fragmentary results do not seem to support those hypotheses since the

MSLQ scales only predict success in limited ways. Moreover, it is surprising that those scales

are independent of age; this is not what we expected based on our experience, observations of

these classrooms, and interviews with these students.

Further research should seek to expand the sample size so additional factors can be

explored and more powerful statistical models can be tested. For example, the MSLQ has several

robust subscales that make up the two primary scales and a larger sample size would enable us to

determine if some of those scales may be more useful or informative in determining correlates or


predictors of student success. Further research should also expand beyond the first, introductory

Spanish course as some gains may not be apparent or even possible in only one semester.

Conclusion

In this study of 100 students enrolled in five sections of an introductory Spanish course,

we explored the impact of changing the mode of delivery to a hybrid mode for two sections. The

mode of instruction did not significant impact any of our learning outcomes, confirming the “no

significant difference” phenomenon widely reported in previous studies of online and hybrid

courses. Using a modified version of the Motivated Strategies for Learning Questionnaire, we

discovered a muddled picture that does not fully support our hypotheses that self-regulation has a

significant impact on our learning outcomes.


References

Arnone, M. (2002). 3 Iowa Universities go online to preserve Eastern European language

courses. The Chronicle of Higher Education,48(45).

Avant Assessment. (2013). STAMP 4S (STAndards-based Measurement of Proficiency – 4 Skills)

Spanish Technical Report. Retrieved from

http://www.avantassessment.com/sites/default/files/Spanish%20STAMP%20technical%2

0document.pdf

Bembenutty, H. (Ed.) (2011). Self-Regulated Learning. New Directions for Teaching and

Learning, 126. San Francisco, CA: Jossey-Bass.

Brown, B. W., & Liedholm, C. E. (2002). Can web courses replace the classroom in Principles of

Microeconomics? The American Economic Review, 92(2), 444-448.

Carroll, B (2003). Going hybrid: Online course components increase flexibility of on-campus

courses. Online Classroom, February, 4-7.

Clark, R. E. (2001). Learning from media: Arguments, analysis and evidence. Greenwich, CT:

Information Age Publishers.

Combs, L. (2004). Science education in the web era. Journal of Computers in Mathematics and

Science Teaching, 23(2), 139-149.

Duncan, T. G., & McKeachie, W. J. (2005). The making of the Motivated Strategies for

Learning Questionnaire. Educational Psychologist, 40(2), 117-128

Hokanson, S. G. (2000). Distance education in foreign languages. Rocky Mountain Review of

Language and Literature, 54(2), 85-93.

Hopper, K. (2003). Reasons to go hybrid. Distance Education Report, 7(24), 7.


Kinney, D. P., & Robertson, D. F (2003). Technology makes possible new models for delivering

developmental Mathematics instruction. Mathematics and Computer Education, 37(3),

315-28.

Krawiec, S., Salter, D., & Kay, E. J. (2005). A “hybrid” Bacteriology course: The professor’s

design and expectations; the students’ performance and assessment. Microbiology

Education, 6, 8-13.

Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2010). Evaluation of evidence

based practices in online learning: A meta-analysis and review of online learning

studies. Washington, DC: U.S. Department of Education Office of Planning, Evaluation,

and Policy Development Policy and Program Studies Service.

Oblender, T. E. (2002). A hybrid course model: One solution to the high online drop-out rate.

Learning and Leading with Technology, 29(6), 42-46.

Patterson, J. (2004). For quality and cost-effectiveness build a hybrid program. Distance

Education Report, 8(21), 1-2.

Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1993). Reliability and

predictive validity of the MSLQ. Educational and Psychological Measurement, 53, 801-

813.

Riffell, S., & Sibley, D. (2005). Using web-based instruction to improve large undergraduate

biology courses: An evaluation of a hybrid course format. Computers & Education, 44,

217-35.

Russell, T. L. (1999). The No Significant Difference Phenomenon As Reported in 355 Research

Reports, Summaries and Papers. Raleigh, NC: North Carolina State University.


Salamonson, Y., & Lantz, J. (2005). Factors influencing nursing students’ preference for a

hybrid format delivery in a pathophysiology course. Nurse Education Today, 25, 9-16.

Taylor, R. T. (2012). Review of the Motivated Strategies for Learning Questionnaire (MSLQ)

Using Reliability. Unpublished doctoral dissertation, Auburn University.

Tunison M., & Noonan, B (2001). On-line learning: Secondary students’ first experience.

Canadian Journal of Education/Revue Canadienne de l’Education, 26(4), 495-511.

Weinstein, C. E., Husman, J., & Dierking, D. R. (2000). Interventions with a focus on learning

strategies. In M. Boekaets, P. R. Pintrich, and M. Zeidner (eds.), Handbook of Self-

Regulation (pp. 727-747). San Diego, CA: Academic Press.

Weinstein, C. E., Meyer, D. K., Van Mater Stone, G., & McKeachie, W. J. (2006). Teaching

students how to become more strategic and self-regulated learners. In W. J. McKeachie

and M. Svinicki (eds.) McKeachie’s Teaching Tips, 12th ed. Boston, MA: Houghton

Mifflin.

Young, J. R. (2002). ‘Hybrid’ teaching seeks to end the divide between traditional and online

instruction. The Chronicle of Higher Education, 48(28), A33-A34.

Zimmerman, B. J. (2000). Attaining self-regulation: A social cognitive perspective. In M.

Boekaets, P. R. Pintrich, and M. Zeidner (eds.), Handbook of Self-Regulation (pp. 13-39).

San Diego, CA: Academic Press.


Appendix

the connection between self-regulated learning … · the connection between self-regulated...

Documents