It's All in What You Ask:

Techniques for Enhancing Reflection and Learning in an Online Course

Karen DiazNancy O’Hanlon

LOEX Conference, April 2010

Agenda

• Describe several techniques for encouraging metacognition and improving student learning.

• Discuss how we used these techniques in an online course to enhance understanding of a difficult concept.

• Demonstrate methods that you could apply in your own teaching.

Clicker Test: Are you really in the right room today?

Yes

Probably

Not s

ure

No, b

ut too embarra

ssed...

40%

20%

0%

40%1. Yes2. Probably3. Not sure4. No, but too embarrassed

to leave now.

Speed kills . . . brain cells!"Deep log studies show that, from undergraduates to professors, people exhibit a strong tendency towards shallow, horizontal, 'flicking' behaviour in digital libraries. Society is dumbing down."

Information Behaviour of the Researcher of the Future, 2008. http://www.ucl.ac.uk/infostudies/research/ciber/downloads/ggexecutive.pdf

Fostering Habits of Reflection

• We teach students to skim and make snap judgments about quality.

• Reflection requires more time and conscious effort.

Why? Improved Outcomes

• Learning is enhanced when students engage in metacognitive activities such as self-assessment and monitoring (Lin, 2001).

• Self-monitoring is particularly important in Web-based environments.

• Tools or features prompting students to reflect on learning were effective in improving outcomes (Means et al, 2009).

How? Prompts

• Prompts (questions) are an important tool for encouraging reflection.

• Prompts that ask students to assess what they understand and where they should direct additional effort were effective in research studies on information skills (Stadtler, 2007; Wopereis, 2008).

How? Process Modeling

• Lin (1999) notes that “reflective thinking involves actively monitoring, evaluating, and modifying one’s thinking and comparing it to both expert models and peers.”

• Pace & Middendorf (2004) suggest that instructors can help students overcome learning obstacles (bottlenecks) by reflecting in depth on the steps an expert would take to accomplish a particular task and then modeling it for students.

Techniques: Summary

1 -Instructor models process

2 - Student practices task

3 -Instructor prompts student to compare, reflect

Reflection 1

Think about your own teaching experiences.

Is there a concept or skill that your students find particularly difficult to grasp?

How could you provide opportunities for students to assess their own progress?

Our Approach: Action Research

• Identify instructional problem• Conduct literature review

PLAN

New Course Content: Movie

Recognizing Bias: • Provides a definition of bias.• Offers a rationale for learning the skill of

assessing bias.• Introduces a cognitive strategy, represented

as a mnemonic, MAPit.

Modeling the Process

Deconstructing the Message

Recognizing Bias Movie

http://liblearn.osu.edu/movies/bias.htm

New Course Content: Quiz

• Recall: 4 multiple choice questions that assess

recall of content from the movie. • Application: 2 multiple choice questions with

links to websites, to assess students’ ability to apply the MAPit strategy.

• Reflection: 2 open ended questions (metacognitive prompts) intended to encourage reflection by students.

Question 1 (no change from Autumn 2008 to Winter 2009)

The letters MAP in the MAPit strategy for recognizing bias stand for:

1) Medium, Audience, Purpose2) Message, Author, Purpose3) Message, Audience, Practicality

Bias QuizKn

owled

ge

Question 2 (no change from Autumn 2008 to Winter 2009)

Worthwhile factors to consider in analyzing a message for bias are whether it is: (Select ALL correct options)

Fact or opinion Sources cited or not cited Right or wrong Fair or selective

Bias QuizKn

owled

ge

Question 3 (no change from Autumn 2008 to Winter 2009)

Why does an author's credibility matter when looking for bias?

1) Authors with deep knowledge of a field, and who have withstood scrutiny of reviewers, are less likely to be biased.

2) Authors who have been reviewed a lot are likely to say what they think the reviewers want to hear, so may be more biased.

3) Authors who are new to a field bring an important fresh perspective that is less likely to be biased.

Bias QuizCom

preh

ensio

n

Question 4 (no change from Autumn 2008 to Winter 2009)

What factors can help you determine if the purpose of a publication is partisan? (Select ALL that apply)

Where it gets its funding How many members it has It's own mission statement Who is on its board of directors

Bias QuizCom

preh

ensio

n

Question 5 (no change from Autumn 2008 to Winter 2009)

Using the strategies described in the movie, connect to the following site and analyze it for bias. Drug Policy Alliance

Which statement is most true about this site?

1) It is unbiased 2) It is biased 3) It is commercial

Bias QuizAppli

cation

Question 6 (no change from Autumn 2008 to Winter 2009)

Using the strategies described in the movie, connect to the following site and analyze it for bias. Who's Watching What You Eat?

Which statement is most true about this site?

1) It is unbiased 2) It is biased 3) It is commercial

Bias QuizAppli

cation

Question 7 (Autumn 2008)

How well do you understand what is needed and how prepared do you feel to recognize bias in other information you encounter? (To receive credit, your answer must include at least one complete and clear sentence.)

Bias QuizRe

flection

Question 7 (Winter 2009)

How well do you understand what is needed and how prepared do you feel to recognize bias in other information you encounter?

1) I understand the concepts related to, and feel confident that I will be able to recognize bias in all other information sources.

2) I understand the concepts needed to recognize bias, but think it may still be difficult to recognize bias in some information sources.

3) Not all the concepts related to recognizing bias are easy for me, and am not confident in my ability to recognize bias in all information sources.

4) I do not feel prepared at all to recognize bias in information sources.

Bias QuizRe

flection

Question 8 (Autumn 2008)

What do you think you should do to improve your own abilities for recognizing bias? (To receive credit, your answer must include at least one complete and clear sentence.)

Bias QuizRe

flection

Question 8 (Winter 2009)

What do you think you should do to improve your own abilities for recognizing bias?

1) I need to remember to apply the MAPit strategy in the future.

2) There are one or two elements in the MAPit strategy that I know are challenging for me, and need to pay attention to those when I try to detect bias.

3) I need to pay more attention to elements from information sources (such as author credentials, or "about us" links).

4) I should spend time practicing going to web sites and applying the MAPit strategy to get good at it.

Bias QuizRe

flection

Bias Quiz - Reflection

Capstone

Question 8 (Autumn 2008 and Winter 2009)

What is the PRIMARY purpose of this site? (Pick one option below and type it into the answer box):

• Advocacy (sway opinion or get support for a cause) • Reference/Information (provide balanced view of a topic) • Commercial (sell a product; offer product support)

Applica

tion

Capstone

Question 11 (Autumn 2008 and Winter 2009)

Evaluation of Web Sites--Content Bias/Balance An advocacy site may attempt to sway the reader's view one way or another, while a reference/information site will provide a balanced, objective look at the topic. Examine some of the pages on the site before answering the following questions.

Also, remember the MAPit strategy for recognizing bias that was introduced in the movie you viewed during week three. Be sure to consider these factors when you make a decision.

Choose the best description of the site's content from the options listed below..

• Biased (slanted) • Balanced (objective)

Applica

tion

Capstone

Question 24 (Autumn 2008 and Winter 2009)

Write a paragraph covering all of the points listed below. Use complete sentences. Your answer will be judged on accuracy, clarity of expression, and completeness.

For the site that you found with your search (Questions 22, 23), indicate:

• Who is the author? Provide some background information about the author. • What is the purpose of this site? Find a statement of purpose on the site and

quote from it. • When was this site last updated? Indicate where you found this information. • Does the site's content appear balanced (objective) or biased (slanted)? Provide

support for your answer.

Analysis

Bias Quiz - Performance

Capstone - Performance

Capstone – Qualitative look at Performance

• Case Studies• Terminology– Reflective– MAPit (instructed)

Applied to another course

• Provide students with an article to read– How We Failed the Net Generation by William

Badke• Quiz them over it

Article Quiz

Question 1

This article critiques the current state of information literacy and offers suggestions to improve information literacy education. Who is the intended audience of this article?

a) Those teaching information literacy at the University level. b) University students trying to understand information literacy. c) Those teaching information literacy at the K-12 level.

Compr

ehen

sion

Article Quiz

Question 2

Why might a library database present a better alternative to Google?

a) Library materials are reviewed and regulated. In contrast, no one checks the reliability of Web pages. b) Libraries are better funded than most Web pages. Therefore, they are more accurate. c) The author states that Google is the best way to search for library materials.

Know

ledge

Article Quiz

Question 3

Where does the author think students learn most of their Internet search skills?

a) At home. b) At their hometown libraries. c) In their K-12 education.

Know

ledge

Article Quiz

Question 4

Why does the author think it is important to teach stronger Web search skills?

a) In the author's opinion, the Web is really only good for finding music or blogs. b) Information provided by search engines is limited and often lacking precision when compared to library databases. c) The author is a librarian and wants to promote libraries.

Know

ledge

Article QuizQuestion 5

After reading this article, pick which of the following assumptions the author makes that you MOST agree with:

Reflec

tion

I trust the quality of the first page of Google results. 2%

I consider library resources to be a viable option for my research needs. 16%

Using library resources is more alien and frustrating than using Google. 20%

No one ever explained what kind of information I was dealing with when I threw words in a Google search box. 6%I don't agree with any of the author's assumptions. 6%I thought "someone" was controlling the Web. 0%Simplicity in information-seeking comes at a cost. 26%

I generally don't think about checking the reliability of Web sites. 6%

When I need a scholarly article I use Google (or a search engine) instead of a library database. 18%

Your Turn to Reflect

I underst

and what r

eflec...

I need to

know more

ab...

I will

probably

never b

uil...

I feel s

omething diff

eren...

20% 20%20%

40%1. I understand what reflective

questions look like and will be able to apply them to my assignments.

2. I need to know more about the use of reflection and metacognition before trying to use it.

3. I will probably never build reflective questioning into my assignments.

4. I feel something different than any of these alternatives.

References• Lin, X., Hmelo, C., Kinzer, C.K., & Secules, T.J. (1999). Designing technology to

support reflection. Educational Technology Research and Development 47 (3), 43-62.

• Lin, X. (2001). Designing metacognitive activities. Educational Technology Research and Development 49, 23-40.

• Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2009). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies. Washington, D.C.: U.S. Dept. of Education, Office of Planning, Evaluation, and Policy Development. Retrieved from: http://www.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf

• Pace, D. & Middendorf, J. (Eds.) (2004). Decoding the disciplines: Helping students learn disciplinary ways of thinking. New Directions for Teaching and Learning, 98. San Francisco: Jossey Bass.

• Stadtler, M. & Bromme, R. (2008). Effects of the metacognitive computer-tool met.a.ware on the web search of laypersons. Computers in Human Behavior 24 (3), 716-37.

• Wopereis, I., Brand-Gruwel, S., & Vermetten, Y. (2008). The effect of embedded instruction on solving information problems. Computers in Human Behavior 24 (3), 738-52.