sloan bl workshop04192010final
DESCRIPTION
TRANSCRIPT
For more information:“Next Generation Course Redesign” in the Nov/Dec issue of Change
Next Generation Course Redesign (2010) Peter Lang Publishing
“The Promise of Blended Learning,” AA&CU News
Goals for todayShare why we thought that
it was important to have a serious look at how we teach undergraduates at UNT
Provide a brief overview of the blended learning approach
Describe how the Next Generation Course Redesign™ Project works at UNT
Demonstrate two NGen courses
Challenge you to think about the optimum blend of pedagogies for an instructional unit
Why use a blend of pedagogies in the redesign?
The “Perfect Storm”Bad News
High DFW rates Demographics (higher and more diverse
enrollments)Financial (tuition cannot keep exceeding CPI)Accountability
Good NewsKnowledge of learningEmergence of digital tools
What Do We Know About College Students and How They Learn?
Brain ResearchThe role of neural networksEvery network can be traced back to an experiential
referentOur students have “grown up digital”
Comfortable in a multimedia information-ubiquitous environment
Social networking is second natureCognitive development is an important goal
Most entering freshmen are at Kegan’s 3rd stageCollege experience must encourage a “discipline of
inquiry”
We know that, if we provide an active learning experience that allows students to engage with the content, each other, and instructors, they can and will think critically and develop cognitively.
Goals of the UNT NGen ProjectTo improve student learning outcomes in
large enrollment undergraduate coursesTo have a university-wide impact through the
establishment of a Community of Practice and the creation of redesign that is both sustainable and replicable
(Translation) Goals:Students think, work hard, like what they are
doing, get good grades that mean something, and graduate
Doesn’t cost more and uses less spaceFaculty have fun
UNT’s Transformational QEP Goal
Next Generation Course Redesign™ is a Process…Faculty teams redesign 4-6 courses each year
(two-year commitment)“Choreographed” Process
RetreatsMonthly meetings with teams of faculty and
staffInstitution-wide forumsEnd-of-pilot meetingsEnd-of-project meetings
Next Generation Course Redesign™The redesign
occurs within an interdisciplinary Community of Practice
NGen Faculty Fellows have the option of being designated Senior Faculty Fellows
Creating a Culture of Reflection
Promulgate the philosophy of “never ending redesign”
Schedule small group meetings with the majority of time devoted to reflection and discussion
Provide opportunities for ongoing virtual discussion
Promote assessment-driven change
Next Generation Course Redesign™ also has products. NGen Courses have:Clearly articulated
student learning outcomes that include outcomes representing higher level learning.
Test items that match the student learning outcomes.
An assessment plan that shows how the test items map to the outcomes.
Common Assessment Across All NGen CoursesPreference for Course
Format Survey (NGen vs. Traditional)
Attitude Toward Subject (pre/post)
LEP Survey of Cognitive Development (pre/post)
Raw score distributions
(The Blend) NGen Courses have:Large group
lectures (0-30% of contact hours)
Small group experiential learning (30-60% of contact hours)
Media-rich interactive online environment (30-50% of contact hours) These are the “building blocks” for an
NGen course.
In NGen, Lectures are best used to:Create interest and motivation and provide
assurance that the students can be successfulClarify and expand upon, rather than deliver,
contentModel the acquisition of knowledge in the
field, e.g., “How does a historian/chemist/sociologist approach a research question?
Present the critical lower level concepts to provide scaffolding for higher level concepts
In NGen, Experiential Learning Activities are best used to:Introduce an emotional component (brain-
based learning)Analyze, evaluate, and synthesizePresent and defend newly-acquired
hypothesesProvide collaborative, cooperative, and
academic controversy activities that encourage thinking critically from multiple perspectives
Dilemma!If I use experiential learning in my class, how
am I going to “cover” all of the material?
In NGen, Online Learning Activities are best use to:Acquire lower level learning to free up time
for in-class experiential learningChunk content to overcome working memory
limitsProvide low-stakes assessments such as
quizzes for practice and confidence buildingProvide psychomotor experiencesProvide concrete experiences that are guided
and efficient
Next Generation Course Redesign™UNT offers 19 NGen Courses
Art History, Biology I, Computer Applications, U.S. History I & II, Human Development, World Literature, Principles of Language Study, Business Communications, Occupational Health, Music Appreciation, Developmental Math/College Algebra, Intro to Communications, Modernism and the Visual Arts, Organic Chemistry, American Government I, World Literature Since the Renaissance, and Sociology of Disasters
With five more in development (Motor Development, Survey of Mathematics, Global Marketing Concepts, American Government II, and Individuals and Society)
RFP has been issued to select up to eight courses to start the redesign process in the fall
Nicole Dash, PhDAssociate Professor, SociologyDirector, UNT Core Academy
Why Redesign? Large Enrollment No Textbooks 90% Lecture Students struggling to connect what they
learned with real life situations
Redesigned Model 10% Lecture: First two class periods
only 50% Online: 9 Online Modules 10% Online Activities and Projects 30% Small Group Experiential
Learning Activities
What the class looks like to a student:
Lecture:First two face to face classes only
Introduces how the course will runDiscusses the schedule and who attends class
whenFocuses on explaining online components Introduces subject
Disaster photos and impact discussions Basic definitions More complicated theoretical issues
Online 9 online modulesExamsDiscussionsGroup Sign UpsWiki Project
Experiential LearningSmall GroupsUse information learned online
Develop a Warning MessageStratified Monopoly/Disaster Scenario ExerciseMock City Commission Meeting
Back to Monopoly… How does the Monopoly game relate to SLOs?
Assesses two SLOs: 6.2 Investigate how social structures (such as race,
class, and gender) generates disaster vulnerability and privilege
6.5 Assess your own social position and resulting vulnerability and privilege
Ties the results of the Monopoly game to a disaster scenario
Requires students to step out of their own shoes first, and then apply that to their own lives
Preliminary FindingsStarting to Analyze Data:
Majority preferred Next-Gen Model to traditional model 57% Overall preferred Next-Gen to F2F 61% of Successful Students preferred Next Gen 64% of Unsuccessful Students preferred F2F All students who attended class and engaged in
activities passed the classUnsuccessful students tended to consistently
miss projects and assignments
Next Generation Course Redesign™: Example
ENGL 2210 World Literature I
Dr. Tracey Gau
NGen Senior Faculty Fellow
QEP: The N-Gen Course Model
A rigorous outcome-based assessment plan
Ongoing redesign based on assessment results
A specific mix of instructional strategies, including:Small-group Experiential ActivitiesMedia rich, Interactive Online ExercisesLarge-group Lectures
A Sample N-Gen Course: World Literature
Course Assessment Blueprint
Outcomes based assessment plan
Coordinates SLOs, instruction, and assessment items
Ensures that what is measured is what is valued
Shows types of assessment instruments used to measure each outcome
©2008 T.M.Gau Dr. Tracey Gau Assessment Plan for N-Gen ENGL2210: World Literature I
Test Items and Difficulty Level
Domain Course Learning Goals(Institutional) General Student Learning Outcomes
(Departmental)Specific Lesson Learning Outcomes
(Classroom)
Low
Med
Hi Tot
CR
World Literature -- Ancient through Renais-sance
1. Demon-strate an awareness and recogni-tion of the scope and variety of works of literature
1.1 Recall and recognize the historical sequence of major literary figures, texts, and movements within the Ancient, Middle, and Renaissance periods
1.1.1 Recognize that the history of representative epics are oral or written compositions
6 2 8
1.2 Identify conventional literary genres, elements, and devices
1.2.1 Identify major characters and figures that appear in more than one text
6 2 8
1.3 Employ discipline specific vocabulary in order to recognize the relationship between form and content
1.3.1 Apply genre characteristics to representative texts
4 4 8 1
1.4 Relate literary or cultural concepts, principles, terms, strategies, and styles to a range of literature
1.4.1 Identify and connect literary or cultural concepts as they directly relate to representative texts
5 3 8
World Literature Test Plan (Blueprint)
All course materials © 2009, Dr. T.M. Gau
©2008 T.M.Gau Dr. Tracey Gau Assessment Plan for N-Gen ENGL2210: World Literature I
Test Items and Difficulty Level
Domain Course Learning Goals
(Institutional)General Student Learning Outcomes
(Departmental)Specific Lesson Learning Outcomes
(Classroom)
Low
Med
Hi Tot
CR
World Literature -- Ancient through Renais-sance
2. Read critically and analytically
2.1 Analyze, evaluate, interpret, synthesize representative texts from the Eastern and Western traditions and relate them to their literary and cultural contexts
2.1.1 Relate cultural qualities of a hero to a representative character
3 2 2 7 1
2.2 Compare and contrast major literary figures, their situations, decisions
2.2.1 Compare the Eastern depiction of fate, virtue, and heroism to the Western depiction in representative texts
2 3 2 7 1
2.3 Connect various periods, texts, authors, and characters
2.3.1 Differentiate between concepts, such as absolutism and relativism, using examples from representative texts
3 2 2 7 1
2.4 Evaluate the ideas presented in a text, their implications, and their relationship to ideas beyond the text
2.4.1 Evaluate characters’ decisions and actions in the context of their various cultures and worldviews
2 2 3 7 1
World Literature Test Plan cont’d (Goal 2)
©2008 T.M.Gau Dr. Tracey Gau Assessment Plan for N-Gen ENGL2210: World Literature I
Test Items and Difficulty LevelDomain Course Learning Goals
(Institutional) General Student Learning Outcomes(Departmental)
Specific Lesson Learning Outcomes
(Classroom)
Low
Med
Hi
Tot
CR
World Literature -- Ancient through Renais-sance
3. Construct informed, organized and coherent written responses to literary texts
3.1 Formulate a central interpretive idea about the texts;3.2 Develop ideas logically and coherently with adequate supporting textual examples;3.3 Present ideas clearly and concisely;3.4 Observe the standard conventions of formatting, citation, grammar, and punctuation
Written Composition(See Constructed Response
Rubric)
4. Actively discuss ideas with others
4.1 Present (in discussion posts or small groups) information or conclusions that help other students summarize, synthesize, and integrate the conceptual material
4.2 Establish a relevancy between literature and how it affects one’s life, personal values, ethical behavior, aesthetic judgment, and problem solving
4.2.1 Relate readings to life and academic experiences4.2.2 Make interdisciplinary connections, such as with history, philosophy, art, rhetoric, business, politics
TOTAL EXAM ITEMS 31
20
9 60
5
World Literature Test Plan continued (Goals 3 and 4)
Structure of N-Gen N-Gen World LitWorld Lit
All course materials © 2009, Dr. T.M. Gau
SLOs begin every lesson
Cognitive Categoriesand
Difficulty Levels
Low: Literal and Factual Knowledge• Recall, identify, understand
Med: Conceptual and Procedural Knowledge• Interpretive and Inferential• Apply, analyze, interpret
High: Meta-cognitive Knowledge• Evaluate, construct, create
All course materials © 2009, Dr. T.M. Gau
Formative Assessment:Online Activities
All course materials © 2009, Dr. T.M. Gau
Formative Assessment:Tools of Mastery
All course materials © 2009, Dr. T.M. Gau
Formative Assessment:Games of MasteryAll course materials © 2009, Dr. T.M. Gau
Formative Assessment:Self Tests
All course materials © 2009, Dr. T.M. Gau
Interaction and Collaboration: Debate
All course materials © 2009, Dr. T.M. Gau
Summative Assessment: Constructed Responses
All course materials © 2009, Dr. T.M. Gau
All course materials © 2009, Dr. T.M. Gau
All course materials © 2009, Dr. T.M. Gau
Best Practices: Assessments Assignments that encourage students to employ critical
thinking strategies
Rubrics or performance criteria that are made available to the students
Scaffolding activities that promote higher cognitive development
Providing students with ample opportunities for self assessment
All course materials © 2009, Dr. T.M. Gau 48
Summative Assessment:Multiple Choice Items
Offer high degree of reliability & validityVary in difficulty level:
Low (literal and factual) Medium (interpretive and analytical) High (metacognitive)
Correspond to Assessment Plan and SLOs Course Goal General SLO Specific SLO
Use Item Analysis to improve test items Conform to item-writing guidelines
Match test items to cognitive level of Learning Outcome Use Prompts to Address High(er)-Level Cognitive Category Use Specific Items Shells in Writing Multiple Choice Item Sets
All course materials © 2009, Dr. T.M. Gau
SLO 2.1 According to Machiavelli, what two animals must a ruler imitate and what meaning does the analogy represent? A. Fox and Tiger; in addition to being cunning, a ruler must protect himself from traps. B. Fox and Cougar, in addition to being cunning, a ruler must be able to act alone. C. Fox and Lion; in addition to being cunning, a ruler must use force as well as laws to rule.
Rate The Item
(LOW .93/.90)
1. Low2. Medium3. High
SLO 2.1 Which piece of evidence that Iago presents to Othello has the GREATEST effect of conjuring up doubt about Desdemona’s fidelity? A. Cassio’s erotic dream B. Cassio’s drunken brawl with Roderigo C. The handkerchief he sees Cassio give to Bianca
1. Low2. Medium3. High
Rate The Item
(MEDIUM .93/.90)
SLO 2.1 What is the BEST interpretation of Iago’s advice to Roderigo to “put money in thy purse”? A. Invest your time, money, and effort in joining Iago in revenge against Othello B. The marriage between Desdemona and Othello is frail and can easily be broken C. Because of woman’s changeable nature, Desdemona will tire of Othello and seek a new lover
1. Low2. Medium3. High
Rate The Item
(HIGH .63/.64)
SLO 2.1 Which one of the following is the MOST COMPLETE description of what Iago represents? A. Vice Figure B. Communal Evil C. Parasitic Flatterer
1. Low2. Medium3. High
Rate The Item
(MEDIUM .89/.83)
Item Analysis: Why it’s necessaryto validate m/c items
Item analysis of summative selected-response (MC) and constructed (written) response tests due at end of each semester
Flawed multiple-choice items are up to 15 percentage points more difficult than standard items (items that conform to the standard principles) when testing the same content,
with median passing rates that are about 3.5 percentage points lower for flawed items as compared to standard items
(Downing, 2005).
Item Prop. Disc. Point Prop. Endorsing PointNo. Correct Index Biser. Alt. Total Low High Biser. 1 .61 .51 .42 A .17 .30 .05 -.27
B .08 .15 .02 -.19 C .61 .35 .86 .42 * D .14 .19 .06 -.14 2 .21 .18 .18 A .07 .09 .05 -.08 B .22 .26 .18 -.09 C .49 .49 .45 -.02 D .21 .14 .32 .18 * 4 .73 .41 .39 A .73 .50 .91 .39 * B .02 .06 .00 -.16 C .12 .23 . 04 -.24 D .13 .21 .05 -.19 5 .82 .38 .43 A .02 .05 .01 -.15 B .82 .59 .96 .43 * C .05 .09 .01 -.16 D .10 .26 .02 -.33 6 18 .05 .07 A .18 .15 .20 .07 * B .10 .18 .04 -.22 CHECK A was specified C .41 .45 .33 -.08 D works better D .31 .20 .43 .19
CONSTRUCTED RESPONSE STUDENT SCORES
St ClarityStructure Logic
ReferEvid Sub tot
Grammar Sent Struct Dev sub tot
1 3 2 2 1 8 0 3 2 72 3 3 3 3 12 2 2 2 63 4 3 2 3 12 3 2 3 84 3 3 2 2 10 2 3 2 75 2 2 1 1 6 1 1 2 46 2 2 3 3 10 2 2 2 67 4 3 3 3 13 3 2 3 88 4 4 4 3 15 3 3 3 99 3 4 2 2 11 3 3 2 8
…..…..38 2 3 2 2 9 2 3 2 739 1 2 1 1 5 1 1 2 440 2 3 3 3 11 2 2 2 6
M 2.65 2.75 2.28 2.38 10.05 2.13 2.23 2.2 6.60Mean for content = 10 = 63% Mean for conventions = 6.6 = 55%
Do the average scores for each element make sense in terms of the intent of the outcomes they are measuring? (validity evidence)
Do the student responses appear to reflect what was taught in terms of the outcomes they are measuring? (validity evidence)
Did the prompt and the question seem to work? That is, did it produce a high degree of response?
Assessment System Model
Instruction Assessment Course Design Test Development
VALIDITY EVIDENCE
Learning Goals and Outcomes
Scores that are valid for theirintended interpretation
and use
SURVEY 1: Survey of Student Attitude Toward Subject of the
Course Survey designed to gather information on how the student feels about the subject
matter of the course the student is beginning, finishing, or has just finished
Administered Pre and Post
21 items on a five point scale that are specific to the “subject”- not teacher effectiveness- not course effectiveness
Inference = improved attitude correlates with N-Gen format and course redesign
13 items positive
8 items negative .
Specific Results for World Lit:Student Attitude Toward Subject
Items that increased:+.190 This subject is worth knowing+.254 Knowing this subject makes me more employable+.638 I know a lot about this subject+.317 This subject is useful to my everyday life
Items that decreased:-.079 This is a difficult subject for me-.095 Learning this subject requires a lot of hard work-.079 This subject is difficult to understand-.143 This is a complicated subject-.032 I will have no application of this subject in my profession-.032 I am scared by this subject
All course materials © 2009, Dr. T.M. Gau
SURVEY 2: Preference for Course Format:
N-Gen or FTF
This survey consists of one survey question with two answer choices (N-Gen and FTF) with a short written response in which the student explains the reason(s) for his/her choice.
Instructor records an S (Successful = A,B,C) or U (Unsuccessful = D,F,W,I) after end of course. S and U can be typed into the online student- response data file at end of the course.
Two weeks prior to final exam, spring semester year
Example of student preference for course format (Ngen vs FTF)
If you were to start this course over again, would you prefer a traditional face-to-face format, or would you prefer the N-Gen redesign format you are experiencing? Please tell why.
CoursN=282
PrefN-gen
PrefFTF
TotalNumb Succ
Total Numb Un-succ
Succprefer N-Gen
Succprefer FTF
Un-succs prefer N-Gen
Un-succ prefer FTF
Count 172 110 201 81 149 52 51 30
% 61% 39% 71% 29% 74% 37% 63% 37%
Categories of ResponsesFormat Category Typical CommentN-Gen Pace I liked that I could do most of the work at my own pace when I had time to do it. If I wanted
to go a little slower, I could.
Flexibility
This course allowed me to work out my hectic schedule. Being able to submit quizzes and stuff online made my life easier.
Learning
I like to learn from a bunch of different sources at once and this course really allowed me the chance to do that. You got stuff from online sources and you got some face to face interaction and I think I ended up learning more in this course than I would have otherwise.
Practice I always liked the fact that you could go back and take the quizzes over and over again until you got them right. I really feel like that helped me a lot in the class.
FTF Manage I prefer a traditional face-to-face lecture because I would often forget about online assignments and I think my grade suffered. I need more structure in my courses so, I need to come to campus more often!
Learning I seem to absorb the information better in a traditional class format when I'm taking notes during a lecture. It just suits my learning style better.
People I can't seem to learn without a teacher lecturing to me and me taking notes. Maybe it something about the interaction in the communication. I guess I just like being around people.
Technical
I don't trust submitting my quizzes online. The internet didn't seem to be reliable. Sometimes the website would be slow or wouldn't work at all. SO frustrating!
Final Score Distribution
To allow a comparison of means from year to year and pre N-Gen to N-Gen
Due 1 week after end of each semester year 1 & 2
Test for statistical significance
Success Rates: World Literature I
70%63%
69% 68% 65% 68% 71%
84%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
ENGL 2210 Large Enrollment ClassesSuccess Percentages Across Semesters
Fall 04 Spg 05 Fall 05 Spg 06 Fall 06 Spg 07 Fall 07 Spg 08
All course materials © 2009, Dr. T.M. Gau
Departmental AdvantagesAddresses problems of course drift, inconsistent learning
experiences, and inefficient use of faculty/course deliveryIncreases the quality of the department’s offeringsProvides a basic framework that meets university,
departmental, and course objectivesOffers a pedagogical training for new instructorsImplements best practices of both face-to-face and online
teaching and learningPromotes higher-level learning in introductory coursesMeets assessment standards of reliability and validityAllocates more time for faculty to work on other projects
All course materials © 2009 Dr. T.M. Gau
Vision and Objectives after RedesignSustainability and ReplicationOther UNT instructors teaching the courseAdoption of courseware by outside universities
1. World Literature I: Ancient through Renaissance
2. World Literature II: Since the Renaissance
Designer: Lead touchpoint resource Provides training to instructors Updates and maintains course Responds to feedback for course improvements
Cost: $90 per student (course fees)
All course materials © 2009 Dr. T.M. Gau
AssignmentSee handout with goals and SLO’s
Select goal and break into groups of 4-6Brainstorm combination of the three
pedagogies to reach the SLO’sReport
Discussion
Dr. Niki [email protected]
Dr. Brenda [email protected]
Dr. Tracey [email protected]
Dr. Phil [email protected]
For more information:Nov/Dec issue of Change
Next Generation Course Redesign™ Forthcoming from Peter Lang Publishing
Email to join Course Redesign NING