e19.2250 research on simulations and games for education jan l. plass, dmdl
TRANSCRIPT
E19.2250 E19.2250 Research on Simulations and Research on Simulations and
Games Games for Educationfor Education
E19.2250 E19.2250 Research on Simulations and Research on Simulations and
Games Games for Educationfor Education
Jan L. Plass, DMDL
Game GenresGame GenresGame GenresGame Genres
Goals for Games for LearningLiteracyFocus on social learning, identityExploration and knowledge creationLegitimate Participation in community of practiceApprenticeship (reciprocal)Learning through design & creating new artifacts
Game ResearchGame ResearchGame ResearchGame Research
Playtesting–Research GoalsGuide Design of New GamesPlaytest, Usability of Game PrototypesEfficacy, Evaluation of Completed GamesAdvancement of Theory
Games ResearchGames ResearchGames ResearchGames Research
PlaytestingObtain user opinions about a game Questions concern levels of difficulty, fun, engagementFollows in-house QA beta-testingTypically involves 100...103 participants
Games ResearchGames ResearchGames ResearchGames Research
Usability ResearchUsability Research: evaluate perceived and actual effectiveness of the design Provide guidance for specific design decisions using a variety of methods involving the target audienceTypically involves 101...102 participants
Games ResearchGames ResearchGames ResearchGames Research
Efficacy ResearchEvaluate the ability of the game to have the intended outcomes–cognitive, affective, or skills outcomesTypically involves >103 participants
Class ActivityClass ActivityClass ActivityClass Activity
What are the function and process of an Expert evaluation/ Heuristic review?
What research questions are addressed?What data is collected?What happens with the results?
PlaytestPlaytestPlaytestPlaytestResearch Design
Focussed Research QuestionsSampling of ParticipantsPlayer Group Sizes and CompositionLocation of Testing (Lab, field)StandardizationReference DataHeuristics (experienced-based methods)
Group ActivityGroup ActivityGroup ActivityGroup Activity
Play the Angles Game. Devise 3 research questions for playtesting of this game (15 min., Groups of 3–4)
What research questions are most important?What function would the answer serve?How could the question be addressed?
PlaytestPlaytestPlaytestPlaytest
Research MethodsQuestionnairesInterviewsThink-Aloud ProtocolsVideo ObservationsUser Logs (Events, User actions)
PlaytestPlaytestPlaytestPlaytestQuestionnaires
Learner/Player ProfileLearner VariablesLearner ResponsesConsider
Types of questionsQuestion contentQuestion wordingResponse formatQuestion placement and sequencing
PlaytestPlaytestPlaytestPlaytestInterviews
Structured, Semi-Structured, or Open-EndedSemi-Structured
How much did you like this game?
Would you play this game again in the future?
What did you like most about this game?
What did you like the least?
Did you make enough progress?
PlaytestPlaytestPlaytestPlaytest
Think-Aloud ProtocolsLearner utterances during play (more on that next week)
PlaytestPlaytestPlaytestPlaytestVideo Observations
Video Recording of Player Behavior and Game PlayAllows for later analysis of a variety of issues
PlaytestPlaytestPlaytestPlaytestUser Logs
Instrumented simulation or game writes game events and user actions to a log file or databaseNeed to specify what should be recorded
Highest Level: Every detail, allowing play back of gameLowest Level: Only key events are recorded
PlaytestPlaytestPlaytestPlaytestUser Logs
id--Uniquely identifies each row of the tablegame-A string identifiying the name of the game (with it's version number)treatment-"Conceptual" or "Arithmetic"userid-the user's id. This is not required to be unique so it's up to us to make sure we don't reuse these, timestamps can help disambiguateusername-The user's name that they typed in themselvestimestamp-The date/time the rows was inserted into the table, rounded to the nearest whole secondlocation--The chapter&level or screenname where the event took placewho-The name of the entity who issued the event (for example, "ANGLE EDF", or "RIGHT ANGLE BUTTON")what-The content of the message (for example, "SELECTED", or "Correct")extra- Additional values pertinent to the message, but is often unused
PlaytestPlaytestPlaytestPlaytestUser Logs
CyberBuch Böll Survey Monday, November 18, 1996
12:15:38 PM Start using stack
12:15:38 PM Environment: Annotations: TXTE PIC MOV Comprehension: Off Vocabulary: Off Forced lookup: On
12:15:38 PM User Name Clarke XXXX
12:15:40 PM Open Card 1
12:18:44 PM ärmlich TextEngl
12:18:54 PM ärmlich Picture
12:19:04 PM döst Picture
12:19:16 PM döst TextEngl
12:19:49 PM See Picture
12:19:56 PM See TextEngl
12:20:11 PM Wellenkämmen TextEngl
12:20:20 PM Wellenkämmen Movie
PlaytestPlaytestPlaytestPlaytest
Research SetupLab or Field settingEquipment to use for studyRole of ExperimenterInternet access
PlaytestPlaytestPlaytestPlaytestPlaytest Research Plan
Research Questions Research Design (player groups & composition, location)Participant profile criteriaSampling PlanMethod (Survey Instruments, Procedure, Technical Setup)Analysis of Data
Group ActivityGroup ActivityGroup ActivityGroup Activity
Develop your own research plan for playtesting of the Angles Game! (15 min., Groups of 3–4)
Research Questions Research Design (player groups & composition, location)Participant profile criteriaSampling PlanMethod (Survey Instruments, Procedure, Technical Setup)Analysis of Data
StatisticsStatisticsStatisticsStatistics
Descriptive StatisticsFrequency DistributionMeasures of Central TendenciesVariabilityCorrelationsMean ComparisonStatistical Power and Effect Size
Out of Class ActivityOut of Class ActivityOut of Class ActivityOut of Class Activity
Play a game!Play a computer/console game at home
Study a game!What research questions do you find interesting exploring in the game you played?
Cognitive Design Cognitive Design FactorsFactors
Cognitive Design Cognitive Design FactorsFactors
Evidence-centered Design Define educational goals of gameDefine outcomes (e.g., desired knowledge of learners)Define and identify acceptable evidence for learningDesign activities to learn and provide evidence of learning
Research FrameworkResearch FrameworkResearch FrameworkResearch Framework
Elements of InquiryElements of Inquiry Approaches to Approaches to ResearchResearch
Design Process of Design Process of ResearchResearch
Alternative Knowledge Claims
Qualitative Questions
Strategies of Inquiry Quantitative Theoretical Lens
Methods Mixed Methods Data Collection
Data Analysis
Report
Validation
Research FrameworkResearch FrameworkResearch FrameworkResearch Framework
Alternative Knowledge ClaimsPostpositivist knowledgeSocially constructed knowledgeAdvocacy/Participatory Knowledge Pragmatic Knowledge
Research FrameworkResearch FrameworkResearch FrameworkResearch FrameworkStrategies of Inquiry
QualitativeEthnographiesGrounded TheoryCase StudiesPhenomenological ResearchNarrative Research
QuantitativeExperimentsQuasi-ExperimentsCorrelational
Mixed Methods
Research FrameworkResearch FrameworkResearch FrameworkResearch Framework
MethodsSurveys & QuestionnairesField observationsOpen-ended interviewsKnowledge TestsPsychometric ScalesLog filesBiometricsEEG, EKG, EMG, NILS, GSR
OverviewOverviewOverviewOverviewLearning Game Design Model (Plass, 2010)
Formalize IdeasEvaluate Results
Test Ideas
Physical PrototypeDigital Prototype
Brainstorming
Design Documentation
ProductionRefine and PolishQuality Assurance
Educational Effectiveness
No problems Problems with Design
Analyze Requirements
Generate Learning
Ideas
Generate GamingIdeas