internet based assessments via collaborative problem solving: the assessment and teaching of 21 st ...
DESCRIPTION
Internet based assessments via collaborative problem solving: The Assessment and Teaching of 21 st Century Skills (ATC21S™) Project Patrick Griffin Executive Director, ATC21S. PROJECT GOVERNANCE. Patrick Griffin , University of Melbourne, Executive DirectorATC21S - PowerPoint PPT PresentationTRANSCRIPT
Internet based assessments via collaborative problem solving: The Assessment and Teaching of 21st Century
Skills (ATC21S™) Project
Patrick GriffinExecutive Director, ATC21S
Patrick Griffin, University of Melbourne, Executive DirectorATC21S
Michael Stevenson, Cisco Vice President - Global Education, Former ATC21S Board Chair (2009–2010)
Shelly Esque, Intel Vice President - Legal and Corporate Affairs , Current ATC21S Board Chair (2011–2012)
Anthony Salcito, Microsoft Vice President - Education, Former ATC21S Board Chair (2010–2011)
Esther Care, University of Melbourne, ATC21S International Research Coordinator
Ministerial representatives from…
– Australia:
– Finland:
– Singapore
– USA:
– Costa Rica:
– Netherlands:
EXECUTIVE BOARD
PROJECT GOVERNANCE
PROJECT GOVERNANCE
ADVISORY BOARD
Patrick Griffin, University of Melbourne, Executive Director ATC21S (Chair)
Andreas Schleicher, Organization for Economic Co-operation and Development (OECD)
Seamus Hegarty, International Association for the Evaluation of Educational Achievement (IEA)
Irina Bokova (Director General UNESCO)
Ray Adams, Technical Director PISA 2003-2012
Marc Durando, European Schoolnet
Esther Care, University of Melbourne, ATC21S International Research Coordinator
Stuart Elliott, National Academy of Sciences
David Forster, International Testing Commission
Robin Horn, World Bank
Eugenio Eduardo Severin, Inter-American Development Bank
National project managers from each founder country and associate country
Task force member from each company:
Katrina Reynan, Director, Cisco
Martina Roth, Director, Intel
Greg Butler, Director, Microsoft
Traditional assessments may not be suited to measure many 21st century skills
Goal is to develop new assessment approaches matched to new C21 skills and to advise systems, schools and teachers on the use of assessment data to help students develop higher order performances
Focus: the Assessment and Teaching of 21st Century Skills
Phase 1Conceptualise C21 skills
and education output needs
Phase 2 Skill
Identification and
hypotheses
Phase 3 Development
and coding via coglabs
Phase 4 Pilot studies and trials
Phase 5 Dissemination
scale and policy
CONCEPTUALIZING THE SKILLS
Assemble experts Define methods
Explore practical and technical needs in the classroom Create a new framework
Five working groups were established: Defining 21st-century skills: Ms. Senta Raizen, WestEd Methodological issues: Dr. Mark Wilson, University of
California, Berkeley Technological issues: Dr. Beno Csapo, University of
Szeged, Hungary Classrooms and formative evaluation: Dr. John
Bransford, University of Washington, and Dr. Marlene
Scardamalia, University of Toronto Policy frameworks and new assessments: Dr. Linda
Darling-Hammond, Stanford University
21ST-CENTURY SKILLS DEFINEDATC21S started by internationally defining 21st-century skills as
four broad categories.
Ways of thinking
Ways of working
Tools for working
Living in the world
Collaborative Problem Solving
Learning through Digital
Networking
Problem Solving
Critical thinking
Problem solving
Learning to learn
Metacognition
Communication
collaboration
Information Literacy
ICT Literacy
citizenship
Life and Career
Personal responsibility
Social Responsibility
hypotheses
generalisations
Rules
Patterns
Elements
Cognitive Social Intellectual Capital
Social Capital
Producer
Consumer
Task Regulation
Knowledge Building
Social Regulation
Participation
Perspective
Ways of thinking
Ways of working
Tools for working
Living in the world
Collaborative Problem Solving
Learning through Digital
Networking
Problem Solving
Critical thinking
Problem solving
Learning to learn
Metacognition
Communication
collaboration
Information Literacy
ICT Literacy
citizenship
Life and Career
Personal responsibility
Social Responsibility
hypotheses
generalisations
Rules
Patterns
Elements
Cognitive Social Intellectual Capital
Social Capital
Producer
Consumer
Task Regulation
Knowledge Building
Social Regulation
Participation
Perspective
Collaborative problem-solving
ATC21S Assessment and Teaching of 21st Century SkillsThe University of Melbourne • Cisco • Intel • Microsoft • www.atc21s.org
Components
Collaborative problem solving
Social skills
Participation
Perspective taking
Social regulation
Cognitive skills
Task regulation
Knowledge building
Ways of thinking
Ways of working
Tools for working
Living in the world
Collaborative Problem Solving
Learning through Digital
Networking
Problem Solving
Critical thinking
Problem solving
Learning to learn
Metacognition
Communication
collaboration
Information Literacy
ICT Literacy
citizenship
Life and Career
Personal responsibility
Social Responsibility
hypotheses
generalisations
Rules
Patterns
Elements
Cognitive Social Intellectual Capital
Social Capital
Producer
Consumer
Task Regulation
Knowledge Building
Social Regulation
Participation
Perspective
Participation skills over levelsLow Peripheral participation
Low subjective responsibility for outcomes of collaboration, leading to lurking behavior
Simple epistemological beliefs (knowledge is perceived as fixed and to be transmitted from teacher/textbook to learner)
Middle Activity in scaffolded environments Responding to cues in communication Medium subjective responsibility for outcomes of
collaboration Developed epistemological beliefs (knowledge is
perceived as fixed, but can be elaborated through communication and collaboration)
High Initiating and promoting interaction Activating and scaffolding others in participation Ensuring equal participation rates among group members High subjective responsibility for outcomes of
collaboration Sophisticated epistemological beliefs (knowledge is
perceived as fluid, constructed, and inherently social/collaborative in nature)
Perspective taking skills
Low Low levels of empathy High egocentric bias Social projection (expectation of others as highly similar to
oneself) Ignoring contributions from others Contributions are not tailored to participants
Middle Medium levels of empathy Medium level of egocentric bias Receptive ability (being able to understand what others want to
convey, e.g. from overhearing) Contributions from others are taken into account Contributions are moderately tailored to recipients
High High levels of empathy Low or no egocentric bias Contributions from others are embraced and contextualized with
respect to collaborators’ opinions and skills Eliciting contributions from others (e.g. through questions) Contributions are tailored to recipients (audience design)
Social regulation skills
Low Low tolerance for ambiguity Competitive or individualistic social value orientation Low readiness to negotiate joint understanding Tendency to withdraw after conflict arises
Middle Cooperative social value orientation Attempts to negotiate joint understanding Conflicts will be avoided Initiation of compromises
High Pro-social attitudes Strategies for conflict resolution Conflicts are regarded as productive tensions Initiation of successful compromises
Task regulation skills
Low Trial and error hypothesis testing Unorganized sequence of solution attempts Little or no goal setting Variety of taskwork mental models will be ignored
Middle Forward search through a problem space Organized sequence of solution attempts Setting of unspecific goals Variety of taskwork mental models will be taken into
accountHigh Reflective regulation
Forward and backward search through a problem space Strategic oversight over collaborative strategy Setting of specific goals Variety of taskwork mental models will be harnessed
productively
Knowledge building skills
Low Knowledge telling Sharing of information Isolated contributions Lack of argumentation patterns
Middle Critical analysis of information Building on input from others Adding artifacts Forming of incomplete arguments
High Knowledge transforming Integration and synthesis of multiple artifacts Forming of complete, proper arguments (explanatory
coherence)
Developin assessment Tasks for ATC21S
Founder countries
Associate and founder countries
Draft -- Concept check for reality
Panel – cognitive laboratory for codes
Pilot – for administration
Trials – for calibration
Tasks and Assignments to assess ICT Literacy
Poetry(Graphic Organizers, Creating and Listening to Audio and video)
Arctic Trek (Collaborative Notebook, Information Foraging)
ATC21S
Twenty first century skills
Collecting the Data
Coding and scoring
Cognitive Behaviour
U2L001A = presence of chat before any action
Social Behaviour
U2L00 4A = All positions hav been covered (providing player had access to 3 balls)
U2L006A = sewuential placement of balls -6 combinationsLMRLMRRMPRMLRMLLMRLMRRMLLLMMRRRRMMLL
Calibration and Interpretation
Element Amplification Low Middle HighParticipation 1.Action Activity within environment No or very little activity in
environmentActivity in scaffolded context Activity in scaffolded and
unscaffolded contexts2.Interaction Interacting with, prompting and
responding to the contributions of others
Acknowledges communication
Responding to cues in communication
Initiating and promoting interaction or activity
3.Task completion Undertaking and completing a task or part of a task individually
Maintains presence only (lurking)
Identifies and attempts the task
Perseveres in task as indicated by repeated attempts or multiple strategies
Perspective taking 4.Responding adaptive responsiveness
Ignoring, accepting or adapting contributions of others
Contributions or prompts from others are ignored
Contributions or prompts from others are taken into account
Contributions or prompts of others are adapted and incorporated appropriately
5.Audience awarenessMutual modelling
Awareness of how to adapt behaviour to increase suitability for others
Contributions are not tailored to participants
Contributions are modified for recipient understanding in the light of feedback
Contributions are tailored to recipients (audience design)
Social regulation6.Negotiation Achieving a resolution or
reaching compromiseNo attempts to negotiate joint understanding
Comments on differences in perspective
Negotiates through differences in perspective
7.MetamemorySelf concept
Recognising own strengths and weaknesses
Notes own performance Comments on own performance in terms of appropriateness or adequacy
Comments on own performance in terms of appropriateness or adequacy in the context of the task
8.Transactive memory Recognising strengths and weaknesses of others
Notes performance of others
Comments on performance of others in terms of appropriateness or adequacy
Comments on performance of others in terms of appropriateness or adequacy in the context of the task
9.responsibility Initiative Assuming responsibility for ensuring aspects of task are completed by the group
Undertakes activities largely independently of others
Completes activities and reports to others
Assumes group responsibility as indicated by use of second person plural or accepting others’ contribution
11. Resource management Managing resources or people to complete a task
Uses resources (or directs people) without consultation.
Suggests that people or resources be used in part of a task
Allocates people or resources to a task through to completion
Element Amplification Low Middle High
Knowledge Building
20. Knowledge acquisition Follow path to gain knowledge Acquires knowledge as a result of being given it directly
Deliberate single actions to acquire knowledge
Knowledge acquired through multiple purposeful actions
12. Rules “If …then” (planning and executing)
Formulating a course of action to address a problem or task
Activity is undertaken with little or no prior formulation for a course of action
Identifies short sequences of actions for a specific task
Identifies potential multiple sequence routes for a complex task
19. Relationships (representing and formulating)
Making connections between elements of knowledge
Focused on (acts/shares) isolated pieces of information
Building on input and information from others
Integrates and synthesises of multiple pieces of information
18. Hypothesis “what if…” (reflecting and monitoring
Changing from one line of reasoning or course of action to another as information or circumstances change
Maintains a single line of approach
tries multiple options in light of new information or lack of progress
Reconstructs and reorganizes understanding of the problem in light of ne information or opinion.
Task Regulation
15. Collecting Elements Explore and understand Does not recognize the need for further information
Identifies the need for specific information related o immediate activity
Identifies need for varied information related to multiple activities.
13. Systematicity Implementing possible solutions to a monitoring progress
Trial and error hypothesis testing in an unorganized sequence of solution attempts
Forward search through a problem space with an organized sequence of solution attempts
Forward and backward search through a problem space with reflective solution attempts
17. Tolerance for ambiguity
Accepting ambiguous situations and exploring options within these
Maintains only a presence in situations where there is ambiguity
notes ambiguity and suggests options
Explores the problem space
14. Organising (Problem analysis)
Analysing and defining a problem in familiar language (i.e. Making the problem more manageable and meaningful)
Problem is stated as initially represented either explicitly or implicitly
Problem is divided into sub problems
Problem is divided into sub problems and their inter dependence is recognised
Dimensionality
- cognitive
Cognitive Strand Level Descriptions
Level 5 Refined strategic application & problem solving At this level, students can recognise previous errors and transfer and use that knowledge to more complex sub task pages. They recognise resources which they concluse have caused previous errors and avoid these. They are developing a level of confidence in dealing with the task and this allows them to correct or override their partner’s responses/answers/ actions. Students share appropriate resources with their partner and use consistent strategies that have they have identified as effective in previous steps in the problem solution. Any planning and most strategies being implemented are within a team context.
Level 4 Systematic and methodical At this level, students begin to solve problems through a collaborative process and use more effective co-working strategies. By now they can simplify the problem and review previous sub task implementations several times to determine whether the rule is transferable in subsequent task pages. Exchanging information between partners and using the shared information efficiently . Their sequencing and trial and error actions require few attempts and are hence completed in an optimal amount of time/attempts. Actions appear to be well thought out and planned and each action appears purposeful They can identify cause and effect and use suitable strategies to gain a correct path solution. The student directs and guides the partner in appropriate actions and resources.
Level 3 Strategic Planning Organising & Executing At this level the student can complete lower difficulty tasks correctly and independently. They are able to complete the final step or sub task and assist their partner in this process. They increase their working together by planning strategies of working, goal setting and sharing their resources with their partner. The student tends to focus on the relevant resources and disregards those that posed no benefit in previous trials. The student is able to conclude mutually agreeable answers when required. In multi-page tasks the student teds to identify successful approaches and resources and apply the same rules repeatedly. The student continues to intensify the sequential investigations and systematic behaviour in subsequent task pages that have increased in difficulty. This leads to attempts to test hypotheses together with the partner and to the development of increasingly thorough trial and error behaviour.
Level 2 directed approach Systematic Trial & Error At this level, the student identifies possible cause and effect of actions and demonstrated an initial understanding of the task concept and begins testing hypotheses and rules. The student uses successively enhanced sequential trials or systematic exploration to increase this knowledge. they may try to alter the techniques they use to investigate or explore the change to the task they are making. They start to share information with their partner, and to identify differences of resources and information that each partner has. They seek information about their partners resource set and provide information about their own to the partner. They reorganise information and begin to distinguish between the resources relevant the irrelevant to the problem space.
Level 1 Trial and Error Exploration Students at this level are primarily exploring the task environment, clicking on various resources often in a random fashion. They may engage in singular / unilateral approaches to trial and error in an attempt to build knowledge of the problem space. They participate individually in the activity and attempt to problem solve through an apparent unsystematic guessing approach. Their strategy is limited to following the specific instructions provided unable to proceed without instructions. They tend to repeat errors or reproduce unproductive actions with no clear indication of advancing through the task for several attempts. They may acknowledge their partner, in the context of discussing the problem space in a descriptive context or for the purpose of gaining more information.
social
Social Strand Level 5- Cooperation, Shared Goals and Appreciation
At this level, the student works collaboratively with partner for knowledge building and problem solving solutions. The student engages with the partner early in the task and there is discussion of planning and strategy. Feedback from partner is incorporated and/or adapted to identify solution paths or modify incorrect ones. Where there are conflicts (e.g. missing information or inaccurate information from partner), the student manages them successfully to reach common understanding / agreement with partner before proceeding on a possible solution path.
Level 4 Mutual Commitment At this level, the student works collaboratively with partner throughout the problem solving process. He/She promotes meaningful interaction with partner, exchanging information/ resources and approaches as well as incorporating ideas from partner. The student also reports his/her own activities to partner. Differences in points of view (e.g. inaccurate information / irrelevant resources) are managed much more effectively and there are attempts to reach consensus to progress on the activity.
Level 3 Resolving Differences At this level, the student becomes aware of the partner role early in the collaborative problem solving process and recognises the need to engage with partner for the problem solving task. The student initiates and promotes interaction with partner; sharing resources, information and ideas. The student attempts to clarify differences (e.g. irrelevant resources from partner) in order to reach agreement (not necessarily consensus) to progress on the task. The student acts as though they are aware of his/her own strengths and challenges in collaborative problem solving.
Level 2 Awareness of Partnership At this level, the student works independently on some parts of the activity but is also beginning to develop an awareness of partner role which can impact on the chances of completing the task successfully. He/she interacts with partner and requests resources / information. The student also starts to tailor communication with partner to improve mutual understanding of the communications exchange. The student may have conflicts with his/her partner but there is an attempt to clarify differences.
Level 1 Clarification and Independence At this level, the student is working independently on the activity using only his/her own ideas. There is limited interaction with partner and this is mainly prompted by task instructions. The student acknowledges communication cues by partner. However, the student’s communication with partner elicits requests for clarification and restatement Disagreements lead to conflicts that are not resolved by the end of the activity.
Developmental Progressions
Social Strand Cognitive StrandLevel 5- Cooperation, Shared
Goals and AppreciationLevel 5 Refined strategic
application & problem solving
Level 4 Mutual Commitment Level 4 Systematic and methodical
Level 3 Resolving DifferencesLevel 3 Strategic Planning
Organising & Executing
Level 2 Awareness of Partnership
Level 2 directed approach Systematic Trial & Error
Level 1 Clarification and Independence
Level 1 Trial and Error Exploration
Feedback and Reporting
Reports
Reporting to StudentsLearning Readiness
Real time reports will be available, based on empirically developed learning progressions. They will identify a point of readiness to learn for each student.
This type of report will be linked to teaching interventions associated with readiness to learn as indicated by the black bar in the spine of the chart.
Instructional groups
Teacher development
Professional development
modules
1. Defining and Assessing 21st Century Skills
2. Using a Developmental Model
3. ATC21S Assessments: Getting Started
4. Interpreting Reports5. Teaching and
Learning 21st Century Skills
The System
An integrated system– Developmental progressions – standards referenced – Assessment tasks with automatic scoring– Teaching intervention and professional development– Reports for students, teachers and systems
Next stepsAccess and Use
Validation studiesExpanded resources – mainstream
curriculumOther 21st century skills
• ATC21S Assessment and Teaching of 21st Century Skills• The University of Melbourne • Cisco • Intel • Microsoft • www.atc21s.org
Transfer and Utility
To what degree will 21st century skills such as collaborative problem solving or ICT Literacy in Learning Networks facilitate student learning?
– Are these skills teachable and learnable?– Will the skills enhance learning in the classroom?
To what degree might enhanced 21st century skills contribute to adaptiveness of graduate students to the workforce?
ATC21s New Users?
Follow up!
Volume 2: The methods
Volume 3: The re
search
Implications
Conventional Assessment
• Correctness if the important part of the response
• Response can indicate reasoning ability of a specific level
• Difficult to interpret various incorrect responses
• Additipnal data is lost in the focus on the correct response
Future Applications
Task design
I. Design tasks to capture specific components of cognitive ability
I. working memoryII. Processing speedIII. Pattern recognitionIV. Systematicity’
II. Background data capture allows more user friendly and engaging asssessments
I. Reduce test anxietyII. counteract teaching to the testIII. Capture multiple variablesIV. Reduce test time needV. Increase assessment efficiencyVI. Better, richer information for teachers
III. Applications in serious game industry
Roll Out and Dissemination
What is in the system?
• Two models - cloud and portable
• Prototype assessment tasks
• Developmental progressions
• Reporting and feedback for schools, teachers and students
• Professional development modules
• Specifications for developers
Portable and Cloud Systems
Portable version Cloud version
Local adaptations Centralised control/management
Decision-making autonomy Centralised decision-making
Customisation enabledEmbed or link within existing systems
Local adaptationsAddition of tasks
No customisation
Local support structure Centralised support structure
Flexible registration system Rigid registration system
FlashLess susceptible to differences across
browsersNot currently supported on iPad*
HTML 5More susceptible to differences across
browsers
PISA 2015
• Collaborative problem solving to be tested in OECD countries in 2015. PISA will test human to computer interaction
• Collaborative Problem solving involving human to human maybe in 2018