keynote designing assessment, assessing instructional design
DESCRIPTION
Assessment plays a vital role in delivering, evaluating, monitoring, improving and shaping learning experiences on the Web, at the desk and in the classroom. In the process of orchestrating educational technologies instructional designers are often confronted with the challenge of designing or deploying creative and authentic assessment techniques. The talk provides examples of the conceptual development and implementation of assessment approaches in three different areas: Needs Assessment: At the outset of an instructional design project, we work with stakeholders to gather data that helps us to reach the audience effectively and design user-friendly interfaces. Typical techniques are focus groups, surveys, qualitative interviews, personas and scenarios. Impact Assessment: Once the program or project is launched, we seek to understand how learners access online material or move through the curriculum, which helps us improve their experience. Data sources comprise Web analytics, social media metrics, learning analytics, surveys and interviews. Classroom Assessment: In the classroom, we aim to implement assessment techniques that support students’ critical thinking abilities and transfer learning skills. This includes peer-to-peer assessment, rubrics, portfolios and problem-based learning.TRANSCRIPT
Designing Assessment,Assessing Instructional Design
TCC Hawaii 2014Stefanie Panke
Aloha Ya’ll!
From Pedagogical Concepts to Practical Applications
AACE Special Interest Group ‘Assessing, Designing, and Developing E-Learning’
http://panke.web.unc.edu/tcc2014
“Practically everybody in the academic community gets assessed these days,
and practically everybody assesses somebody else”(Astin, 2012).
“We need to get past the emulation society, where instead
of learning you spend all your energy on convincing others
what you know”
“Assessment is the fabric of education, if
you change assessment, you change everything”
Ambjorn Naeve, Tel-Map Group Expert Meeting, Lisbon 2011
Learning and assessment are two sides of the same coin, and they strongly
influence each other (Gulikers, Bastiaens & Kirschner, 2004).
Assessment = Student
Feedback
“Effective learning occurs when students receive feedback, i.e. when they receive information on what they have (and have not) already learned. The process of generating such information is assessment” (Entwistle, 2000).
Assessment = Learning
Organization Catalyst
“We view assessment as the gathering of information concerning the
functioning of students, staff and institutions of higher education. The
basic motive is to improve the functioning of the institution and its
people”(Astin, 2012).
Measure vs. Treasure
From ‘assessment of learning’ towards ‘assessment for learning’
Knowledge imparted by the instructor (input)
Competencies students can apply (output).
“To change student learning in the direction of competency development, authentic competency-based instruction aligned to authentic competency-based assessment is needed” (Gulikers, Bastiaens & Kirschner, 2004).
“For learning objectives not easily measured by multiple choice or true/false questions, assessment and evaluation can be time-consuming and difficult” (Anders, 2012).
The growth of virtual conferences has high
potential as a tool for a more authentic assessment
in higher education graduate programs (Menchaca, Ho & Hoffmann, 2013).
Five dimensions of authentic assessment: (a) the task, (b) the physical context, (c) the social context, (d) the results(e) the criteria.
Gulikers, Bastiaens & Kirschner (2004)
“The perception of what authenticity is may vary among individuals as a result of educational level, personal interest, or amount of professional experience” (Gulikers, Bastiaens & Kirschner, 2004).
Threshold Concepts (Meyer & Land 2003, 2005):• Transformative • Integrative • Oftentimes bounded • Probably irreversible• Potentially troublesome
Designing creative assessment helps students unfold emerging understanding of key concepts.
Moving on…
Examples &Tools
(1) Needs Assessment (2) Impact Assessment (3) Classroom Assessment
Needs Assessment
• Stakeholder input• Example: Website
Redesign Process• Workshop Series:– Audience– Content– Categories– Navigation
Audience
• Goal: Identify potential audiences for the online resource
• Exercise: Personas
• Structure: Participants create personas in groups (approx. 20 minutes), present and vote on most important / fitting representations of their audience.
Content
• Goal: Identify organizational output, recurring website elementsExercise: Content Toolkit
• Visual building blocks • Create representation of
different assets (books, news,user profiles, articles….)
Results
Categories
1. Review existing categories
2. Label sample content
3. Taxonomy on the fly
Examples &Tools
(1) Needs Assessment (2) Impact Assessment (3) Classroom Assessment
Impact Assessment
• Formative evaluation of material, learning environment, curriculum
• Example: Learning Analytics– MPA competence
assessment process
Data is the new oil – are learners the new sardines?
Analytics Dashboards
“Deciphering trends and patterns from educational big data” (Horizon Report 2013).
“The measurement, collection, analysis and reporting of data about learners and their contexts” (SOLAR 2014).
Competence- based
CurriculumRedesign
• Analytical rubrics facilitate the assessment of student performance in central learning outcomes.
• Collaborative effort of instructional designer, program leaders and faculty.
Rubrics
Rubrics
Examples &Tools
(1) Needs Assessment (2) Impact Assessment (3) Classroom Assessment
Classroom Assessment
• Support students’ critical thinking abilities and transfer learning
• Example: Portfolio– Infrastructure– Scaffolds– Peer Learning– Assessment– Evaluation
Portfolio Components
• WordPress Multisite• Sample Portfolio• Portfolio Guide• WordPress Training• Competence Memo Template• Peer Feedback and Milestones in PUBA 746• Portfolio Assessment Rubric• Portfolio Evaluation (Questionnaires, Focus
Groups, Interview Guides)
Benefits of e-portfolios • Fostering digital
literacy /multimedia storytelling (Wakimoto & Lewis, 2014).
• Supporting career development (Reese & Levy, 2009).
• Strengthening organizational ties (cf. Lorenzo & Ittelson, 2005; Reese & Levy, 2009).
• Encouraging reflection (Roberts, Maor & Herrington, 2013).
• Promoting self-regulation (Abrami et al, 2008, Meyer at al, 2010).
• Improving knowledge management (Chang, Tseng, Liang & Chen, 2013).
• Acknowledging diversity and transfer learning (O’Toole, 2013).
Types of Portfolios (Hewett, 2004)
• Documentation portfolios: growth toward learning goals
• Process portfolios: phases of the learning process
• Showcase portfolios: accomplishments and competences
What nuggets can you use in your work or personal environment?
Thanks you – have a good conference!