introduction to questionmark’s assessment …€¢assessment technologies and portal solutions...
TRANSCRIPT
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
A s s e s s f o r S u c c e s s
Questionmark Conference 2018 Savannah
Introduction to Questionmark’sAssessment Management Technologies
Rory Northam
Product Evangelist, Questionmark
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Assessment Solutions enabled by Questionmark Assessment Management System Overview
▪ Collaborative Authoring Demonstration
▪ Blended Delivery Demonstration
▪ Reporting and Analytics Demonstration
▪ Integration
Q&A
Agenda
Slide 2
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
• Founded in 1988
• Offices in US, UK, Canada; Partners worldwide
• Assessment technologies and portal solutions that enable organizations to measure knowledge, skills, and attitudes securely for certification, regulatory compliance and improved learning outcomes
• ISO/IEC 27001 Certified (Learn more at www.questionmark.com/trust)
Background
• 1100 Enterprise customers worldwide
• Corporate, Government; Higher Education; Certification Providers; Awarding Bodies
Customers
About Questionmark
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Assessment Management System
Collaborative, secure authoring of surveys, quizzes, tests, and exams
Reporting to analyze results and inform stakeholders
Blended delivery to support a broad range of devices and security needs
Multilingual: 36 Languages
Global 24/7 Support
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
OnDemand OnPremiseOnDemand for Government
Questionmark Deployment Options
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Measure and Enhance Learning
•Measure knowledge, skills and attitudes –before, during and after learning
•Correct misconceptions and increase knowledge retention
Qualification and Certification
•Deliver reliable, defensible tests to certify employees
•Ensure sound personnel decisions
Regulatory Compliance
•Ensure knowledge of essential laws, regulations and health/safety guidelines
•Demonstrate compliance and reduce risk
Sales/Technical Channel
•Competence/product expertise of sales and technical professionals
•Increase revenues, customer satisfaction
•Reduce costs
Credentialing
•Efficiently create and maintain valid, reliable tests
•Generate revenue from exams and practice tests
Higher Education
•Increase student satisfaction
•Improve learning
•Reduce Cheating
Why our customers use assessments
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Browser-based, collaborative environment▪ Role-based access control
Manage item bank▪ Organize items by topic, subtopic and
metatag▪ Item version control▪ Import items, assessment and other
content
Author items and assessments
Publish assessments for delivery
Authoring and Item Banking
Slide 8
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Slide 9
Many item types to choose from…
Multiple choice
Multiple Response
Matching
Ranking
Pull-down List
Select-a-blank
Matching
Job Task Analysis
Knowledge Matrix
Survey Matrix
Likert scale
True/False
Yes/No
Text Match
Drag-and-Drop
Hotspot
Numeric
Essay
Fill-in-the-blanks
Explanation
File Upload
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Demonstration
Slide 10
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Question-level feedback
Slide 16
Question with participant’s answer
Corrective feedback
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Feedback based topics of questions
Slide 17
Corrective feedback points participant to appropriate learning
resources
“Topics” of questions
answered in assessment
Score on topic
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Provide users role-based access to content
Provide users role-based access to content
Roles & Permission Management
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Item version history and rollback
Slide 19
Revision History
Audit trail of edits and option to “rollback”
Easily “track changes” made in collaborative authoring or review
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Search description, stimulus, choices, status, meta tags,
dates and more
Item SearchSearch item bank to find items that contain certain words and/or tags
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Multilingual Interfaces
Slide 22
Arabic ( (العربية Bulgarian (Български) Catalan (Català) Chinese, Simplified (简体中文)
Chinese, Traditional (繁體中文)
Croatian (Hrvatski) Czech (Čeština) Danish (Dansk) Dutch (Nederlands) English Finnish (Suomi) French (Français)
German (Deutsch) Greek (Ελληνικά) Hebrew ( (עברית Hindi (हिन्दी) Hungarian (Magyar) Indonesian (Bahasa
Indonesia) Italian (Italiano) Japanese (日本語) Korean (한국어) Norwegian Bokmål
(Bokmål) Persian ( (فارسی Polish (Polski)
Portuguese, Portugal (Português)
Romanian (Română) Russian (Русский) Serbian (Српски) Slovenian (Slovenščina) Spanish (Español) Swedish (Svenska) Thai (ภาษาไทย) Turkish (Türkçe) Ukrainian (Українська) Urdu ( (اردو Vietnamese (Tiếng Việt)
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Translation Management for Multilingual Assessments
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Blended, Secure Delivery
Slide 26
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Author Once… Deliver Anywhere
Slide 27
Single results set
Available Securely
Author once Deliver to any devicein context
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Responsive Design, Blended Delivery
One Assessment…
…many devicesand browsers
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Demonstration
Slide 30
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Print your own bubble sheets
Participants enter responses
Scan and upload for scoring
Analyze and report on results
Printing and Scanning:Deliver assessments online or on paper
Slide 32
Deliver assessments on paper or online Print your own question and bubble sheets Use your own scanner to read responses Upload scanned bubble sheets for scoring Scanned assessment results available for reporting
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Delivering high stakes assessments
Quizzes & Surveys
Diagnostic Tests
Low Stakes Tests
Medium Stakes Tests
High Stakes Exams
S e c u r i t y R e q u i r e m e n t s
Sta
ke
s o
f A
ss
es
sm
en
t
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
“Save as you go”
Results saved at the server “as you go” so IT failures don’t lose data
Hope for the best; prepare for the worst
Slide 34
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Secure browser for high-stakes assessments
Reduce content theft, cheating; improve exam integrity
Questionmark SecureOne Setting – Multiple Platforms
Slide 35
Windows Mac iPad
PrintingScreen captureTask switching
Control/system keysRight-click / context menu
optionsView source
Accidentally exiting exam
Prevents….
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Require Proctoring
Slide 36
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Convenience of online exam delivery at home or workplace under the watchful eye of a proctor▪ Works with most standard webcams/audio
▪ Live proctor can start, pause, and stop exam
More info:▪ Website
▪ Your account manager
Online Proctoring
Slide 37
www.questionmark.com/proctoring
Powered by
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Accessibility
Also support for…- Non-mouse navigation; alternate pointing devices- Screen readers
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Questionmark BadgingRecognize Assessment Outcomes with Verifiable, Portable Digital Credentials
Slide 39
Integrated with industry-leading digital credentialing platform
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Single-Sign-On
• SAML
Launch and Track via AICC or SCORM
• SuccessFactors, Cornerstone, etc.
Learning Tools for Interoperability (LTI)
• Blackboard, Canvas, D2L, Moodle, etc.
Web services API
• QMWISe
OData APIs
• Results API
Integration
Slide 40
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Publish to LMS
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Reporting and Analytics
Slide 42
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Reporting and Analytics
Slide 43
What can we learn from assessments?
Did they like it?
Did they learn it?
Can they do it?
Can we prove it?
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
What you can learn from Questionmarkreporting and analytics
How are our Items and Assessments doing? How are our Participants doing?
Did the learning investment impact
performance?Where are the skills and knowledge gaps in my
organization?
Are my questions and tests valid and reliable?
What did participants think of the learning
experience?
What did participants learn from the training
or course?
How do I coach participants to improve
performance?
How did one group of test takers compare to
another?
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Item and Test Analysis Item Analysis report Test Analysis report Score list report Test center analysis report Assessment completion time report Question frequency analysis report Demographic report Question type report Assessment Overview report Assessment results over time report
Participant-focused reports Score list report Coaching Report Transcript Report Grade Book Report
Survey Reports Survey Frequency Analysis Job-task-analysis (JTA)
▪ JTA Summary▪ JTA dimension by demographic
Course Evaluation▪ Course Summary Report▪ Instructor Summary Report▪ Class Summary Report▪ Class Detail Report
Types of reports available
Slide 45
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Score List Report
Slide 46
Demonstration
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
What it measures:▪ % answered correctly
How it is expressed:▪ Number from 0.0 to 1.0
How to interpret:▪ Too high = too easy▪ Too low could mean…
Too difficult
Confusing/ambiguous
Miscoded
What it measures:▪ Correlation between
item score and test (assessment) score
How it is expressed:▪ Number between
-1.0 and +1.0.
How to interpret:▪ Generally…
< 0.2 low
> 0.3 good
Item Analysis: Reliability of Question Results Data
Difficulty index (aka P “value”) Item correlation discrimination:
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Slide 48
Difficulty/P-Value
Correlation -Discrimination
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Slide 49
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
John Kleeman’s
Item Analysis for Beginners
Thursday, March 8, 13:30 – 14:45
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Slide 51
Coaching ReportCompare against
benchmarks or other participants
Participant
Topiclevel
Overall
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Job Task Analysis (JTA) Reports
Slide 52
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Course SummaryInstructor Summary
Class Summary Class Detail
Four Course Evaluation Reports:
Slide 53
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Built-in Reports Custom Reports and Dashboards
Microsoft ExcelPowerPivot
BI Tools
Big Data Applications: Results API
Slide 54
R e s u l t s W a r e h o u s e
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
•Planning
• Installation
• Integration
Deployment
•Assessments
• Importing items
•Templates / Look & feel
Authoring
• Item reports
•Other custom reports
Reporting
•Testing and Assessment Portals
•People Data syncing
•Results Data syncing
•Self Registration
•Auto Scheduling
•Assessment Clustering
•Certificate Generation
•Distributed Administration
•Assessment Catalogs
Solutions & Integration
Consulting / Solution Services
Slide 55
w w w. q u e s t i o n m a r k . c o m / c o n s u l t i n g
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Summary of key benefits
Collaborative Authoring
• Secure
• Browser-based
• Available globally
Blended Delivery
• Responsive Design
• Browser support
• Tablets/smartphones
• Accessibility
• Secure delivery for high stakes exams
Reporting and analytics
• How are my participants doing?
• How are my tests doing?
• Results API
Multilingual
• Translation Management
• Screens/Controls in 36 languages
• Right-to-left languages
Integration
• Supports industry standards
• APIs
• Connectors
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
What questions do you have?
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
www.questionmark.com/whitepapers
Recommended eBooks and White Papers
Slide 59
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
A s s e s s f o r S u c c e s s
Questionmark Conference 2018 Savannah
Follow us…
Thank you for attending!
We hope to see you at future webinars
Keep up to date at blog.questionmark.com!