Educational Standard Correlation Services for Digital Libraries

Holly DevaulJonathan OstwaldJohn Weatherley

Digital Learning SciencesUCAR

Anne DiekemaJennifer Bailey

CNLP Syracuse University

Presentation Overview

We will discuss a suite of educational standards assignment and alignment services that support cataloging and discovery of educational standards information in the NSDL – First half - how these services work, scenarios for

use, and expected performance levels based on recent evaluations

– Second half - more technical in nature; aimed at software engineers and web developers focusing on how the APIs work

CASAA- Computer Assisted Standards Assignment and Alignment

NSF NSDL Service project with two goals Support the assignment of standards to

learning resources– Content Assignment Tool (CAT)

Provide a cross-walk between the different state and national standards – Standards Alignment Tool (SAT)

Motivation

No Child Left Behind Act (NCLB) Teachers must document how their teaching is

aligned to standards States required to develop assessments that

evaluate student achievement relative to standards

Teachers want access to standards information for accountability reporting

Opportunity for digital libraries to provide this service

Content Assignment Tool (CAT)

Natural Language Processing technology

assesses resource content, suggests relevant

educational content standards

Semi-automatic approach– CAT suggests relevant standards to the user– User makes final determination

System learns from vetted assignments Web tool plus REST API for easy integration with

other applications

Web-based access to CAThttp://cat.nsdl.org:8280/casaa

User name- guest

Password- p

Integration with theDigital Collection System (DCS)

Accesses resource content via URL

Utilizes data from catalog record -url, grade level

-description, keywords (optional)

Is configured to query only NSES

Suggestions provided in context of cataloging

Visual cues help track suggested and human selected standards – human makes final decision on what to assign

Sta

nd

ard

s

Selected Suggestion

Selected but not suggested

Suggestion

CAT Evaluation

Compared human-assigned NSES standards to CAT suggested standards

Research questions:– Does the CAT Service return standards that a

human would choose for assignment? – Do the standards human catalogers have

chosen occur near the top of the ranked list of CAT Service suggestions?

– Does the inclusion of metadata improve standards assignment via the CAT Service?

National Science Education Standards (NSES)

Hierarchical in structure, classified by grade band and content area and can be grouped as follows:– inquiry - (nature of science; Standard A)– subject - (life, physical, Earth and space

science; Standards B, C and D)– applied - (science and technology, personal

and social perspectives, and the history and

nature of science; Standards E, F, and G).

Does the CAT Service return standards that a human would choose for assignment?

CAT Service suggestions include 28-51% of the standards humans assigned

This offers a time savings in the initial assignment of the standards by presenting the cataloger a smaller subset of standards to choose from, that includes a substantial number of the preferred standards.

Do the standards human catalogers have chosen occur near the top of the ranked list of CAT Service suggestions?

Analysis was conducted to assess the placement of the human-assigned standards in the ranked list of suggestions from CAT

If metadata included, 83% of the standards suggested at rank 1 were also chosen by a human; then drops off quickly

Subject standards performed better than applied or inquiry

We speculate that the language used in the resource corresponds better with subject standards than with applied or inquiry which require more subjective interpretation

Does the inclusion of metadata improve standards assignment via the CAT Service?

YES- if no metadata included, 60% of the standards at rank 1 were also chosen by a human

In addition, including metadata expands the utility of this tool to suggest standards for non-text-based resources that have descriptive metadata associated with them

See technical paper for full details of this analysis

Standards Alignment Tool (SAT)

Dynamic automatic correlation of standards Combination of various algorithms to find

equivalent (correlated) standards Allows great flexibility for indexing and

search Takes standard text or ASN standard PURL REST API for easy integration with other

applications/systems

Web-based access to SAThttp://sat.nsdl.org:8380/casaa-sat/

User name- guest

Password- p

SAT-DLESE Discovery integration

Testbed collection with NSES standards cataloged at 4th level

Select state of interest to view corresponding state standards side-by-side

Proof-of-concept for technical integration; alignments not yet vetted

http://sat.dlsciences.org

State standards are displayed side-by-side national standards

SAT Evaluation

Looked at NSES to state mapping for five states (OH, MA, MN, CO, NY)

NSES standards distributed proportionately across inquiry, subject and applied subgroups and across grade level (n=49)

Queried SAT for state alignment to NSES Evaluated first 5 suggestions for each

standard (strong, moderate, weak correlation) or none at all

Preliminary analysis

Issues– Lack of equivalent state standards – Differing granularity of scope of content

A mechanism to better discriminate no match and multiple matches is needed

For this alignment (NSES-state) human mediation is still necessary (as with CAT)

Other alignment pathways (state-to-state) need to be evaluated to get the full picture

Additional machine learning may improve results

Part Two – The Technical Bits

Aimed at software engineers and web developers who wish to utilize the API’s and integrate CAT and SAT functionalities with existing applications

CAT API in DCS: Topics

CAT / DCS Interaction The Educational Standards Hierarchy API call: suggestedStandards Constraining the search for suggestions Other uses of API in DCS RPC vs REST Services

DCS Communicates with CAT via Web Service

Selectable / Suggested Standards are leaf nodes

Leaf

Standards Hierarchy

suggestStandards request

CAT search for suggestions guided by info supplied to the API

Returns StandardsList suggested by the CAT Service

CATDCS

Urls, Constraints

StandardsList

•ASN Identifier•Benchmark

Constraining Suggestions

author – (NSES) topic – (Science) maxResults – (10) startGrade endGrade keywords

– Description– (weighted) keywords

RESTRepresentational State Transfer

CAT Service Response

CAT API Requests

suggestStandards – Suggests aligned standards for a specific lesson plan

saveStandards – Save standards to a specific assignment

getStandard – Retrieve the details for a specified standard

getStandards – Retrieve the details for a specified list of standards

CAT API Requests (continued…)

removeStandards – Remove standards from a specific assignment

getAssignments – Retrieve the standards for a specific assignment

getAllAssignments – Retrieve all assignments (including the standards for each assignment) for a specified user

getAuthors – Retrieve all assignments (including the standards for each assignment) for a specified user

getTopics – Retrieve a list of all topics available in CAT

SAT Service: Topics

DLESE use case: Display state standards that are aligned to NSES

API call: suggestedStandards Displaying the suggested standards in the

DLESE interface SAT API overview Development tips

DLESE Request for State Standards

User selects a state in the DLESE UI DLESE provides ASN identifiers for NSES

standards– suggestStandards request accepts ASN identifiers

or text SAT returns a list of suggested standards as

XML– Response contains text suitable for display in the UI

DLESE UI displays the standards to the user

DLESE Communicates with SAT via Web Service

SuggestStandardsREST Request

Standardslist returned as XML

SATService

DLESELibrary

Application

• ASN Idor

• Text

NSES Standards Supplied by DLESE, State Standards Returned by SAT

NSES standards supplied by DLESE as ASN Ids

State standards returned by SAT service as text

suggestStandards request

One or multiple standards can be supplied in the request

Multiple standards handled one of two ways:– Combine the text of all the standards into a single

query and return a single result set - or -

– A query for each standard and return as many result sets as there are standards

DLESE used the latter– DLESE has multiple standards cataloged for each resource– Display best match for each standard

Supply grade range to refine results

SAT suggestStandards Response

DLESE Display

State standards displayed side-by-side with NSES standards

When a state is selected by the user, request is made asynchronously using AJAX– Provides time for SAT to respond and gives the

user feedback with a spinner graphic

Selected state is remembered throughout the user’s session

SAT API Requests

suggestStandards – Suggests aligned standards for specific standard

saveStandards – Save standards to a specific assignment

removeStandards – Remove standards from a specific assignment

getAssignments – Retrieve the standards for a specific assignment

getAllAssignments - Retrieve all assignments (including the standards for each assignment) for a specified user

SAT API Requests (continued…)

getAuthors - Retrieve a list of all authors (states) available in SAT

getTopics - Retrieve a list of all topics available in SAT

getStandard - Retrieve the details for a specified standard

getStandards - Retrieve the details for a specified list of standards

Development Tips

Use AJAX to retrieve and display standards at point of request

Retrieve and cache standards alignments ahead of time if needed

Use JSP/PHP to retrieve and display the results in a Web setting

REST Service URLs

CAT http://cat.nsdl.org:8280/casaa/service.do

SAThttp://sat.nsdl.org:8380/casaa-sat/service.do

Contact us for more information

Anne Diekema diekemar@syr.edu Holly Devaul devaul@ucar.edu Jonathan Ostwald ostwald@ucar.edu John Weatherley jweather@ucar.edu