Standards or Semantics for Curriculum Search?

Byron Marshall and René Reitsma

Oregon State University

Martha N. Cyr

Worcester Polytechnic Institute

JCDL 2007

TeachEngineering is funded by the National Science, Technology, Engineering, and Mathematics Education Digital Library (NSDL) program,

under National Science Foundation grant nos. 0226322 and 0532709.

2Standards or Semantics for Curriculum Search?

Agenda

• TeachEngineering• The Goal: Help teachers find curricula that

support assigned local educational standards• Standards Alignment:

– What is being done and why?– Two problems: granularity and sparseness

• Standards + Semantics:– Network-based approach– Preliminary results

3Standards or Semantics for Curriculum Search?

TeachEngineering

• Collaborative project:– Colorado School of Mines, Duke, Oregon State

University, University of Colorado at Boulder & Worcester Polytechnic Institute

– American Society for Engineering Education– NSF National Science Digital Library

• Engineering curriculum (FIPSE and GK-12)

• Free, Web-based search

http://www.teachengineering.com/about.php

4Standards or Semantics for Curriculum Search?

The Task: Identifying Useful Curricula

• Assigned Educational Standards– Thousands of individual standards across

multiple states and national groups– Increasing pressure to teach to standards

• Curriculum is often mapped to standards in one state at most: an opportunity for value-added search in a digital library

• One seemingly useful approach: identify equivalent standards across states

5Standards or Semantics for Curriculum Search?

Transitive Results Direct Results

Query: What lesson plans support my Massachusetts

standard?

Standards-Enabled Search: Crosswalking at TeachEngineering (TE)

• TE created a network of transitive mappings (state std. national std. state std.) by manually mapping 4 sets of state standards to several national models (3386 statenational alignments)

• Curriculum is mapped to standards for one state so users can search for standards-relevant items developed in different states

Lesson Plan

Lesson Plan

CO. State Std.CO. State Std.

Lesson Plan

National Std.National Std.

Lesson Plan

Lesson Plan

Lesson Plan

MA. State Std.MA. State Std.

6Standards or Semantics for Curriculum Search?

Curriculum Alignment for Search

• TE did its manual mapping but always planned to rely on others to map the standards:– JES&Co and the University of Washington– CNLP (Center for Natural Language Processing) at

Syracuse University– Standards-based search from collections such as

DLESE and TeachersDomain.org– For-profit initiatives, e.g., nettrekker.com,

academicbenchmarks.com, statestandards.com.– Ongoing collaborations between some of the above

projects and organizations

7Standards or Semantics for Curriculum Search?

Problem 1: Granularity -- an Example

• The TE system contains some materials developed in Colorado which are related to triangles and the Pythagorean theorem

• But, no documents are returned for a TE search based on the standard:– MA Math. Geom. 8.G.4: Demonstrate an

understanding of the Pythagorean theorem. Apply the theorem to the solution of problems.

• Why? The Colorado standards do not specifically mention the Pythagorean theorem; there exists no CO-MA standards crosswalk

8Standards or Semantics for Curriculum Search?

Problem 2: Sparseness in the TeachEngineering Std-to-Std Mappings

States

CO MA NC OK Overall

Standards 117 691 415 725 1948

National Alignments 24 370 237 266 897

National Alignments/Std. .21 .54 .57 .37 .46

(transitive) State Alignments 34 235 240 266 775

(transitive) State Alignments/Std. .29 .34 .58 .37 .40

(projected) State Aligns./Std. (U.S.) 4.73 5.55 9.47 6.04 6.53

We found only .4 transitive alignments on average for each standard. Projecting this for the US (.4/3 * 49) each state standard would have fewer than 7 mapped standards anywhere in the country.

9Standards or Semantics for Curriculum Search?

Looking Ahead:Solutions We are Exploring at TE

• Term-based search can help teachers find those Pythagorean lessons – tedious for the user? spotty results?

• Commercial services; e.g., Academic Benchmarks, will align standards to lessons– cost? proprietary/manual methodologies? static

mappings to changing standards? millions of unprocessed documents?

• Learning-based systems (e.g., CNLP’s CAT & SAT tools)– Can comprehensive and reliable training data be

developed and maintained?– State of the art provides very useful support tools for

semi-automatic alignment

10Standards or Semantics for Curriculum Search?

Our Agenda: Exploit Multiple Linkages to Establish Relevance

Educational Standard

Educational Standard

Lesson Plan

Lesson Plan

Lesson Plan

Lesson Plan

TermTerm

Lesson Plan

Lesson Plan

TermTerm

More pathways may help find more items and improve ranking

More Terms

More Terms

Lesson Plan

Lesson Plan

Other Educational Standards

Other Educational Standards

AAAS Strand Maps

Std. Correlations

Other Associations; e.g., Curriculum Structure, Ontologies, or Resource Similarity

Other Associations; e.g., Curriculum Structure, Ontologies, or Resource Similarity

Lesson Plan

Lesson Plan

Lesson Plan

Lesson Plan

11Standards or Semantics for Curriculum Search?

Preliminary Results: Expanding Queries Using Standards Alignments

• Adding terms found in aligned standards improved our human-judged accuracy.

• Using terms from a standard resulted in 11 correct (as judged by a person) lesson plans in the top 15 returned items.

• When we used terms from the standard and from other aligned standards, accuracy improved to 14/15.

Query: What lesson plans support my

Massachusetts standard?

NC State Std.

NC State Std.

MA. State Std.MA. State Std.

crustearth

Lesson Plan

Lesson Plan

Lesson Plan

Lesson Plan #1

#2

#3

#4

earthquake

12Standards or Semantics for Curriculum Search?

Preliminary Results: AAAS Plate Tectonics Strand Map

AAAS 4.C.4 Grades 9-12

Including terms from all of the parts of this strand map added some interesting dimensions to our results set such as

•The Energy of Music•Seeing and Feeling Sound Vibrations•Riding Radio Waves

These are not the first thing we think of, but then again….

13Standards or Semantics for Curriculum Search?

Wrap Up

• Teachers need curriculum that supports assigned educational standards. How can we help them get to the documents they really want?

• Standards mappings are a start but granularity differences and sparseness are real problems.

• To address this important user task, hybrid solutions leveraging both standards and term-based semantics are promising.

14Standards or Semantics for Curriculum Search?

Thanks! Questions?

Standards or Semantics for Curriculum Search?

Byron Marshall and René Reitsma

Oregon State University

Martha N. Cyr

Worcester Polytechnic Institute

15Standards or Semantics for Curriculum Search?

Preliminary Results: Noun Phrases

• Term-relevance is a well-explored phenomenon but which method of choosing index terms is best in this context?

• Based on preliminary work we observe that using noun phrases can boost the relevance score of good items.

0

50

100

150

200

250

300Noun phrases increased

the relative score of some high quality selections

Noun phrases increased the relative score of some

high quality selections

Scored using noun phrases (NP)Scored using Controlled Vocabulary (CV)

Rel

evan

ce R

ank

Scor

e

0

50

100

150

200

250

300Noun phrases increased

the relative score of some high quality selections

Noun phrases increased the relative score of some

high quality selections

Scored using noun phrases (NP)Scored using Controlled Vocabulary (CV)

Rel

evan

ce R

ank

Scor

e