concrete approaches to quality assessment: moving beyond peer review august 6, 2003 howard burrows...

Concrete Approaches to Quality Assessment:

Moving Beyond Peer ReviewAugust 6, 2003

Howard BurrowsAutonomous Systems Institute

Lee, NH

Outline

1. Hardcore philosophical underpinnings

2. Evaluating peer review

3. Alternatives to peer review

4. Justifying decisions about quality

Hardcore Underpinnings

There is a dialectic between ontology and epistemology.

John Burke Grammar of Motives

Ontology, Epistemology, Dialectic

• Ontology – what is taken to exist (what needs a name)

• Epistemology – what do you know about it

• Dialectic – thesis, antithesis, synthesis

Dialectic betweenOntology and Epistemology

• What counts as data when you form your beliefs

• The relation between evidence and justice.

• Are your beliefs “sensible” and “reasonable”

• The relation between metrics and evaluation

Outline

1. Hardcore philosophical underpinnings

2. Evaluating peer review

3. Alternatives to peer review

4. Justifying decisions about quality

Evaluating Peer Review

• Selecting reviewers; providing incentives

• Informed objectivity; conflict of interest

• Review criteria; comparing across panels

Ronald N. KostoffUS Office of Naval Research

Selecting Reviewers: Peer vs Non-Peer

Balancing interests:

• Economic- government, academia, industry, society

• Intellectual- engineers, scientists, policy agencies

• Practical- curiosity, problem based, market driven

Informed objectivity

• “Good old boy” consensus

• Natural bias often subtle, needs diversity

• With paradigm shifts, experience becomes a liability

US National Science FoundationReview Criteria

• Intellectual merit – creative and original concepts

• Broader impacts

– benefits to society

Outline

1. Hardcore philosophical underpinnings

2. Evaluating peer review

3. Alternatives to peer review

4. Justifying decisions about quality

The Age of Unreason

“Changes are not what they used to be.”from book “The Age of Unreason”

Charles Handy, 1989

a. Change by discontinuous leapsb. Learning from the past dangerousc. Evolution yes, but allow for revolution

NASA’s Earth Science Information Partners

Courtesy Don Collins of the DAAC Alliance

Federalism

• Central coordination, local autonomy Tiered governance (US Federal vs States) Yield power to center (only reluctantly)

• Heterogeneous, diverse communities Data centers, academics, government, and industry

• Interdependence & minority interests Match and balance different values Take into account intensity of interest The whole is greater than the parts.

Outline

1. Hardcore philosophical underpinnings

2. Evaluating peer review

3. Alternatives to peer review

4. Justifying decisions about quality

Justifying Decisions about Quality

• Concept spaces and mapping

• Fact-value continuum?

• Fate of Knowledge (social construction)

Concept spaces and mapping

Fact-value continuum?

• Can we distinguish fact claims from value judgments?

• Are there really objective as opposed to subjective distinctions?

• Amartya Sen introduces value judgments in economics

Fate of knowledge

• Knowledge is social

• Cognitive processes are social (reasonable)

• Actions based on knowledge are justified through social processes.

Vision

• Concept-based science communication

• Personalized Learning Environment

• Economic Environment supports learning

Promising developments

• Semantic web not words; rather “meaningful” data, concepts, and ideas.

• Science draws meaning from data; and has changed the way it justifies this.

• The semantic web offers to improve or supplant “peer” review (and education).

• The semantic web provides a “marketplace” for learning.

Profiles for discussion

• People

• Content

• Technology

• Values and priorities

• Economic Environment

Evolution to a Semantic Web

• HTML – Generic display of hypertext

• XML – Generic display of data

• RDF – Triples (subject verb object)

• OWL – Taxonomy, inference rules, and proofs

Changes in Science

• 1930 –Reason and Sense (logical positivism)

• 1960 – Beyond Reason (linguistic turn)

• 1990 – Social Construction (peer review)

• 2020 – Back to Reason and Sense

“Peers” and “Educators”

• Ontology – Locally coherent naming structures

• Realism – “Reifying” the structures

• Under standing – Logical foundations

• Knowledge – Justifying decisions (personal beliefs)

Business Plan and “Marketplace”

• Sales agents – Know-bots negotiate deals

• Value chains – Multiple entities in assembly line

• Public choice – Market forces and “fair” voting rules

• Learning Economy – From food chain to ecosystem

Back to the Vision

• Concept-based science communication

• Personalized Learning Environment

• Economic Environment supports learning

Problems in Public Funding

National Archives -

GPRA -

Curriculum -

2% GNP -

What data is valuable?

Is science making progress?

What is “educated”?

How much to spend?

The Age of Unreason

“Changes are not what they used to be.”from book “The Age of Unreason”

Charles Handy, 1989

a. Change by discontinuous leapsb. Learning from the past dangerousc. Evolution yes, but allow for revolution

Category I Earth Science Information Partners

Courtesy Don Collins of the DAAC Alliance

MPEG-7: Metadata for Content Description

DataSignal

Features Model Semanticsstructure

Objects

Events

Actions

People

Labels

Relationship

Clusters

Classes

Collections

Probabilities

Confidences

Color

Texture

Shape

Motion

Camera motion

Regions

Segments

Mosaics

Relationship(Spatio-

temporal)

Images

Video

Audio

Multimedia

Formats

Layout

SemanticModelsFeaturesStructureData

Courtesy John Smith, IBM

Data, Community, Public Use

Farmers USDA

$$$ on pest control

Yes,treat your crops today

Johns Hopkins

$$ onfire antsstudies

NCDC

GHRC

USGS

$ onreal timedata

IBM

UsersApplication solutionprovider

Science dataprovider

Backbone datacenter

Technologyprovider

Real-timeriskmap

Historicaldata

Software

Real-timedata

Real-timedata

Courtesy Yuan-Chi Chang, IBM

Other Cross Discipline ESIP’s

Science

• Oceanography (2)

• Terrestrial Studies (4)

• Climate (3)

• Technology (3)

Public Use

• Education (3)

• Regional Policy (4)

• Public Health

• Media

• Legal

Community I

• “A united system of knowledge is the surest means of identifying the still unexplored domains of reality. It provides a clear map of what is known, and it frames the most productive questions for future inquiry.”

E. O. WilsonConsilience: The Unity of Knowledge

Community II

• “It is the disorder of the scientific community—the laminated, finite, partially independent strata supporting one another; it is the disunification of science—the intercalation of different patterns of argument—that is responsible for its strength and coherence..”

Peter GalisonImage and Logic, 1997

Federalism

• Central coordination, local autonomy Tiered governance (US Federal vs States) Yield power to center (only reluctantly)

• Heterogeneous, diverse communities Data centers, academics, government, and industry

• Interdependence & minority interests Match and balance different values Take into account intensity of interest The whole is greater than the parts.

Federations Preserve Heterogeneity

• Economic- government, academia, industry

• Intellectual- academic disciplines, policy agencies

• Practical- problem based, market driven

Economic Model

• Public funding caps out at less than 3% of the GNP

• Dynamic “pricing” is needed to demonstrate value to end-users

• “Market forces” or other methods of public choice provide robust mechanism to rank options

Trading Zones

• Economic- government, academia, industry

• Intellectual- engineers, scientists, policy agencies

• Practical- curiosity, problem based, market driven

Peer Review

Nurturing cooperation vs competition

• Economic- government, academia, industry

• Intellectual- engineers, scientists, policy agencies

• Practical- curiosity, problem based, market driven

Back to the Vision

• Concept-based science communication

• Personalized Learning Environment

• Economic Environment supports learning

Business Plan and “Marketplace”

• Sales agents – Know-bots negotiate deals

• Value chains – Multiple entities in assembly line

• Public choice – Market forces and “fair” voting rules

• Learning Economy – From food chain to ecosystem