concrete approaches to quality assessment: moving beyond peer review august 6, 2003 howard burrows...
TRANSCRIPT
Concrete Approaches to Quality Assessment:
Moving Beyond Peer ReviewAugust 6, 2003
Howard BurrowsAutonomous Systems Institute
Lee, NH
Outline
1. Hardcore philosophical underpinnings
2. Evaluating peer review
3. Alternatives to peer review
4. Justifying decisions about quality
Hardcore Underpinnings
There is a dialectic between ontology and epistemology.
John Burke Grammar of Motives
Ontology, Epistemology, Dialectic
• Ontology – what is taken to exist (what needs a name)
• Epistemology – what do you know about it
• Dialectic – thesis, antithesis, synthesis
Dialectic betweenOntology and Epistemology
• What counts as data when you form your beliefs
• The relation between evidence and justice.
• Are your beliefs “sensible” and “reasonable”
• The relation between metrics and evaluation
Outline
1. Hardcore philosophical underpinnings
2. Evaluating peer review
3. Alternatives to peer review
4. Justifying decisions about quality
Evaluating Peer Review
• Selecting reviewers; providing incentives
• Informed objectivity; conflict of interest
• Review criteria; comparing across panels
Ronald N. KostoffUS Office of Naval Research
Selecting Reviewers: Peer vs Non-Peer
Balancing interests:
• Economic- government, academia, industry, society
• Intellectual- engineers, scientists, policy agencies
• Practical- curiosity, problem based, market driven
Informed objectivity
• “Good old boy” consensus
• Natural bias often subtle, needs diversity
• With paradigm shifts, experience becomes a liability
US National Science FoundationReview Criteria
• Intellectual merit – creative and original concepts
• Broader impacts
– benefits to society
Outline
1. Hardcore philosophical underpinnings
2. Evaluating peer review
3. Alternatives to peer review
4. Justifying decisions about quality
The Age of Unreason
“Changes are not what they used to be.”from book “The Age of Unreason”
Charles Handy, 1989
a. Change by discontinuous leapsb. Learning from the past dangerousc. Evolution yes, but allow for revolution
NASA’s Earth Science Information Partners
Courtesy Don Collins of the DAAC Alliance
Federalism
• Central coordination, local autonomy Tiered governance (US Federal vs States) Yield power to center (only reluctantly)
• Heterogeneous, diverse communities Data centers, academics, government, and industry
• Interdependence & minority interests Match and balance different values Take into account intensity of interest The whole is greater than the parts.
Outline
1. Hardcore philosophical underpinnings
2. Evaluating peer review
3. Alternatives to peer review
4. Justifying decisions about quality
Justifying Decisions about Quality
• Concept spaces and mapping
• Fact-value continuum?
• Fate of Knowledge (social construction)
Concept spaces and mapping
Fact-value continuum?
• Can we distinguish fact claims from value judgments?
• Are there really objective as opposed to subjective distinctions?
• Amartya Sen introduces value judgments in economics
Fate of knowledge
• Knowledge is social
• Cognitive processes are social (reasonable)
• Actions based on knowledge are justified through social processes.
Vision
• Concept-based science communication
• Personalized Learning Environment
• Economic Environment supports learning
Promising developments
• Semantic web not words; rather “meaningful” data, concepts, and ideas.
• Science draws meaning from data; and has changed the way it justifies this.
• The semantic web offers to improve or supplant “peer” review (and education).
• The semantic web provides a “marketplace” for learning.
Profiles for discussion
• People
• Content
• Technology
• Values and priorities
• Economic Environment
Evolution to a Semantic Web
• HTML – Generic display of hypertext
• XML – Generic display of data
• RDF – Triples (subject verb object)
• OWL – Taxonomy, inference rules, and proofs
Changes in Science
• 1930 –Reason and Sense (logical positivism)
• 1960 – Beyond Reason (linguistic turn)
• 1990 – Social Construction (peer review)
• 2020 – Back to Reason and Sense
“Peers” and “Educators”
• Ontology – Locally coherent naming structures
• Realism – “Reifying” the structures
• Under standing – Logical foundations
• Knowledge – Justifying decisions (personal beliefs)
Business Plan and “Marketplace”
• Sales agents – Know-bots negotiate deals
• Value chains – Multiple entities in assembly line
• Public choice – Market forces and “fair” voting rules
• Learning Economy – From food chain to ecosystem
Back to the Vision
• Concept-based science communication
• Personalized Learning Environment
• Economic Environment supports learning
Problems in Public Funding
National Archives -
GPRA -
Curriculum -
2% GNP -
What data is valuable?
Is science making progress?
What is “educated”?
How much to spend?
The Age of Unreason
“Changes are not what they used to be.”from book “The Age of Unreason”
Charles Handy, 1989
a. Change by discontinuous leapsb. Learning from the past dangerousc. Evolution yes, but allow for revolution
Category I Earth Science Information Partners
Courtesy Don Collins of the DAAC Alliance
MPEG-7: Metadata for Content Description
DataSignal
Features Model Semanticsstructure
Objects
Events
Actions
People
Labels
Relationship
Clusters
Classes
Collections
Probabilities
Confidences
Color
Texture
Shape
Motion
Camera motion
Regions
Segments
Mosaics
Relationship(Spatio-
temporal)
Images
Video
Audio
Multimedia
Formats
Layout
SemanticModelsFeaturesStructureData
Courtesy John Smith, IBM
Data, Community, Public Use
Farmers USDA
$$$ on pest control
Yes,treat your crops today
Johns Hopkins
$$ onfire antsstudies
NCDC
GHRC
USGS
$ onreal timedata
IBM
UsersApplication solutionprovider
Science dataprovider
Backbone datacenter
Technologyprovider
Real-timeriskmap
Historicaldata
Software
Real-timedata
Real-timedata
Courtesy Yuan-Chi Chang, IBM
Other Cross Discipline ESIP’s
Science
• Oceanography (2)
• Terrestrial Studies (4)
• Climate (3)
• Technology (3)
Public Use
• Education (3)
• Regional Policy (4)
• Public Health
• Media
• Legal
Community I
• “A united system of knowledge is the surest means of identifying the still unexplored domains of reality. It provides a clear map of what is known, and it frames the most productive questions for future inquiry.”
E. O. WilsonConsilience: The Unity of Knowledge
Community II
• “It is the disorder of the scientific community—the laminated, finite, partially independent strata supporting one another; it is the disunification of science—the intercalation of different patterns of argument—that is responsible for its strength and coherence..”
Peter GalisonImage and Logic, 1997
Federalism
• Central coordination, local autonomy Tiered governance (US Federal vs States) Yield power to center (only reluctantly)
• Heterogeneous, diverse communities Data centers, academics, government, and industry
• Interdependence & minority interests Match and balance different values Take into account intensity of interest The whole is greater than the parts.
Federations Preserve Heterogeneity
• Economic- government, academia, industry
• Intellectual- academic disciplines, policy agencies
• Practical- problem based, market driven
Economic Model
• Public funding caps out at less than 3% of the GNP
• Dynamic “pricing” is needed to demonstrate value to end-users
• “Market forces” or other methods of public choice provide robust mechanism to rank options
Trading Zones
• Economic- government, academia, industry
• Intellectual- engineers, scientists, policy agencies
• Practical- curiosity, problem based, market driven
Peer Review
Nurturing cooperation vs competition
• Economic- government, academia, industry
• Intellectual- engineers, scientists, policy agencies
• Practical- curiosity, problem based, market driven
Back to the Vision
• Concept-based science communication
• Personalized Learning Environment
• Economic Environment supports learning
Business Plan and “Marketplace”
• Sales agents – Know-bots negotiate deals
• Value chains – Multiple entities in assembly line
• Public choice – Market forces and “fair” voting rules
• Learning Economy – From food chain to ecosystem