AScILSAssessing Science Inquiry and

Leadership Skills

Research Questions:• How do activities implemented by support programs (especially research and mentoring) influence: - skills in science inquiry and scientific team leadership and membership

- beliefs in efficacy and collective efficacy regarding these skills

- stage-appropriate education and career outcomes?

• Are these influences similar for minority and non-minority students?

“Inquiry” in the context of AScILS

Lisa HunterAssociate Director, Education & Human Resources, Center for Adaptive Optics

AScILS Research Team Member

Inquiry Strand

Advisor/Mentor Focus Group, 9/13 & 9/15/05

What do we mean by “inquiry”

The processes or methods that scientists use to gain an understanding of the world, and to create a body of knowledge (“research”)

Which “Science Inquiry Skills?”

CfAOSurvey

Intern Advisor/Mentor Meeting (9/13 & 9/15)

Education & Assessment Lit.

AScILS AlumniSurvey

AScILS qualitativeStudies (interviews)

Team Members’Prior Experience

Current AScILS Inquiry Strand Focus: Which science inquiry skill(s) or knowledge should be assessed?

Elements of scientific inquiry

• Skills or processes (what scientists do)

• Knowledge about scientific inquiry (what scientists know)

Scientific Inquiry Processes

Chinn, et. al.

“Cognitive processes”

America’s Lab Report

“Scientists’ activities”

Generating research questions

Posing a research question

Designing studies Formulating hypotheses

Making observations Designing investigations

Explaining results Make observations, gather & analyze data

Developing theories Building and revising scientific models or theories

Studying research reports Evaluating, testing, or verifying models

1. Chinn, Clark, et. al. (2002). Epistemilogically Authentic Inquiry in Schools. Science Education, 86:2.

2. National Research Council (2005). America’s Lab Report: Investigations in High School Science. Washington, DC: The National Academies Press.

Knowledge about scientific inquiry or the nature of science

Lederman, et. al

“Nature of science aspects”

Tentativeness (of scientific knowledge)

Empirical basis

Subjectivity (influenced and driven by presently accepted theories)

Creativity

Observations and inference

Social and cultural influences

Nature of theories (how and why they change)

Lederman, N.G. et. al. (2002). Journal of Research in Science Teaching, 39(6), 497-521.

Why we are interested in “inquiry”

• Many (10 of 14) of the AScILS study programs include research experiences, and is one of the 3 criteria for inclusion as a study program

• Students engage in authentic researchexpect gains in inquiry (or research) skills and understandings about inquiry

• NOTE: Student researchers clearly gain specific technical skills, but this is outside the AScILS focus

How can we assess interns’ gains in inquiry skills and/or

understandings about inquiry?

If we could assess inquiry skills, there are many interesting questions

• Do students gain inquiry skills and understandings about inquiry more in a research experience than in classes?

• Do gains depend on the type of research experience?

• Is there a connection between gains in inquiry skills and students’ confidence (or self-efficacy)? Students’ identity as a scientist? Persistence in science?

What is known?

• Knowledge base on assessing inquiry skills gained from authentic research is very limited:– Papers reporting self-reported gains from interviews and

surveys• “Thinking and working like a scientist” is a major gain/benefit

from research experiences*

– Papers reporting on how research experiences impacted students understandings about the nature of science

– Significant amount of work on assessing inquiry skills at K-12 classroom level

*Seymour, Elaine, et. al. (2004). Establishing the Benefits of Resarch Experiences for Undergraduates: First Findings from a Three-Year Study. Science Education, 88: 493-534.

AScILS Inquiry Strand

• Determine the most fruitful inquiry skill to assess

• Design a performance assessment (or simulation) to assess those skills

Which “Science Inquiry Skills?”

CfAOSurvey

Intern Advisor/Mentor Meeting (9/13 & 9/15)

Education & Assessment Lit.

AScILS AlumniSurvey

AScILS qualitativeStudies (interviews)

Team Members’Prior Experience

Current AScILS Inquiry Strand Focus: Which science inquiry skill(s) or knowledge should be assessed?

Scientific Inquiry Processes

Chinn, et. al.

“Cognitive processes”

America’s Lab Report

“Scientists’ activities”

Generating research questions

Posing a research question

Designing studies Formulating hypotheses

Making observations Designing investigations

Explaining results Make observations, gather & analyze data

Developing theories Building and revising scientific models or theories

Studying research reports Evaluating, testing, or verifying models

Scientific Inquiry Processes

Chinn, et. al.

“Cognitive processes”

America’s Lab Report

“Scientists’ activities”

Engineering processes

Generating research questions

Posing a research question

Identifying a problem or design need

Designing studies Formulating hypotheses Designing solutions within constraints

Making observations Designing investigations Implementing

Explaining results Make observations, gather & analyze data

Testing

Developing theories Building and revising scientific models or theories

Explaining solution, including limitations and tradeoffs

Studying research reports Evaluating, testing, or verifying models

Studying research reports

As a result of doing research with your group, which skill(s) are most important

for your students to develop?Chinn, et. al.

“Cognitive processes”

America’s Lab Report

“Scientists’ activities”

Engineering processes

Generating research questions

Posing a research question

Identifying a problem or design need

Designing studies Formulating hypotheses Designing solutions within constraints

Making observations Designing investigations Implementing

Explaining results Make observations, gather & analyze data

Testing

Developing theories Building and revising scientific models or theories

Explaining solution, including limitations and tradeoffs

Studying research reports Evaluating, testing, or verifying models

Studying research reports

ASSESSMENT BASICS

AScILS Style

Jerome ShawAssistant Professor

Education Department

ASSESSMENT

as a PROCESS of

REASONING from EVIDENCE

Data

Evide nc e

Infe re nc e s

Source:

National Research Council. (2001). Knowing what students know: The science and design of educational assessment. Committee on the Foundations of Assessment. Pelligrino, J., Chudowsky, N., and Glaser, R., editors. Board on Testing and Assessment, Center for Education. Division of Behavioral and Social Sciences and Education. Washington, DC: National Academy Press.

ASSESSMENT MANTRA

• STANDARDS– what?

• TASKS– how?

• RUBRICS– how well?

Scientific Inquiry Processes

“Cognitive Processes”

Chinn et al.

“Scientists’ activities”

America’s Lab Report

Engineering processes

Generating research questions Posing a research question Identifying a problem or design need

Designing studies Formulating hypotheses Designing solutions within constraints

Making observations Designing investigations Implementing

Explaining results Make observations, gather & analyze data

Testing

Developing theories Building and revising scientific models or theories

Explaining solutions, including limitations and tradeoffs

Studying research reports Evaluating, testing, or verifying models

Studying research reports

Source:

Sandoval, W. A., & Reiser, B. J. (2004). Explanation-driven inquiry: Integrating conceptual and epistemic scaffolds for scientific inquiry. Science Education, 88, 345-372.

Explanation Rubric: Sample Criteria

Thoroughness and Clarity of Explanations

Use of Data

Ruling Out Alternative Explanations

Documenting the Limitations of Your Explanations

Scoring Guide: Using Evidence

Score

Response uses objective reason(s) based on relevant evidence to argue for or against a choice.

4

Response accomplishes level 3, AND goes beyond in some relevant way, e.g., questioning or justifying the source, validity, and/or quantity of evidence.

3

Provides major objective reasons AND supports each with relevant and accurate evidence.

2

Provides some objective reasons AND some supporting evidence, BUT at least one reason is missing and/or part of the evidence is incomplete.

1

Provides only subjective reasons (opinions) for choice; uses unsupported statements; OR uses inaccurate or irrelevant evidence from the activity.

Source:Science Education for Public Understanding Program (1995) as presented in… National Research Council. (2001). Classroom assessment and the National Science Education Standards. Committee on Classroom Assessment and the National Science Education Standards. J. Myron Atkin, Paul Black, and Janet Coffey (Eds.). Center for Education, Division of Behavioral and Social Sciences Education. Washington, DC: National Academy Press.

Which “Science Inquiry Skills?”

CfAOSurvey

Intern Advisor/Mentor Meeting (9/13 & 9/15)

Education & Assessment Lit.

AScILS AlumniSurvey

AScILS qualitativeStudies (interviews)

Team Members’Prior Experience

Current AScILS Inquiry Strand Focus: Which science inquiry skill(s) or knowledge should be assessed?