nova evaluation report 1996-2002 presented by: dr. dennis sunal

NOVA Evaluation Report 1996-2002

Presented by:Dr. Dennis Sunal

Higher Education faces significant changes.

Data has been collected as part of a formative evaluation effort in the continuous enhancement of NOVA structure and functions to met its objectives.

Ten questions and data follow.

1. What are the characteristics of the institutions, faculty, and students who matriculate in NOVA courses?

- Principles from the National Science Education Standards

- Principles from the National Council of Teacher of Mathematics

- NOVA Professional Development Workshop Components

National Science Education Standards-Five principles

1. Change throughout the entire science education system.

2. What students learn is influenced by how they are taught.

3. The actions of teachers are influenced by their perceptions of science.

National Science Education Standards-Five principles, cont.

4. Student understanding is constructed through individual and social processes.

5. Teaches actions are influenced by the relationships with their students.

National Council of Teacher of Mathematics-Six principles

1. Math education requires equity. 2. The curriculum must be coherent, and

well articulated across the grades. 3. Effective math requires the teacher to

understand students prior knowledge, and continue to challenge and support the students to learn.

National Council of Teacher of Mathematics-Six principles, cont.

4. Students must learn math by actively building new knowledge from experiences and prior knowledge.

5. Assessment needs to support the learning of math.

6. Technology influences math that is taught, and enhances students’ learning.

NOVA Development Workshop Components-Twelve skills

1. NASA strategic enterprises2. Inquiry-based learning3. Innovative instructional strategies4. Assessment strategies that

facilitate learning5. Teaching science for all Americans6. New curriculum goals

NOVA Development Workshop Components-Twelve skills, cont.

7. Interdisciplinary approaches8. Using technology to facilitate learning9. Overcoming barriers10.Action research for college faculty11. Writing proposals for NOVA12. Connections to NSES, NCTM, ISTE

standards.

2. To what extent is the NOVA Model being disseminated and implemented by other faculty at NOVA institutions or at other institutions?

100 publications in journals, manuscripts, and presentations at national and regional conferences.

Regional and local workshops, consultancies, and other professional activities.

3. How congruent is the NOVA program with recommendations found in major reports on the preparation of teachers?

The NOVA Model for Professional development includes eight major elements aimed at changing how science and math are taught in K-12 classrooms by influencing “how” content is taught at the college level.

Eight major elements

1. Continuous professional development.

2. Sustained collaboration among interdisciplinary teams of faculty and administrators.

3. Courses based on national standards for science, math, and technology.

4. Courses utilize content based on NASA Strategic Enterprises.

Eight major elements, cont.

5. Program improvement at NOVA institutions through ongoing research.

6. Instructional strategies that are inquiry based and center on student interaction.

7. Extensive use of technology. 8. Collaborations among institutions in

the NOVA network.

4. What are the participant reactions to the NOVA professional development model? (Two part question)

Part A: Participant Reactions to the NOVA program

PositiveUseful workshopsNOVA course online modules has

been considered time well spent in professional development.

4. What are the participant reactions to the NOVA professional development model? (Two part question)

Part B: The change process for NOVA faculty

Commitment to the project. Belief in the model. The use of NASA resources. Monetary support for innovations. Introduce innovative teaching and/or

interdisciplinary courses to faculty.

Part B, cont.

Inspiration that came from changes in state education requirements and/or content standards.

Team of teachers attend a Phase I NOVA workshop.

Staff development.

5. How has the implementation of the NOVA Model impacted the collaborative work and organizational climate of science, mathematics, engineering, and education faculties? (Two part question)

Part A: Collaborative work changes When a team member leaves the

university. Teams no longer exist at the

institution. Faculty turnover and retirement. New faculty has not been integrated

into the team.

Part A, cont.

Collaboration continues with institutionalization of courses.

Team approach is vital to course development.

5. How has the implementation of the NOVA Model impacted the collaborative work and organizational climate of science, mathematics, engineering, and education faculties? (Two part question)

Part B: Organizational climate changesThe NOVA course has become the

“favorite lab.” The collegiality and impact on faculty

and students. Collaboration between the College of

Education and the science disciplines in the College of Arts and Sciences.

Part B, cont.

Time barriers. Institutional support barriers. Funding was a barrier. Conflicts

between or within department conflicts and the lack of mutual respect.

Students who had weak backgrounds, lacked motivation, or were fearful of math and science.

6. What new and modified courses are being offered as a result of NOVA?

135 courses were developed and offered at 84 NOVA institutions in the topic areas of :

-Biology-Life science-Life in space-Environmental science

7. To what extent does the NOVA course content integrate data and information that are unique to NASA, its fundamental questions, and its missions?

The NOVA courses integrate at least one of the five NASA Strategic Enterprises as a fundamental course component: ATE, BPRE, ESE, HEDS, and SSE.

8. What aspects of the NOVA professional development model are effective in creating and sustaining intended faculty knowledge and skills for reform action to take place? (Three part question)

Part A: What are the faculty perceptions of effective elements in the NOVA professional development process?

Cognitive apprenticeshipFacilitators of learningHigh self-efficacyKnowledge of pedagogy


Part B: Does action research conducted as a regular part of faculty teaching facilitate development of effective courses in science and mathematics?

Multiple techniques concerning assessment: journals, on-line tasks, rubrics, observation, performance tasks, interviews, and discussion groups, etc.

Part B, cont.

Faculty members may need additional support to engage in action research.

Investigating effective ways to teach science for improved student learning.

The need for instructors to create a “friendly learning environment” for students with poor attitudes.


Part C: What factors relate to sustaining change brought about through the NOVA professional development process?

Continued interaction between departments or disciplines.

Teams should have ‘people with similar ideas.’

9. What aspects of professional development affect course disciplinary content, classroom pedagogy, and specific disciplinary pedagogy related to student learning outcomes? (Three part question)

Part A: Course disciplinary contentNOVA course had significantly impacted

student attitudes to science as inquiry and to the study of climate.

Students taking the NOVA course had significantly higher teaching efficacy.

Students in the NOVA course also displayed more positive attitudes to geology.

Part A, cont.

Students in the NOVA course demonstrated significant gains in content knowledge.

Students valued the hands-on, collaborative approach to learning science.


Part B: Classroom pedagogyMore interactive methods to enhance

student learning. Some classes were team-taught by

two or more faculty. Good collaboration helped shift

faculty from teacher-centered teaching to student-centered approaches.

Part B, cont.

The team approach provided the necessary expertise, collaborative preparation, and sharing of the workload.

98% of the new courses were using technology of some sort in course delivery.


Part C: Specific disciplinary pedagogyHands-onLess or little lectureLots of group workStudent inputA diversity of teaching stylesThe content was “experienced–not told”

Part C, cont.

NOVA instructor was motivational. Students content knowledge was

improved. Technology enhanced learning. Course was clearly connected to

understanding the content.

10. Does scientific and mathematics literacy among students increase as a result of the faculty change through NOVA? (Five part question)

Part A: What are students’ perceptions of effective learning in science, mathematics or engineering classrooms resulting from the NOVA course?

Hands-on approachThe goal of teaching was student

understanding, not just covering content.

Part A, cont.

Awareness of students’ prior knowledge.

Having the ability to explain things on the students level.

Teaching in diverse ways. The teachers had the ability to

“make science come to life.”


Part B: What changes in attitude toward science and mathematics are found in the students who experienced the NOVA course?

Becoming comfortable with various technology aids.

Positive student attitudes. An increase of minority students who

participated in the NOVA classes.


Part C: What changes in teaching efficacy and learning outcomes resulted from students taking the NOVA course?

An increase in students’ content knowledge.

An increase in scientific processing skills. A greater understanding of the

mathematical concepts. Increased self-efficacy in science.

Part C, cont.

An increased ability and willingness to teach science and mathematics.

An increase in the students’ ability to use technology.

Enhanced critical thinking skills. A high level of interest in the

activities and were able to relate the content to real life.


Part D: What changes in the nature of science and nature of mathematics, related to the literacy goals in the national standards, are found in students taking the innovative NOVA courses?

Undergraduate students in NOVA reform-oriented classes demonstrated a significantly higher end of course growth and level of understanding of the nature of science when compared to students in other science, “non-reformed,” courses.

Part D, cont.

An increased interest of females in science and mathematics literacy.

The learning climate was significantly more positive in the NOVA courses as compared to traditional NOVA courses.


Part E: What are the indicators of change in students in NOVA course related to knowledge and performance of standards-driven reforms as a K-12 teacher of science or mathematics?

Improvements in the achievement of science content assessments.

Students had a high level of interest in the activities and were able to relate the content to real life.

Part E, cont.

Students’ self efficacy has increased.

6. The Effect of the NOVA Model on University Courses

A total of 135 courses were developed and offered at 84 NOVA institutions. 24% - biology, life science, life in space, or environmental

science 21% - earth and space science 20% - combined categories of mathematics, mathematics and

technology, mathematics and science, or statistics 16% - chemistry alone or chemistry, physics, or physical

science 10% - nature of science, inquiry science, or integrated

mathematics and science 5% - mathematics or science methods for preservice teachers 4% - engineering based for preservice teachers.


10060%

5001%

400

8%30015%

200

16%

NOVA Phase I Course Levels by Percent of Total


Course Level for Modified and Created Phase I Courses

0

20

40

60

80

100

L-100 L-200 L-300 L-400 L-500

No. Courses 81 21 20 11 2

7. The Impact of the NASA Mission, Data and Information, and Fundamental

Questions on the NOVA Courses

NASA Strategic Enterprise Representation

0

10

20

30

40

50

60

70

80

ATE BPRE ESE HEDS SSE

Institution 18 3 74 39 51



Percent of NOVA Courses Making Connections with each NASA

Enterprise

10%

1%

40%21%

28%

ATE

BPRE

ESE

HEDSE

SSE



During workshop sessions, the research program of each NASA Strategic Enterprise is described and participants are provided: NASA Strategic Plan NASA Implementation Plan for Education, 1999-2003 NSES & NCTM standards NRC’s Project 2061 & Science for All Americans NSTA’s College Pathways to the Science Education

Standards ISTE’s standards



Many institutions have 3 or more connections to NASA Enterprises

Number of NASA Enterprises implemented remains similar across institutions by Carnegie Classification.



An assessment instrument identifying NASA resources indicated: 78% were able to readily identify NASA

Enterprises relevant to their proposed course 70% had course materials that clearly

indicated the NASA Enterprises relevant to the courses being taught

91% stated they had used NASA resources to develop college level activities



Examples of use of NASA resources: University of Tennessee at Chattanooga offers

2 courses using NASA materials through Phase III NASA Field Center projects

Brescia University used spreadsheets involving NOVA data as part of course activities

University of the Incarnate Word integrated NASA research data and NASA Internet sources in laboratory investigations

8. Aspects of NOVA Professional Development Model that are Effective in

Creating and Sustaining Change

Sunal, Hodges, Sunal, Whitaker, Freeman, & Edwards (2001) examined faculty conceptions about effective teaching and learning at 30 universities.

Information was gathered on faculty attitude, knowledge, teaching efficacy, course planning, and classroom interactions.

Results showed that faculty who described themselves as facilitators of learning and those with higher self-efficacy were more likely to follow through on implementing change.

Successful implementation of change was also dependent upon knowledge of pedagogy and course design.

a. What are the faculty perceptions of effective elements in the NOVA professional development

process?



Nine important criteria emerged for successful implementation of change. For example:

interaction between science/mathematics and education faculty

administrative and collegial support a sense of common purpose and similar goals among faculty interpersonal skills and trust between faculty developmental and incremental change importance of action research component network of faculty as a support mechanism

a. What are the faculty perceptions of effective elements in the NOVA professional development

process?



Fifty-two action research plans were analyzed: Eight were in the novice-apprentice range Thirty-two in the apprentice-proficient range Thirteen in the proficien-distinguished range

Most were weak on the data analysis component Some did not devise actual research questions or

hypotheses Most were a mixture of qualitative and quantitative; most

common was pre-posttest design

b. Does action research conducted as a regular part of faculty teaching facilitate development of

effective courses in science and mathematics?



Emphasis of action research resulted in the use of a range of techniques for student assessment: • journals • on-line tasks • performance tasks • rubrics • observations • interviews • concept maps • discussion groups • discourse analysis • video and audio tape • self-evaluation • peer review





Most institutions saw students as the only source of data or unit of analysis

Research focused on student content knowledge, student skills, student attitudes, etc.

Cleveland State University used an approach where instructors focused on their own teaching: Two instructors constructed portfolios that included

personal reflections, philosophies, methods, student surveys and suggestions for improvement





Cyclical nature of change is indicated in action research



Planning

Acting

Reflecting

Observing

Followed by

Does it answerthe problem?

Leads to

Results in



Sixteen action research papers presented at the NOVA Leadership Development Conferences, 2001 & 2002: 13 described improvement in science or math content

knowledge, attitudes to the subject, process skills, and teaching efficacy

Others dealt with instructor issues such as classroom environment and teaching strategies

Studies of student outcomes showed improved student outcomes as a result of new courses





2002 LDC surveys showed 91% agreed that NOVA workshop provided opportunities to learn about NASA resources

Site Evaluation Visit interviews showed that continued interaction between disciplines was a valuable form of professional development

There was a relationship between a shift to a more student-centered approach and the effectiveness of NOVA teams

c. What factors relate to sustaining change brought about through the NOVA professional development

process?

9. Impact of the NOVA Model on Classrooms and Students

Most common title was interdisciplinary course in science Second most typical was a course in a single discipline Third - “NOVA Project” Fourth - integration of science, mathematics, and

technology Fifth - mathematics course for preservice teachers Sixth - connection between pedagogy and science or

mathematics Others involved nature of science, inquiry-based science,

etc.

a. Course Disciplinary Content


Exemplary action research projects

San Francisco State University Kansas State University Alaska Pacific University Brescia University



San Francisco State UniversityNOVA Geology Course

Higher scores in attitudes toward science as inquiry and nature and study of climate

Better connections between concepts



Kansas State UniversityNOVA Geology Course

Students had significantly higher teaching efficacy More positive attitudes toward geology



Alaska Pacific University

Significant gains in content knowledge Gains did not depend on gender, class standing, or major Students with lower initial scores made more gains



Brescia University

Significant gains in content knowledge No significant difference in students’ science literacy

although NOVA groups scored higher Students valued hands-on, collaborative approach Revised methods improved course in second year



68% were using inquiry-based lessons Majority of faculty have shifted to more student-centered

approaches and others are in transtition Team approach was seen as essential 98% used technology of some sort 80% of instructors had high full-scale scores on the ESTEEM

Science Classroom Observation rubric Content-specific pedagogy subscale was lowest indicating

more NOVA workshop time should be spent on pedagogy

b. Classroom Pedagogy


100% indicated national standards were referred to in development of course

Students reported hands-on methods with little or less lecture, lots of group work, motivational instructors, improved content knowledge, and enhancement of learning by technology

Three NOVA institutions experienced poor implementation of the NOVA model Instructors were not original NOVA team members

c. Specific Disciplinary Pedagogy


Overall, typical NOVA courses emphasized:construction of knowledge through hands-on, inquiry-

based activitiesfocusing on solving relevant problemssupport from technologyenhancement from cross-college faculty

collaboration



Doctoral dissertation studies showed:Modeling, active engagement, project-based

activities, and cooperative learning positively affected science teaching efficacy (Staples, 2002)

Higher achievement in elementary students of teachers experienced in NOVA (Staples, 2002)

Cross-college faculty collaboration is effective but time consuming (Hodges, 1999)


10. Impact of the NOVA Model on Science and Mathematics Literacy of University

Students

Integration of lab & lecture to promote a hands-on approach

Goal should be understanding, not just content Awareness of prior knowledge Ability to explain on students’ level Use diverse ways of teaching Make science “come to life” NOVA model “improved content knowledge, attitude and

confidence”

a. What are students’ perceptions of effective learning in science, mathematics, or engineering

classrooms resulting from the NOVA course?


Students

Increased content knowledge Appreciation and enjoyment in becoming comfortable with

technology aids High rates of employment and admission to graduate

school Positive impact on minority students Overall improvement in attitude toward science and

mathematics More confidence in future ability to teach science

b. What changes in attitude toward science and mathematics are found in the students who

experienced the NOVA course?


Students

Increased self-efficacy in science Greater understanding of mathematical concepts Increased ability and willingness to teach science and

mathematics Improvement in attitudes toward the language of mathematics Decrease in misconceptions Long-term retention of science content Improvements in achievement on science content assessments A variety of ways to teach mathematical content were learned

c. What changes in teaching efficacy and learning outcomes resulted from students taking the NOVA

course?


Students

Overall increase in science and mathematics literacy Increase seen in minorities and females Increased understanding of the nature of science More positive learning climate Positive impact on attitudes toward mathematics Long-term effects such as incorporation of relevant nature

of science topics in secondary content

d. What changes in the nature of science and nature of mathematics, related to the literacy goals in the

national standards, are found in students taking the innovative NOVA courses?


Students

Overall achievement improvement in all areas of science and mathematics understanding related to national standards

Improvement in achievement on science content assessments

More comfortable attitudes toward the idea of teaching science

Improved Personal Science Teaching Self-Efficacy Comfort with constructivist learning environments

e. What are the indicators of change in students in NOVA courses related to knowledge and

performance of standards-driven reforms as a K-12 teacher of science and mathematics?

nova evaluation report 1996-2002 presented by: dr. dennis sunal

Documents

nova institutions

nova courses

nova program

nova model

national standards

national council of

learningteaching science

perceptions of science