editors' notes

EDITORS’ NOTES

This volume of New Directions for Evaluation discusses evaluation of sci-ence, technology, engineering, and mathematics (STEM) programs, withspecial emphasis on evaluation of STEM educational initiatives. STEM eval-uation has always been important given the issues facing public schools andthe economic and social considerations of STEM fields. But because thesefields today face a variety of concerns, this discussion of STEM evaluationis particularly timely. Advances in evaluation may contribute to STEM fieldsby helping programs address the challenges they face. This volume presentsmultiple viewpoints and state-of-the-art examples and methodologicalapproaches in the hope that its chapters will contribute to understanding ofSTEM evaluation, STEM education, STEM educational evaluation, and eval-uation in general.

Issues and Challenges Facing STEM Education andEvaluation

Many view the fields of science, technology, engineering, and mathematicsas critical for the future of the nation and the world as a whole. To competein today’s advanced global economy, it is essential to develop a workforce thatis literate in science, technology, engineering, and mathematics (National Sci-ence Board, 1999). To achieve this goal, world-class educational opportuni-ties in mathematics and science at all levels must be a major goal. This is animportant goal not only for economic reasons but also to enhance individ-ual life opportunities for employment, active citizenship, and the pursuit ofhappiness. It is important for all students, including those who have not tra-ditionally had opportunities to participate in STEM fields, to have a chanceto learn the knowledge and skills they will need in a technologically orientedfuture. Issues of social justice and economic opportunity play an importantrole in this future. STEM-related fields continue to be a major source of jobgrowth, and as a result those who are knowledgeable in STEM fields willhave more economic opportunity. The future well-being of our countrydepends on how well we educate our children in mathematics and science(National Science Foundation, 2002).

Unfortunately, the United States faces many challenges related to STEMeducation. Helping students achieve in mathematics and science remains apersistent challenge. According to international comparisons of science and

NEW DIRECTIONS FOR EVALUATION, no. 109, Spring 2006 © Wiley Periodicals, Inc.Published online in Wiley InterScience (www.interscience.wiley.com) • DOI: 10.1002/ev.174 1

2 CRITICAL ISSUES IN STEM EVALUATION

mathematics achievement, the United States has suffered from mediocreachievement for decades (National Center for Education Statistics, 1996,2004). For example, in the last three international comparisons of mathe-matics and science achievement, U.S. achievement scores were at or nearthe international average. Of more concern is the fact that even the top-performing students in the United States score below the best students ofother countries (National Center for Education Statistics, 2004). The UnitedStates also suffers from an ongoing achievement gap between majority andminority students, despite numerous attempts at addressing the problem.As declared by the National Commission on Mathematics and ScienceTeaching for the 21st Century (2000), America’s students must raise theirperformance in mathematics and science if they are to succeed in today’sworld. Among the underlying explanations for the poor performance of U.S.students in mathematics and science are a number of problems: underpre-pared science and mathematics teachers, ineffective instructional practicesused by science and mathematics teachers, science and mathematics teach-ers who teach out of field, difficulty recruiting and retaining qualified sci-ence and mathematics teachers, too few students taking advancedcoursework, science and mathematics curricula that tend to lack depth ofcoverage on advanced topics, and too few schools offering challenging sci-ence and mathematics curricula to all students. All of these problems areultimately important to society, and the field of STEM education evaluationcan play an important role in addressing them.

It is also imperative to develop the pre-K–12 teacher workforce(National Research Council, 2001). Fundamental questions are often raisedabout recruiting and training new teachers as well as professional develop-ment of current teachers. Improving teacher preparation in postsecondaryinstitutions is central to sustaining and deepening quality education for allstudents (National Commission on Teaching and America’s Future, 1996)because there is a close relationship between student achievement andteacher knowledge and teaching skills (Hill, Rowan, and Ball, 2005). Overthe years, the National Science Foundation (NSF) has found that studentlearning is complex and depends on successful interaction of many factors,among them school leadership, resources and partnerships, policy and infra-structure, strategic decisions and intervention, sustainability, and outcomesand evaluation (Kim and others, 2001; Webb and Weiss, 2000). Given thismore integrated view of education, many STEM education evaluation find-ings should be of interest to people in a number of fields.

Expanded Views of Evaluation

This volume was also designed to advance notions of what it means to engagein STEM education evaluation. In recent years, there has been renewed inter-est in methodological issues, thanks to the federal government’s focus on

NEW DIRECTIONS FOR EVALUATION • DOI 10.1002/ev

3EDITORS’ NOTES

program evaluation and its attempts to promote experimental designs to bet-ter attribute causation to program activities. In this volume, we describe newapproaches to STEM education evaluation that take a broader, more con-temporary view of what it means to engage in evaluation.

STEM education evaluation has traditionally used many methods andphilosophical approaches to evaluate programs; however, there is now a polit-ical movement to focus on experimental approaches to evaluation. Althoughthis movement affects all fields of evaluation, it acutely affects STEM educa-tion evaluation. The U.S. Department of Education (2003) has placed prior-ity on “scientifically” based evaluation methods, advocating particularevaluation methods (especially experimental approaches) over others. Specif-ically, there is a belief that evaluations using more experimental designs arebetter for determining project effectiveness. This emphasis on “scientific” evi-dence corresponds with the national educational accountability movement,which began with developing standards for education such as those in thefields of science and mathematics (National Council of Teachers of Mathe-matics, 1989, 1991, 2000; National Research Council, 1996) and was fol-lowed by the passage of the No Child Left Behind Act (2001). In 2002, theOffice of Educational Research and Improvement was reconstituted as theInstitute of Education Sciences, reflecting the intent of the federal governmentto advance the field of education by making it more rigorous and supportingmore evidence-based education. All of these events point to a significantnational movement toward using evidence to evaluate programs, a movementwith important implications for the field of evaluation.

The priority on experimental design raises many questions about howbest to evaluate programs. In the last few years, there have been advancesin evaluating STEM education programs that could help the field as a wholeaddress evaluation issues. They are in mixed-methods evaluation, multisiteevaluation, cultural competency, issues of equity and justice, and evaluationcapacity building. Taken together, these perspectives go beyond the currenttraditional experimental view of scientific evaluation and expand the notionof what it means to evaluate STEM education programs. In this volume, wepresent innovative approaches to STEM education evaluation and advancenew notions of evaluation for science, technology, engineering, and math-ematics. We also include pertinent examples and perspectives from thefield’s leading experts that help readers see how they can apply these newapproaches to STEM evaluation in practice.

Advances in STEM Evaluation

This issue of New Directions for Evaluation includes contemporary perspec-tives and state-of-the-art methods in STEM education evaluation. Eachchapter describes an advance for the field along with unique theoreticalviews and practical methods for both STEM and non-STEM evaluators alike.


4 CRITICAL ISSUES IN STEM EVALUATION

The advances described here help to address the range of challenges STEMfields face, while also enhancing the notion of what it means to engage in“scientific” evaluation. The advances encourage the field to take a broadview of what it means to be scientific and follow the well-known procedureof having the research question drive the methodology, rather than equat-ing rigor with a particular methodology.

Included here is current work in program evaluation from some of thefield’s top STEM evaluators. In the first chapter, Conrad Katzenmeyer, for-mer NSF program officer in the Directorate for Research, Evaluation, andCommunication, and Frances Lawrenz bring a federal perspective to thechallenges facing STEM evaluation. Katzenmeyer served at the NSF for sev-eral decades and sets the context for the volume by describing NSF’sapproach to STEM education, including the work done to develop new ap-proaches to evaluation. In the second chapter, Frances Lawrenz and Douglas Huffman describe advances in STEM education evaluation meth-odology, guiding the reader through the methodological pluralism thatdominates the field. This chapter is based on experience evaluating numer-ous NSF-funded STEM programs. To expand the notions of traditionalSTEM evaluation, Donna M. Mertens and Rodney K. Hopson encourage thefield, in Chapter Three, to consider how cultural competency issues can beapplied in STEM educational evaluation. The fourth chapter, by Jennifer C.Greene, Lizanne DeStefano, Holli Burgon, and Jori Hall, deals with thecutting-edge issue of a values-engaged approach to evaluation with a focuson equity and justice. Douglas Huffman, Frances Lawrenz, Kelli Thomas,and Lesa Clarkson then describe advances in evaluation capacity buildingin STEM education. This fifth chapter includes a new model of evaluationcapacity building that is currently part of an NSF grant to develop collabo-rative evaluation communities in urban schools and to recruit and developnew STEM education evaluators. Arlen R. Gullickson and Carl E. Hanssendescribe advances in multisite evaluation in Chapter Six. Using a large-scale, multisite STEM evaluation as an example, they describe the relation-ship between evaluation use and results in multisite settings. The finalsynthesis chapter, by Lawrenz, Huffman, and Thomas, lays out a criticalreview of the material presented in each of the chapters and specific take-home messages for both STEM and non-STEM evaluators. The chapterincludes an overarching review of the main issues that emerged and theirrelevance. It also includes recommendations for future theoretical andempirical work in STEM education evaluation.

Overall, this volume of New Directions for Evaluation documentsadvances in the field of STEM evaluation that have important implicationsfor evaluation generally and for STEM evaluation specifically. We hope theadvances help not only to move the field to consider new methods andmethodologies for engaging in evaluation but also to reconsider ideas ofwhat it means to engage in scientific evaluation.


5EDITORS’ NOTES

References

Hill, H. C., Rowan, B., and Ball, D. L. “Effects of Teachers’ Mathematical Knowledge forTeaching on Student Achievement.” American Educational Research Journal, 2005,42(2), 371–406.

Kim, J., Crasco, L., Smith, R., Johnson, G., A., Karantonis, A., and Leavitt, D. AcademicExcellence for all Urban Students: Their Accomplishments in Science and Mathematics.Norwood, Mass.: Systemic Research, 2001.

National Center for Education Statistics. Pursuing Excellence: A Study of U.S. Eighth-Grade Mathematics and Science Teaching, Learning, Curriculum, and Achievement inInternational Context. (NCES 97–198). Washington, D.C.: U.S. Department of Edu-cation, 1996.

National Center for Education Statistics. Highlights from the Trends in International Math-ematics and Science Study (TIMSS) 2003. (NCES 2005–005). Washington, D.C.: U.S.Department of Education, 2004.

National Commission on Mathematics and Science Teaching for the 21st Century. BeforeIt’s Too Late: A Report to the Nation. Washington, D.C.: U.S. Department of Education,2000.

National Commission on Teaching and America’s Future. What Matters Most: Teachingfor America’s Future. Washington, D.C.: U.S. Department of Education, 1996.

National Council of Teachers of Mathematics. Curriculum and Evaluation Standards forSchool Mathematics. Reston, Va.: NCTM, 1989.

National Council of Teachers of Mathematics. Professional Standards for Teaching Math-ematics. Reston, Va.: NCTM, 1991.

National Council of Teachers of Mathematics. Principles and Standards for School Math-ematics. Reston, Va.: NCTM, 2000.

National Research Council. National Science Education Standards. Washington, D.C.:National Academy Press, 1996.

National Research Council. Educating Teachers of Science, Mathematics, and Technology:New Practices for the New Millennium. Washington, D.C.: Committee on Science andMathematics Teacher Preparation, National Academy Press, 2001.

National Science Board. Preparing Our Children: Mathematics and Science Education inthe National Interest. (NSB 99–31). Washington, D.C.: National Science Board, 1999.

National Science Foundation. Mathematics and Science Partnership (MSP). (Programsolicitation, NSF-02–061). Washington, D.C.: National Science Foundation, 2002.

No Child Left Behind Act. H.R. 1, 107th Congress, 1st Session, 2001.U.S. Department of Education. Scientifically Based Evaluation Methods. (Federal Registry,

vol. 68, no. 213). Washington, D.C.: U.S. Government Printing Office, 2003.Webb, N., and Weiss, I. Annual Report of the Study of the Impact of Statewide System Ini-

tiatives. Madison: National Institute for Science Education, University of Wisconsin-Madison, 2000.

Douglas HuffmanFrances LawrenzEditors

DOUGLAS HUFFMAN is associate professor of science education in the School ofEducation at the University of Kansas.

FRANCES LAWRENZ is Wallace Professor of Teaching and Learning in the Collegeof Education and Human Development at the University of Minnesota.


editors' notes

Documents