evaluating nsf epscor initiatives: tracking and assessing the development of epscor-supported...
TRANSCRIPT
EVALUATING NSF EPSCOR INITIATIVES:TRACKING AND ASSESSING THE DEVELOPMENT OF EPSCOR-SUPPORTED RESEARCH CAPACITY
Julia Melkers, Associate ProfessorSchool of Public Policy, Georgia Institute of Technology
Eric Welch, Associate ProfessorDepartment of Public Administration, University of Illinois at Chicago
Prepared for presentation at the National Research CouncilWashington, DC
December 17, 2012
Evaluation frames are based on the understanding that:
Building research capacity is a complex and continuous process. Tracking magnitude of changes over time is critical to
capturing effects (interim effects). Given its mission and the structure of the
awards, EPSCoR has the potential to affect research capacity in range of ways.
Capacity is reflected in (and affected by): State politics and science objectives Institutions Infrastructure Human capital Collaborative relationships Resources Among others....
InstitutionalizatiInstitutionalizationon
Development of Development of early collaborative early collaborative outputsoutputs
Increased Increased production & other production & other impactsimpacts
Movement toward Movement toward sustainabilitysustainability
InstitutionalizatiInstitutionalizationon
Development of Development of early collaborative early collaborative outputsoutputs
Increased Increased productionproduction
Movement toward Movement toward sustainabilitysustainability
Dimensions of Research Capacity
Capacity as infrastructure Lab equipment Cyberinfrastructure Institutional processes and policy
Capacity as human capital Scientists Postdocs Graduate Students
Capacity as integration Integrating different expertise and perspectives on the
research problem (disciplinary, translational, etc) Capacity as productivity
Grant activity that extends and builds upon award work
Using Theories of Change in Evaluation:Example Project Logic Model
Sample Guiding Evaluation Questions: Collaboration
Collaboration: What are the barriers and facilitators to collaboration in [state] EPSCoR? within and between sites/groups? interdisciplinary and cross institutional? early career engagement? mid career role? Native and rural community partners? private and non profit sector integration? international partners? critical success factors?
Sample Guiding Evaluation Questions: Impacts
Research Impacts: What are the knowledge impacts of [EPSCoR] research and activities?
Stakeholder Impacts: How has [EPSCoR] developed interactions /tangible relationships with external partners/stakeholders over the course of the project?
Faculty and Staff Impacts: What research and management capacity is increasing as a result of [EPSCoR] work?
Student Impacts: What benefits and capacities do students accrue from [EPSCoR] ?
Institutional Impacts: How has the research capacity of [state] institutions changed via [EPSCoR] ?
Data Sources
Example: Some Observable Outcomes Science
Knowledge advancement and integration
Research productivity
Research impacts Increased know-how
Faculty/Researchers Production Advancement Enhanced or new
collaborative ties Student engagement Leadership Visibility
“Next generation” Student learning
and other impacts STEM interests and
advancement Institutional
capacity New programs,
processes Systems Support staff Culture
Measuring Research Activities & Outcomes
Collaborative Development
Intermediate outcomes that precede academic production
Emerging collaborative relationshipsIntegration of students in researchKnowledge transfer and ImpactsResearch-relevant behavioral changesNature of linkages and interactions
Operationalized through Network Analysis
Pre-publication interaction Faculty-faculty & faculty-student
collaboration Changed conceptual/ theoretical/
methodological approaches Changed research outlets and
communities Mechanisms/modes for boundary-
spanning linkages and outcomesMeasurable
through network analysis
Example: Student Interaction
All Collaborative Ties – Graduate Students Only Statistics – All Ties Number of nodes Number of ties Average degree centrality E-I Discipline
48 77 .42 -.299
Legend Square = Core institution Circle = Other institutions Green = environmental science Blue = computer science/engineering
Orange = social science Black lines = within discipline Red lines = between disciplines Node size = larger nodes reflect more semesters supported by EPSCoR
Collaboration Ties # ties Worked with on a grant proposal 14 Presented research to external stakeholders or external partners 20 Presented research to an academic audience 69 Presented research to a nonacademic audience 29 Co-authored a working paper 20 Co-authored an academic paper that has been published or is under review 5
Knowledge Impacts
Alaska-Based Orgs (Federal, State, Private, NGO)
International
Top Ten Subject Categories
(2007-12 EPSCoR Publications# Records
Subject Category
1 37 Environmental Sciences & Ecology
2 27 Ecology3 18 Geology4 17 Environmental
Sciences5 12 Environmental
Studies6 12 Zoology7 11 Physical
Geography8 10 Geosciences,
Multidisciplinary
9 8 Evolutionary Biology
10 7 Biochemistry & Molecular Biology
Other EPSCoR States
Other States
Rafols, Porter and Leydesdorff (2009)
Cognitive Sci.
Agri Sci
Biomed Sci
Chemistry
Physics
Engineering
Env Sci & Tech
Matls Sci
Infectious Diseases
Psychology
Social Studies
Clinical Med
Computer Sci.Business & MGT
Geosciences
Ecol Sci
Econ. Polit. & Geography
Health & Social Issues
Global Map of Science, 2007221 SCI-SSCI Subject Categories
Knowledge Impacts: Disciplinary Placement & Knowledge Sources
Reflections on EPSCoR Outcomes Research
Integration across disciplines – new questions, approaches Added value of collaborative engagement -- generation of new ideas and
new collaborative relationships Knowledge impacts -- National and global visibility and impacts
STEM workforce Early career affiliates – early grants, network reach, production, time to
tenure, mobility and competitiveness, mentoring and advice relationships Students – research networks, apprenticeship structures, mentoring
relationships, production, next career step Research – related support staff, sustained research relationships, new
grant awards Institutions
EPSCoR offices – communication, administrative structures, management Smaller institutions – research activity, developed capacity and
integration, statewide representation in decision making (eg. state S&T plan)
Curricular issues – new courses, content Social Impacts
engaged stakeholder organizations – economic development, social and economic objectives, support for education and research by stakeholders
Issues Relevant to Evaluation of EPSCoR EPSCoR projects and settings vary.
Disciplinary variance Partner relationships Under-represented populations Changes over time (historically)
Multiple and competing programmatic goals Lack of clarity/changing evaluation needs from EPSCoR Data consistency, across states and over time. Increased attention to strategic plans Unpacking role/impacts of senior leads vs other
faculty/researchers Institutional variation in data availability (esp. small
schools). Institutional effects not always evident Futures are unclear -- difficulty/lack of support to track
students and faculty post-award. Issue of relationships across awards and cumulative effects
on a state are not captured.