evaluating outcomes of federal grantee collaborations patricia mueller, chair aea conference 2014...
TRANSCRIPT
![Page 1: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/1.jpg)
Evaluating Outcomes of Federal Grantee Collaborations
Patricia Mueller, ChairAEA Conference
2014Denver, CO
The National Center on Educational Outcomes-National Assessment Center is supported through a cooperative agreement between the University of Minnesota, National Center on Educational Outcomes (NCEO) (#H326G110002) and the U.S. Department of Education, Office of Special Education and Rehabilitative Services (OSERS), Office of Special Education Programs. The materials do not necessarily reflect the position or policy of the U.S. Department of Education or Offices within
it. Project Officer: David Egnor.
![Page 2: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/2.jpg)
Our Session Goals
Federal Grantee Landscape
Grounding the Evaluation
Roles of Logic Model and Collaboration Map
Grantee Experiences
![Page 3: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/3.jpg)
Who’s in the Audience? Your goals?Evaluators working with Federal Programs?
EducationHealthScience
Evaluators Measuring Collaborative Efforts?
Others?
![Page 4: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/4.jpg)
Federal Grant LandscapeCollapsing of Programs
Grants to Cooperative Agreements
Requirement for “Collaboration”
Performance Measurement
Annual Reporting
![Page 5: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/5.jpg)
National Assessment Center National Center on Educational
Outcomes
Mission:
To support the development of inclusive assessment in a rapidly changing state (and
consortium) assessment system environment in order to promote improved educational results for
students with disabilities.
![Page 6: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/6.jpg)
National Assessment Center Activities
KnowledgeDevelopment
Leadership and
Coordination
Technical Assistance
andDisseminatio
n
![Page 7: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/7.jpg)
Collaboration: Research and Evaluation
Theoretical Model, Evaluation Questions and Design (Judy)
Logic Model and Collaboration Mapping (David)
What’s it Mean to us? Grantee Perspective (Laurene)
Discussant, Q & A (Vitaliy)
![Page 8: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/8.jpg)
Grounding the Evaluation: Literature Review, Design
Development & ComponentsAEA Conference 2014
Judy Lee
![Page 9: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/9.jpg)
Goals
1)So, you want to measure Collaboration. Where to begin?
2)How do others measure Collaboration?
3)With all this information, how does it fit your agency’s needs and goals?
![Page 10: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/10.jpg)
Where to begin: Literature Review
• Journal articles
• Program and evaluation-related web
sites
• PowerPoint presentations
GOAL: Gain broad understanding
![Page 11: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/11.jpg)
Where to begin: Program Documents
• Evaluation questions, performance
measures
• Team meetings, planning discussions
• Logic models
GOAL: Gain program-specific
understanding
![Page 12: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/12.jpg)
Where to begin: Ask Questions
What do you want to know about collaboration?
When do you want to ask questions?
Who do you want to ask?
How do you want to ask these questions?
Why does each question matter? How will findings be applied,
utilized?
GOAL: Narrow focus for evaluation
![Page 13: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/13.jpg)
Review Existing Tools and Frameworks
Collaboration
Levels ofCollaboratio
n Outcomes
DimensionsOf
Collaboration
Levels of Collaboratio
n
Levels of Integration
Collaboration
for the 21stCentury
![Page 14: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/14.jpg)
Review existing tools, cont’d Levels of Collaboration Scale
The Collaboration Checklist
Wilder Collaboration Factors Inventory
Collaboration Implementation Rubric
CFSA Collaboration Planning and Assessment Tool
Strategic Alliance Formative Assessment Rubric (SAFAR)
IDEA Partnership Success Rating Scale
Communities of Practice: Collaboration Assessment Rubric (CoPCAR)
![Page 15: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/15.jpg)
Apply information (an example)
NCEO was tasked by funder to partner with others.
While each has own focus, they are expected to collaborate with one another.
The purpose of the evaluation was to measure: 1.Strength/types of collaboration across Centers; 2.Strength/types of collaboration among Centers and key partners;3.Quality/benefits of the collaboration across Centers; and4.Quality/benefits of the collaboration among Centers and key
partners.
![Page 16: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/16.jpg)
Evaluation Questions
1) What do clients understand about the Center and what it does?
2) What does the Center understand about their partners’ role, mission and work?
3) To what extent is the Center engaged in activities that support collaboration?
4) What are the benefits/advantages of the collaboration?
5) What is the quality of joint TA?
![Page 17: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/17.jpg)
Evaluation Methodology
• Mixed methods approach
• Process data collected annually
• Outcome data: benefits and quality of
collaboration
![Page 18: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/18.jpg)
Data Collection Design & Instrumentation
1) Collaboration mapping
2) Online survey
3) 1:1 interviews with key partners and clients
4) Case studies with clients receiving various
TA
![Page 19: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/19.jpg)
Recap: To measure collaboration…
1)Review literature, program documents, meeting minutes, logic model ask questions!
2)Investigate existing tools, frameworks &
methods
3)Apply to your agency’s needs and goals
![Page 20: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/20.jpg)
Logic Model and Collaboration Mapping
AEA Conference 2014David Merves
![Page 21: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/21.jpg)
Collaborative Logic Model
• Collaborative Logic Model: separate and distinct
• Purpose: guide theoretical framework(s), methodology, instrumentation and timing
• Activities: align with levels of collaboration
![Page 22: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/22.jpg)
NCEO Partner Collaborative Logic Model
NCEO Partner Collaborative Logic Model Purpose: The overarching goal of collaboration at the National Center on Educational Outcomes is to provide national leadership and coordination in developing educational assessments and accountability systems that appropriately monitor educational results for all students, including students with disabilities, English language learners (ELLs), and ELLs with disabilities.
Activities
Outcomes Impacts Inputs
![Page 23: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/23.jpg)
NCEO Collaboration Logic Model
Inputs Activities Outcomes Impacts
![Page 24: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/24.jpg)
Levels of Collaboration Survey
![Page 25: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/25.jpg)
Social network analysis First used in 1934 by Joseph Moreno in New York City
Schools. New York Times called it, “Psychological Geography”
Money flow between organizations, how people obtain employment, transmission of infectious diseases, decision-making
The use of social network analysis (SNA) to measure strengths in community collaboration offers several advantages
25
![Page 26: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/26.jpg)
MI3-CEN Levels of Linkage
![Page 27: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/27.jpg)
Parent Involvement - all links
![Page 28: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/28.jpg)
Parent Involvement 3+ links
![Page 29: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/29.jpg)
Digital Dashboard
![Page 30: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/30.jpg)
What’s it Mean to us? Grantee Perspective
AEA Conference 2014Laurene Christensen
![Page 31: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/31.jpg)
NCEO’s Focus on CollaborationPartners employ common strategies and language
Partners develop common products, tools, and services
Non-duplication of efforts
Cost efficiency
![Page 32: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/32.jpg)
Benefits of CollaborationCo-sponsorship of TA products, tools, and services
Assisting clients no matter whom they call (efficiency of ‘triage’)
Communicating activities using similar language across TA providers
Sharing resources/funds for cost-effective, complimentary efforts
Cost of TA with centers operating jointly versus separately
Increased availability of resources
![Page 33: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/33.jpg)
Collaboration ChallengesDifficulties associated with initial stages of evaluation
Staff members’ lack of time
Delayed accomplishments
Duplication of efforts
Working with diverse organizations
![Page 34: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/34.jpg)
Design assess-ment of NCEO
collaborations –
November 2013-
February 2014
Develop collaboration logic model – February 26, 2014
Refine data
collection tools –
April-May, 2014
Collect collabora-tion data – ongoing
Develop additional
data collection
tools – Fall, 2014
Collaboration Data Collection Schedule
-
![Page 35: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/35.jpg)
Current Data Collection Technical Assistance Documentation Tool
Where in the World Tool
Collaboration Survey for Partners
Website linkage analysis
![Page 36: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/36.jpg)
Technical Assistance Tool Questions
![Page 37: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/37.jpg)
Collaboration Survey for Partners
Communication
Cooperation
Coordination
Collaboration
![Page 38: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/38.jpg)
![Page 39: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/39.jpg)
Current Data Collection Technical Assistance Documentation Tool
Where in the World Tool
Collaboration Survey for Partners
Website linkage analysis
![Page 40: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/40.jpg)
DiscussantQ & A
AEA Conference 2014Vitaliy Shyyan
![Page 41: Evaluating Outcomes of Federal Grantee Collaborations Patricia Mueller, Chair AEA Conference 2014 Denver, CO The National Center on Educational Outcomes-National](https://reader037.vdocuments.site/reader037/viewer/2022110320/56649cae5503460f94971d0e/html5/thumbnails/41.jpg)
Thank you!For additional information contact:
Pat Mueller, Judy Lee &
David Merves
Evergreen Evaluation & Consulting, Inc.
Laurene Christensen &
Vitaliy Shyyan
National Center on Educational Outcomes