virginia g. piper trust presentation - dr. jeff sheldon - january 4, 2016

STRATEGIC LEARNING, RESEARCH AND EVALUATION AT THE VIRGINIA G. PIPER CHARITABLE TRUST

Jeffrey A. Sheldon, Ph.D.4 January 2016

Virginia G. Piper Charitable Trust

Overview

• Global Vision for Strategic Learning, Research, and Evaluation

• Creating and Supporting a Context for Learning and Knowledge Management

• Approaching and Assessing the Impact of Investments

• Planning and Refining Grant-making

• Work Approach 1: Fostering Empowerment

• Work Approach 2: Fostering Self-Determination

• Projected Collaborative Accomplishments

• Facing Challenges

Virginia G. Piper Charitable Trust - 2

Jeffrey A. Sheldon, Ph.D.

Global Vision for Strategic Learning, Research, and Evaluation• A supportive context for learning and evaluation (i.e., culture, leadership, systems and structures, communications, and

teams) is created and sustained.

• Learning across the Trust is in real-time, is communicatively grounded, and based on Adult Learning Theory.

• Everyone within the Trust is involved in learning such that the Trust becomes a community of learners.

• Generating knowledge for learning:

• Through evaluation and research.

• Gathered from individuals within and without the Trust.

• Gathered through the Trust’s “collective wisdom.”

• Gathered from grantees.

• Gathered from external sources (e.g., peer reviewed and gray literature, and from other similarly engaged philanthropic and human service organizations).

• Evaluation capacity is built within the Trust so that everyone understands its inherent importance, everyone becomes better consumers of evaluation, everyone can contribute within their specific role(s), everyone can teach others about evaluation, and everyone can contribute to organizational learning.



Global Vision for Strategic Learning, Research, and Evaluation

• Grantees evaluation capacity is built so they become integral to the overall research, evaluation, and learning enterprise by being better consumers and producers of evaluative knowledge.

• A community of learners is created within each initiative so grantees can learn from each other and the Trust can learn from them.

• A rigorous research agenda is developed within each initiative, there is publishing in peer reviewed journals and other media as appropriate, and presentations are made at conferences (e.g., American Evaluation Association).

• Strategic learning, research, and evaluation are integral in a complex ecological system of interrelated parts.

• Strategic learning, research, and evaluation informs the planning and refinement of the Trust’s grant-making endeavors.



• Organizational learning is the best way to address critical issues facing organizations today because they must be ready to receive and act upon new information for change to occur (Bowen, Rose, & Ware, 2006).

• Contextual factors influence how and to what extent learning will occur in an organization (Cousins & Leithwood, 1986; Cousins & Shulha, 2002).

• Culture, leadership, communication, teamwork, and systems/structures are considered to be the elements of organizational context necessary for organizational learning (Shulha & Cousins, 1997).

• Transformational leadership, supportive structure, and culture are the key factors promoting the learning process (Lam & Pang, 2003).

• Evaluation contributes to individual, group, and organizational learning (Preskill & Torres, 1999; Russ-Eft, Atwood, & Egherman, 2002).

• An organization’s culture and context influence the extent to which evaluation occurs in support of learning and decision-making (Preskill & Torres, 2000).



Creating and Supporting a Context for Learning and Knowledge Management


Creating and Supporting a Context for Learning and Knowledge Management• Is the Trust’s culture supportive?

• Examples of collaboration and problem-solving behaviors:

– There is respect for different perspectives and opinions.– Staff is continuously looking for ways to improve processes and initiatives.– Opportunities are provided to think about and reflect on work.– There is more concern about how work contributes to the success of the Trust rather than on individual success.– Problems or issues are generally viewed as opportunities to learn.

• Examples of risk-taking behaviors:

– Mistakes are viewed as opportunities for learning.– There is continuous questioning abut how the Trust is doing, what it can do better, and what is working.– There is a willingness to take risks in the course of doing the Trust’s work.– There is a commitment to being innovative and forward looking.– There is confidence that mistakes or failures will not have a negative consequences.

• Examples of participatory decision-making behaviors:

– Individuals trust themselves and each other.– Individual capacity to learn is the Trust’s greatest resource.– Data/information is used to inform decision-making.– Asking questions and raising issues about the Trust’s work is encouraged. – Offering dissenting opinions and alternative viewpoints is encouraged.



• Is the Trust’s leadership transformational by supporting learning and knowledge management?

• Officers believe that the Trust’s success depends upon learning from daily practices.

• Officers support the sharing of knowledge and skills among staff members.

• Officers provide the necessary time and support for systemic, long-term change.

• Officers use data/information to inform decision-making.

• Officers take on the role of coaching, mentoring and facilitating learning.

• Officers help staff understand the value of experimentation and the learning that can result from such endeavors.

• Officers admit when they don’t know the answer to a question.

• Officers make realistic commitments for staff (e.g. time, resources, and workload).

• Officers understand that staff has different learning styles and learning needs.

• Officers are more concerned with serving the Trust than with seeking personal power or gain.

• Officers are open to constructive feedback from others.

• Officers model the importance of learning through their own efforts to learn.




• Does the Trust have systems and structures in place that will support learning and knowledge management?

• Examples of an open and accessible work environment:

– There is little bureaucratic red tape when trying to do something new or different.– The Trust is designed to allow for easy and frequent communication amongst staff.– There are few boundaries that keep staff from working together.– All staff are available to participate in meetings.

• Examples of rewards and recognitions system and practices:

– There is recognition or rewards for learning new knowledge and skills.– Staff are recognized or rewarded for helping solve Trust problems.– The current reward/appraisal system recognizes, in some way, team learning and performance.– Staff are recognized or rewarded for helping others learn.– Staff are recognized or rewarded for experimenting with new ideas.

• Examples of the relationship between the Trust’s work to its goals:

– Staff understands how its work relates to the goals and mission of the Trust.– Staff performance goals are clearly aligned with the Trust’s strategic goals.– Staff meets work deadlines.




• Do the Trust’s communications support learning and knowledge management?

• Examples of availability:

– Information is gathered from staff, grantees, board members, community partners or other stakeholders to gauge how well the Trust is doing.

– Currently available information tells the Trust what it needs to know about the effectiveness of initiatives.– There are adequate records of past change efforts and what happened as a result.

• Examples of dissemination:

– There are existing systems to manage and disseminate information for those who need and can use it.– Staff is cross-trained to perform various job functions. – Officers and staff have access to the information needed to make work-related decisions.– Technologies are used to communicate with one another.– When new information that would be helpful to others is learned or discovered it gets disseminated to

those individuals.




• Does the Trust perform work in teams?

• The Trust operates via, or is transitioning towards, a team-based structure.• Work at the Trust is sometimes conducted as part of a working group that is or could be identified as a

“team.”• Teams are an effective way to meet Trust goals.• Training is provided on how to work as a team member.

• Is The Trust engaged in evaluation?

• The integration of evaluation activities into the Trust’s work has enhanced the quality of decision-making.

• It has been worthwhile to integrate evaluation activities into the Trust’s daily work practices.• There would be support if the Trust tried to do more evaluation work.• Evaluation helps the Trust provide better initiatives.• Doing more evaluation would make it easier to convince Trust staff of needed changes.• This would be a good time to begin, renew or intensify efforts to conduct evaluations.• There are evaluation processes in place enabling the Trust to review how well changes it makes are

working.



Approaching and Assessing the Impact of Investments

• Each of the Trust’s funded programs is unique therefore evaluation design needs a theory-driven approach based on two models:

• Action Model Theory-driven process evaluation - It is prescriptive in that it explains the action that is required to solve a social problem by highlighting the components and activities program designers and key stakeholders see as necessary for program success.

• Change Model Theory-driven outcome evaluation - It is descriptive in that it explains why the problem will respond to the action by highlighting the assumptions about causal processes through which an intervention is supposed to work.

• Action Model + Change Model Integrated Theory-driven process/outcome evaluation.



• Action Model (Process)

• Intervention and service delivery protocols.

• Implementing organization: assess, enhance, and ensure its capacity.

• Program implementers: recruit, train and maintain both competency and commitment.

• Associate organizations/community partners: establishing collaborations.

• Ecological context: seek its support at micro and macro levels.

• Target population: identify, recruit, screen, serve.

• Change Model (Outcomes)

• Intervention/treatment

• Determinants

– Mediators– Moderators

• Goals/outcomes:

– Distal– Intermediate – Proximal



Approaching and Assessing the Impact of Investments

WWK

Principal

Project Manager

PTO/PTAOfficers

Computer Instructor

Parents

Effects intrinsic motivation

leading to use

TeachersStudents

Champs

Positive Letter

Negative Letter

Incentive Letter

Phone

E - Mail

Personal

Follow-up

Follow-up Letter

Internet training

No effect on intrinsic

motivation leading to non-

use

Direct influence/communication:

Direct action:

Two-way influence/communication:

Example of an Action Model

Or

Recruitment Program



Example of a Change Model

Safer children

ReduceReluctance

ReduceIgnorance

ReduceDenial

ReduceEmbarrassment

ReduceTime

Wired With WisdomParent Recruitment Program

(LettersPhone calls

E - mailsPersonal contact)

IncreaseExperience

IncreaseConcern

Successfulrecruitment =

use of Wired With

Wisdom

Well managed family internet environment &

safety plan

Reduce Risk

Increase Expectations

Increase Mental

Readiness



Approaching and Assessing the Impact of Investments• Effective Evaluations Using the Theory - Driven Approach:

• Future action directedness

– Useful to stakeholders.– Assessing merit, worth, or value is a means rather than an end.– Provides useful information for stakeholders to improve current or future programs.

• Scientific and stakeholder credibility

– Follow scientific methods and principles to optimize validity and reliability.– Responds to stakeholders’ values, views, concerns, and needs.

• Holistic Approach

– Intrinsic value.– Considers program context.

• Compels evaluator to be thoughtful before acting because she thoroughly understands the program.

• Helps tailor evaluations to answer the most important questions (i.e., parsimony) and informs methodological choices.

• Heightens evaluation sensitivity and validity.

• Helps evaluator meet American Evaluation Association professional evaluation standards – Utility, Feasibility, Propriety, Accuracy.

• Fosters cumulative wisdom.



Approaching and Assessing the Impact of Investments• Face-to-face meetings with stakeholders (working group or intensive interview).

• Develop a systematic understanding of stakeholder views, needs, and values.

• Facilitate conceptualization of a program:

• Stakeholders reflectively examine what they are doing.• Stakeholders identify elements that are essential for achieving program goals.• Stakeholders articulate causal relationships.• Theorizing methods – backward reasoning (start with intended outcomes) to inputs, forward reasoning, or both.• Increases stakeholders’ buy-in and support for evaluation.

• Work with stakeholders to focus the evaluation design (e.g., questions, data, methods, analysis, etc.).

•Gather credible evidence using rigorous methods based on contingency (e.g., qual., quant., or mixed).

•Work with stakeholders to examine data and justify conclusions.

•Work with stakeholders to ensure utilization and lessons learned are widely disseminated.

• Utilization of the evaluation can include:

• Conceptual (knowledge produced used for understanding/education).• Instrumental (knowledge produced used for decision-support).• Process (making use of the logic of the evaluation).• Symbolic (knowledge used to justify a priori decisions).



Planning and Refining Grant-making• Identify the issues/problems that need to be solved based in part on previous grant-making experience and criteria internal to the Trust.

• Articulate assumptions about how the problems can be addressed with available resources:

• Existing knowledge of the issue.

• The activities of others in the field.

• Previous grant-making experience.

• How the Trust’s support can address the problem and achieve desired results.

• Develop theoretical action and change models or logic models to express assumptions; these are updated throughout the planning/refinement process.

• Scan the field to assess needs and identify gaps by taking stock of the social, political, and economic context—where it is currently and where it may be heading.

• Goal: Identify areas or strategies where additional resources can have significant impact.

• Methods: Secondary research, discussions, gathering and convening, surveys, and network mapping.

• Topics: Populations of concern, community needs, context, key players, strategies, other funders, successful strategies/what works, and gaps.



Planning and Refining Grant-making• Take stock of internal capacity through a frank and thorough assessment.

• Develop a clear understanding of the Trust’s capabilities so that it can align what is needed with what it can realistically contribute.

• Internal capacity includes: Finances, human assets, intellectual capital, social capital, comparative advantage/influence.

• Validate assumptions through a "quality review" with other stakeholders to ensure that the logic is sound, assumptions are accurate, and relevant influential factors are accounted for.

• Implement, evaluate, and plan again as an ongoing, iterative process.



Work Approach 1: Fostering Empowerment

• A long-term process individuals, organizations, and communities undertake for themselves rather than something done to or for them (Lather, 1991, 1992; Claridge, 1996)

• Empowered outcomes: individuals or aggregate bodies of individuals engaging in behaviors that permit effective pursuit of planned social change, for example, working with others, managing resources, learning decision-making skills, influencing policy, accessing government, sharing leadership, obtaining needed resources, being networked to others, etc. (Schulz et al., 1995; Zimmerman, 2000).

• Three interrelated components (Zimmerman, Israel, Schulz, & Checkoway, 1992).

• Interactional: Developing knowledge, skills, and strategies to become leaders/advocates for change.

• Intrapersonal: Appraisal/critical awareness of, and the motivation and agency to change one’s life circumstances (i.e., self-efficacy).

• Behavioral: Individual-level actions requiring leadership that benefits individuals and communities by influencing social policy.



Work Approach 2: Fostering Self-Determination

• A general, organismic theory of human motivation explaining the relationship between three sets of motivational processes—intrinsic, extrinsic, and amotivational —and self-determined behaviors (Ryan & Connell, 1989).

• To be self-determined, experience well–being, and have optimal health (e.g., self-esteem, self-actualization, vitality, positive affect, and work engagement) people have to satisfy three basic psychological needs endowed by nature (Koestner, Ryan, Bernieri, & Holt, 1984):

• Competence: efficacious performance of behaviors based on positive feedback and communication.

• Autonomy: choice in how to perform a task is provided, the task is recognized as being important, and feelings about the task are acknowledged.

• Relatedness: meaningful connections to others and a sense of belonging; the opposite is alienation and inauthenticity.

• The collective research on SDT suggests that when basic psychological needs are supported, intrinsic motivation and integrated extrinsic motivation are most likely to be evident.



Projected Collaborative Accomplishments Within 3 – 6 Months

• Contextual assessment conducted with internal Trust team to determine readiness for organizational learning and evaluative inquiry using quantitative (e.g., the ROLE) and qualitative (e.g., observation and content analysis) methods.

• Qualitative research methods will be used to determine knowledge needs and ascertain knowledge, skills, experiences, and attitudes about evaluation by group (see below) and across the Trust:

• Interviews/focus groups of recent grantees within each initiative (random sample). • Interviews with all current grantees (full census)/formulate action and change models and evaluation plans.• Interviews with board of trustee members (full census, n = 8).• Interviews with outside board committee members (full census, n = 4).• Interviews with officers (full census, n = 3).• Interviews with staff (full census, n = 13).

• Information gathered through quantitative and qualitative research will be collaboratively analyzed to make meaning of the data, synthesized, and reported back to each group along with a cross-group report.

• Will have canvassed other charitable trusts/foundations with similar initiatives to inform the Trust’s best practices in research, evaluation, and strategic learning.

• A framework for evaluation, research, and strategic learning that maps onto the Trust’s strategic plan as informed by data and input from Trust leadership, staff et al. is at or near completion.



Overcoming Challenges using the Getting To Outcomes Model• Focus: The problem related to pediatric vision we’re trying to solve or unmet need to be fulfilled based on

secondary or external data.

• Target: Identify goals, target population, and desired outcomes of a pediatric vision screening and follow-up care initiative.

• Adopt: Find existing programs and best practices regarding pediatric vision screening and follow-up care worth adopting (must be evidence-based or a promising practice based on preliminary research).

• Adapt: Modify the program or best practice to fit children’s vision screening and follow-up care needs.

• Resources: Assess capacity (e.g., staff, financing, etc.) to implement the initiative.

• Plan: Make a plan for getting started by developing action and change theoretical models, and formulating and prioritizing evaluation questions.

• Monitor: Track planning and implementation process.

• Evaluate: Evaluate initiative’s success in achieving desired results by using rigorous scientific methods to answer evaluation questions.

• Improve: Make a plan for continuous quality improvement from the results of evaluative inquiry.

• Sustain: Consider how to keep the initiative going if it is successful.



virginia g. piper trust presentation - dr. jeff sheldon - january 4, 2016

Documents