volume iv, number 3/4 1998 free quarterly newsletter from the … · 2017-09-01 · mastery, mental...

RESEARCH PROJECT 1 VOLUME IV, NUMBER 3/4 1998

Heather B. WeissDirector, HFRP

The notions of “learning organiza-tions” and “organizational learn-ing” are very popular these days.

Chris Argyris and Donald Schön, whofirst discussed organizational learning inthe 1970s, promote the idea of double-loop learning. This learning results inchanges in organizational values and pro-motes a fundamental transformation ofthe organization such that it is more suc-cessful and productive.1 Peter Senge, whohas popularized organizational learningwith his book, The Fifth Discipline, de-scribes five characteristics of learning or-ganizations: systems thinking, personal

From the Director’s DeskVolume IV, Number 3/4 1998 Free Quarterly Newsletter

TH E EVA L U AT I O N

EX C H A N G E :Emerging Strategies in Evaluating Child and Family Services

HARVARD FAMILYRESEARCH PROJECT

©1999 President and Fellows ofHarvard College

Published byHARVARD FAMILY RESEARCH PROJECT

Harvard Graduate School of Education38 Concord Avenue

Cambridge, MA 02138Tel: (617) 495-9108Fax: (617) 495-8594

E-mail: [email protected]: http://gseweb.harvard.edu/~hfrp

All rights reserved. This newsletter may not bereproduced in whole or in part without writtenpermission from the publisher. The HarvardFamily Research Project gratefully acknowl-edges the support of The Annie E. Casey Foundation, The Council for Excellence inGovernment, The F.B. Heron Foundation, TheHitachi Foundation, the Ewing Marion Kauff-man Foundation, and the W.K. Kellogg Foun-dation. The contents of this publication aresolely the responsibility of the Harvard FamilyResearch Project and do not necessarily reflectthe views of the funders.

mastery, mental models, building sharedvision, and team learning.2

As we struggle to face the large-scalechanges affecting our work — account-ability, devolution, information technol-ogy, a new century — the dynamism,flexibility, and opportunity implied byan organizational learning approach iscompelling. But what does it take for or-ganizations that serve children and fam-ilies to become “learning organiza-tions”? What constraints do they face?How do they actually do it? Who needsto be involved?

This issue of The Evaluation Ex-change is the first of two which examinethese questions. We have brought togeth-er a variety of perspectives that we hopewill stimulate thinking and discussionon this important topic. Our Theory andPractice section summarizes a paper thatWilliam Morrill and I authored on howto develop and promote useful learningin public programs. The article suggeststhat organizational learning in the policyarena must involve the creation of a“learning community” of policymakers,researchers, evaluators, and practitionersand sets forth for discussion a model forimplementing a learning approach in thepublic sector. In our Promising Practicessection, we include two articles aboutapproaches for creating a public policylearning community that actively in-volves citizens. William Novelli of the

IN THIS ISSUE:LEARNING ORGANIZATIONS

♦ Useful Learning for Public Action..............................2

♦ Making the Case for Children ...............................5

♦ Civic Participation and theCitizens Research Method........7

♦ Interview with Patricia McGinnis .....................8

♦ Building State and LocalEvaluation Capacity: The U.S. Department of Justice’s Evaluation Strategy ....................................10

♦ Studying Learning Initiatives:The Experience of theEmpowerment Zones/Enterprise CommunitiesNational Learning Initiative ...................................11

♦ State Policy DocumentationProject: Data Collection and Dissemination...................13

♦ Lessons Learned......................14

♦ New and Noteworthy ..............15

♦ Electronic Mailbox..................16

1 Argyris, Chris & Schön, Donald A. (1996). Orga-nizational learning II: Theory, method, and prac-tice. Reading, MA: Addison-Wesley Publishing.

2 Senge, Peter M. (1990). The fifth discipline: The artand practice of the learning organization. NewYork, NY: Currency Doubleday.

As America struggles with global-ization and devolution, alloca-tion of scarce resources, and

growing public demand for accountableand cost-effective services, the demandfor and the use of information to guidedecision-making should be at an all-time high. They arenot. Despite sub-stantial investmentthat has been madein knowledge de-velopment over theyears, experienceshows that informa-tion about publicprograms has oftennot been used forpolicy formation,program design, orthe high perfor-mance managementof existing services(Darman, 1997;Levitan, 1992).

Now is an opportune time to beginworking to make our knowledge devel-opment investments, including evalua-tion, more useful in the country’s ef-forts to solve its basic problems. To agreat extent, the conditions, experience,and ingredients necessary to generateand use knowledge are largely in place.The pervasive, multi-level push forgreater accountability has resulted in anew demand for a wide array of high-quality information to support policy-makers and program managers as wellas growing practical experience with in-dicators. There is also growing knowl-edge about how to build the capacity ofpublic managers, community groups,and non-profit organizations to use per-formance and other data from manage-ment information systems in an ongoingway to improve performance and out-comes (Usher, 1993; Gray and Associ-ates, 1998). Additionally, domestic pol-icy initiatives, such as standards-driveneducational reform, managed care, andthe privatization of human services, areforcing a critical examination of how

best to invest limited research and eval-uation dollars. The evaluation field it-self has also changed, moving from a focus on single, large-scale experi-ments to more flexible, participatory,and community-based studies. Finally,the recent devolution of responsibility

for welfare pro-grams to the stateand local levels(and the flexibilityand governancechanges implied bythis shift) providesan unparalleled op-portunity to devel-op the multi-levelcapacity to gener-ate and use infor-mation to improveoutcomes for ournation’s people sig-nificantly.

To move to atruly relevant sys-

tem of information, we must strive to in-stitutionalize knowledge developmentand use so that programs can continu-ously improve and be more accountable.Private-sector experience suggests thatwe must create “learning organizations.”A learning organization is able to create,acquire, and transfer knowledge as wellas modify its own behavior to reflect newknowledge and insights (Garvin, 1993).To be successful “learners,” organiza-tions must be willing and have the ca-pacity to use data on a regular basis, as-sess implications, and make changes ac-cordingly.

Experience with past informationbuilding endeavors points to eightlessons that can guide the creation ofpublic sector learning organizations:

1) Investing in knowledge buildingshould be strategic in concept, target-ed to maximize impact.

2) Learning must consider the incre-mental and iterative nature of the pol-icy and decision-making process and

The Evaluation Exchange 2 HARVARD FAMILY

THEORY AND PRACTICE

Useful Learning for Public Action

National Center for Tobacco-Free Kidswrites about implementing an informa-tion campaign to inform citizens andpolicymakers about children’s health.HFRP researcher Serene Fang writesabout the Citizens Research process, atechnique to better inform and engagecitizens in understanding and influenc-ing the policy-making process. OurQuestions and Answers section presentsa conversation with Patricia McGinnis ofThe Council for Excellence in Govern-ment about how accountability and orga-nizational learning in the public sectorcome together. As interest in and exper-imentation with learning endeavorsgrow, being able to evaluate their effec-tiveness will be important. In our Evalu-ations to Watch section, Andy Mott ofthe Center for Community Change andVicki Creed of Learning Partners discussan approach they used to evaluate learn-ing through the National Learning Initia-tive of the Empowerment Zones/Enter-prise Communities Project. In one of ourSpotlight sections, Lisa Plimpton of theCenter for Law and Social Policy dis-cusses work that CLASP is doing to doc-ument state policies on welfare reform.In another Spotlight article, MichaelQuinn Patton of The Union Institute andRicardo A. Millett of the W.K. KelloggFoundation look at criteria that distin-guish casual/informal notions of lessonslearned from “high quality” lessonslearned. In our Beyond Basic Trainingsection, Robert Kirchner of the U.S. Department of Justice discusses workthat his agency has been doing tostrengthen evaluation capacity at thestate and local levels. Finally, in our Newand Noteworthy and Electronic Mailboxsections, we provide information on cur-rent resources to assist those interested inlearning organizations.

Organizations are increasingly askedto undertake many challenges with fewerresources and under greater scrutiny. Or-ganizational learning theory and prac-tice may help us to better understand andimplement productive and relevant waysin which to do so. ♦

For information on submitting articles toThe Evaluation Exchange, please call (617) 495-9108 or visit our Web site:http://gseweb.harvard.edu/~hfrp/eval

Now is an opportunetime to begin workingto make our knowledgedevelopment invest-ments, including evalu-ation, more useful inthe country’s efforts to solve its basic problems.


thus should be structured to flow ascontinuously as possible at all levels.

3) Collaboration among the research/evaluation, the policy, and the practi-tioner communities is necessary.

4) An improved learning system mustmake reasonable judgments about thecontent and timing of expectations forlearning activities and the conse-quences that flow from results. Itneeds to emphasize low-risk learning,especially at the outset.

5) Selection of learning methodologiesmust be judicious, both in terms oflearning objectives and timing.

6) Investment in a variety of data collec-tion strategies, including an integra-tive MIS in support of operations,needs to be considered.

7) Efforts will require strong federalleadership and support and shouldbuild on information and indicatorwork already underway.

8) More integrated funding of evalua-tions and programs will be importantin developing continuous learningsystems that improve outcomes.

The first step in building a learningsystem is developing the commonground that unites the three communi-ties relevant to the development ofpublic sector learning organizations:the knowledge-building community,the policy and managerial community,and the professional services(“doing”) community. This commonground includes:

• Development of a Collaborative Learn-ing Agenda. Multiple forces push agen-cies to develop learning agendas, in-cluding the Government Performanceand Results Act (GPRA) and the con-gressional reauthorization process. Alearning agenda should be built arounda learning process that emphasizes se-quential learning, iterative and partici-patory processes, use of the full rangeof learning methodologies, and atten-dance to means whereby learning willbe communicated for both policy andprogram improvement purposes. A

longer-range objective of building theresearch agenda and the evaluationstrategy should be the creation of learn-ing organizations.

• Performance Goals and Measures.Given the explosion of interest in perfor-mance indicators and standards due tothe accountability movement, both thepolicy and the learning communitieshave an opportunity and interest in com-ing together for a discussion about whatgoals are being sought and what mea-sures will be used tojudge how well theyare being achieved.

• Allocation of Re-sources. Financialsupport will benecessary to imple-ment the researchagenda and the newlearning approaches. All parties willwant to move sequentially with more re-sources and then assess the results.

• Accountability Consequences. Allstakeholders must have reasonable ex-pectations for and promises about theircollective efforts. There should also beclarity and agreement about the conse-quences of the learning process from theoutset.

• Education and Communication. Thethree communities must work togetheron an education and ongoing communi-cations component. Actors who havebeen talking at and sometimes past eachother need to communicate clearly witheach other, and the groups included inthe dialogue should be broadened.More attention needs to be paid to thecreation of a dissemination strategy inorder to maximize the many potentialp ayo f f s f r o m r e s e a r ch a n d evaluation.

In order to begin the discussionabout how to improve knowledge devel-opment and learning in the public sec-tor, we propose a model for a continu-ous learning system. This model seeksto create continuous opportunities forthe development and use of relevant in-formation; for encouraging correctiveactions, risk taking, and participation;and for recognition and rewards for per-

formance improvement. It is intended tostimulate thinking about a learning sys-tem and will be revised based on com-ment and through examination of ef-forts to implement systems like it at thefederal, state, and local levels. Themodel envisions a continuous, five-stage learning process:

1) Engage key stakeholders in strategicplanning and set learning agendaand performance goals and stan-dards. Activities in this step include

obtaining resourcesand commitment tolearning, specify-ing performancegoals, identifyingresearch and evalu-ation questions andgaps, and designingan overall learningagenda.

2) Learn from experience and relevantresearch and incorporate lessons intoprogram/policy design. Activitieshere include assembling resources,specifying outcome and process mea-sures and data to support them, net-working to share successful innova-tions and identify common problems,and identifying technical assistanceneeds and providers.

3) Engage in innovation, monitoring,and evaluation. Activities includecontinuously testing new ideas andapproaches, designing evaluation, andmonitoring and assessing process andprogress with performance measures,evaluation, and data.

4) Learn from evaluation and compar-isons with others and make coursecorrections. Activities include usingmonitoring and evaluation informa-tion for corrections and improvement,using benchmarking to examineprogress of the program and/or fieldof practice/policy, and assessing andapplying knowledge from relevantbasic and applied research.

5) Transfer lessons for program respeci-fication and identify knowledge gapsfor further research and experimenta-tion. Activities here include identify-ing gaps for further research and

The first step in build-ing a learning system is developing commonground…


transferring knowledge for continu-ous improvement across the networkand service/policy field.

There are examples of organizationsthat have worked to apply such a learningmodel. They suggest the feasibility ofsuch learning models and offer cases touse to see if such approaches appeal toboth the policy and learning communi-ties. Over the next few years, as devolu-tion and accountability are further im-plemented, there will no doubt be moreexperiences and opportunities to imple-ment a learning model such as this. Atthe same time, a series of related andsupporting actions could be taken to bet-ter connect knowledge, policy, and prac-tice. These include:

• Systematic learning agendas could beadded to the strategic planning, annualperformance plans, and performancemeasures now incorporated at the feder-al level in GPRA and similar GPRA-likerequirements at the state level.

• Similar learning agendas could be con-tinued as newly added to reauthorizationlegislation.

• Public sponsors could offer increasedregulatory flexibility in return for clearoutcome accountability to provide in-centives for innovation.

• Foundations and corporate philan-thropy could make a significant contri-bution to the development of institu-

tional capacity for learning at thegrassroots level.

• While retaining their own identities,public agencies and private foundationscould be far more effective in the aggre-gate by being considerably more openwith each other and specific about theirindividual learning agendas.

• Public and private agencies could investmore systematically in the developmentof knowledge about what works.

The framework presented here hasmany of the key features that we be-lieve are necessary for a new learningand knowledge-building strategy that

Continued on page 14


from around the country who have ex-pressed willingness to be involved in theissue of reducing the tobacco industry’spowerful hold on children. We keepNAN members engaged by providinguseful information on key Congression-al votes, tobacco money in politics, ini-tiatives to protect kids, and activities andcontacts in their local areas. NAN mem-bers are asked to write to their electedrepresentatives, write letters to the edi-tor, speak out at local meetings, and take

other action, as ap-propriate.

Before creatingthis network, weconducted substan-tial research to findout more about peo-ple who are, or arelikely to become,consumer activists:specifically, whatmotivates them tog e t i n v o l v e d ,whether they careabout the issue of

children and tobacco, and how we mightencourage them to take on a more activerole. We knew from polling data thatthere was interest in protecting childrenfrom tobacco, but we had to turn publicopinion into citizen action.

We began by analyzing a large mar-keting database (Simmons) that reportson all manner of product usage, mediahabits, and other consumer behaviors.This database also profiled what it de-fined as consumer activists, and estimat-ed this group to be approximately 20percent of the adult U.S. population.Consumer activists were defined asadults who had, in the past year, contact-ed a radio or TV station, magazine ornewspaper editor, or elected official; metpersonally with an elected official; ac-tively participated in a local civic issueand/or volunteered for an environmentalor conservation group; or addressed apublic meeting.

We then conducted a phone survey toconfirm the profile and assess levels of

issue (for example, new legislation onchildren’s health insurance), or do notsee how it applies to them (they may bechildless, or assume that children haveaccess to health care if the parents trulywant it).

• Contemplation. During this phase, indi-viduals will consider greater involve-ment in an issue, such as seeking moreinformation.

• Action. In order tomove from contem-plation to concreteaction, people mustsee the benefits ofthe activity to them-selves and/or to so-ciety. It is impor-tant that the “costs”of involvement bekept to a minimum,as even the slightestbarriers may act asdeter ren ts . I n-creased social pres-sure can also help to move individualsfrom contemplation to action.

• Maintenance. Once people do get in-volved in an issue, it is important to re-mind them that they are helping to makeprogress, and to recognize their efforts.This stage of social advocacy is too oftenneglected, and activists can be lostthrough this lapse of attention. Progressreports, report cards, rewards, and re-minders for activists are critical for help-ing to maintain their involvement.

Children’s advocates need to deter-mine exactly what they want the publicto do to help solve a particular problem,then look for ways to move individualsalong this type of continuum. With re-search, analysis, and strategic thinking,the framework can be filled in.

Our Campaign for Tobacco-FreeKids did this when building our Nation-al Action Network (NAN), a growingdatabase of some 14,000 individuals

Too often, the political rhetoricsurrounding efforts to improvechildren’s health and welfare

does not accurately reflect the actionstaken to carry out these goals. Localand state governments are often limitedby inadequate federal policies or bud-gets in what they can do to help fami-lies. That is why grassroots activists areso important if we are substantially toimprove the results for children andfamilies. Doing so, however, will re-quire that advocates learn how to “makethe case for children” more effectively;they must motivate activists to helpstimulate coverage of children’s issuesin the mass media (thereby raising theprofile of these issues and increasingpressure for social change), and raisethe stakes for elected representativesand government officials.

So, how can we do a better job ofmaking this case? First, we must recog-nize that change must be broad in orderto be deep. That is, we must strive si-multaneously for environmental change(through public policy and agenda set-ting); for change at the community level(by affecting norms, expectations, andpublic support); and for changes in indi-vidual behavior (through skill teaching,positive reinforcement, and rewards).Accomplishing broad-based change re-quires communication at both the massmedia and interpersonal levels in orderto extend the reach and frequency — orpervasiveness — of key messages. Italso necessitates galvanizing manychange agents to create the critical massneeded to bring about substantivemovement.

Prochaska and DiClemente’s modelof individual behavior change providesclues about how to spur individuals to-ward greater involvement. The model in-cludes four stages:

• Pre-contemplation. In this first stage,which includes most people, individualsdon’t see the relevance of getting in-volved. They are either unaware of an

PROMISING PRACTICES

Making the Case for Children

So, how can we do a

better job of making

this case? We must

recognize that change

must be broad in order

to be deep.

• Engage activists emotionally by makingthem perceive a threat, or by generatinganger and/or moral outrage.

• Portray the fight as winnable; let themknow that others are involved and aresucceeding.

• Time messages and calls to specific ac-tions at intervals when these actions willhave the greatest impact.

• Provide factual information, especiallyon the scope of the problem and threatsfrom Congress.

• Encourage activists to personally ap-proach others, since other potential ac-tivists react favorably to this.

Clearly, it is important to know youraudience. In the case of children’shealth, key audiences includes policy-makers, “influentials” (who vary de-pending upon the subject area), activists,and the general public. In the case ofyouth smoking, youth advocacy is an im-portant strategy for us, and young ac-tivists play a particularly critical role in

capturing the atten-tion of policymak-ers and the public.Ch i ldren of tenmake the strongestadvocates for issuesthat directly affectthem, because theyhave first-hand ex-perience with theproblems they face

(and are therefore credible spokespeo-ple), and because they elicit an emotion-al response from those in a position to ef-fect change.

At the same time, it is critical not tooverplay the “child card” when advocat-ing for children. These images are indanger of being overworked, so all mes-sages pertaining to children must be be-lievable and real in order to maintaintheir effectiveness.

One helpful tool is to put a face on theproblem by focusing on a “villain.” Inthe case of youth smoking, highlightingthe tobacco industry’s unethical behav-ior in targeting children in its marketingcampaigns has been extremely effective.Not only does it draw an emotional re-sponse from the public and policymak-

concern about our own issue and thelikelihood of people getting involved init. Finally, using a qualitative researchtechnique, one-on-one interviews withindividuals who fit the consumer activistprofile, we probed to find what informa-tion and ideas were most appealing, andgained comprehension and reaction to anumber of message concepts. We foundthat activists are most likely to get in-volved when an issue affects them per-sonally; when they perceive a blatantwrong, untruth, or injustice; or whenthey observe a threatening situation. Ourresearch also showed that activists preferto work within an existing organizationor effort, rather than initiating actionthemselves. Many are driven to action bypersonal appeals from people they know.

In testing message concepts with con-sumer activists to see how they would re-spond, we found that we had to be care-ful when presenting messages that Con-gress was beholden to the tobaccoindustry and might act in the industry’sfavor. This is because activists are mostmotivated to act when they think theiractions will make a difference, and afeeling of hopelessness (Congress is inthe industry’s pock-et) worked againstthis motivation. Wealso found that ac-tivists did not re-spond positively togeneral messagesabout the health ofchildren or savinglives, but preferredmessages with aspecific call to action. Interestingly, intesting messages that assailed specificindividuals from the tobacco industry(e.g., a woman who is the senior vicepresident for external affairs for PhilipMorris), we found that respondents werenot comfortable with such personal at-tacks, despite their contempt for the in-dustry as a whole.

Overall, our research taught us thefollowing:

• Localize the issue to motivate activists,who respond to issues that directly affecttheir families and communities.

• Focus on kids in order to engage activists(smoking among adults is not as emo-tionally engaging).


…activists are mostmotivated to act whenthey think their actionswill make a difference.

ers, but the media are oriented towardstories with built-in controversy, and thisone has a clear villain that guaranteesheated debate.

These tools and lessons can be adapt-ed to any children’s issue. It is importantis to keep in mind that many agents ofchange are needed to achieve significantresults and improve the lives of children.Do not rely on mass media alone. Doyour research, identify your target audi-ences, and engage the activists who aremost likely to help you achieve yourgoals. ♦

William D. NovelliPresidentNational Center for Tobacco-Free Kids

THE EVALUATOR’S INSTITUTE

The fourth Evaluator’s Institute willbe held in Washington, DC from July19-24, 1999. The Institute brings to-gether internationally recognizedleaders in the practice of evaluationto plan and teach courses on evalua-tion issues. This year’s offerings in-clude courses on basic programevaluation; sampling; implementa-tion analysis; cost-benefit and cost-effectiveness analysis; evaluatingcommunity collaborations and build-ing their capacity; developing pro-gram theory; using qualitative meth-ods; learning from data; needs as-sessment; graphical displays;evaluating social programs and poli-cies; efficient and effective data gath-ering; and reducing and managingevaluation anxiety. Courses range inprice from $375 to $995; past partic-ipant and institutional discounts areavailable. The deadline for courseregistration and payment is July 1,1999. Those interested should con-tact the Institute at (703) 448-8324.Additional information about the Institute is available through the Institute’s Web site:http://www.erols.com/cwisler

Before his now-famous 1995 essay,Bowling Alone, political scientistRobert Putnam published a book

t i t l e d M a k i n gDemocracy Work:Civic Traditions inModern Italy, inwhich he comparedlocal Italian govern-ments in the Northto those in theSouth after Italytransferred powerand responsibilityfor many of its ser-vices from a centralized government tonewly-created regional governments.Putnam contended that local govern-ments in the North tended to be moresuccessful and efficient than local gov-ernments in the South because citizensin the North possessed a long historicaland cultural tradition of participating inassociational life that citizens in theSouth lacked.

At a time when our own governmentis engaged in a “devolution revolution,”passing down many of its social welfarefunctions from the federal to state andlocal levels, Putnam now makes the casethat Americans are behaving less likeNorthern Italians and more like the civi-cally-isolated Southern Italians. He ar-gues that Americans’ participation inconventional voluntary associations (thePTA, Elks Clubs, bowling leagues) hasbeen declining for the last 40 years, andthat this trend has contributed to an over-all weakening of the civic engagementand social capital of local communities.The widespread interest in his theorymay be an indication of how others havealso noted that our engagement with ourcommunities, and indeed, with democra-cy itself, is no longer what it should be.

The Harvard Family Research Projecthas been exploring methods for address-ing citizen engagement in policies thataffect local communities. One suchmethod is the Citizens Research process.Citizens Research is a variation of theCitizens Jury process developed by Ned


Crosby at the Jefferson Center in Min-nesota. Both share characteristics ofother recently developed processes such

as Richard Sclove’sCitizens’ Panel,Michael Briand’sCommunity Con-vention, and theDanish ConsensusConference. All ofthese techniquesseek to increase theopportunity for de-liberative democra-cy to have an effect

on the policy-making process, especiallyfor policy-making done at the local level.Specifically, these methods share the fol-lowing characteristics:

• Citizens from a community are selectedthrough a random survey and arrayed ina group that is representative of the com-

munity in terms of demographic charac-teristics and relevant attitudes.

• The group is charged with a specific taskregarding an issue or policy problem

• Over a series of meetings, the group iseducated about the issue by experts andwitnesses representing a range of views.

PROMISING PRACTICES

Civic Participation and the Citizens Research Method

…our engagement withour communities, andindeed, with democra-cy itself, is no longerwhat it should be.

For Additional Information:

The Citizens Jury MethodThe Jefferson Center7101 York Avenue SouthMinneapolis, MN 55435Tel: (612) 926-3292Fax: (612) 926-3199http://www.usinternet.com/users/jcenter

The Citizens Research MethodThe Citizens Research Group1650 Franklynn Dr.Furlong, PA 18925Tel: (215) 794-5475Fax: (215) [email protected]

The Citizens’ Panel MethodThe Loka InstituteP.O. Box 355Amherst, MA 01004Tel: (413) 582-5860Fax: (413) 582-5811http://www.amherst.edu/~loka

The Community Convention MethodThe Community Self-LeadershipProjectCampus Box 308Trinidad State Junior CollegeTel: (719) 846-5240Contact: Michael Briand

The Civic Practices Network(http://www.cpn.org/sections/tools/models/deliberative_democracy) main-tains a Web site on deliberativedemocracy which provides an intro-duction to basic concepts, links to ad-ditional sources, and a suggestedreading list.

For Further Reading:

Briand, Michael K. Building deliber-ative communities. A report preparedfor the Pew Partnership for CivicChange.

Crosby, Ned. (April 1996). Creatingan authentic voice of the people.Paper presented at the Midwest Polit-ical Science Association, April 1996.

Lemann, Nicholas. (1996). Kickingin groups. The Atlantic Monthly, vol.277, no. 4.

Putnam, Robert D. (1996). Thestrange disappearance of civic Amer-ica. The American Prospect, no. 24.

Putnam, Robert D. (1995). Bowlingalone. Journal of Democracy, vol. 6,no. 1.

Sclove, Richard E. (1994). Technolo-gy, society, and democracy: Newproblems and opportunities. A reportprepared for the John D. and Cather-ine T. MacArthur Foundation.


• The group is given adequate time to de-bate and deliberate together after whichthey are asked to make a collective rec-ommendation.

• Recommendations are shared with thepublic.

A key source of legitimacy for thismethod lies in the extent to which thegroup represents the larger community.In Crosby’s words, the selection processcreates “an authentic voice of the peo-ple” for the community that is selected.Another key strength lies in the citizens’experience of informed and collectivedeliberation.

For evaluation purposes, this methodfocuses attention on an important aspectof program assessment. While othermethods may be employed to answer thequestion of whether goals are being at-tained in a program’s design, implemen-tation, and effectiveness (and if not, whynot), this method provides informationas to whether the program itself reflectsthe public will of the community inwhich it is situated.

Some people have argued that biggovernment contributed to the weaken-ing of American civic engagement be-cause it fostered the perception that alarge and professionalized bureaucracytakes away the opportunity, perhaps eventhe need, for civic initiative. The CitizensResearch or Citizens Jury process canserve as a bridge for engaging a commu-nity with few other forums for public dis-cussion, in issues that will increasinglybe decided not at the level of federal gov-ernment, but at the theoretically more re-sponsive level of local government. ♦

Serene FangResearch AssistantHFRP

Substantial changes have takenplace in the public sector recently.One of the most important has

been the focus on results-based account-ability. At the federal level, accountabil-ity has been codified in the GovernmentPerformance and Results Act (GPRA) of1993, which requires federal agencies todevelop strategic plans and report annu-ally on their performance. At the sametime that account-ability systems arebeing firmly set inplace, interest in thenotion of “organiza-tional learning” inthe public sectorhas grown. Learn-ing organizationsseek to improveoutcomes through adynamic processthat enables them tocollect, utilize, and act on data in an or-ganizational climate that supports andfosters change. Organizational learning,like accountability, has the potential toimprove policy and program decisionmaking. We asked Patricia McGinnis,President and CEO of The Council forExcellence in Government, to discussthe potential of and constraints to publicsector organizational learning in the cur-rent climate of accountability.

1) What does organizational learningmean in the context of public organi-zations?

The concept of organizational learning isnot terribly different in the public andprivate sectors. It means building uponwhat works and exhibiting a willingnessto change, probably continuously, inorder to be more effective. Operationally,organizational learning in both sectorshas the same components, although thecomponents may look a little different,but there is, to some extent, a greatercomplexity in the public sector in thearea of measurement and even in the areaof setting goals. For organizational learn-

ing to occur, the organization needs tohave very clear goals, priorities, and mea-sures. You need to have an alignment ofgoals and measures for people, for bud-geting and financial management, and forassessing organizational results. You alsoneed — and this is tremendously impor-tant — clear communication among peo-ple working in the organization and withstakeholders, partners, and customers. In

the case of the pub-lic sector, thismeans not just cus-tomers, but the citi-zens you are serv-ing, whether theyare direct customersor not.

Alignment ofgoals is extremelyimportant. In manypublic organiza-tions, I’ve noticed

that the personnel system, the budgetsystem, the financial management sys-tem, and now the GPRA strategic plan-ning system, are operating on separatetracks. There is no consistency amongthe goals and measures for each of thesesystems, and alignment of these is es-sential for organizational success. Forexample, people in teams need to havespecific goals and measures that areconsistent with the larger organization-al goals and strategic planning; theirrole in meeting an organization’s ulti-mate goals needs to be very clear.

When it is very clear what you aretrying to accomplish, then people can bevery smart and creative about how to or-ganize themselves to accomplish well-understood goals. This means more thanjust the work of teams within an organi-zational structure; it means thinkingabout how to reach out to those outsidethe organization and make them part ofthe team. In this sense, the public is veryimportant. The public is not involvedenough now, and we see the conse-quences of that in the lack of public con-fidence and trust in government. I thinkthere is a communications gap between

Organizational learn-ing, like accountabili-ty, has the potential toimprove policy andprogram decisionmaking.

Have You…Completed our reader survey?

If you were not able to complete thereader survey we included in our last

issue, please take a few minutes to doso now. Your responses will help us tobetter address your needs for evalua-

tion information. The survey is alsoavailable on our Web site:

http://gseweb.harvard.edu/~hfrp.

QUESTIONS AND ANSWERS

Interview with Patricia McGinnis


Congressional committee. But it is alsoimportant to recognize that in trans-portation, safety is a very clear goal forwhich there are some fairly good mea-sures, even though measurement can stillbe improved. It is also a goal that is verycompelling to the public. Thus, trans-portation has the in-gredients to be suc-cessful. In some or-ganizations, wherethe outcomes arenot so easily mea-sured and not soclear or compelling,it is a little moredifficult. It is useful,however, to havemodels that dowork, and weshould focus on agencies like the De-partment of Transportation to get theprocess right and learn from that.

To do this right requires more thanjust smart planning; it requires creatinga team spirit around priorities and goals.Right now, a lot of agencies are worriedthat Congress will punish them in theappropriations process, so they are notnecessarily willing to put everything outon the table in the way that they wouldif there were a real partnership. I thinkwe have a long way to go on that, and Iam not sure how it is going to work outin GPRA, but it has the right compo-nents and the right players, with theCongress having a major role. The op-portunity is there.

3) What is necessary for public organi-zations to be accountable and to belearning organizations?

I think that in terms of organizationaldynamics, there must be teamwork,communication, and trust. People arecoming from different perspectives, andthey need to build a certain trust thatthey are working on the same agenda.You need clear goals. A lot of goals thatI see are not clear; they are very processoriented. You need goals that focus onthe end results and on the results that areimportant to the public interest. Then,you need measures that really do mea-sure effectiveness.

You need leadership that communi-cates and motivates. When you see a suc-cessful organization, you generally see

the government and the broader public.It is hard for organizations to deal withthe public; it is easier for them to workwith their direct stakeholders, those or-ganizations really interested in what theyare doing and the customers they are di-rectly serving. But if organizations lookat what the public is saying about thethings they care about, they can learnand perhaps reach out to involve the pub-lic more effectively. This year in the In-novations in American Governmentawards, we saw several examples of or-ganizations that were really engaging thepublic in setting priorities and imple-menting their programs. Their work wasimproved because of this, and I thinkthey were getting more support from thepublic and legislatures.

2) Are accountability and organizationallearning at odds?

They are very compatible, but not thesame. Accountability can be seen as mea-suring results against clearly stated goals.Organizational learning involves goalsand measurements, but as you look atwhat is happening, you are open to chang-ing the goals and priorities and adjustingthem, based on what you are learning andthe information you are seeing. Hopeful-ly, accountability encompasses organiza-tional learning but I do not think, at thispoint, that the flexibility implied by orga-nizational learning is often included inthe concept of accountability.

It is very important in a system likeGPRA to create a greater understandingand trust between managers in the gov-ernment agencies and in the Congress sothat there is an interest in learning aboutboth what is working and what is notworking and to avoid a “gotcha” mental-ity. We need to build upon what worksand try to adapt or change what is notworking. I think this is happening insome cases, such as in the transportationarea. The Department of Transportationis using the strategic planning process totry to bring the various transportationmodes together and to focus on specificgoals and measures. The authorizingcommittee is working very constructive-ly with the Department to learn from thisinformation and to use it to improve thedesign of these programs. This processhas worked because of leadership andtrust between the Department and the

that kind of leadership. In terms of in-formation management, you need to givemanagers and teams data about what isworking, in a very useable and accessibleway. Technology, particularly the Inter-net, is making that easier. It is also mak-ing what is going on a lot more transpar-

ent, not only tomanagers but alsoto the people beingserved and others.

It is very impor-tant to recognizeand celebrate suc-cess, particularly ofteams and whole or-ganizations. I tendto think that astrong focus on indi-vidual incentives, in

terms of bonuses and other financial in-centives, can be counterproductive. It isbest not to focus on an individual who istrying to show that he or she has donesomething spectacular. You want individ-uals who work effectively as members ofteams and whose goals are not individualgoals but organizational ones.

I believe that if you have a good sys-tem of measurement and have definedthe right results, you can trade flexibilityfor accountability using those measures.It is important not to micromanage andnot to second guess how teams are oper-ating, but rather to allow them to be cre-ative. In fact, it is important to encouragethem to take some risks — and even tofail, and to learn from their failure.

It is really encouraging that this sub-ject is being discussed so widely and isbecoming a fundamental part of theprocess of planning and management. Iam very encouraged by the passage ofGPRA, even though we do not yet knowhow well it will work. There is a lot ofpotential, and the fact that the public isreally demanding results helps. If legisla-tors, managers, and frontline deliverersof services can all focus on results, thenI think we will see much greater valuegiven to organizational learning. ♦

Jessica ChapelEditorial AssistantHFRP

Karen HorschResearch AssociateHFRP

To do this rightrequires more thanjust smart planning; itrequires creating ateam spirit around priorities and goals.


In response to Attorney General JanetReno’s call to “find out what worksand spread the word,” the Bureau of

Justice Assistance (BJA), a componentof the Office of Justice Programs (OJP),U.S. Department of Justice, is imple-menting a comprehensive, multi-tieredprogram evaluation strategy. This strate-gy is designed to disseminate findings oflocally produced evaluations of BJA-funded projects; stimulate program eval-uations at the state and local levelsthrough evaluation capacity building,grant programs, and technical assis-tance; strengthen BJA’s partnership withstate Administrative Agencies (SAAs),practitioners, and researchers; and builda cooperative framework for interagencycoordination.

The Initiative has six components:

Effective Programs Initiative: Underthis initiative, begun in FY 1997, BJApublishes a series highlighting innova-tive programs which have undergone in-tensive evaluation at the state and/orlocal levels. It publishes these results inan Effective Programs Monograph se-ries. The first of these, on improving thenation’s criminal justice system, waspublished in December 1997. The sec-ond in the series will focus on creating anew criminal justice system for the 21stcentury.

Tools for the Criminal Justice Com-munity: Launched in August 1998, thisinitiative provides an “ElectronicRoadmap for Evaluation,” an automatedversion of a comprehensive evaluationhandbook (Assessment and EvaluationHandbook Series) for planners, re-searchers and evaluators, and local prac-titioners. The Evaluation Technical As-sistance Web site, which is part of BJA’sHomepage (www.bja.evaluationweb-site.org), provides extensive new evalua-tion resources and materials, and is thefirst of its kind developed by a publicagency for general use.

Enhancing State and Local Assess-ment and Evaluation Capabilities:Through a federal, state, and local part-nership, assessment and evaluationhave been increasingly institutional-ized. There has been an increase in theprofessional staff assigned exclusivelyto evaluation duties (34 states now havesuch staff), yearly funds allocated toevaluation have increased over time,and the number of states funding exter-nal, independent evaluation grows eachyear.

The State Evaluation Development(SED) Program: Through this Pro-gram, BJA provides technical assis-tance to jurisdictions needing directhelp with capacity building. Evaluationtechnical assistance and training forstate and local practitioners, usually fo-cused on specific program areas, is de-livered in regional workshops, single-state settings, and multi-state work-shops. Through the establishment of astate-to-state exchange program, BJAhas supported evaluation capacitybuilding initiatives by enabling expertsfrom one jurisdiction to visit and pro-vide technical assistance to another ju-risdiction or by sending practitioners toother offices to learn its practices. SEDalso produced a series of nine publica-tions, Innovative State and Local Pro-grams, focusing on state and localstrategies to improve the ability to ob-tain, publish, and disseminate docu-mentation of promising practices. Thepublications were the result of nineworking meetings that led to the docu-mentation of 191 programs. Notably,those participating in the process haveapplied and institutionalized what theyhave learned in their own jurisdictions.As a separate activity to capture highquality programmatic information onstrategy implementation across pro-grams within specific program areas,BJA, in conjunction with state plannersand the National Institute of Justice

(NIJ), has produced a series, Highlight-ed State and Local Programs. Almost550 programs in three major programareas have been documented, and thereports are available on BJA’s Web site.

The Byrne Evaluation PartnershipProgram: To increase the quality anduse of evaluations conducted by stateand local agencies, BJA created theevaluation partnership program as partof its Edward Byrne Memorial Stateand Local Enforcement AssistanceProgram. The evaluation partnershipprogram seeks to enhance collabora-tive evaluation among SAAs and cre-ates a mechanism for enhancing thedesign, implementation, measurement,evaluation, and dissemination of infor-mation in high-priority program areas.Under this program, SAAs have princi-pal responsibility for evaluation in co-ordination with funded program man-agers and evaluators at universities orother research organizations. SAAsapply to the program either individual-ly or as coapplicants with other stateand local agencies involved in plan-ning, coordinating, administering,and/or evaluating criminal justice pro-grams funded under the Byrne Formu-la Grant Program.

Annual Criminal Justice Researchand Evaluation Conference: BJA andNIJ, along with other OJP offices, spon-sors an annual conference on researchand evaluation in the criminal justicearea. Through plenary sessions, panelsand training workshops, criminal justiceevaluators, researchers, practitioners,and policymakers come together to shareexperiences and knowledge. ♦

Robert A. KirchnerSenior Advisor for EvaluationBureau of Justice AssistanceU.S. Department of Justice

BEYOND BASIC TRAINING

Building State and Local Evaluation Capacity: The U.S. Department of Justice’s Evaluation Strategy

• capacity-building through developmentof skills and leadership of local partici-pants in rural EZ/EC communities;

• continuous improvement of the pro-gram, through strengthened citizenparticipation, feedback, and account-ability; and

• research and evaluation developmentthrough local documentation and learning.

CPC conducted a documentation ofthe pilot phase of the project (January1996 - June 1997). This documentationincluded findings related to the early im-plementation of the EZ/EC programbased on citizen monitoring and assess-ment of selected goals in the pilot com-munities, and findings related to theLearning Initiative approach.

To understand the Learning Initiative,researchers examined the extent to whichthe three broad goals were addressed.Learning Teams were asked to addressthe following questions:

1) How have you grown personally as aresult of involvement with the Learn-ing Initiative?

2) What have you learned through thisprocess related to the Citizen Learn-ing Team Model? What works andwhat doesn’t? What have you learned“about” or “how to do” while work-ing with the Learning Team in yourcommunity?

3) What have been the key findings ofyour Citizen Learning Team aboutthe EZ/EC program? How did youreach these findings? To whom didyou communicate them? What hasbeen the response? What differencehas it made so far? From the Citi-zens Learning Team perspective,what are the major successes andweaknesses of the EZ/EC programin your community?

part-time paid coordinator as well as ac-cess to the Regional Researcher. Theseteams were responsible for:

• meeting with Regional Researchers andstaff to discuss the most appropriate

ways to evaluatethe progress andsuccess of theEZ/EC program;

• gathering infor-mation locally tohelp in assessingprogress made to-ward meeting theprogram’s goalsin the communityand in identifyingsuccesses and ob-stacles to success;

• identifying lessonsfrom the commu-nity’s experiencethat should beshared with others;

• participating in workshops to share thoselessons with other communities and withprogram leaders;

• sharing the findings with others in theirown community through follow-uplearning and educational activities.

Team members decided which of theEZ/EC’s program’s local goals weremost important to them and, using aflexible evaluation tool called a “Learn-ing Wheel” (see diagram, next page),each did its own monitoring, analysis,and reporting. The Learning Teamswere expected to work with and sharetheir findings with local EZ/EC boards,while maintaining sufficient autonomyto evaluate and discuss their findingsfreely and openly.

The Learning Initiative had threebroad goals:


The Empowerment Zones and En-terprise Communities (EZ/EC)Program, established by the Clin-

ton Administration in 1993, is the mostcomprehensive federal program in recentyears aimed at relieving severe stress inrural and urbanareas. The LearningInitiative, devel-oped in 1995 andbased at the Com-munity PartnershipCenter (CPC) of theUniversity of Ten-nessee-Knoxville,was a project de-signed to pilot aparticipatory evalu-ation process in 10of the 33 rural Em-powerment Zonesand Enter pr isecommunities. Thisproject was part ofan overall move-ment toward com-munity-based mon-itoring, which wehave termed Community-based Monitor-ing, Learning, and Action (CMLA).CMLA helps people understand theviews and values they share, workthrough their differences with others, de-velop longer-term strategies, and takecarefully researched and planned actionsthat fit their contexts, priorities, andstyles of operating. It is based in the tra-ditions of community organizing, citizenmonitoring, participatory action re-search, and popular education.

The Learning Initiative was radicallydifferent from traditional evaluations thatrely on outside experts and that often ex-amine and report on the success or failureof a program very late in its implementa-tion or after its completion. The founda-tion of the Learning Initiative was com-posed of Learning Teams comprising be-tween 8 and 20 local volunteers located ineach site. Each of these Teams also had a

EVALUATIONS TO WATCH

Studying Learning Initiatives: The Experience of the EmpowermentZones/Enterprise Communities National Learning Initiative

CMLA helps peopleunderstand the viewsand values they share,work through their differences with others,develop longer-termstrategies, and takecarefully researchedand planned actionsthat fit their contexts,priorities, and styles of operating.

4) What would you recommend to feder-al, state, and local governments aboutthe EZ/EC program? What would yourecommend to state and local govern-ments about the Citizen LearningTeam Model? What would you rec-ommend to someone else who wishedto start a Learning Team in their com-munity? What would you recommendto the University about the CitizenLearning Team Model for Phase II?

In assessing the capacity-building ac-tivities of the Learning Initiative, re-searchers looked at skills that had beendeveloped among team members; newand expanded networks that had beencreated; the extent to which team mem-bers had broader and more criticalawareness of the political dynamics and

culture of their communities; how confi-dence among members had been built;and changes in the ability of members totake on new roles. To understand the im-pact that the Learning Teams were hav-ing (much of this was too early to assess),researchers documented early policychanges, the role of Learning Teams inmaintaining citizen participation andpublic accountability, the effectivenessof different structures, and the LearningInitiatives’s contribution to continuouslearning and improvement at the nation-al level. Finally, to assess the influenceon research and evaluation, researchersdocumented noteworthy team accom-plishments and innovations in research,lessons on research design and approach-es to measuring and monitoring, andlinkages with the Regional Researchers.

Researchers used a variety of datasources to understand the Learning Ini-tiative. These included documents, inter-views, and participant observation. Themain source of data for the analysis of thecapacity-building component of theLearning Initiative were the LearningTeams themselves. Learning Teams pro-duced and presented reports during thethird cross-site meeting, which includedanswers to the questions given earlier inthis article. In addition, Regional Re-searchers worked with the LearningTeams to develop qualitative case studiesthat provided in-depth accounts of theearly implementation and progress of theEZ/EC program and the Learning Initia-tive. Two national cross-site sessions wereheld in which participants heard and ana-lyzed case study presentations from eachsite and worked together to identify keythemes, findings, and recommendations.Participants included Learning Team co-ordinators and team members from eachcommunity, Regional Researchers, FordFoundation staff, USDA Policy and Plan-ning staff, consultants, and CPC Learn-ing Initiative staff.

Although the EZ/EC Learning Initia-tive pilot project has been completed, theLearning Teams have continued on a vol-untary basis in some communities. Aspoor people feel the pinch of policychanges and budget cuts, their desire tolearn what can be done opens manydoors for enlarging community basedconstituencies engaged in participatorylearning and education. Understandinghow activities such as the EZ/EC Learn-ing Initiative can help develop such ca-pacity is critical to ensuring that voice isgiven to those traditionally excludedfrom the policy process. ♦

Andy MottExecutive DirectorCenter for Community Change

Vicki CreedLearning Partners

John Gaventa, Fellow at the Institute ofDevelopment Studies, University of Sus-sex, was the founder and initial directorof the Learning Initiative. Vicki Creedfollowed John Gaventa as director of theLearning Initiative; she also has her ownconsultant practice, Learning Partners,in Knoxville, Tennessee.


Form a Learning Team

How will we work together?Build a team.

What are our goals?

Listen to thecommunity.

What do we need to learn

and why?Choose priorities for

monitoring and measuring.Identify indicators

of success.

Who will do what and when?Create a plan.

Collect information.

What are we learning and what

does it mean?Analyze and assess.

What differences have we made?

Apply the Learning Cycleto ourselves and

our work.

How do we celebrate our victories?

Recognize, honor, play.

What next?Reflect, plan, act.

Phases for

Learning Teams

How will we find out about what we need to learn?Develop ways to collect

information.Refine our methods through field tests.

How do we make changes with what

we’ve learned?Develop a strategy.

Take action.


The Center for Law and Social Policy(CLASP) and the Center on Budgetand Policy Priorities (CBPP) under-

took the State Policy Documentation Pro-ject (SPDP) in 1997 to document and ana-lyze policy choices made by the states intheir TANF cash assistance, food stamp,and Medicaid programs. Enactment of thePersonal Responsibility and Work Oppor-tunity Reconciliation Act (PRWORA) of1996 and the Balanced Budget Act of1997, particularly the creation of the Tem-porary Assistance for Needy Families(TANF) block grant, significantly alteredthe relationship between the federal andstate governments in providing assistanceto needy families. The federal legislationboth offers substantial new discretion forstates and imposes new requirements. Thegoal of the SPDP is to provide a detailedand accurate source of state policy infor-mation on these programs for advocates,policymakers, and researchers. The projectwill collect and disseminate data on writ-ten policy (statute, regulation, and policymanual), rather than on how policy is im-plemented or its impacts.

In the project’s first year, staff andCLASP and CBPP developed five surveys,three on TANF policies, one on Medicaid,and one on food stamps, with a total ofmore than 1,000 questions. Most questionswere framed with multiple choice or yes/noanswers to facilitate comparison of policiesacross states. The surveys also includeopen-ended questions to allow for full ex-planation of state policies. The survey in-struments contain substantial explanatorytext to ensure that all respondents fully un-derstand the questions. The SPDP providesmore detail than other sources of informa-tion on state policy choices, because initial“threshold” questions are followed by a se-ries of follow-up questions. For example,in addition to listing the circumstancesunder which a state extends cash assistanceto families that have reached a time limit,the SPDP survey asks whether the durationand renewability of extensions vary by cir-cumstance, and if there are any circum-stances under which families can be dis-

qualified from receiving extensions.The topics in the TANF survey include

application issues, categorical eligibility,financial eligibility, time limits, work ac-tivities and requirements, sanction provi-sions, child care, child support, programadministration, entitlement, housing assis-tance interaction, and behavioral require-ments. There is a separate survey on TANFpolicies with implications for reproductivehealth and teen parents, such as familycaps, minor living arrangements, rules,family planning, abstinence education, andteen school and training requirements.

The food stamps survey topics in-clude application procedures, financialand categorical eligibility, immigrants’eligibility, sanctions for non-compliancewith work requirements, collections ofover issuances, the three-month timelimit, and employment and training ac-tivities. The Medicaid survey topics arethe new eligibility category for familieswith children, family composition rules,transitional Medicaid assistance, sanc-tions, time limits, effects of TANF pro-cedures on Medicaid, Medicaid for SSIchildren, and Medicaid for immigrants.

The volume of data SPDP is collectingis large. For example, the survey section onTANF applications contains subsectionson policies on availability and processingof applications; the extent to which Medic-aid, food stamps, and other benefits’ appli-cations are linked with cash assistance ap-plications; requirements imposed on appli-cants; rules on personal responsibilitycontracts and employability plans appli-cants must sign; programs that divert ap-plicants from cash assistance with one-time payments or other services; and theavailability of emergency assistance.

In order to collect the data, SPDP con-tracted with “reporters,” technically andsubstantively proficient policy analysts, inthe 50 states and the District of Columbia.Each reporter completes the surveys; SPDPthen compiles and cleans the data, andsends reports on the survey answers to stateofficials at the TANF, Medicaid, or foodstamps agency for confirmation. The final

step is to resolve any discrepancies in inter-pretation of written policy between the sur-vey respondents and state officials. In thesecases, the written policy is the final author-ity. SPDP data are stored in databases, andan interface was developed that allows re-porters to enter survey answers directly intothe databases via the Internet.

Information on these state policies is inhigh demand. The scope of issue areascovered and the level of the detail of the in-formation from the SPDP surveys are notavailable from other sources. For example,state-level advocates and policymakerslearn about innovative policies being pro-posed and implemented in other states andare able to see how their policies compareto those across the nation. Researchersevaluating welfare related programs, par-ticularly those looking at sites in differentstates, will use the policy information toinform their evaluation findings. Membersof the media at both the national and statelevels frequently request information fromSPDP for stories on welfare reform policytrends. To further disseminate its informa-tion, SPDP is developing a Web site. SPDPreports will be advertised as they becomeavailable through networks of state re-porters and partner organizations, throughmailings to state officials, and by postingnotices to electronic listservs for state andfederal policymakers, researchers, policyanalysts, and advocates.

SPDP is designing database reports forindividual state summaries of the policiesin each of the survey areas; state reportswill be posted to the Web site individuallyas data are verified. Eventually, 50-statetables will be produced to allow for com-parison of policies around the country.While making the data available willallow others to do state-specific, regional,and national analyses, SPDP staff willwrite and distribute issue-specific sum-maries based on the full range of issuescovered by the five surveys. ♦

Lisa PlimptonResearch AssociateCLASP

SPOTLIGHT

State Policy Documentation Project: Data Collection and Dissemination

The ill-defined, trivial, and over-generalized ways in which thephrase “lessons learned” has

come to be used challenges evaluators todistinguish between casual/informal no-tions of lessons learned and what we call“high quality” lessons learned.

High-quality lessons learned containknowledge that can be applied to futureaction and can be derived from screen-ing according to specific criteria:

1) practice wisdom and experience ofpractitioners;

2) experience from program partici-pants/clients/intended beneficiaries;

3) evaluation findings – patterns acrossprograms;

4) basic and applied research;

5) expert opinion;

6) cross-disciplinary connections andpatterns;

7) assessment of the importance of thelesson learned; and

8) strength of the connection to outcomeattainment.

The idea is that the greater the num-ber of supporting sources for a lessonlearned, the more rigorous the support-ing evidence, and the greater the trian-gulation of supporting sources, the moreconfidence one has in the significanceand meaningfulness of a lesson learned.Lessons learned with only one type ofsupporting evidence would be consid-ered a “lessons learned hypothesis.”Nested within and cross-referenced tolessons learned will be the actual casesfrom which practice wisdom and evalua-tion findings have been drawn. A criticalprinciple here is to maintain the contex-tual frame for lessons learned, that is, tokeep lessons learned grounded in theircontext. For ongoing learning, the trickis to follow future supposed applicationsof lessons learned to test their wisdom inaction.

Michael Quinn PattonThe Union InstituteUtilization-Focused Evaluation

Ricardo A. MillettDirector of EvaluationW.K. Kellogg Foundation


Sources:

Darman, Richard. (December 1,1996). Riverboat gambling. NewYork Times Magazine. 116-117.

Garvin, David A. (July-August 1993).Building a learning organization.Harvard Business Review. 78-91.

Gray, Sandra Trice and Associates.(1998). Evaluation with power. Wash-ington, DC: Independent Sector.

Levitan, Sar A. (June 1992). Evalu-ation of federal social programs: Anuncertain impact. Occasional Paper1992-2, June 1992. Washington,DC: Center for Social Policy Stud-ies, George Washington University.

Usher, Charles L. (1993). Buildingcapacity for self-evaluation in familyand children’s services reform efforts.American Evaluation Association

will better inform and guide public ac-tion: collaborative buy-in; a transparentand public system of performance mea-surement and accountability; opportu-nity for innovation, rigorous experi-mentation, and program and policy re-design; and processes and mechanismsto share lessons and learning. This ap-proach has the potential to produce in-formation that is more likely to be usedfor public policy decisions and re-source allocation as well as to improveprogram and policy outcomes than thecurrent system of episodic evaluationand experimentation now does. In theera of accountability, we believe thatthis approach has the potential to en-gage all the necessary players from thenational to the community level in thelearning process — and thereby createa learning community. ♦

Heather B. WeissDirector, HFRP

William A. MorrillSenior FellowMathtech, Inc.

The Harvard Family Research Project, locatedat the Harvard Graduate School of Education,anticipates a job opening for a full-time Re-search Analyst starting in July/August 1999.This is an excellent opportunity for a profes-sional interested in contributing to the field ofevaluation and after-school programs.

Responsibilities: Work on several evaluation-related research projects, including projectsfocused on after-school programs and results-based accountability among non-profits. Mayalso work on national evaluation newsletter,the Evaluation Exchange. Conduct researchand collect/analyze data using qualitative andquantitative methods. Conduct extensive in-terviews with national and community-basedorganizations, advocacy and research organi-zations, state and local policymakers, founda-tion representatives, and other stakeholders.Analyze documents and assist with develop-ment of interview protocols and codingschemes. Write reports, journal articles, ana-

lytic documents, and meeting presentations.Conduct literature reviews. Participate in pro-posal development; assist with other specialprojects. Occasional travel required.

Requirements: Master’s degree, 3+ years re-lated research/evaluation experience. Knowl-edge of/experience with current evaluationtheory/practice, public policy, and/or educationrequired. Knowledge of after-school programspreferred. Excellent writing and verbal com-munication skills, and ability to work indepen-dently essential.

To Apply: Send a cover letter and resume to:Tacy Corson, Associate Director for Adminis-tration, Harvard Family Research Project, 38Concord Avenue, Cambridge, MA 02138.

Note: This position is contingent on grantfunding.

Harvard University is an Equal OpportunityEmployer.

Continued from page 4SPOTLIGHT

Lessons Learned

JOB OPENING:


Argyris, Chris. (1993). Knowledge foraction: A guide to overcoming barriersto organizational change. San Fran-cisco, CA: Jossey-Bass. This bookhelps readers understand why individu-als and organizations are unable tolearn from their action, and presentsthe steps that must be taken to createchange. Part One discusses how to un-cover the roadblocks to improvement;Part Two suggests ways to diagnose andintervene to create learning; and PartThree discusses how to use key learn-ings to solve problem situations. Tel: (415) 433-1740. http://www.josseybass.com/catalog

Argyris, Chris & Schön, Donald A.(1996). Organizational learning II:Theory, method, and practice. Reading,MA: Addison-Wesley Publishing Co.This book expands on the ideas and con-cepts of the authors’ ground-breakingfirst book, Organizational learning: Atheory of action perspective (1978). Theauthors address four principal questionsthat cut across the field of organizationallearning: Why is an organization a learn-ing venue; are real-world organizationscapable of learning; what kinds of learn-ing are desirable; and how can organiza-tions develop their capacity for desirablekinds of learning? Tel: (800) 447-2226.http://www.awl.com

Greenwood, Davydd J. & Levin,Morten. (1998). Introduction to actionresearch: Democratizing the researchprocess. Thousand Oaks, CA: SagePublications. This publication providesboth new and experienced researchersand practitioners with the informationand skills necessary to conduct actionresearch. It begins with a brief presenta-tion of two cases of action research andcontinues with a discussion of the histo-ry, philosophy, social change agenda,methodologies, ethical arguments for,

and fieldwork tools of action research.Tel: (805) 499-9774. http://www.sagepub.com

Hess, Peg M. & Mullen, Edward J.(Eds.) (1995). Practitioner-researcherpartnerships: Building knowledgefrom, in, and for practice. Washing-ton, DC: National Association of So-cial Workers Press. The authors in thispublication discuss ways to forge moresuccessful and effective partnershipsbetween the worlds of social work prac-tice and social work research. Using di-verse examples, they discuss the crucialcharacteristics that contribute to thesurvival and enhancement of collabora-tive relationships as well as key issuesand obstacles. The book captures andreflects the complexities of actual part-nerships in research and practice. Tel: (800) 227-3590. http://www.naswpress.org/publications.

Senge, Peter M. The fifth discipline:The art and practice of the learningorganization. (1990). New York: Dou-bleday. Senge describes the five disci-plines that are necessary for a learningorganization: systems thinking; per-sonal mastery; mental models; buildingshared vision; and team learning. Usingexamples, Senge discusses how totransform organizations into learningorganizations.

Senge, Peter, Kleiner, Art, Roberts,Charlotte, Ross, Richard B., & Smith,Bryan J. (1994). The fifth disciplinefieldbook: Strategies and tools for build-ing a learning organization. New York:Doubleday. The authors provide theviewpoints for implementation and de-velopment of the ideas presented in theFifth discipline. Included are exercisesthat promote the development of each ofthe five disciplines. http://www.fieldbook.com ♦


NEW AND NOTEWORTHY

We regret that we cannot providecopies of the materials listed below.Please contact the publisher or authoring organization directly.

ANNOUNCING: RESULTS-BASED ACCOUNTABILITYPUBLICATIONS FROM HFRP

Results-Based Accountability CaseStudiesNew evidence from recent studies byHFRP shows that RBA can be a pow-erful tool to initiate and enhance sys-tems of continuous improvement andlearning. These case studies describeRBA efforts in eight states: Florida,Georgia, Iowa, Minnesota, North Car-olina, Ohio, Oregon, and Vermont.Each case study identifies the processof designing and implementing RBAsystems. Every report includes a de-scription of state and local RBA efforts,their history, the development process,the uses of the RBA systems, and thelessons learned. The unique experi-ences of the eight states in the studiescan provide valuable insights to thoseresponsible for RBA efforts elsewherein the nation.

Aiming for Accountability: LessonsLearned from Eight StatesThis report highlights some of the im-portant lessons in designing and de-veloping RBA based on the insightsgained from the studies of eight states.Included is information about howthese states have overcome some ofthe challenges in developing effectiveRBA systems, and what the character-istics of promising RBA efforts are. Theexecutive summary of this report is anew addition to our collection ofReaching Results Briefs, and discuss-es the potential and limitations of RBAsystems based on HFRP’s eight casestudies.

Each of the state reports is $5.00plus shipping and handling, andLessons Learned from Eight Statesis $7.00 plus shipping and handling.A special discount is available if or-dering all eight of the state reportsor all nine in the series. To obtainthese publications, please contactthe HFRP publications departmentat (617) 496-4304 or look at our Website for an order form. You may alsoe-mail your order: [email protected]

The Fifth Discipline Fieldbookhttp://www.fieldbook.comThis site offers strategies, tools, and con-nections for building a learning organi-zation. Visitors can learn more about theFieldbook and download or read unpub-lished material or segments from futureFieldbooks. The site also provides infor-mation on study groups, Internet links,and events and talks related to learningorganizations.

Stanford Learning Organization Web (SLOW)http://www.stanford.edu/group/SLOWSLOW is an informal network of indi-viduals interested in the nature and de-velopment of learning organizations.This site offers connections to other Websites, information about articles, books,videos, and papers related to learning or-ganizations.

Learning Organization Mailing List and Archivehttp://world.std.com/~lo/LOinfo.htmlThis international electronic discussiongroup brings together those working tobuild learning organizations. To sub-scribe, send e-mail to [email protected]. Ignore the subject lineand begin the message with two lines:subscribe learning-orgend

The Learning Organizations Home-pagehttp://www.albany.edu/~k17686/learnorg.htmlThis site provides information on learn-ing organization strategies for innova-tion, change, and growth as well as on-line articles and information about re-sources.


Help us keep intouch with you!

Please make sure we have your

most recentmailing address.

Also, please sendus the names and addresses

of any colleaguesor friends who

should be on ourmailing list.

Nonprofit Org.U.S. Postage

PAIDBoston, MAPermit No.

57868HARVARD FAMILY RESEARCH PROJECT

Harvard Graduate School of Education38 Concord AvenueCambridge, MA 02138

THE EVALUATION EXCHANGE: Emerging Strategies in Evaluating Child and Family Services

ELECTRONIC MAILBOX

volume iv, number 3/4 1998 free quarterly newsletter from the … · 2017-09-01 · mastery, mental...

Documents