differencemaking

a

A toolkit to help local authoritiesdemonstrate where and howElected Member Developmenthas community impact

The toolkit supports submission for“Level Two of the North West Charterfor Elected Member Development”

The North West has an excellent reputation,both regionally and nationally, for ElectedMember development and the North WestCharter for Member development has beena key part of the regional strategy. The NorthWest Charter was developed by a smallgroup of members appointed to North WestEmployers.

The Regional Minister, Beverley Hughes MP,has supported the annual celebration eventfor the last four years which is recognition ofthe importance of the Charter, and has said:

“I think, firstly, it (the Charter) hascompletely changed the perception thatexisted hitherto about the importance ofmember development. When I first becamea councillor in 1986 that was not on theagenda at all and you really didn’t get anyinduction or very much, let alone continuoustraining, so it’s completely transformed theunderstanding about the importance of that.

And, secondly of course, I think particularlyhere in the North West the Employers’Organisation has shown a really importantlead in how best to do that because of theprogrammes that they have incepted hereand I think it has been a template that therest of the country is following.”

Beverley Hughes, Minister of State forChildren, Young People and Families andMinister for the North West

The Charter begins with the followingstatement “At the heart of any local authoritydedicated to meeting the needs of its

community there needs to be a commitmentto the development of its Elected Members“and this toolkit addresses the very difficultquestion of “What impact does Memberdevelopment have on communities?”

Evaluating impact is often left on the “hard todo pile” and this toolkit provides a very timelyand useful guide to help authorities reviewtheir practice. This toolkit has been producedspecifically to support authorities that chooseto have their Charter status reviewed at theLevel Two exemplar status, but the principleswill be useful for any authority interested inlooking at the impact of their memberdevelopment work.

As a region we have continued to raise thebar and, working in partnership with theNorth West Improvement and EfficiencyPartnership (NWIEP), we have developed anElected Member Gateway to provide supportto authorities and share learning. This toolkitis an important part of developing modelsand real life examples of evaluating theimpact of development.

Our approach has been to work alongsideauthorities as they have developed theiranalytical thinking and asked themselvesdifficult questions about the impact of theirwork, and particular thanks go to Blackpool,Halton, Lancashire Councils and Lake DistrictNational Park Authority who have been ourpathfinder partners.

The Charter was developed ‘for and by’Members and North West Employers’Members have continued to shape and leadthis work.

“I was involved in the very first CharterAssessment in 2000 and I am proud that theNorth West Charter has continued to flourishand, more importantly, we as Members havecontinued to improve the process through

working with, and listening to, authorities.The new Level Two Review Process has beendeveloped in response to the conversationswe had with authorities who wanted a reviewwhich was challenging and ambitious and asa Member I can’t think of anything moreimportant than making sure the work we doand the investment we make in developmentis benefiting the people in our communities.As a result of the work we have donecollectively over the past eight years, Ibelieve our Members across the North Westare now much better equipped to deal withtheir varied roles and are more confident todeal with the many challenges that the roleof Councillor presents them within the everchanging and challenging environment oflocal government in the 21st century.”

Cllr Alan Dean, Chair of North WestEmployers.

“Members have often been asked whatdifference their development programmeshave made in actually servicing theircommunities. Has the developmentprogramme really made a difference? Thisgap is now filled with the evaluation toolkit.Real life examples that memberdevelopment has made can now be shared.And the benefit of the investment in time andresources laid open for all to see.”

Councillor Tim Stoddard, Chair of NWIEP

Our thanks to Liz Richardson, ResearchFellow from the Centre for Local Governanceat Manchester University, who wrote thetoolkit and brought her insight and academicrigour to the work.

Liz McQueChief Executive

North West EmployersNovember 2008

foreword

1

evaluationtoolkit

Why read this toolkit?Background on the North West Member Development Charter

Development Charter Review Process at Level Two

What this toolkit is about?

How this toolkit relates to the North West Charter Review

Process at Level Two

Making impact evaluation do-ableDon’t be scared of impact evaluation

Evaluation is part of celebration of achievement

Better to do something than not do evaluation at all

How much effort and time will evaluation take?

The Charter Level Two model

Three practical lessons

Practical Lesson No 1. Think differently about‘evidence’ and how to get itWhat is meant by evidence?

Being careful with statistics

Fitting evidence gathering into busy lives

3

6

7

8

10

11

12

5

4

Practical Lesson No 2. Don’t go it alone –sources for evidenceMembers’ own experiences

Material and information from elsewhere in the organisation

Testimony from partner organisations and external stakeholders

Member Development Officers’ own knowledge –

uncovering hidden evidence

Sample list of questions and topics for a ‘Research Interview’

Practical Lesson No 3. Tell the story fromthe beginningUse the Level Two application to tell the story of your success

Getting to the happy ending

Happy endings for whom? Authority or community level impacts?

Tell the story from start to finish: outcomes must link to

strategic priorities

Linking Member development to Strategic Priorities‘Suck it and see’ – evaluation where the initial development

needs, aims or objectives are not clear

Conclusion and checklist

14

15

16

17

18

20

22

23

24

28

31

26

2

Background on the North WestMember Development CharterIn 2000 North West Employers established theNorth West Charter to raise the profile of Memberdevelopment in the North West and reward thoseauthorities that had demonstrated theircommitment. The North West MemberDevelopment Charter at Level One is now wellestablished across the region and 38 of the 46authorities in the North West have been awardedthe Charter, which equates to 82% of the wholeregion.

Members and Officers are keen to getaccreditation, and appreciate the framework itoffers for thinking about Member development.

The Charter at Level One covers the essentials:commitment to Member development at thehighest levels; development of flexible learningand development options; tailoring to individualneeds; widest possible access to (and take up of)development opportunities; and having theresources in place.

The Charter at Level One sets out six practicepoints which an authority has to put in placebefore the Charter can be awarded:

• Having a statement of intent.• Ensuring that all Members are made aware oftraining and development opportunities.

• Having a process to identify individualdevelopment needs which involves Members.

• Having an Officer allocated to assist Membersand groups in identifying needs and providinginformation on resources.

• Having a strategy to meet the training anddevelopment needs of Elected Members.

• Implementing the strategy locally.

More information can be found in Guidance NotesFor Local Authorities Working Towards AchievingThe North West Charter for Elected MemberDevelopment, available on North West Employerswebsite at:

www.nweo.org.uk//ElectedMembers/MemberCharter/Sign+Up+and+Contact

Why read this toolkit?

3

To maintain the Charter standards, authorities are reviewedafter three years and demonstrate that they are still committedto Member development and still meet the six principles.In addition to confirming that authorities are still on track,authorities also expressed a desire to have a Review Processthat was stretching and that recognised where authorities hadsignificantly raised their game. As a result a new exemplarstandard has been introduced to complement the existingstandard.

Authorities now have the choice to be reassessed either atLevel One, against the original standards of the Charter, orat Level Two, the ‘exemplar’ level. To achieve Level Two,authorities must first provide evidence in two areas: ‘impact’and ‘commitment to improvement’. More information and anapplication form can be found in North West Charter forElected Member Development Charter Review Process,available at:

www.nweo.org.uk/ElectedMembers/MemberCharter/Sign+Up+and+Contact

Review Processat Level Two

4

Technical Note 1. The what of evaluation: frombasic to ‘gold standard’

One of the best known models for learning anddevelopment evaluation has been designed by DonaldKirkpatrick. His model has the four levels of learningevaluation:

Level 1. Reaction of student - what they thought and feltabout the training.

Level 2. Learning - the resulting increase in knowledge orcapability.

Level 3. Behaviour - extent of behaviour and capabilityimprovement and implementation/application.

Level 4. Results - the effects on the business orenvironment resulting from the trainee's performance.

Some people now add a fifth level:

Level 5. Return on investment - which looks at thefinancial payback of learning.

The Kirkpatrick model is very useful for picking out whattypes of impact you are assessing. Kirkpatrick Level 1 isseen as the most basic and might include ‘happy sheet’feedback forms. Kirkpatrick Levels 4 and 5 are seen asthe ‘gold standard’ of evaluation.

The North West Charter Level One assesses Memberdevelopment up to Kirkpatrick Level 3.

The North West Charter Level Two is at Level 4 in theKirkpatrick model. Therefore this toolkit is designed tohelp people tackle evaluation at Kirkpatrick Level 4.

What this toolkitis aboutThe Level Two Charter Review Process poses a keyquestion:

“What impact does Member development have oncommunities?”

Local Councillors become involved and devote many longhours in order to represent their areas. Ultimately, thisrepresentation should contribute to making things betterfor the people in those areas. Member development addsto the responsibilities and hours that Members put in,therefore it should also add to their ability to representtheir areas. Showing that Member development hascreated benefits or real change for communities istherefore extremely important.

This toolkit has been produced because:

• People can often forget that community impact is theultimate reason they do Member development.

• Showing the link between Member development andcommunity impact can feel extremely difficult to do.

This toolkit gives practical advice about how to show thislink. It is based on the real-life experiences of five NorthWest local authorities, backed up by relevant material.

The toolkit includes:

• Guidance to explain the process• Technical Notes- providing detailed information forofficers and member thinking about how they evaluatetheir work.

• Case study examples – to give you real life examples

5

How this toolkit relatesto the North West Charter ReviewProcess at Level TwoThe Review process at Level Two first requiresauthorities to re-affirm their progress againstLevel One standards. Level Two then has twosections:

Section One: The impact of Memberdevelopment on the work of the local authorityand community.

Section Two: That the authority is committed tocontinuous improvement in Memberdevelopment.

This toolkit only looks at Section One: Theimpact of Member development on the work ofthe local authority. Authorities should completethe application form using the Guidance.

This toolkit is designed to complement theapplication form guidelines. It looks at howpeople can pull together evidence to answer thequestions on the form to meet the criteria. Itoffers guidance on how best to present the workdone in Member development to achieve theexemplar Level Two standard.

6

Don’t be scared of impactevaluation

If you are nervous about communityimpact evaluation then you are notalone. One Member DevelopmentOfficer from a North West authoritytold us:

“When I read through the booklet itlooks simple enough, but I thoughtabout how to evidence everything,and how to link everything back tostrategic priorities and I was mindboggled. I haven’t got the foggiest.”

Evaluation is part of celebration ofachievement

Examples of how Memberdevelopment has had real benefitsfor communities are part of a recordof achievement for Members and theauthority. If we want to celebratesuccess and publicise achievement,then you can make evaluation usefulfor these purposes. Impactevaluation then becomes a way toanswer the question: “what have youdone that you are proud of? “

The process of doing impactevaluation can be rewarding forthose involved: “We’ve done somuch, we don’t realise.” (MemberDevelopment Officer, North Westauthority)

Better to do something than not doevaluation at all

A question that is often asked aboutlearning and development iswhether it offers value for money(Level Five in the Kirkpatrick Model.)Nearly all of the authorities wetalked to shied away from this valuefor money question. For example,one authority said their MemberDevelopment Steering Group hadchosen “not to go that far” onevaluation. However, the sameauthority continued by showing acanny awareness of costs, hadcompared costs to other forms ofprovision and explained how theycould sustain the developmentactivity in the most cost-effectiveway.

Value for money (VFM) is a classicarea where people are nervous ofsaying something because they feel

it is not up to the right standard.Where authorities try for extremelyhigh level evaluation they can end updoing nothing, rather than doingsomething at a lower level that isstill useful. A VFM assessment doesnot necessarily need to be a fullcost-benefit analysis. Explaininghow a community enterpriseprovided refreshments for a learningevent at a lower cost thancommercial or in-house caterers isa good enough start, and shows aspin-off community benefit ofMember development.

Sometimes ‘the perfect is the enemyof the good’. This is shown in thisquote from an authority struggling tostart their Level Two application:

“Previously, we looked for really bigimpacts, but we just need a seriesof individual examples.” (Officer,North West authority)

North West Member DevelopmentCharter Review Process at Level Twois a chance to try something out inorder to improve. The message isthat doing something is better thannot doing anything when it comes toimpact evaluation.

Making impact evaluation do-able

7

Technical Note 2. The how ofevaluation: from basic to ‘goldstandard’

The Kirkpatrick model explains what youevaluate at different levels, but how do youdo the evaluation? There are also differentlevels in how evaluation is done, from themost basic to the most sophisticated. Thisis sometimes known as a hierarchy ofevaluation methodologies.

Here is a simple guide to the differentlevels of evaluation methods, starting withthe most basic:

• After the event – a one-off description orassessment of what happened as a resultof training.

• Before and after – where the situationbefore the learning is measured, and thenmeasured against after the learning, andthe results compared to assess theimpacts.

• Comparison – where Members who havechosen to do learning and developmentare compared against those who did notparticipate either from the same group,or a similar group. This could takeadvantage of situations where an issueaffects a large number of Members, butnot all Members affected could beinvolved in learning about that issuefor whatever reason. The similar groupmay also be from outside the authority.Sometimes known as a quasi-experimental method.

• Experimental – where Members arerandomly selected to do a learning anddevelopment activity, and the results arecompared against Members who wererandomly selected not to do the activity.

‘After the event’ evaluation is usually seenby evaluators as the most basic level, andexperimental methods are seen as the ‘goldstandard’. The North West MemberDevelopment Charter Review Process atLevel Two is a ‘before and after’ evaluation.This toolkit focuses on ‘before and after’evaluation methods.

How much effort and time willevaluation take?

The idea behind this toolkit is to offerpeople do-able and practical solutions thatthey can achieve within already busyworkloads. Proving the community impactsof Member development potentially couldbe a huge undertaking. But you do notneed to do the ‘Rolls Royce’ version to beable to achieve Level Two. This toolkitadvises on quick, simple, low-cost ways togather evidence that can be integrated intoactivities that are already happening. Youdo not need a piece of research costinghundreds of thousands of pounds to applyfor Level Two.

The Charter Level Two model

The North West Charter Review at LevelTwo has been built from the experience andknowledge of the many practitioners in theregion working in the learning anddevelopment field, and shaped throughconversations with Members and Officersfrom authorities that have successfully metthe standards set out in the North WestCharter. In developing the Level Two NorthWest Employers looked to the very best inclass and acknowledge the ideas takenfrom the tried and tested approach used inthe National Training Awards.

The National Training Awards aresponsored by City and Guilds, Learning andSkills Councils (LSC), and CharteredInstitute of Personnel and Development(CIPD). They are a well-regarded nationalaward dedicated to raising standards intraining and development across allsectors. This includes private sectororganisations, public bodies as well asvoluntary organisations.

Winners of National Training Awardsinclude large companies such as BMW,Debenhams and Intercontinental HotelsGroup, small family-owned businesses, andpublic sector organisations including theBritish Army. As one Elected Memberpointed out to us, many different sorts oforganisations understand the importance oftraining and development:

“I was in the army and we did training fromthe first day until the last.”

More information is available at:http://www.nationaltrainingawards.com/

8

Three Practical Lessons9

The rest of this toolkit looks atthe three practical lessons welearnt from talking to NorthWest authorities:

• Think differently aboutevidence.

• Don’t go it alone.• Tell the story from thebeginning.

Practical Lesson No. 1 looks atwhat counts as evidence andways to collect evidence.Practical Lesson No. 2 looks atwhere these types of evidencecould be sourced. PracticalLesson No. 3 then looks moreclosely at what is meant by‘community level impact’, andhow to put the evidence togetherinto a good application.

Practical Lesson No 1.Think differently about evidence and how to get itPractical Lesson No 1 looks at what countsas evidence and ways to collect evidence.

What is meant by evidence?

To be awarded the Level Two Charter theapplication must provide convincingevidence that “your investment in Memberdevelopment had had a direct impact onthe work of the local authority andcommunity” (North West Charter on ElectedMember Development Charter ReviewProcess. The guidance notes ask authoritiesto show a link between impacts andMember development interventions chosenusing a wide range of different sources ofevidence.

A successful application must containappropriate evidence in support of anyclaims as to the impact and benefits of thelearning. A range of different types offactual evidence to strengthen and bringyour entry to life should be shown.One type of factual evidence is statisticalevidence, e.g. measuring performanceoffers robust evidence of impact.

However, factual evidence does not simplymean statistics and figures. For somethingto count as evidence, it does not need tohave been counted. Factual evidence alsoincludes:

• Examples or case studies of impacts oractivities now being carried out moreeffectively.

• Observations of changes that aredocumented.

• Documentary evidence, such ascommittee minutes, memoranda, orexternal assessment reports.

• Quotes from named individuals who cancompare the before and after situation,and focus on the benefits,e.g. from Members, Officers, residents,local interest groups, or outside agencies.

• Testimony from outside named people –partner organisations, local press,external stakeholders, regulators, serviceusers, constituents, etc. – commenting onthe improvements and benefits, perhapsby comparing other similar organisationsor individuals.

• Other awards or accreditations which linkdirectly to the learning described withinthe entry, such as Investors in People,Charter Mark.

Statistics seem to some people to be the‘best’ and most robust type of evidence.This is not always the case.For example, some statistics are based onlarge numbers of people surveyed, butthey are still based on people’s opinionsor perceptions of outcomes, rather than‘objective’ evidence of the outcomesthemselves. Even where objectivemeasures are used, it often takes asophisticated evaluation to show thatlearning and development ‘caused’ ameasurable impact as seen in the figures.The use of case studies and so on canclarify the process by which Memberdevelopment contributed to creatingcommunity outcomes, in a way that figuresalone cannot.

Using many different types of evidence andsources of evidence is also known astriangulation, explained in Technical Note 3.

10

Technical Note 3. Triangulation

Triangulation means bringing together different types ofdata, or sometimes different ways of looking at data, toanswer a research question. In essence it means usingdifferent ways to find out answers to the same question.This is in order to both verify your information, and to addrichness, depth and breadthto evidence.

There are four different types of triangulation:methodological triangulation, data triangulation, investigatortriangulation and theory triangulation.

• Methodological triangulation means combining differentresearch methods.

• Data triangulation means combining data from more thanone source.

• Investigator or analyst triangulation involves more than oneresearcher looking at the data.

• Theory triangulation means looking at the data fromdifferent theoretical positions.

Being careful with statistics

Consider the following examples:

Fact 1 Elected Members in Blackpool have undertakensuccessful training on licensing due to the Licensing Act 2003transferring responsibility for liquor licensing from theMagistrates’ Court to local authorities.

Fact 2 100% of licensing Panel hearings have been heard anddetermined within the statutory time limits. There have been lessthan 1.5% appeals.

Fact 3 The Blackpool Evening Gazette reported that alcohol-related violence in the town centre was down by 40% (January2007). The number of drink-fuelled violent incidents was downfrom 247 in Dec 2005 to 153 in Dec 2006.

It seems credible that Facts 1 and 2 are related – training enableda smooth process in the hearings. But when we get to Fact 3, it isnot immediately convincing that the licensing training and thereduction in incidents are linked, or that one thing led to the other.There may have been several other interventions taking place thatcontributed to a reduction in drink-related violence in Blackpool,for example more targeted policing, or a safety campaign. Thereduction may simply have happened naturally, e.g. due to a fall invisitors to the town centre.

Therefore, although in principle statistical evidence is the mostrobust, extreme caution must be shown when using figures toprove that Member development ‘caused’ an improvement in thecommunity.

Using other sorts of evidence, such as examples or case studies,can help demonstrate how one thing led to another in a way thatstatistics cannot. It can track an impact from start to finish,explaining how it came about and the role of Member developmentin creating the impact. One significant example or case study thatdirectly links Member development to impact may well be moreconvincing than a raft of slippery statistics.

Many people are confused about examples, quotes, testimony andcase studies. These types of evidence are often referred to as‘anecdotal’, and not ‘proper;’ evidence. This is not the case.Presented well, ‘anecdotes’ can be excellent material. Informal‘story telling’ should not be dismissed, but seen as solid qualitativeresearch. A range of different sorts of evidence can be used toadd colour to the application. The Level Two application does notneed to be dry; it should give a sense of the challenges theauthority faced, the excitement in developing success, and people’spride in the outcomes they have achieved.

11

Fitting evidence gathering into busylives

One issue that many Members and Officerswill face in gathering evidence of impact isa lack of time to do extra work on top ofthe ‘day job’.

Evidence gathering for the Level Twoapplication should not be a significantadditional burden for an authority. It is vitalto create low-cost, low intensity but highvalue ways to gather evidence if this workis to be done. It should also givesomething back, for example by helpingMembers to feel proud of what had beenachieved.

Existing feedback systems should beadapted to include feedback on communitylevel impact. Impact evaluation should beintegrated into normal activity wherepossible. Most Member developmentevents are already evaluated for learningand enjoyment; could an additional 30minutes be added in to collect examplesof impact from earlier Memberdevelopment activities? It can easily bebuilt into Member development meetingsas an update and feedback item. Manylearning programmes have celebrationevents attached, and these double as asource of rich evidence of what peoplehave done as a result of Memberdevelopment.

Evidence gathering does not need to be aformal activity, done in a particular setting,or a particular way. It can happen

anywhere. Some of the best research isdone out and about on location or on theway to the next meeting.

Much of the evidence you will need tocollect involves talking to other people.Collating information might include: aninformal conversation with Members;inviting Officers to your meetings, going ona hunt for data and documents; ‘de-briefing’ Members or Officers informally ona one-to-one basis; ringing externalstakeholders and partner organisations.All verbal information should be writtendown at the time if possible or evenrecorded, and checked back with theperson quoted for accuracy.

The example below illustrates some ideasone North West authority had about waysto gather evidence.

Ideas for evidence gathering:Lancashire County Council

The authority has been doing ‘Bus Tours’for Members around the authority area toraise Member awareness of the diversity ofLancashire and local community issues.Their ideas for finding out what communityimpacts of the Tours have been include:

- doing an exercise on ‘3 things I wantfrom the day/3 things I took away on thecoach there and back from Tours- taking a Dictaphone on Tours andinterviewing Members on the day aboutwhat they have learnt from the Tour andwhat they intend to do as a result, thenusing this to review progress

- encourage Members to put feedback onthe authority’s Members’ websites.“We’ll encourage people to put thingsonto our Members website, so it’ll beuseful in lots of ways.” (Officer, NorthWest authority)

- phone calls afterwards to residents andcommunity groups who were involved inthe Tours – how had closer contact withMembers helped? What had happenedor changed since?

- asking Officers from the CommunityEngagement Team and other servicedelivery staff - have the Members theyengage with locally acted differently? Ifso how and what has been the result?

- revisiting projects and establishmentsthat hosted the Tours – haverelationships with Members changed?If so, how?

- asking party whips and CabinetMembers if and how they think Memberdevelopment has been linked todecisions made about related issues

“It’s time saving to integrate evaluationinto learning. We should find this outanyway – why people are going on theTours and what do they get out of them.We need a clearer idea about whatMembers want out of the Tours so thatwe can develop future ones. I’m going totake a Dictaphone on future Tours, youget some ‘real gems’ when people talkoff the cuff, but when you get theevaluation forms it’s more stilted.”(Officer)

12

Practical Lesson13

Practical Lesson No 1 looked at what counts asevidence and ways to collect evidence. PracticalLesson No 2 looks at where these types of evidencecould be sourced. The key message here is to useas many different sources of evidence as possible:from Members themselves; from other departmentswithin the authority; from other organisations andexternal stakeholders; and ‘hidden’ knowledge ofMember Development Officers.

“You get so blinkered. We’re office bound so we’renot out and about, so you have no appreciation ofwhat’s happening around you.” (Officer, North Westauthority)

Members’ own experiences

The key information resource for impact evaluationis Members themselves. The North West MemberDevelopment Charter Review for Level One requiresthere ‘to be a process to identify individualdevelopment needs which involves Members.’

Two of the indicators for this are:- Have one-to-one interviews for Elected Memberstaken place once or twice in the last three years?

- Have 75% of Members got a PersonalDevelopment Plan (PDP)

The Level One Charter Review Process also asks ifMembers have reviewed their PDPs throughPersonal Development Reviews (PDRs). These one-to-one interviews and associated PDRs are apotential source of ‘data’ for the impact evaluation.It is appropriate to ask Members about theirachievements so far in putting learning into practiceduring a learning needs review.

Practical Lesson No 2.Don’t go it alone – sources for evidence

14

An example of how to use Personal DevelopmentReviews to collect impact evidence: Halton BoroughCouncil

Each Member in the authority has a six monthly review, whichasks Members to identify 3 skills they have acquired and give anexample of how they have been able to demonstrate the variouscompetencies in a particular module and rate themselves on ascale of 1 - 5. The Member Development Officer supportsMembers to think through how they have demonstrated thecompetencies.

One of the key aspects of this process is its confidentiality,therefore individuals are not identified. The information can beused to generate anonymised individual examples of success, oridentify people who would be willing to give permission or provideexamples for the Level Two application.

The authority already inputs the information from the reviews to adatabase. Reports from the database are used to generate andidentify themes for further development needs. Beyond theindividual level, this type of database could potentially be used tocollate evidence of community impact by theme.

Whether a database is used or not, existing information fromPDR documentation can easily be loosely categorised to showfollow-through from learning by theme. This could include howMembers behaved differently after development, and any impactsthey reported. This type of categorisation can transform acollection of individual level ‘anecdotes’ into solid evidence. Itmakes good use of information that most authorities alreadycollect or could easily collect, if they do not already.

Group feedback

The individual PDR information can be supplemented by groupfeedback on community impacts of learning. A group discussioncan be integrated into existing get-togethers, such as MemberDevelopment Steering Group meetings, or as a slot at learningevents about what happened since the last learning event. Thisgenerates positive feedback about the effectiveness of learningand development that stimulates further learning.

It could also be used as a way of convincing other Members of thebenefits of participating in learning – what’s in it for me (WIIFM)?An open discussion could highlight obstacles people have inputting learning into practice and seeing the relevance.

Material and information from elsewhere in theorganisation

In the research for this toolkit we found that authorities had manysources of useful evidence from within their own organisation. Butwe found that often these sources had not been considered initiallyby the department responsible for Member development. Peopletended to focus on the information held by their own departments.

These other sources included:

• Evaluations commissioned by or done for other departments.

“We had an evaluation by the IDeA. I did know about it but Ihadn’t related it to the application [for Level Two Charter].”(Officer, North West authority)

• Management information processed by a differentdepartment, eg customer surveys.“We could use the number of hits we’ve had on our ICTsystem.” (Officer, North West authority)

“The Development Control customer survey.” (Officer, NorthWest authority)

“The minutes of the Improvement Board and the PerformanceAssessment report.” (Officer, North West authority)

• Informal and undocumented knowledge of other Officers.

“We should talk to the Community Development team. They’vegot loads of good examples, like that planning application.”(Officer, North West authority)

“The planning and partnership team do a lot with parishes,they’re mostly talking about Local Development Frameworksand housing.” (Officer, North West authority)

• Submissions for other Awards, e.g. Charter Mark.

15

Testimony from partner organisations and externalstakeholders

Partner agencies such as the Police, Fire and Health alsohold potentially useful information about communityimpacts where Members have contributed. Authoritieshave tended not to consider these sources for evidence ofcommunity impact, and indeed these sources can bedifficult to obtain or fit into an application.

Another way to add evidence from external stakeholdersis to use testimony from partner organisations and externalstakeholders.

One of the forms of evidence discussed in Practical Lesson1 was testimony from people outside the organisation aboutthe work the authority has done and the benefits produced.Testimony from partner organisations and externalstakeholders support the Charter application in a similarway to references for someone applying for a job. Yourreferees should comment directly on their experience andperceptions of community impacts. Testimony should referto any changes in Members’ behaviour and skills that mayhave led to the community improvements, where possible.However, not all external referees will be aware of what liesbehind improvements. Those people can still contribute tothe application by giving testimony about the end results.

One example of testimony was in a successful Level Twoapplication, where Member development had improvedMembers’ ability to support voluntary and communitygroups in their area. Members had done training and hadthen been able to signpost groups to additional funding.The application was supported by testimony from one of thecommunity organisations that had been supported.

The example from Blackpool Council in the box belowillustrates how imaginative use of testimony can be used tosupport an application.

Imaginative use of testimony: BlackpoolBorough Council

Sources used to support Blackpool’s successfulLevel Two application were:

Survey questionnaires with a number of externalstakeholders including the Police, Fire Authority,Solicitors, Blackpool Pub Watch and the BlackpoolHotel and Guest House Association.

In order to gain more extensive and qualitativefeedback from the main stakeholders in the process,one-to-one discussions were held with representativesfrom the Police and Fire Authorities.

Visit from an authority facing similar issues to look atbest practice being used by Blackpool.

Parliamentary Scrutiny Councils’ report referring toBlackpool’s work.

Comments from the Secretary of State for Culture,Media and Sport in the House of Commons, payingtribute to Councillors such as those in Blackpool.

An Editorial article in a relevant trade publication.(Licensed Trade Journal)

1

2

345

6

16

Member Development Officers’ own knowledge -uncovering hidden evidence

During the research for this toolkit, Members andMember Support Officers described their work to anoutsider unfamiliar with it in an interview. Simply sittingdown and telling someone else about their workbrought many pieces of evidence and examples to thesurface that may have remained hidden. WhenMembers and Officers are involved in somethingintensively it can mean they ‘can’t see the wood for thetrees’. Some of the “little gems” go unnoticed by thoseinvolved. Officers doing Member development oftenhave lots of relevant knowledge for the Charter LevelTwo application, but do not realise, or have not thoughtabout things in this way. Some of the Officers co-ordinating the applications said they found it easier totalk about the work and community impacts than to putit down on paper initially.

“We need to think laterally, we need to think differentlyabout it. There’re so many things we hadn’t thoughtabout.” (Officer, North West authority)

For all of these reasons, one way to strengthen theCharter Level Two application is to use a version of aresearch interview.

The aim of this exercise is to uncover evidence thatOfficers supporting Member development might have,but do not know they have. It can help to brainstormideas and pieces of evidence for the application in anew way.

A research interview is different to having an internalmeeting of relevant people to discuss the application.The interviewer is someone who is unfamiliar with yourwork and has a list of questions or topics. This providesa level of external challenge, and the questions offer adifferent way of thinking. Someone who is unfamiliarwith the work will need a higher level of detail whichthen helps generate the examples and evidence base.

The interviewer can be from within the organisation, orfrom outside. They do not necessarily need experiencein conducting interviews and they do NOT need to be atrained researcher.

The interview takes 1-2 hours. The interviewer shouldtake notes that they feed back, and/or the interviewshould be recorded, and/or notes should be taken bysomeone not involved in asking or answering thequestions. The list of questions should be a loose guideto the topics you want to cover. The interview should besemi-structured, ie flexible, and extra questions askedwherever appropriate.

17

Sample list of questions and topics for a‘research interview’

• Describe the political context of the authority.

• How would you characterise the organisationalculture of the authority?

• How did the Charter Level One application comeabout?

• What did the authority do and how successful wasit overall?

• What has changed since the Charter Level Oneapplication?

• Where is Member development at now?

• What are the main issues and priorities now forMember development?

• Please give examples of why this is, what theissues are fromconstituencies/authority/neighbourhoods/Membersurgeries, etc.

• What is the authority doing to address Memberdevelopment priorities? Please give general andspecific examples of learning.

• Which other Officers and organisations are youworking most closely with?

• What evaluation systems do you have in place?

• What are you most proud of in Memberdevelopment? Why? Please give specificexamples and details.

• If you’d have known at the beginning you weregoing to create these achievements, whatmeasures would you have put in place, and howwould you have tracked impacts?

• Who else would have noticed these changes?

• What are three most significant community-levelchallenges facing Councillors? How do these linkto the current Member development described?

• What are the aims of learning – what did you hopewould happen? What was the theory behind it?

• How do the aims of Member development link tothe community challenges and successes justdiscussed?

• How are Members involved in the Level Twoapplication?

• Name three ways that Member developmenthelps tackle the authority’s corporate priorities.

• Has the authority dealt with the NWEO assessors’comments from the Level One assessment orreview? If yes, how?

• What other awards does the authority have/or isapplying for?

18

Practical Lesson19

Practical Lesson No 1 looked atwhat counts as evidence and waysto collect evidence. PracticalLesson No 2 looked at where thesetypes of evidence could be sourced.Practical Lesson No 3 now looksmore closely at what is meant by‘community level impact’, and howto put the evidence together into agood application.

Use the Level Two application totell the story of your success

Successful Level Two applicationsgive the assessment team a real feelfor the journey that that authorityhas been on. They do this by tellinga coherent story of what happened.

The application form is broadlystructured to match how Membersand Officers might naturally telltheir story. The core elements ofthe story, or application, are set outbelow.

Elected Members helping to tackleissues for the community

• What were the original issues orproblems for the community andauthority?

• What was done to tackle theissues?

• How are things different now forthe community, and for theauthority?

• What role did Members play intackling these issues?

Links to learning and development

• What were the gaps in Members’knowledge, skills, capacity andexperience in relation to theseissues?

• How and when were these gapsidentified?

• What learning and developmenttook place to fill these gaps?

• How did the learning anddevelopment go? •Have therebeen adaptations along the way?

• How was the learning offereddesigned to meet the gaps inMembers’ skills, knowledge,capacity and experience?

• Did you have a clear idea of whatmight come out of learning anddevelopment when you started?

• What has come out of thelearning – what difference hasthere been for Members’knowledge, skills, capacity andexperience?

• How did learning help Membersto tackle the community issues ina more effective way than before?

Links to strategic and corporatepriorities

• How does this work fit withstrategy and corporate priorities?

Practical Lesson No 3.Tell the story from the beginning

20

Another way to think about the application is to think about the theorybehind what was done, as shown in the example below.

Theory behind application

Community cohesion challengesin the authority

Members help address challengesthrough community leadership role.

Lack of comprehensive Memberawareness of cohesion issues

Awareness raising of Members tostrengthen their ability to contributeto cohesion. Through fact finding

tours and on-line learning

Member awarenessincreased

Cohesion strengthen-edResident feedback

Evidence

Corporate plan

PDRs

Member developmentprogramme material

PDRs, Learning eventfeedback, Member action

plans

Resident feedback, Projectinfo, Member evidence

Link to Learning &development process

Identifying the communityneed

Identify thedevelopment need

Learning anddevelopment offered

Results of learning

Corporate Impact

The reasoning

The rest of this section looks at how to tell a coherent story, in practice

21

Getting to thehappy endingThinking about your application as astory or a narrative can help make astronger application. We havealready looked at how to bring theapplication to life by adding colourthrough examples, case studies andtestimony. We have alreadydiscussed telling the story fromseveral different points of view.

A good story explains what happened next,and makes it clear why one thing leads toanother. The Level Two application shouldflow from the need identified, to the aims ofthe development, to the developmentoffered, to the learning results (evaluationand review), to the impact. The personreading or assessing the application shouldbe able to understand where the storycomes from and where it goes - ‘that’s whywe…’ ‘and because of that we…’, ‘then wedecided to…’, ‘as a result differences areevident in…’. For the Level Two Charter,the story should have a happy ending forthe authority or the community, not justthe Members. Who and/or what else, apartfrom the Members who did thedevelopment, have benefited and how?

The worst type of story is where you areleft not knowing what happened at the end.Cliff-hangers are good in soap operas,but not Charter applications. A commonmistake is to stop short at the learningMembers have gained or even how theybehave differently. More awareness of anissue is a first step, but the applicationneeds to go further to the communityimpact level. What happened next?What difference has it made to the livesof ordinary people? The example in thebox below is of a happy ending for localpeople from Member development.

Happy endings in HaltonBorough CouncilThe initial information offered by theauthority was that training had enhancedMembers’ competencies so that that theycould play their part in meeting the needs ofthe community. This was an ambitious aim,but what was the evidence to support it?There were quotes from Elected Membersabout the learning gained generally fromtraining: “The training gave me a very goodinsight”, “I am more confident knowingabout different resources available”. So, ahappy ending for Members. But whatdifference had the extra confidence made toneeds in the community? Where was thehappy ending for the community?

Two of the local community organisationsdescribed their views of the benefits. Theysaid that their local Member was now:“always very involved”; “more inclined to dothings”; “more active, wanting to participateand help the group achieve things”. Theorganisations felt that they receivedMember support whenever they asked forit, and sometimes before. There was oneexample of a social group for older peoplewho had already been awarded severalhundred pounds after their local Memberhad advised them where to access funding,using information from Memberdevelopment training.

Ultimately, this was a happy ending for theauthority too, as it contributed to achievingone of the corporate priorities to create aBorough where: “People respect and carefor each other and share a sense ofresponsibility and pride.”

The box below is an example of a cliff-hanger where the authority had plans forfinding out what happened next.

A cliff-hanger: LancashireCounty CouncilCommunity cohesion is a priority for theauthority. One goal is to strengthen

community leadership on communitycohesion – to understand the issues andneeds of all sections of the population andbe able to represent all the population, todeal with competing community prioritiesand divisions and distrust.

One of the actions in the cohesiondevelopment plan to grow the capacity ofleadership to counter myths andmisunderstandings was awareness-raisingwith Members. This included thedevelopment and rollout of ‘Tours’ of theauthority to better understand the issues,talk to projects, link to residents inneighbourhoods facing the issues, liaisewith Officers and showcase initiatives thatwere working well.

Members “got more out of it than theyimagined, or we envisaged.” (Officer)Members came back from the Tours with abetter understanding of diversity issues andof the cohesion plan for the area. “I have abetter understanding of the local issues”(Member).

The Tours were a comfortable or safesetting to explore difficult, new and complexissues. One Member said: “I found theTour very helpful and enlightening…. Youcan learn what solutions have been usedto problems and whether the sameapproach would benefit the area yourepresent. It makes you more outwardlooking and more efficient as arepresentative for your own area.”

Councillors are now “equipped to betterserve their local community.” (Officer)But how have they done this? The nextstep is to see what happens as a result:“Awareness-raising isn’t enough, what’sit leading to? We need concreteexamples.” (Officer)

The authority will collect evidence aboutimpacts by tracking Members’ plans forfollow up action, and by contacting theprojects and Officers involved in hostingthe Tours.

22

Happy endings for whom?Authority or community level impacts?

The Charter Review Process for Level Two asks for examplesand evidence of direct impact [of Member development] onthe work of the local authority and on the community

There are essentially two different levels of impact that could besubmitted in the application: impact on the work of the authority;impact on community outcomes. Where possiblethe application should offer detail of community level impact.However, solid evidence of impact on the authority is alsoextremely valid, and may be easier to evidence fully.

An example from Blackpool illustrates the differencebetween authority level impact and community level impact.

Authority versus community level impacts:Blackpool Borough Council

Blackpool has the second highest number of licensedpremises of any authority in the country, and has achallenging job to balance their needs for economic developmentin a deprived area, heavily reliant on a night-time economy, withtheir need to create and maintain a safe, clean and vibrant towncentre, and reduce alcohol harm. Responsible licensing by theauthority plays a key role in managing these competing demands.

The Licensing Act 2003 transferred responsibility for liquorlicensing from the Magistrates’ Court to local authorities. As thiswas a new statutory responsibility, and given the nature of thelocality, this represented some major and significant challengesfor Blackpool. Blackpool Council, through Legal and DemocraticServices, ran a series of training and information workshops;some joint sessions with Members and Officers; did a ‘mockhearing’ for Licensing Committee members; and produced a guideto legislation called ‘Last Orders’.

The bulk of evidence in the Charter Level Two application showedthat where this Member development had been effective wasaround its impact on the work of the authority. Blackpool Councilargued very convincingly that the local authority’s licensingprocesses have been operating successfully, including a widerange of types and sources of evidence:

• Every one of the 153 Licensing Panel hearings that had beenheld had been heard and determined within the statutory timelimits.

• Low number of appeals – despite the high quantity of LicensingPanel hearings, to date there have only been two appeals to theMagistrates’ Court. Neither received comment that Panelmembers did not fulfil their role or that poor decision-makingwas an issue.

• Positive feedback about the quality of the hearings fromexternal stakeholders including the Police, Fire Authority,Solicitors, Blackpool Pub Watch and the Blackpool Hotel andGuest House Association. Overall, the feedback was extremelypositive with respondents stating that Panel hearings ran in afair, competent and efficient manner; procedures wereconducted well; all parties present understood their roles andresponsibilities; Members seemed to demonstrate a goodknowledge of legislation; and decisions throughout were clearlycommunicated.

• Strong working relationships between Members across parties,as well as with Officers, relevant stakeholders such aslicensees, the Police and Fire Authority.

• Blackpool to be seen as an example of best practice of anauthority with high numbers of licensed premises, eg localpress reports, parliamentary reports, study visits from otherauthorities.

During the assessment visit, further details emerged about howeffective licensing seemed to be on impacts at the communitylevel. These impacts were harder to demonstrate, but theassessment team felt they gave additional strength to theapplication and welcomed the extra information.

First, the Fire Authority gave an example of one licensed premisesthat had breached fire safety regulations. The Fire Authority wasable to report this to the Licensing Committee, which thenenforced legislation by suspending the license for two months.This caused the company an £80,000 loss of profits, and “sendsout a message” to other licensees. The idea was that “we’re notout to punish”, but the feedback from the licensees’ forum is thatlicensees are “more on their toes”, and this is “making it safer”for the community.

Second, the training had added to the engagement of residentsin the licensing process, with local people starting to show agreater understanding of their rights to make representations.As a result of training, Members were able to informconstituents about appropriate grounds for objections toapplications. Even if a licence was granted despite objections,the public could now monitor the conditions of the licence andinform the committee if the conditions were broken in order torevoke the licence. Evidence for this was offered via officialdocumentation (Parliamentary Scrutiny Council report).

23

Tell the story from start to finish: outcomesmust link to strategic priorities

The Level Two application must show how the Memberdevelopment activity and its outcomes link to the authority’scorporate priorities, the priorities of the authority and itspartners, e.g. the Local Strategic Partnership (LSP) and theSustainable Community Strategy. There needs to be a ‘goldenthread’ running through from what happened on the ground(the finish) to wider strategy and targets (the start).

Where links are clear

Sometimes Member development is structured aroundstrategic priorities, and therefore the links are clear andsimple to explain, as the example below illustrates.

Seeing is believing: linking strategic prioritiesto development in the Lake District NationalPark Authority (LDNPA)

One example of how Member development is linked tostrategic priorities comes from the LDNPA. Their ‘Seeing isBelieving’ events offer Members the chance to get out andabout in the National Park Authority (NPA) area and seewhat’s happening on the ground. The events are focused on theLDNPA’s Vision (under the theme of a prosperous economy);the corporate plan priorities, eg to promote sustainable ruraldevelopment; as well as the Cumbria Economic Plan set out byCumbria Vision. One of the challenges for the NPA is “theculture and perception of the NPA”.

Previously, there has been a presumption against planningapplications: “We had a reputation for saying ‘no’, for notallowing economic development. We need to considerapplications differently. It’s a culture change for us and ourcustomers.” (Officer)

One ‘Seeing is Believing’ event was around the theme of‘prosperous economy’ and involved Members visiting parishcouncils and local businesses, which was one of the first timesbusinesses had the opportunity to speak to a NPA member.Members enjoyed the events, but, more importantly, therehave been changes as a result of these development activities.One of the meetings with the parish councils did not go well.Parishes were unhappy about changes which meant they were

now excluded from planning application site visits. As a resultof this discussion the policy was changed to include parishesback in.

This positive example was presented by the LDNPA at a veryearly stage of the Level Two application. It’s possible to seehow this example of community impact could be taken muchfurther. The ‘so what?’ questions are: What difference doeshaving parish council members along to planning applicationsite visits make? How does this help the Lake District becomea more prosperous area?

What if?

What if the Member development steering group could findout that:

- A greater proportion of planning applications fordevelopment were being passed than previously.

- Objections to planning applications in those parisheshad gone down.

- The time to agree the application had speeded up.- There was an example of a successful application for adevelopment that brought business into the area.

- The successful application could be compared to a similarapplication that had previously been turned down.

- There was evidence from the Economic Plan that planningapplications to extend existing dwellings created morewealth or assets for local people.

Where links are not clear

Sometimes there is no immediate, direct or obvious linkbetween Member development activity and corporate orstrategic priorities. During a visit to one authority, we asked:

Q: In what three ways does Member development helpimplement the community strategy?

A: “That is a good question.” (Member)

The authority had not been challenged to think about Memberdevelopment in this way before.

Where the link is not clear, more effort is needed to uncoverthe relationship between development, community outcomesand the corporate priorities. The hypothetical example in thebox, using a mix of real material from different North Westauthorities, shows how this could be done.

24

25

Linking Member development to strategic priorities

26

The Borough Council has a wellestablished Member developmentprogramme which is focused on thegeneric development needs of Members.The programme includes learning anddevelopment on topics including: timemanagement; speed reading; IT skills;public speaking; decision making;leadership; chairing meetings; andnegotiating and persuading skills.

The Democratic Services Team organised adebriefing session with Elected Memberswho had participated in learning anddevelopment. Members came up withthree examples each of how Memberdevelopment had helped them make moreof a difference in the communities andneighbourhoods they represented. Oneexample was:

“Because of the training on publicspeaking, chairing meetings, leadershipand negotiating, Members have been moreconfident in consulting the community onrestructuring of Children’s Services. Inthe first set of meetings the community

was hostile because they felt that facilitieswere being taken away. Members haveused the skills acquired in training toexplain that the idea is to integratecommunity centres and children’s centresso that children have more opportunities,not less. They weren’t frightened to gointo a potentially hostile environment thistime.”

The debrief session then took the fiveCorporate priorities of the authorityidentified in the Corporate Plan:

Living Places To improve, preserve andmaintain those neighbourhoods and placesthat local people consider valuable forrecreation, conservation, commerce anddaily living.

Living Communities To provide a flexibleand responsive organisation that supportsthe delivery of services in an efficient andeffective manner through continualimprovement, regular challenge andlistening to local people.

Living Economy To promote investment,create jobs and build modern sustainablecommunities.

Living Safely To work with others to makethe area a safer place to live, work and visitby helping to reduce crime and the fear ofcrime in the local community.

Living Healthily To ensure that everyone inthe community has somewhere to live andis able to access local opportunities for ahealthy and fulfilling lifestyle.The group categorised the examples of theimpact of development on communitiesunder the Corporate Plan headings. Theydecided after some discussion to put theexample of Children’s Services underLiving Healthily.

The Officer leading on the Level Twoapplication took this evidence back tothe office. They did a further search ofpolicy documents, and identified thatthis example also helped to meet someof the action points in the Social CareDirectorate’s Business Plan.

Linking Member development tostrategic priorities:a real hypothetical example

27

‘Suck it and see’ – evaluation where the initial developmentneed, aims or objectives are not clear

In an ideal world it would be possible to evaluate Memberdevelopment by checking if it had met its original goals.These goals would be set out before Member developmenttook place, then the measurement systems would be put inplace beforehand to track those intended outcomes andimpacts. Unfortunately, the evaluation of Memberdevelopment does not always happen in an ideal world.

One issue is that at the start of a process, Members andOfficers might not always know for definite what they wantout of it, or what might happen as a result. Sometimes peoplewant to try things out, or ‘suck it and see’, without being clearwhat they will get out of it, as one Member explained:

“You do your PDP, identify a gap in your knowledge, then yougo on a course. But sometimes it is woolly at the beginningabout why you would want to fill the gap in knowledge. Youmight not realise until later on what you got out of it.”(Member, North West authority)

Another issue is that sometimes there are unexpectedoutcomes from Member development - both good and bad–that were not envisaged at the beginning. Or that themember of staff who designed a learning programme hasmoved on, and the new person in post is not clear what goalswere intended. In real life, the original aims of learningoccasionally just get lost or forgotten,when you get to the end of a process and ask yourself:“Why did we start this again? What were we trying to achieveall those months ago!” In everyday life, things can justhappen without being part of a plan, as in this authority:

“We did equalities training. We’re not sure where the trainingcame from. We have no idea why we got it. [But] it was reallyuseful, people liked it” (Member, North West authority)

In all these situations it is then not possible to evaluateMember development by checking if it achieved its originalgoals. For some authorities that have faced these situations,they have felt it made evaluation difficult:

“We wanted to use induction, but we couldn’t quantify ormeasure the impact of specific Member development;

evaluation wasn’t built in to induction at the beginning so it’sdifficult. We’ve got nothing to compare induction against frompreviously. We knew Members had developed but we couldn’tpin down why or how.We need to be clear on our objectives before [developmenttakes place] so we can see when we look back what we haveachieved. We have to have resources to do training, but it’snot clear [how we can justify this]. It’s hard to go backwardswith evaluation” (Officer, North West authority)

If these situations occur, there are things an authority can do.It is acceptable to do a ‘suck it and see’ evaluation. TechnicalNote 4 explains more about suck it and see evaluation – whichis also known as ‘goals-free evaluation’.

Technical Note 4. Goals-based versusgoals-free evaluation

Have the goals of a policy, programme or project beenachieved?

This is one of the most frequently asked questions in policyevaluation, and is sometimes referred to as Goals-basedevaluation. Before and after methods of evaluation (ie pre-and post- intervention designs), using both quantitative andqualitative research methods, are used to answer questionssuch as these.

However, sometime programmes or activities will haveunintended outcomes. Policy makers and evaluators are ofteninterested in the unintended consequences or outcomes of apolicy, programme or project. These unintended outcomes maybe beneficial or harmful.

Goals-free methods of evaluation are used for this. Theydetermine the actual effects or outcomes of some policy,programme or project, without necessarily knowing what theintended goals might be. Goals-free policy evaluation usessimilar methods as goals-based evaluation, with a greateremphasis on qualitative methods of research (eg in-depthinterviews, participant-observation, focus groups).For more detail see the Civil Service guidebook on policyevaluation, known as the ‘Magenta Book’, at:http://www.policyhub.gov.uk/evaluating_policy/magenta_book/chapter1.asp

It is important to consider doing goals-free evaluation so that positive outcomesdo not get missed. In the example ofequalities training mentioned abovesome exciting things happened as aresult, despite the original developmentneed and aims of the development notbeing entirely clear at the start, as shownin the example below.

An example of positive outcomesfrom trying something out

In one North West authority during theequalities training a discussion came upabout the rapid changes in the populationdue to incoming migrants from ‘A8’countries such as Poland. It was realisedthat little was known in the authorityabout migrant workers, or that therewere over 15,000 Polish residents nowliving in the town. As a result it wasagreed to do further investigation of theneeds of migrant workers throughworkshops. This was not planned andwas not one of the original goals of thetraining. It was a positive, unintended,consequence of training that no-one wasquite sure why it had happened in thefirst place.

The illustration below uses a mix of realmaterial from different local authoritiesand shows how it might be possible topiece together a story retrospectively.

‘Suck it and see’ evaluation: anillustration

Every year there is an introductory-daytraining event for new Members aboutthe planning system. The Democraticand Legal Services Officer who arrangedthe event knows that this training takesplace every year, and that it is generallyuseful for raising awareness. But she is

not sure exactly why it was initiated inthe first place, what it links to, or whatoutcomes it is specifically designed toachieve. She was not able to attend theday.

One day in a corridor in the Town Hall,the Officer bumps into one of theMembers who attended the training,Cllr Jones. During her chat withCllr Jones it turns out that one of thediscussions was extremely lively.

The session was on how the Council cansuccessfully promote good designthrough Local Development Frameworks.Members talked about complaints theyhave received from constituents about an‘ugly’ new mixed use development near alocal shopping centre. Not only that,there have been messages fromresidents posted on the local web-baseddiscussion board questioning the ethicalconduct of Councillors in relation toplanning, for example: “It is an utter freefor all here now, and there may be thosewho would be wondering whethersomebody, somewhere, wasn't on thetake big-time.”, and also on the validityand transparency of consultation:“Consultation on most new stuff in thisborough is usually nothing more than ajoke.”

Cllr Jones mentions that Cllr Smithwas planning to “sort this all out”. TheOfficer realises this could be greatevidence for the North West CharterLevel Two application under ‘TheDevelopment Offered’ section, so shedocuments what Cllr Jones has just saidin her notebook.

Initially the Officer tries to find data onresidents’ perceptions of the ethicalconduct of Members through the

residents’ survey, but realises it will betoo difficult to make the link to Memberdevelopment, and the question isn’tincluded anyway. But then she tracksdown the following information:

1 Evidence about the situation beforethe learning/the identified need

- approximate numbers of residentcomplaints about design indevelopment through surgeries inthe previous three months

- archived messages from thelocal web-forum about the ‘ugly’development

2 Link to corporate and strategicpriorities

- quotes from the Corporate Planabout the need to: “Reinvent ourtown centres: Making bold decisionsto create distinctive, high-quality,safe and healthy environments thatbalance the old and the new.”

- that this process will be done with“the whole community involved fullyin decision-making about the futureof the Borough.”

3 Evidence about the impact

- a phone call to Cllr Smith to discoverhe has already set up a meeting withPlanning to create a ‘design code’ thatall new developments should meet,and that he’s posted back on the localweb-forum board telling people whatis happening.

The Officer then sets a reminder onher electronic calendar to get updatesin three months’ time:

- from the local message board- from Cllr. Smith- from the planning dept

28

Conclusion and Checklist29

Conclusionand checklistPut simply, evaluating the impact of Member development meanschecking if residents in the community are getting, seeing andfeeling the benefits of their elected representatives doing morelearning and development. The 2007 Councillors’ Commission,chaired by Dame Jane Roberts, reinforced the point that localCouncillors playing a community leadership role is more importantthan ever, and that the challenges are more critical now than inprevious years:

“Councillors, as the elected representatives of the wholecommunity, are uniquely entitled to act as the interface betweenrepresentative and participatory democracy. Some are good atdoing this already, although they would probably not recognise thefact when described in such dry terms. They shuttle betweentaking up individual cases, working with single-issue local groupsall the way through a variety of other activities to formal decision-making by Councillors serving on Cabinets or quasi-judicialcommittees.

The package of skills which this demands from Councillors –individually or working as a team – is extensive: listening;negotiating; probing and scrutinising; handlingcontradiction; lobbying; campaigning; community development;conflict resolution; mediation; and, of course, decision making.”

There are still serious gaps to overcome between communityperceptions and awareness of the work of local Elected Members,and Members’ own views about whether they are doing a good job.Evaluation might be able to help bridge those gaps in perceptionsby further demonstrating how Members’ work helps improve thelives of their constituents.

Our key messages are: that Members and Officers shouldkeep a focus on the community when looking at the results ofMember development; and that they can enjoy finding out whatdifference was made by all their hard work on training, mentoring,e-learning, study visits and the rest.

Member development is now well established. Perceptions ofMember development have come a long way from being seen as anadd-on or afterthought, or as an excuse to attend junkets or fancyconferences. However, we remain perhaps a long way off fullyunderstanding or recognising the true extent of the potential valueof Member development, not just for Members themselves but forcommunities. More advanced evaluation is one tool to enable thisvalue to be identified and appreciated.

We hope this toolkit is useful in your work to evaluate the impact ofyour Member development, and if you need any more informationplease contact Ruth Ashworth at North West Employers onrutha@nweo.org.uk.

Checklist� Look at the North Western Local Authorities' Employers'

Organisation (NWEO) guidelines on the review process for the

North West Member Development Charter.

� Decide whether to be reassessed at Level One or Level Two.

� Commit to doing impact evaluation. Investigate how this could

be used to celebrate success and publicise achievements.

� Narrow down the most effective areas of Member development

on which to focus evaluation.

� Brainstorm a list of possible sources of factual evidence,

including statistics, testimonials, case studies and quotes.

� Find ways to integrate evaluation into existing systems, and

create low cost, high value ways to collect additional evidence.

� Check Personal Development Reviews are suitable for

gathering evidence, and adapt if necessary.

� Search out other sources of information inside and outside the

authority, including gathering testimonials from external

stakeholders, and uncovering ‘hidden gems’ of evidence within

your own organisation.

� Complete the NWEO application and develop your own story

or narrative.

� Make sure the story tells readers about ‘what happened next’

following Member development, and to and for whom.

� Link your story to strategic priorities.

30

North West Employers6th Floor Delphian House Riverside New Bailey Street Manchester M3 5AP

Tel: 0161 834 9362 Fax: 0161 831 7268 Web: www.nweo.org.uk

© Copyright 2008 North West Employers and North West Improvement Efficiency Partnership