1
SOCIAL RESEARCH ASSOCIATION IRELAND/IRISH EVALUATION NETWORK
SEMINAR
GOOD PRACTICE IN COMMISSIONING RESEARCH AND EVALUATION
10TH JANUARY 2006
Contents of Presentation
2
a “supplier” perspective
nature of evaluation
typical stages in procurement process
common challenges, needs
some issues for future
Aspects of “Supplier” Perspective
3
evaluation more than research
external supplier perspective (not internal procurement)
a commercial consultancy perspective (not research grants)
multiple types, contexts
a small country perspective
emphasising common issues
Multi-dimensional Nature of Evaluation
4
Managing the team
Stakeholder Involvement
Research, Analysis
Time, resources, budget
Client, user relations
EVALUATION
A Balancing Act
Typical Stages in Procurement
5
2. Terms of Reference,
Brief
6. Reporting
1. Policy issue,
topic need
3. Invitations, Tendering
4. Selection, Contracting,
Inception
5. Managing, Undertaking,
Analysing
Common Challenges, Good Practice
6
1. Policy issue, topic, need lack of clarity lack of client consensus
clarity, consensus spell out objectives
2. Terms of Reference, Brief
problems of no.1 flow in here over-specifying methodology “piece of string” problem
don’t rush the ToR focus on objectives
3. Procurement, Tendering poor procedures overly formalised competitive
procedures which prevent dialogue
4. Selection, Contracting, Inception
disproportional procedures rush to get started
proportionality don’t rush the Inception
Challenges Good PracticeStage
Common Needs, Challenges and Good Practice (Cont’d)
7
5. Managing, Undertaking, Analysing
unrealistic deadlines absence of data data collection v analysis multiple stakeholders “scope creep”
good project management role of Steering Committees? avoid surprises
6. Reporting emphasis on written “tomes” difficulty finalising reports “reversing” compromises into
reports
separation of consultants’ recommendations from policy decision
Challenges Good PracticeStage
Procurement Procedures, Tendering (balancing competition and the need for dialogue)
8
need for better procurement skills, e.g. training, guidelines
inviting too many bidders/overuse of e-tenders website?
need for guidance on scale
circulation of all replies to all bidders!
ability, availability of commissioner representatives
see also www.fitzpatrick-associates.com (tendering)
Wider Issues for the Future
9
extent of evidence-informed, evaluation culture
the need for research, evaluation to “speak” to policy makers
need for more basic, neutral data collection
balance between “independence” and “relevance”
emphasis on costs of research/evaluation v costs of poor policy decision
more inter-disciplinary research, evaluation (e.g. “economic” v “social”)
over-evaluation of some areas, under-evaluation of others