TERMS OF REFERENCE

INDEPENDENT PROGRESS REVIEW

of the

PNG DISASTER RISK MANAGEMENT PROGRAM (2010-2014)

These Terms of Reference (TORs) are to conduct a review and prepare an

independent report on the implementation of AusAID’s Disaster Risk Management

Program (2010-2014) in Papua New Guinea (PNG). In particular, the Review Team

will:

1. Assess and rate the progress of the PNG DRM Program against AusAID’s

evaluation criteria

2. Provide recommendations that allow AusAID to make strategic decisions on

the future direction of the Program itself while also providing

recommendations on the continuation of funding and nature of support for

specific activities

BACKGROUND:

AusAID’s Papua New Guinea (PNG) Disaster Risk Management (DRM) Program

(2010-2014) commenced in September 2010. Its goal is that of PNG’s own draft

Disaster Risk Reduction and Disaster Management National Framework for Action

2005-2015: ‘To ensure safe and resilient communities in high risk provinces of Papua

New Guinea’

Its purpose is:

To strengthen the DRM sector in Papua New Guinea, at the national,

provincial and sub-provincial levels, to prepare for, mitigate and respond to

the effects of disasters

The anticipated outcomes of the DRM Program are:

1. Increased capacity of organisations, both government and non-government, to

manage disaster risks and respond effectively to disasters through improved

financial and human resources, planning and management systems

2. Increased capacity of selected high risk provinces to manage disaster risks and

respond effectively to disasters through improved financial and human

resources, planning and management systems

3. Strengthened disaster awareness, disaster preparedness and risk mitigation in

targeted, vulnerable communities

2

4. Increased capacity of AusAID Post to effectively respond to disasters in PNG

Through the DRM Program, the following activities have been supported by

AusAID:

1. Improved national and provincial disaster response capability: technical

support for the National Disaster Centre (NDC) and selected high-risk

Provinces (initially West New Britain) to build and improve disaster risk

management and emergency response capacity.

2. A better coordinated response capacity: funding to the United Nation’s

Office for the Coordination of Humanitarian Assistance (OCHA) to improve

coordination for disaster preparedness activities and during an emergency

response.

3. Improved monitoring of major risk factors: continued support for

Geoscience Australia, in partnership with PNG technical agencies, to

strengthen assessment, monitoring and early warning of natural hazards such

as volcanic eruptions, tsunamis and earthquakes.

4. Strengthened community-led disaster awareness and response capacity: continued support to the PNG Red Cross and funding to community groups

and NGOs for disaster risk reduction activities.

5. Enhanced AusAID response: continue to train Emergency Response Team

members and build the capacity of AusAID PNG Post to respond to

emergencies.

PURPOSE OF THE REVIEW:

The Review Team will:

1. Assess and rate the progress of AusAID’s PNG DRM Program against the

following evaluation criteria (refer to Annex A):

a) Relevance: to assess whether the activity contributes to higher level

objectives of the aid program (outlined in country and thematic strategies).

b) Effectiveness: to assess whether the activity achieves clearly stated

objectives.

c) Efficiency: to assess whether the activity is managed to get value for

money from our inputs of funds, staff and other resources, and to

continually manage risks.

d) Impact (where feasible): to assess whether the activity produces positive

or negative changes (directly or indirectly, intended or unintended). The

degree to which the various aspects of impact can be assessed will vary

according to the nature and duration of the activity. Whether impact can be

assessed, or the way impact can be assessed will need to be determined by

the Review Team. Impact will not be rated.

3

e) Sustainability: to assess whether the activity appropriately addresses

sustainability so that the benefits of the activity will continue after funding

has ceased, with due account of partner government systems, stakeholder

ownership and the phase-out strategy.

f) Monitoring & Evaluation: to assess whether the monitoring and

evaluation framework effectively measures progress towards meeting

objectives.

g) Gender Equality: to assess whether the activity advances gender equality

and promotes women (considering the four dimensions of gender equality:

access, decision-making, women’s rights, capacity-building).

h) Analysis & Learning: to assess whether the activity is based on sound

technical analysis and continuous learning.

The Review Team will be expected to draw out lessons from their assessment of each

of the evaluation criterion that may be relevant to ongoing implementation of the

DRM Program.

2. Provide recommendations that allow AusAID to make strategic decisions on the

future direction of the DRM Program itself while also providing recommendations

on the continuation of funding and nature of support for the following activities:

a) Rabaul Volcanological Observatory Twinning Program – due to end in

September 2013

b) Natural Hazard Risk Assessment Mapping Program – due to end in

September 2013

c) Support to OCHA – due to end in December 2012

d) Community based DRR activities – due to begin in 2012/13

e) Support to the National Disaster Centre and West New Britain Province –

initial technical assistance support provided in early 2012

f) Red Cross Disaster Management Program – due to end in January 2013

REFERENCE FRAMEWORKS:

The Review Team will base its analysis on the following Reference Frameworks:

1. AusAID’s Humanitarian Action Policy (2011)

2. OECD-DAC and AusAID criteria for evaluating development assistance

3. AusAID Disaster Risk Management Program (2010-2014) – Performance

Management Framework

4. Other relevant AusAID policies, including the Multilateral Engagement

Strategy and Civil Society Engagement Framework

4

SCOPE OF THE REVIEW:

The Review Team will undertake the following:

Stage 1: Finalisation of TORs and Review Plan

1. Review these TORs – once selected and if required, the Review Team will

revise and update the TORs (to be accepted by AusAID in writing)

2. Develop Review Plan – as outlined on page 4, the Team Leader (in

consultation with other members of the Review Team) will submit a Review

Plan that outlines the methods the Review Team will use to meet the TOR

objectives and scope, including the evaluation questions

Approximate timeframe: 3 days

Stage 2: Document Review

1. Document review – including AusAID DRM Program documentation (design

document, Program Logic/Capacity Building Diagnostic, Performance

Management Framework, Quality at Implementation reports, activity design

and milestone reports, etc)

Approximate timeframe: 4 days

Stage 3: Field Visits and Consultations

1. Field visits – including Port Moresby, Kimbe (West New Britain) and

Kokopo/Rabaul (East New Britain)

2. Stakeholder meetings – including with Government of PNG agencies

(National Disaster Centre, Department of Mineral Policy and Geohazard

Management, Rabaul Volcanological Observatory, National Weather Service,

Provincial representatives from West and East New Britain, etc), PNG and

Australian Red Cross, UN Office for the Coordination of Humanitarian

Affairs (OCHA), Geoscience Australia, NGOs, other donors and members of

AusAID’s Disaster Management and Emergency Response Teams.

Approximate timeframe: 10 days in-country

Stage 4: Reporting

1. Finalise the Independent Progress Review of the AusAID DRM Program,

ensuring that this report addresses the objectives outlined in this TOR

Approximate timeframe: 7 days

5

REPORTING REQUIREMENTS:

1. Review Plan – which outlines the methods and timeframe the Review Team will

use to meet the TOR objectives and scope, including

a) an evaluation design that describes a logical model for assessing the DRM

Program and its activities;

b) a process for information collection and analysis (identify specific

evaluation questions and how these will be answered, including those

identified in Annex A); and

c) any substantive challenges to achieving the evaluation objectives that will

have to be addressed.

The Review Plan should also provide an outline of the roles and responsibilities of

team members and target dates for deliverables. The Review Plan will be no more

than three (3) pages and will be submitted to AusAID for approval two (2) weeks

before the Review Team undertakes field visits.

2. Aide Memoire – presented (and electronically submitted) to AusAID before

departure from PNG. The Aide Memoire will briefly (no more than 5 pages)

discuss the evaluation criteria and recommendations for future implementation of

the DRM Program.

3. Draft Independent Progress Report – due within three (3) weeks of completing

the field visit to PNG. The Team Leader will be responsible for coordinating

inputs from the Review Team and submitting this Report to AusAID. A template

will be provided to the Review Team by AusAID.

4. Final Independent Progress Report – due within seven (7) days of receiving

AusAID’s written comments on the Draft Report. The Team Leader will be

responsible for coordinating inputs of the Review Team and submitting this

Report to AusAID.

TIMEFRAMES:

The Review Team will be available from October – December 2012 to complete the

requirements of these Terms of Reference. Exact dates will be discussed directly with

the Review Team, once selected.

REVIEW TEAM:

The Review Team will be comprised of the following two (2) individuals:

M&E Specialist (Team Leader)

Discipline Category C, Job Level 3

Required Skill Set:

6

1. Demonstrated experience and understanding of contemporary M&E strategies

and systems, including in a developing country and/or Pacific region context.

Experience in PNG is highly desirable.

2. Previous experience in reviewing and evaluating other DRM/DRR activities,

including those supported by United Nations agencies, development agencies

or other donors, is highly desirable

3. Understanding of AusAID’s M&E policies

4. Extensive experience in the use of participatory and consultative evaluation

methods and techniques, including the collection and analysis of qualitative

and quantitative data

5. Proven ability in effectively managing small teams and utilising the expertise

of each team member in meeting Terms of Reference and contractual

obligations (including reporting)

6. Excellent English writing skills

Disaster Risk Management Specialist

Discipline Category B, Job Level 2

Required Skill Set:

1. Strong familiarity with best practice initiatives, current issues, developments

and trends in the humanitarian/disaster management sector (preferably disaster

preparedness and risk reduction), including in a developing country and/or

Pacific region context. Experience in PNG is highly desirable.

2. Understanding of AusAID’s disaster management/humanitarian policies and

the Hyogo Framework for Action

3. Strong understanding of national disaster management organisations, their

role, structure and any issues they face

4. A strong understanding of cross-cutting issues, particularly incorporating

gender perspectives and ensuring the equal participation of men and women in

humanitarian assistance, HIV/AIDS, environment/climate change, inclusion of

those with special needs, and how these relate to disaster risk reduction and

preparedness initiatives

Review Team members will also possess:

1. Strong liaison, communication, representation and negotiation skills

2. Ability to work effectively in a cross-cultural environment

7

AusAID’s Port Moresby based Disaster Management Team (Second Secretary and/or

Program Manager) will accompany the Review Team for the majority of in-country

field visits and consultations. It is expected that a Government of PNG representative

will also accompany the Review Team.

REMUNERATION:

The Adviser Remuneration Framework is a set of market-based, long and short term

remuneration rates applying to AusAID-funded advisers that are commercially

contracted by AusAID directly or through a managing contractor. Further information

on the Adviser Remuneration Framework can be found on AusAID’s website.

Considering the required skills and experience required to meet the obligations under

these Terms of Reference, the following remuneration for these positions are:

M&E Specialist (Team Leader)

Discipline Category C, Job Level 3

Disaster Risk Management Specialist

Discipline Category B, Job Level 2

As clearly outlined in the Adviser Remuneration Framework, AusAID will provide

other costs, such as:

Fixed daily travel rates (or per diems) are prescribed by AusAID for each

country, which are in line with equivalent rates paid to AusAID staff

undertaking short term travel. Per diems will only be available for travel in

PNG.

Adviser support costs are expenses incurred by the adviser that are directly

related to their assignment. These costs are reimbursed by AusAID at cost and

up to a prescribed limit set by AusAID in the contract. Support costs may

include, but not be limited to:

o Short-term accommodation

o Airfares and other work related transport costs

o Compulsory arrival and departure taxes

o International communication costs

o Medical insurance

o Security costs, where applicable

Accommodation (hotel) costs for short-term advisers are prescribed by

AusAID for each country, which are in line with equivalent rates paid to

AusAID staff undertaking short term travel. Accommodation will only be

reimbursed for hotels in PNG.

For individual advisers, management fees are limited to insurance costs that

are not solely linked to the current assignment. Refer to the Adviser

Remuneration Framework for further details.

8

For advisers engaged through a company, management fees must be clearly

documented and justified. Refer to the Adviser Remuneration Framework for

further details.

9

ANNEX A

These suggested standard evaluation questions for Independent Progress Reports have

been provided as a guide for evaluation teams in developing questions that get the

most value from the evaluation.

They are based on evaluation criteria that provide a comprehensive view of aid

effectiveness. The evaluation criteria are: relevance to who, effectiveness for who..,

efficiency, impact (if feasible) on who? Who benefitted? The aid activity must be

rated against these criteria.

The questions can be used as provided, or can be adapted to be more relevant to the

aid activity, country context and the size of the evaluation. The independent

evaluation team can be asked to adapt the evaluation questions when they develop the

methods design in the Evaluation Plan, including developing evaluation questions that

assess relevant cross-cutting issues.

Specific questions should be developed to assess compliance with the Paris

Declaration on Aid Effectiveness and the Accra Agenda for Action under the relevant

criteria that are relevant to the activity.

Questions for an Independent Progress Report

Relevance

– Are the objectives relevant to Australian Government and partner government

priorities?

– Are the objectives relevant to the context/needs of beneficiaries?

– If not, what changes need to be made to the activity or its objectives to ensure

continued relevance?

Effectiveness

– Are the objectives on track to being achieved? If not, what changes need to be

made to objectives to ensure they can be achieved?

– To what extent has the activity contributed to achievement of objectives?

Efficiency

– Has the implementation of the activity made effective use of time and resources to

achieve the outcomes?

Sub-questions:

Have there been any financial variations to the activity? If so, was value for

money considered in making these amendments?

Has management of the activity been responsive to changing needs? If not, why

not?

Has the activity suffered from delays in implementation? If so, why and what

was done about it?

10

Has the activity had sufficient and appropriate staffing resources?

– Was a risk management approach applied to management of the activity (including

anti-corruption)?

– What are the risks to achievement of objectives? Have the risks been managed

appropriately?

Impact (if feasible)

– Has the activity produced intended or unintended changes in the lives of

beneficiaries and their environment, directly or indirectly?

– Have there been positive or negative impacts from external factors?

Sustainability

– Do beneficiaries and/or partner country stakeholders have sufficient ownership,

capacity and resources to maintain the activity outcomes after Australian

Government funding has ceased?

– Are there any actions that can be taken now that will increase the likelihood that

the activity will be sustainable? Are there any areas of the activity that are clearly

not sustainable? What actions should be taken to address this?

Gender Equality

– Was the activity designed to provide equal participation and benefits for women

and men, boys and girls?

– Is the activity promoting equal participation and benefits for women and men, boys

and girls?

Sub-questions:

Is the activity promoting more equal access by women and men to the benefits

of the activity, and more broadly to resources, services and skills?

Is the activity promoting equality of decision-making between women and

men?

Is the initiative helping to promote women’s rights?

Is the initiative helping to develop capacity (donors, partner government, civil

society, etc) to understand and promote gender equality?

Monitoring and Evaluation

– Does evidence exist to show that objectives are on track to being achieved?

– Is the M&E system collecting the right information to allow judgement to be made

about meeting objectives and sustainability at the next evaluation point?

– Is data gender-disaggregated to measure the outcomes of the activity on men,

women, boys and girls?

– Is the M&E system collecting useful information on cross-cutting issues?

Analysis & Learning

– How well was the design based on previous learning and analysis?

11

– How well has learning from implementation and previous reviews (self-assessment

and independent) been integrated into the activity?

Lessons

– What lessons from the activity can be applied to (select as appropriate: further

implementation/designing the next phase of the activity/applying thematic

practices [i.e. working in partner systems/environment/fragile stages] to the rest of

the program/designing future activities).

Rating scale:

Satisfactory Less that satisfactory

6 Very high quality 3 Less than adequate quality

5 Good quality 2 Poor quality

4 Adequate quality 1 Very poor quality