review of economic model - defra,...
TRANSCRIPT
1
Warwick Economics
Review of Four CAP Finance Models
for DEFRA Review of Economic Models
Ken Warwick
7 June 2013
Foxwood, Pyrford Woods Close
Woking, Surrey
GU22 8QN
3
Review of Four CAP Finance Models for DEFRA Review of Economic Models
Background
1. As part of DEFRA’s internal review of modelling work in the Department, DEFRA’s Director of
Analysis and Chief Economist, Ulrike Hotopp, has commissioned a review of four spreadsheet models
that together contribute to policy making and financial management of CAP spending programmes.
These models were identified as business critical models as part of the Treasury review of Government
models and as part of the department’s own review of its modelling work.
2. The models for review include the RDPE Natural England Future Funding Model, the RDPE EU
budget model and two CAP finance models. The terms of reference (which are set out in full in
Annex 1) ask the reviewer to:
Review the data used including that sourced from Natural England
Review the formulae of the models (verification)
Review the representativeness of the models (validation)
Review the version control and governance of the models
Review the associated documentation and guidance
Review the communication of assumptions and uncertainties of the models’ outputs
Review the use of the model outputs for informing policy thinking and development Review how the models interlink.
3. Given that the models have already been subject to an internal peer review process, this
review focuses less on data and verification and more on the validation of the models, and on
governance, guidance and documentation, use in policymaking and scenario analysis and inter-
dependencies between the models. It also looks forward with recommendations for future
development of the models and any general lessons that these case studies suggest for DEFRA’s
modelling work.
Approach
4. Following an inception meeting on 1 February 2013 and background reading of
documentation provided, introductory meetings were held with model owners and developers to
discuss the models, the way they are used, their basic operation and the policy context. During these
meetings, model owners explained the structure of the model and ran simulations to illustrate model
properties. Following these meetings, a questionnaire was prepared for model owners. Responses to
the questionnaire provided much of the raw material for this report on issues such as data sources,
governance, documentation, policy impact and sensitivity analysis.
5. The reviewer was provided with copies of the four models in full in Excel spreadsheet form to
enable investigation and testing of the models to be undertaken off-site. In a second series of
meetings, in early March, model owners and developers answered technical questions on the
4
structure of the models and expanded on their responses to the questionnaire. Some additional
simulations were run, at the request of the reviewer, and ideas on preliminary findings were
informally discussed.
6. Following further discussion, including with Natural England and with departmental analysts
who had conducted internal reviews, interim findings were presented at a meeting with DEFRA
analysts on 14 March 2013. The remaining work programme was also agreed. Further investigation
and testing of the models were conducted off-site during the week beginning 18 March. Discussions
were also held by phone and email with DEFRA modellers and reviewers and with the Treasury.
Information was also provided by the department by email showing the policy impact of the models
and giving details of the work prepared for the Treasury review of departmental models.
7. The draft report was prepared during the week beginning 25 March and provided to DEFRA on
28 March 2013 for circulation to relevant staff. Following receipt of comments from the department,
the report was finalised in the week beginning 3 June 2013. General lessons for DEFRA’s modelling
work arising from these four case studies will be discussed as part of a workshop on modelling
planned by the department for 21 June 2013.
8. A list of all meetings held and those attending is attached as Annex 2.
Disclaimer
9. It has not been possible within the time scales, methodology and budget agreed for this
project to review every single line of coding in the four models. The findings of this review should be
read in that light. These are large and complex models, each designed differently and requiring
specialist knowledge of CAP mechanisms, EU budget rules and Exchequer arrangements. It is
therefore difficult to be 100 per cent confident that there are no errors in the coding or structure of
the model. Any decisions made by DEFRA on the basis of this review of these models are at the
Department’s own risk.
Structure of the report
10. The conclusions and recommendations of the review of the four models are summarised in
the next section (paras 12-24). The detailed report follows, starting with a description of the main use
and strategic purpose of the models, data sources and the structure of the models (validation and
verification). Building on this review and the detailed comments on the models set out in Annex 3,
some high-level suggestions are offered for best practice in future modelling (para 52).
11. Subsequent sections review arrangements for version control, quality assurance and other
aspects of governance and for documentation, guidance and training. This is followed by a discussion
of the impact of the models on policy making and the use of sensitivity analysis to test the robustness
of the models and to inform policy choice in the face of uncertainty. The report concludes with a
discussion of possible interactions between the models and a review of key risks (paras 117-121).
5
Overall findings and recommendations
12. No significant errors in formulae or design faults in the models have been uncovered that
would prevent the models from delivering as intended. Annex 3 provides a detailed list of minor
issues that have been identified – mainly to do with the coding, documentation, data sourcing and
presentation of the models.
13. Internal reviews have generally verified and validated the models. Some models are
calibrated to other data sources and/or have been successfully used for budget planning purposes,
giving further confidence in the robustness of the models.
14. In general, the two CAP Finance models are more elegantly programmed using more powerful
Excel functions than the RDPE models. This has the benefit of avoiding the need for a lot of manual
re-coding and the attendant risks but it introduces more complexity in the form of nested IF
statements and other conditional logic that make the formulae harder to follow.
15. The RDPE models are stronger in terms of update logs, documentation and governance. Of
the four models reviewed, FFM is the most mature and this is reflected in stronger governance and
better documentation. Other models are newer and/or more ad hoc.
16. All models would benefit from a User Guide which sets out the strategic purpose of the model,
a logic map or flow chart showing the main inter-relationships in the model, a detailed guide to the
model’s structure, and a guide – for both model developers and model users – on how the model
should be updated and used for scenario analysis (including screenshots and data sources).
17. Documentation within the models could also be improved – although all have some kind of
introductory description and promise tab-by-tab textbox guides through the model, notes on
spreadsheet tabs are often incomplete, out of date or missing altogether.
18. The models all seem to be used successfully in policy development with the modellers having
good links to the policy community. The RDPE EU Budget Model is perhaps least integrated into
policymakers’ thinking but nevertheless plays a central role in the finance team’s ability to respond to
changing circumstances.
19. Interaction between the models was considered in the review as an area of potential concern.
The Pillar 1 and Pillar 2 models are for the moment distinct from the other two, but together they
determine the EU funding envelope that will be available for the next programme period for the Rural
Development Programme. The FFM model determines the RDPE budget for Natural England which in
turn feeds into the RDPE EU budget management tool.
20. However, the linkages are not strong and the risks of significant adverse interaction effects
would not seem to be large. The outcome of the EU budget negotiations on Pillar 2, and the
remaining fine detail to be settled for Pillar 1, should be known with certainty before decisions need to
be made on the future funding of RDPE for the next programme period, thereby reducing the risks
attaching to projections of available financing necessary for management of the RDPE budget.
6
21. The most serious concern would appear to be around the dependence of the models on a few
key model developers, the lack of succession planning, and the need for user guides and associated
training, and improved governance.
22. The main lessons from this review for the department in its future work on model
development would appear to be:
There is scope for improvement in documentation of data sources, governance frameworks
and the provision of guides for users and future model developers.
The department should ensure that in future modelling work, the recommendations from the
McPherson review relating to the governance of individual models are fully implemented.
More formal arrangements should be established for version control and sign-off of business
critical departmental models.
Full use should be made of the analytical skills in the department, including statisticians and
Operations Research specialists with high level modelling skills.
There is scope for raising awareness of the use of models in policy making, with clearer
communication to policy officials of the strengths, weaknesses and uncertainties of modelling.
Templates and best practice guides could be prepared on spreadsheet model design, user
guides, including the use of logic maps or flow charts, and high level statements of the
strategic purpose and main use of the model.
Ideally, models should be structured to clearly distinguish data entry from calculated cells and
designed with a user friendly accessible ‘front end’ that will enable scenarios and sensitivity
analysis to be carried out more easily.
It is however important to take a proportionate approach. Some models are developed for
specific one-off purposes and used in real time and it is inevitable that they will have to
develop ad hoc in response to changing demands. In those circumstances, every effort should
be made to log updates and ensure key stage versions of the model are saved and stored. If
models are to be used again, they should be streamlined and fully documented at the
appropriate stage.
23. The review has demonstrated that there is much good practice in modelling in the
department. Good examples have been found of strong governance frameworks, careful
documentation of data and logging of model updates, good within-model annotation, close
integration with policy colleagues, helpful use of sensitivity analysis, and elegant programming of
powerful models with strong capabilities for scenario analysis. There is also ample evidence of an
“appetite for continuous improvement” in the department’s approach to its modelling work.
24. However, DEFRA should consider ways of spreading best practice in modelling more widely in
the department, through for example, the use of best practice guides, templates, more formal
arrangements for analytical peer review, appropriate training for those involved in using models or
their results and a central source of advice on analytical modelling.
7
Main purpose and use of the models
25. This section sets out the main strategic purpose of the four models and how they are used,
drawing on information from various sources, including guidance notes and in-model documentation.
The aim is to give an introductory overview of the models, not available elsewhere (at least not in a
single place), before delving into the detailed review. The department has also provided high-level
information on the models under review to the Treasury as part of its review of quality assurance in
Government analytical models and this is reproduced in Table 1 below.
Future Funding Model
26. The main purpose of the Future Funding Model is to explain the financial implications of the
Rural Development Programme for England (RDPE) during the current programme period. The model
has been set up to calculate Natural England's RDPE Budget requirement across the remainder of the
programme. It allows users to calculate the budget requirement under various scenarios, using the
'variables' which have been incorporated into the model.
27. The model uses known spend and commitments, as well as estimated future uptake, to
calculate a total projected commitment (on a UK financial year resource basis). It then allows for an
element of unspent commitments to give a total projected spend. The total budget requirement,
including Exchequer funding, is reviewed between DEFRA and NE with RDP finance teams and the
overall budget is calculated from funding sources available across the programme rather than relying
on the model to calculate it.
28. Outputs from the Future Funding Model feed into the RDPE EU Budget Model alongside
projections of spend from other agencies delivering rural development programmes and in
combination with assumptions about exchange rates and available budgets.
29. The model is used for scheme design, financial profiling, value for money decisions,
prioritisation of delivery, future financing scenarios and EU negotiations.
RDPE EU Budget Model (or Exchange Rate tool)
30. The main purpose of the RDPE EU Budget Model (or Exchange Rate tool) is to monitor the
amount of EU funding available to the end of the current RDPE programme and the extent to which
this is dependent on forecasts of the euro/sterling exchange rate. The model is one tab in a larger
spreadsheet (Dashboard) which is used for a variety of budget management purposes.
31. The model is used for Exchequer planning, risk management, monitoring affordability and re-
profiling Exchequer and EU funds to ensure the most effective use of EU funds.
8
CAP Finance model – Pillar 1 (Income Support)
32. The model attempts to replicate the Commission’s published figures for the distribution of
Pillar 1 payments (direct payments and market mechanisms) between Member States and in particular
to explore the implications of different scenarios for EU budgetary convergence mechanisms. It takes
the Commission’s proposed model of ‘convergence’ (which drives Member States’ direct payment
rates per hectare towards the average payment rate for the EU) and builds on this to enable the user
to change given parameters for redistributing payments. The aim is to assess the implications of a
given change to the convergence mechanism and the implications for each Member State in terms of
the payment they receive. The model is calibrated to the Commission’s published figures, which acts
as a useful external check on how the model generates outputs.
33. The model is used to explore the implications of different convergence schemes for UK
receipts from the EU Budget and the associated Exchequer financing requirements.
CAP Finance model – Pillar 2 (Rural Development)
34. The main purpose of the model is to examine the distribution amongst EU Member States and
the funding implications for each country of the Commission’s proposed objective criteria for Rural
Development allocations in the 2014-20 EU Budget period. It does not, however, take affordability
into account; that is to say, it could be the case that if UK shares of EAFRD were to increase after
applying objective criteria, existing rates of co-financing with Exchequer funds may not be possible.
35. The European Commission have proposed two possible allocation keys in order to distribute
funds for the 2014-20 Financial Perspective. The distribution in the current period (2007-2013)
reflects to a large extent the historical shares of Member States in the Guarantee, Guidance and
Leader funds that were brought together into a single fund, the EAFRD. The model allows comparison
of different allocation keys and allows these to be benchmarked against historic shares. The model is
used to explore the implications of different allocation keys for UK receipts from the EU Budget and
the associated Exchequer financing requirements.
9
Table1: Model descriptions provided to HM Treasury review
Model name and
type Description Why business critical Summary of QA
Future Funding Model
In-house funding model, operated jointly with Natural England, to determine budgets for the Rural Development Programme for England (RDPE). [Policy Simulation]
The model converts business assumptions and policy targets into financial commitments. RDPE agreements are largely 5 or 10 year agreements. The model provides outputs of financial commitments going forwards over the longer term. Annual budgets are set and future programme funding pressures feed into policies to develop new programme.
The model highlights pressures, can simulate different policy scenarios and give policy impacts of various funding decisions.
Internal peer review and internal audit. The model will shortly undergo an external peer review.
RDPE EU Budget
Model
Exchange Rate Impact Calculator. [Forecasting]
The model assesses the impacts of spend, forecast spend and exchange rates on the EU budget for rural development.
Model is used for critical decisions around an EU allocation. Model outputs are used for risk manag-ement in relation to programme affordability, progress against EU spend targets and the risk of surrendering EU funds.
Internal peer review and internal audit. The model will shortly undergo an external peer review.
CAP Finance Models
Suite of Common Agricultural Policy (CAP) finance models, developed in-house. [Forecasting]
The models analyse CAP
spending over the next EU
budget cycle from 2014-
2020, broken down by
member state. It can be
used to analyse various
alternatives to the
Commission proposals.
The model is not available
publicly. The EU has
produced their own
analysis, but has not
shared their underlying
modelling.
The models are an
essential input to the UK
position in negotiations of
the next EU budget and
various proposals. They
enable the calculation of
estimates of UK and other
EU member state receipts
from the CAP, and thus the
financial impact on DEFRA,
UK taxpayers and farmers.
Delivering the CAP is a
central part of the DEFRA
business plan.
Developed in-house to
respond to the evolving
negotiations the models
have been validated by
comparison to EU
Commission estimates.
They have been subject to
internal quality assurance.
HMT officials have also
analysed these proposals
and are working with
DEFRA on using this
modelling, thus providing a
further cross-check on the
accuracy of the models.
Source: HM Treasury (2013), “Review of quality assurance of Government analytical models: final report”,
Annex D – Departmental Returns.
10
Data
36. Model owners were asked to provide details of the main sources of data used in their models.
The documentation of data sources was also checked during the course of reviewing the models and
some of the published sources were spot-checked. The main findings are outlined in the rest of the
section, which draws heavily on material provided by model owners. However, it is important to
stress at the outset that it has not been possible in the scope of this review to verify all the underlying
data as correct. As with any complex model, there is a risk of incorrect or inconsistent data being
entered and it is recommended that data entry is checked carefully for all the models under
consideration.
Future Funding Model
37. Natural England (NE) input data from their IT system, Genesis, to populate historic spend and
current commitments and to extrapolate future uptake. The forecast uptake includes new business
assumptions such as the average cost of an agreement, number of hectares expected to be signed up
and average cost per hectare. Historic spend figures are agreed with the DEFRA finance team against
that recorded in DEFRA’s financial accounting systems. New business assumptions are agreed with
input from the relevant policy team.
38. References to data sources are included in the spreadsheet in most cases, except for the
business assumptions which come from NE’s business specialists – the NE finance team keeps a record
of this. The assumptions made about new business are probably the biggest chance of out-of-date or
incorrect data being used in the model. The data sources for ‘actual’ figures look like they can be
referenced in a robust and consistent way; but the source for ‘forecast’ assumptions seems to be
more ad hoc and therefore harder to verify.
39. Another potential data-related risk is the way that the data needs to be entered in the model.
This risk could be reduced by changing the structure of the model in ways that would minimise the risk
of data being entered or updated incorrectly. (This is discussed further in the next section).
40. A governance note, an outline note on functionality of the model, a guide to the methodology
of the FFM and a note on new business assumptions have all been prepared and provided to the
reviewer. These are all helpful as guides to new users, but could be more explicit about data sources.
From conversations with the teams in DEFRA and Natural England, the current modelling team would
appear to be on top of the data requirements of the model and to be alive to the importance of
control of data entry. It will be important to ensure that there is continuity in this regard if any team
members move on.
11
RDPE EU Budget Model
41. EU budget figures for the programme period come from the RDPE programme document.
Spend to the end of the previous financial year comes from EU reports; spend for the current year
comes from quarterly claims submitted to the European Commission by the Rural Payments Agency
(RPA). Resource expenditure forecasts are from RDPE delivery bodies (in the case of NE, from the
most recently signed-off Future Funding Model forecasts) and the funding sources are managed within
the overall EU budget and known Exchequer settlements. High, median and low assumptions for the
projected euro/sterling exchange rate are those of independent forecasters and sourced from
Bloomberg’s.
42. References to data sources are included in the headings to the data columns and are also
recorded in a methodology document provided to the reviewer. These data sources will no doubt be
familiar to expert DEFRA users, but it would be helpful to provide more precise references, with
document links or web links where possible, both in the methodology note and within the model.
43. Data entry requirements are much simpler than for the Future Funding Model but it would
again be helpful if columns for data entry were marked as such and any duplication of data entry
avoided.
CAP Finance models
44. For the two CAP Finance models, the main data sources are Commission regulations and
methodological notes, references to which can be found within the respective model. Links to data
sources are within the model and this data has been copied into the spreadsheet. Background and
links to the main source data can be found in the ‘ReadMe’ tab and where relevant in the working
spreadsheets themselves.
45. For the most part, the source of data is clear and the risk of data entry error is less than for the
Future Funding Model. Some of the links provided do not take the user to the precise web page from
which the data is sourced and this should be corrected. Given that the models are seeking to replicate
Commission calculations, there is a risk that the source data used does not match exactly the
Commission’s data – this may be part of the reason for the remaining (minor) discrepancies between
the model’s outputs and the Commission calculations.
46. As with the RDPE models, clearer signposting of cells for data input and avoidance of duplicate
data entry points would make the models easier to follow and minimise the risk of incorrect data
entry.
12
Structure of the models - validation and verification
47. During the course of the review, a significant amount of time was devoted to understanding
the structure of the four models and validating that the models do what is expected of them, in
particular that they adequately represent the financial and other processes being modelled.
Spreadsheets were studied in detail and their structure validated. The accuracy of cell formulae was
also verified through spot checks and simulations of the model, although in the time available it has
not been possible to verify that every formula in the thousands of lines of coding embedded in the
model is error-free.
48. That said, no significant errors in formulae or design faults in the models have been uncovered
that would prevent them from delivering as intended. A detailed list of minor issues that have been
identified is attached as Annex 3, which sets out for each of the four models comments on model
structure, documentation, data sourcing and presentation of the models and some minor errors in
coding or annotation.
49. The reviewer benefited from seeing earlier internal reviews of the four models. The review of
the two RDPE finance models was particularly thorough and made many recommendations for
improvement which should be implemented. Changes as a result of the internal review have already
been made to the RDPE EU Budget Model and are being considered in the context of the latest update
of the FFM model. Changes recommended by the internal reviewer to the CAP Finance models were
mainly to do with presentation and linking to data sources, rather than structure or coding, and most
of these recommendations have been implemented.
50. The fact that earlier internal reviews have already generally verified and validated the models
gives added confidence that the models are sound in terms of their structure and coding. Internal
reviewers confirmed in telephone conversations that they are confident that the models are basically
sound, and the high degree of confidence in the underlying soundness of the models was echoed by
model owners and users. In the case of the CAP Finance models, this comes in part from the fact that
the models are calibrated to other data sources and/or have been successfully used for budget
planning purposes, giving further assurance of the robustness of the models.
51. In general, the two CAP Finance models are more elegantly programmed, using more powerful
Excel functions (for example, SUMIF, LOOKUP and other conditional formulae) than the RDPE models.
This has the benefit of avoiding the need for a lot of manual re-coding if data changes, with the
attendant risks. The trade-off is that it introduces more complexity in the form of nested IF
statements and other conditional logic that make the formulae harder to follow. On balance, the
approach taken in the CAP Finance models is to be preferred as it gives the models greater power as
simulation tools and makes them more robust for new users. But it needs to be accompanied by full
documentation to enable the inexpert user to understand how the model works.
52. Building on the detailed comments on the models set out in Annex 3, the following are some
high-level suggestions for best practice in future modelling:
13
- Avoid mixing data and formulae in the same cell. Input data should be entered in a distinct
area of the spreadsheet and preferably marked or colour-coded as data entry. Duplicate data
entry should be avoided to reduce the risk of error or inconsistency.
- More use should be made of data validation tools, such as checksums or restricted value
ranges, in order to minimise the risk of errors in data input.
- As with data entry, it is good practice to design the model so that any variables or parameters
of particular interest for simulations are identified and shown separately in a highlighted area
of the workbook. The RDPE Budget models would benefit from making this change.
- Models should be designed, as with the CAP Finance models, with a user friendly ‘front end’ in
which key results are displayed and where parameters or variables can be adjusted to run
scenarios and simulations with the model. A baseline or reference scenario should be
preserved to allow ‘before’ and ‘after’ comparisons.
- Layouts – for example rows of country order or columns of dates – should as far as possible be
maintained from one ‘tab’ to another.
- Where totals are to be derived from cell entries conditional on, for example, dates (as in the
case of the Future Funding Model), SUMIF functions should be used rather than manually
selecting the relevant cells. In general, formulae should designed in a way that allows them to
be replicated throughout an entire column (or row).
- Where IF statements and switches are being used, they should be kept as simple as possible.
In the CAP Finance models the limiting case of a parameter or variable being set to zero is
often treated as a special case, when it need not have been programmed as such.
14
Version control, governance and quality assurance
53. The Treasury’s review of quality assurance of Government analytical models stressed the
importance of having strong systems in place for quality assurance and governance of departmental
models:
“Some departments and their arm’s length bodies (ALBs) have a clear and structured approach
to quality assurance and a well-defined governance framework. There is much that can be
learnt from this. Equally, almost all models use developer testing and internal peer review,
demonstrating there is a basic application of quality assurance across the board. A significant
proportion had key elements of the model in the public domain, enabling external scrutiny.
Similarly, the review found an appetite for continuous improvement across government, with
many departments and their ALBs assessing their internal processes alongside the work of the
review“. (HM Treasury, 2013, op. cit.)
54. Many of the recommendations from the Treasury’s review, led by Nick McPherson, relate to
department-wide systems and culture, which are beyond the scope of the current review. The
recommendations relating to individual models are summarised in Box 1.
55. Model owners for the four models under review were asked to provide information on the
procedures for version control and governance and the following assessment is based on their
responses. The position with respect to formal quality assurance processed, as reported to the
McPherson review, is summarised in Table 2 and discussed in more detail in what follows.
Box 1: McPherson review recommendations (relating to individual models)
• Recommendation 1: All business critical models in government should have appropriate
quality assurance of their inputs, methodology and outputs in the context of the risks their
use represents. If unavoidable time constraints prevent this happening then this should be
explicitly acknowledged and reported;
• Recommendation 2: All business critical models in government should be managed within a
framework that ensures appropriately specialist staff are responsible for developing and using
the models as well as quality assurance;
• Recommendation 3: There should be a single Senior Responsible Owner for each model
(“Model SRO”) through its lifecycle, and clarity from the outset on how QA is to be managed.
Key submissions using results from the model should summarise the QA that has been
undertaken, including the extent of expert scrutiny and challenge. They should also confirm
that the Model SRO is content that the QA process is compliant and appropriate, that model
risks, limitations and major assumptions are understood by users of the model, and the use of
the model outputs is appropriate;
15
Table 2: Quality assurance arrangements for the models under review
Future Funding Model
RDPE EU Budget Model
CAP Finance models
Developer testing
Internal peer review
External peer review
Use of version control
Internal audit
Quality assurance guidelines
External audit
Governance
Transparency (published results)
Periodic review
Source: HM Treasury (2013), “Review of quality assurance of Government
analytical models: final report”, Annex D – Departmental Returns.
16
Future Funding Model
56. The reviewer was provided with a one-page note on the governance framework for the Future
Funding Model (FFM), responsibility for which is shared between DEFRA and Natural England as
follows:
- NE’s RDPE Finance Team manages the business assumptions that provide forecasts of the
level of business that is expected to be delivered through each scheme against policy
objectives and priorities agreed with DEFRA.
- DEFRA’s RDPE Finance Team manages the budgets and financial engineering assumptions
within the model providing a means through which to allocate the appropriate level of EU and
UK funds to resource future business requirements.
- The FFM will be subject to a major update at the start of each financial year which will replace
the previous year’s forecasts with the year-end actual position taken from DEFRA’s resource
accounts. NE will also update their business inputs with any adjustments required to reflect
their existing legal commitments moving forward. NE will accompany each annual update
with an explanatory paper that provides supporting justification for the new business
assumptions. Amendments will be discussed in detail by DEFRA and NE before recommending
approval at the NE DLG.
- The FFM will be reviewed throughout the year by both together. Amendments will be driven
by changes in policy, business assumptions or financial constraints. Only material changes will
lead to amendment and possible re-submission of the FFM.
- DEFRA and NE may also make copies of the latest agreed version of the FFM to test
adjustments to their working assumptions to identify possible updates required. These
versions will remain independent of the latest agreed position and only the latest agreed
version will be used for planning purposes.
- Future developments to the FFM will be agreed between the DEFRA and NE RDPE Finance
Teams in consultation with their respective policy and delivery teams and/or in response to
key recommendations that are highlighted in SKI reviews. Future developments to the FFM
will be implemented through a clear project planning timetable and subsequently monitored
by DEFRA and NE through the Business and Finance reviews.
57. Version control is assured through saving electronic files on DEFRA servers according to
agreed version control specifications.
58. In addition to version control and the governance framework, the FFM model has been
subject to other quality assurance processes, as summarised in Table 2. Developer testing of
functionality has been undertaken by internal skills, knowledge and information teams. The model
has been peer reviewed by teams independent of the work of RDPE as well as being subject to internal
audit checks and quality assurance guidelines, and is currently undergoing external peer review. There
is a clearly documented sign-off process at deputy senior reporting officer (DSRO) level and agreed use
of versions with the delivery body. Final budgets based on the model are made public, although not
17
the outputs of the model itself. The Future Funding Model is only two years old and is entering its
second review, so it is too early to comment on the record of periodic reviews over the model’s
lifetime. The first review declared the model fit for purpose with some recommendations for
improvement.
59. The arrangements for version control, quality assurance and governance would appear to be
fit for purpose, and indeed, would appear to be the strongest and most formalised of the four models
under review. There is a formal governance framework and robust procedures for developer testing
and review; and the staff involved have the necessary relevant expertise.
60. However, as with any model shared between two bodies, there is a risk that shared ownership
may lead to responsibility not being clear. Expertise on different parts of the model, and on its
structure, inputs and outputs, is divided between the DEFRA finance team and Natural England. While
this works well with the current team, it might be more difficult with a change in personnel. There
may also be a case for more involvement of the analyst community (particularly statisticians and/or
operations researchers) in the future development of the model to ensure that best practice in
analytical model ling is fully incorporated.
RDPE EU Budget Model
61. Arrangement s for governance and version control are set out briefly in a three-page note
provided to the reviewer on the methodology of the RDPE EU Budget Model. The extract on version
control and governance is reproduced in full below:
“[Version control] is maintained through dating the model with the date (YYMMDD) that the
model was ‘published’ which normally ties into when there has been a relatively significant
update to it, whether that is through changes to delivery body forecasts for the remainder of
the programme, moving funding sources around as a result of requests to make savings or
when exchange rate forecasts are updated.
Governance of the model
This is something that is used internally by the RDPE Finance Team (two members). There is
no formal sign off from one version of the model to the next from outside of that.”
62. The team explained that changes from one version to the next are recorded in the model itself
(starting in cell AV60). When a new RDPE EU Budget Model is agreed, there is an exercise to ensure
that the outputs from the previous model can be reconciled with the outputs from the new version.
This provides a useful cross-check on the workings of the model and helps ensure that the latest
version matches and incorporates the latest information.
63. In addition to version control and the governance framework, the RDPE EU Budget Model has
been subject to other quality assurance processes, again summarised in Table 2 above. As the
exchange rate model is only just over a year old, it is at an even earlier stage in terms of developer
testing and periodic review. But it has been subject to internal review, as well as internal audit and
quality assurance, and been assessed as fit for purpose. It is currently also undergoing external peer
18
review. Change control sits with the Rural Development Programme (RDP) policy team and outputs,
though not the model itself, are reviewed by the RDPE Programme Board. Final budgets based on the
model are made public.
64. The arrangements for version control, quality assurance and governance for the RDPE Budget
model are much less mature than for the FFM. There are also some risks attaching to the fact that the
model has evolved from a tool primarily used for managing budget pressures arising from exchange
rate fluctuations into a more general tool for managing the EU and Exchequer funding streams for the
RDPE EU budget. While the data input for the model is relatively straightforward and the existing two-
person team have a thorough understanding of the model and how it should be used and adapted,
there would be benefit from creating a more formal governance framework around the model, with
more formal oversight and sign-off. Such a step would have the added advantage of giving the model,
and its growing role in financial management, more profile within the department.
65. As with the FFM model, there may also be a case for more involvement of the analyst
community (particularly statisticians and/or operations researchers) in the future development of the
model to ensure that best practice in analytical model ling is fully incorporated.
CAP Finance models
66. Day to day responsibility for running and maintaining the models sits with the model
developer. Overall responsibility rests with the model owner under the oversight of a SRO. The model
developer has leeway in terms of updating the models where minor adjustments improve the accuracy
and/or usability of the models. Such changes are notified to the model owner for Quality Assurance.
Major developments, which usually stem from a need to recalibrate the models’ projections, require
approval from the model owner. Documentation of the models is the model developer’s responsibility
in the first instance with the model owner approving all final documentation. In addition, the SRO
provides oversight and review of the major development of the models.
67. As the importance of the models grew in the lead-up to the EU Budget negotiations, a peer
review was initiated to provide a quality assessment. The peer review was conducted by someone
outside the management chain responsible for the models. Following this, some recommendations
were made which were taken on board in the models’ development. As negotiations on CAP enter the
next phase it is likely that further model development will be needed. Depending on the nature and
significance of the developments a further peer review may be required.
68. For the direct payments (Pillar 1) model versions are named with a yyyymmdd prefix (latest
version is 20130208). The Pillar 2 model is named according to the version number (latest is v10).
Previous versions of the model are kept in separate folders. Each significant iteration of the models is
kept for comparison with later versions
69. In addition to version control and the governance framework, the CAP finance models have
been subject to other quality assurance processes, also summarised in Table 2 above. The models
have been developed over a relatively short time period and to tight time scales, so the extent of
formal governance, review and audit, is somewhat less than for the RDPE EU budget model. However,
19
the models have undergone developer testing and validation against EU Commission results and the
modelling team have worked closely with Treasury officials who have provided a further cross-check
on the accuracy of the models. The models have also been the subject of internal peer review and
quality assurance and are currently undergoing external review. The results are not in the public
domain, but are shared with HMT, as has some of the modelling spreadsheets themselves. It is likely
that the models will have a limited shelf life – the negotiations are all but complete, and the models
have essentially fulfilled their main purpose, though there may be some additional follow up work, for
example to analyse intra-UK allocations.
70. The arrangements for version control, quality assurance and governance for the CAP Finance
models are the least highly developed of the models reviewed here. This is perhaps inevitable given
the need to respond to tight deadlines in sometimes fast-moving negotiations. Although formal
governance, sign-off and audit arrangements were relatively light touch, adequate checks and
balances were assured through clear ownership of the model, close working with the Treasury and the
ability to calibrate results on Commission figures and other Member States’ calculations. The close
involvement of analysts, working with Treasury and DEFRA policy colleagues, is an advantage and
appears to have worked well. In any further development of the CAP Finance models, or similar
models in the future, consideration should be given to putting the governance arrangements on a
more formal basis at an early stage in the process.
20
Documentation, guidance and training
71. The four models under review are all large and complex spreadsheets, developed in-house
and relatively recently. In order to improve transparency, facilitate understanding amongst users,
reduce vulnerability to staff changes and promote continuity, it is highly desirable that such models
are accompanied by appropriate guidance and documentation, including a high-level statement of the
main strategic purpose and use of the model.
72. The importance of a strategic overview in particular was emphasised in the McPherson report:
“Making models as intuitive as possible can help drive transparency. Consultancy and
accounting firms emphasised this point. They pointed to a number of techniques they employ,
which include providing a guide upfront of what the model does, in prose not numbers; clearly
structuring presentation of the model with key findings and graphs; and a logic map of the
model. This makes the model easily accessible to reviewers, and so facilitates scrutiny.”
(HM Treasury, 2013, op. cit.)
73. The reviewer was provided with copies of the documentation available for the four models
under review. For the Future Funding Model, this consisted of a one page note on functionality and a
five-page methodology note (which turned out to relate to a different version of the model to the one
under review). For the RDPE Budget model, a three page note on methodology with a column-by-
column explanation of the model was provided. For the CAP Finance models, all the documentation
sits within the model itself in the form of a ReadMe tab and comments and text boxes embedded
within the workbook. Relevant background material was also provided for the CAP Finance models
and, for all four models, the reviews prepared by internal reviewers provided further useful
background on the role, purpose and structure of the models.
74. None of the documentation provided fully met the exacting standards set out in the extract
from the McPherson report quoted above. All four models would benefit from the drawing up of a
User Guide (or guides) setting out the strategic purpose of the model, a logic map or flow chart of the
main inter-relationships in the model, a detailed guide to the model’s structure, and a guide – for both
model developers and model users – on how the model should be updated and used for scenario
analysis (including screenshots and data sources).
75. In addition, all four models would benefit from better documentation of sources,
explanations of modelling strategy and annotations describing key cells or blocks of calculation within
the model. Although all four models have some kind of introductory description and promise tab-by-
tab textbox guides through the model, notes on spreadsheet tabs are often incomplete, out of date or
missing altogether. The CAP Finance models are already pretty strong in this respect, but even here
there is scope for improvement to ensure that guidance is up to date and complete.
76. Model users should also have access to training in the use of the models. For the RDPE
finance models, there is an arrangement in place via on-the-job training for financial users, including a
full-day workshop with Natural England on the FFM. For the CAP Finance models, the team have
provided introductory sessions on the model to policy colleagues with a view to informing them about
21
possible uses of the model, for example running scenarios on changes to the EU budget. These
arrangements would seem proportionate.
77. Of more serious concern is the dependence of the models on a few key model developers, and
the need for appropriate succession planning and the associated training, particularly in the absence
of detailed user guides. The modelling teams described arrangements for informal training when a
new developer joins the team, including ensuring that handover notes between model developers are
comprehensive, and wherever possible allowing a period of overlap between developers. However,
with all the models being relatively new, these arrangements have not yet been fully tested and it
remains an area of vulnerability, particularly in the event of a sudden staff change.
78. All the teams are aware of this and Natural England Finance are taking steps to ensure cover
internally with some training already carried out to bring another member of the Finance team up to
speed on the Future Funding Model, and the identification of an emergency successor within the NE
RDPE finance team (who is however still to be trained).
22
Policy impact
“An effective process involves ongoing engagement between specialist and policy staff to
ensure there is a shared understanding about the purpose and any limitations of a model. This
should include sensitivity analysis, and the degree of uncertainty about model inputs,
assumptions and outputs.” (HM Treasury, 2013, op. cit.)
79. The models all seem to be used successfully in policy development with the modelling teams
all demonstrating evidence of good links to the policy community. The RDPE EU Budget Model, being
relatively new, is perhaps least integrated into policymakers’ thinking but nevertheless plays a central
role in the Finance team’s ability to respond to changing circumstances.
Future Funding Model
80. The key customers for the output of the model are Natural England, DEFRA finance, DEFRA
managing authority and DEFRA policy teams. The main use of the model has been to inform policy
thinking and development on aspects of scheme design, financial profiling, value for money decisions,
prioritisation of delivery, future financing scenarios and EU negotiations. Decisions on how hard
Natural England should promote UELS schemes, for example, are contingent on what the modelling
shows to be the likely availability of funding. The outputs from the FFM also feed into the RDPE EU
Budget Model and contribute to decisions on the use of EU funding sources and dealing with pressures
on Exchequer finance.
81. The reviewer was provided with examples of FFM model outputs being used in policy
submissions. One showed Ministers agreeing to an adjustment to Natural England’s budget and a
consequent need to revise their new business assumptions on the basis of the FFM model. A second
submission, though not referring to the FFM directly, reported on higher than expected uptake of ELS
and advised Ministers of the budgetary consequences calculated using the model. A key test of the
strength of the model and its use in financial decisions making is that financial delivery has been within
98 per cent of budget in each of the last two financial years.
82. The evidence from interviews and documentation provided suggests that the model
developers working on FFM are closely involved with policy colleagues and ensure the model is used
to influence financial decision making. A potential risk is that, with several policy customers using the
model for different purposes, priorities and responsibilities may not always be clear. More could
perhaps also be done to ensure that, in line with the McPherson report recommendations, the
uncertainties about model inputs, assumptions and outputs are clearly communicated and illustrated
through sensitivity analysis (discussed further in the next section).
RDPE EU Budget Model
83. The key customers for the output of the model are DEFRA finance and DEFRA managing
authority. The main use of the models has been to inform policy thinking and development on aspects
of Exchequer planning, risk management of affordability issues, and the reprofiling of Exchequer and
EU funds to ensure effective use of EU funds
23
84. The reviewer was provided with examples of RDPE EU Budget Model outputs being used in
policy discussion. One showed the model being used to advise Ministers on the amount of
unallocated RDPE funding and making recommendations on how it should be used to manage financial
pressures in the transition period between the current RDPE programme and the new programme.
Another described possible funding scenarios for 2014/15 using projections from the model under a
range of different assumptions about exchange rates and options for re-profiling expenditure.
85. The evidence from interviews and documentation provided suggests that, although the model
is newer, more technical and less widely understood, it is nevertheless used actively in financial
decision making, albeit as an input to discretionary adjustments to spend. The model is used to
illustrate different scenarios and is by its nature a tool for sensitivity analysis around exchange rate
assumptions. More could perhaps be done to communicate the power of the model to other
members of the DEFRA Finance team, as the model could potentially be used for a wider range of
purposes. However, if this were to be done, the model might need to be restructured, as it still
betrays its origins as a tool primarily designed for handling exchange rate uncertainty.
CAP Finance models – Pillar 1
86. The key customers have been colleagues in the DEFRA CAP negotiations team, the EU budget
leads at HM Treasury and the agricultural policy team at UKRep in Brussels. Discussions with Treasury
have confirmed that the modelling team played a key role working alongside DEFRA policy colleagues
and Treasury in support of the UK’s negotiation position in Brussels. Treasury commented that the
UK’s capacity to model different proposed budget mechanisms compared favourably to most other
Member States.
87. Examples of policy impact for the CAP Pillar 1 model have been provided by the team and in
interviews with Treasury. The model was used to examine the impacts on member state shares of
Pillar 1 funding in different scenarios for redistributing EU funds under the Commission’s convergence
proposals. Evidence from documentation and interviews suggest that this was of key importance in
informing the negotiations on the EU Budget outcome. The model was used both to compute the
financial implications for the UK and to give insights into whether other Member States would support
different variants of the convergence mechanism. A document was also produced on possible
scenarios for redistribution to meet multiple objectives for the UK and shared with HM Treasury.
88. Although the current budget negotiation is all but over, there may be elements of the model
that need to be retained for the next round of budget negotiations. In addition, in contrast to the
Pillar 2 model, the Pillar 1 model does not yet have the functionality to look at how direct payments
might be allocated within the nations of the UK. A decision still has to be made on whether to
incorporate this capability into the existing Pillar 1 model, or to develop a separate model rather than
adding additional complexity to the existing model. Either way, there will be a need to incorporate the
elements of the existing model design and strategy in the within-UK modelling.
89. The evidence from interviews and documentation provided suggests that the modelling work
on convergence mechanisms was integral to the UK’s policy position and negotiating stance in the
24
discussions on the EU Budget and convergence mechanisms for CAP direct payments. The model was
used for scenario analysis, and the major uncertainties inherent in seeking to replicate an unknown
Commission calculation seem to have been clearly communicated to policymakers.
CAP Finance models – Pillar 2
90. As for Pillar 1, the key customers have been colleagues in the DEFRA CAP negotiations team,
the EU budget leads at HM Treasury and the agricultural policy team at UKRep in Brussels. Treasury
have also commented favourably on working relationships and the team’s effectiveness.
91. Examples of policy impact have been provided by the team and in interviews with Treasury.
Although the final decisions on allocation mechanisms have moved away from the objective allocation
basis that was the focus of the main modelling effort, the modelling was nevertheless important for
policymakers during the course of the negotiations. Another strength of the Pillar 2 model is its ability
to bring together all the different sources of EU and domestic funding. It provides an overall
assessment of the level of funding available for UK and English rural development programmes. In
addition the model provides clarity around how EU level allocation keys could, if they were applied
domestically, impact on the intra-UK allocation.
92. Although the major decisions at EU level have been taken, some of the fine detail for the
Pillar 2 allocations still has to be settled. Adjusting the model to take account of the ring-fenced
amounts for Pillar 2 allocated to certain Member States in the February 2013 EU Budget settlement is
an area for further possible development, although it is not seen by the team (or Treasury) as a
pressing need. An objective system of allocating Pillar 2 funds was one of the UK’s key objectives in
the negotiations and is likely to be so in future, so work done on modelling this should be maintained
for use to support future negotiations.
93. The evidence from interviews and documentation provided suggests that the modelling work
on objective allocation mechanisms for rural development payments provided essential background to
the UK’s policy position and negotiating stance in the discussions on the EU Budget and CAP rural
development payments. The model was used for scenario analysis, and the major uncertainties
inherent in seeking to replicate an unknown Commission calculation seem to have been clearly
communicated to policymakers.
25
Use of sensitivity analysis
“A perhaps apocryphal, but quite believable, story circulates about an economist’s attempt to
describe his uncertainty about a forecast to President Lyndon B. Johnson. The economist
presented his forecast as a likely range of values for the quantity under discussion. Johnson is
said to have replied: ‘Ranges are for cattle. Give me a number.’ ” 1
94. While the demand for a single number is understandable and point estimates have their uses,
it is nevertheless important, in the face of uncertainty, for model users and policy customers to
understand the possible range of outcomes. Uncertainty could arise from assumptions about the
state of the economic environment or from assumptions used in designing the model, its underlying
parameters or the data used to populate the model.
95. All the models under review are used in one way or another for sensitivity or scenario analysis.
However, by their nature, the models vary in the extent to which sensitivity analysis and the capability
for generating scenarios are built into the model and systematically used. The CAP Finance models are
well designed for scenario analysis, and the RDPE EU Budget Model has sensitivity analysis for
different exchange rate assumptions built into it. By contrast, the FFM model is less well adapted for
use in sensitivity analysis although it has been used to test key assumptions.
Future Funding Model
96. The modelling team identified the most important assumptions underpinning the model as:
costs per unit area/agreement, forecast uptake, convert to spend rates and budget constraints.
97. The FFM lacks sensitivity analysis built into the model, but it has been used to generate
scenarios using different assumptions, for example modelling different uptake levels against the next
programme. The key areas identified by the team where further sensitivity analysis would be useful
are forecasts of uptake in future years and average cost of an agreement. This would seem to be a
minimal list and could be expanded.
98. There is scope for developing the FFM to allow more systematic use of sensitivity analysis.
This would require some restructuring of the model to make it more user-friendly for scenario
analysis, identifying key assumptions and parameters that should be subjected to sensitivity analysis,
establishing a reference scenario as a baseline, and adding a section to the governance paper or user
guide detailing a process for running scenarios and recording their outputs.
RDPE EU Budget Model
99. The RDPE modelling team identified the most important assumptions underpinning the model
as: future year forecasts of resource spend, resource to cash adjustments, availability of Exchequer
funding, and euro/sterling exchange rate forecasts.
1 Quoted by Charles Manski, “Public Policy in an Uncertain World” – CEMMAP lecture at the British Academy,
London, 27 March 2013.
26
100. The RDPE EU Budget Model is by its nature a sensitivity analysis tool, at least with respect to
the exchange rate assumption setting out high, low and median exchange rate variants based on the
forecasts of independent forecasters, provided by Bloomberg’s. The model is also used to explore
different assumptions for allocating sources of funding and re-profiling expenditure.
101. As with the FFM, there is scope for developing the RDPE EU Budget Model to allow more
extensive and systematic use of sensitivity analysis. This would require extending the capability of the
model for scenario analysis beyond exchange rates, which in turn would some require restructuring of
the model. As with the FFM, the model could be improved by making it more user-friendly for
scenario analysis, identifying key assumptions and parameters, establishing a reference scenario as a
baseline, and setting out in a user guide the process for running scenarios and recording their outputs.
CAP Finance models - Pillar 1
102. For the pillar 1 model, the modelling team identified the main assumptions underpinning the
model as: the accuracy of the baseline data; the accuracy of the method of calculating convergence
and phasing payment s over seven years; and assumptions about the way that cuts to Pillar 1 in later
versions of the budget are split between direct payments and market measures
103. Given that the model sets out to replicate an (unknown) Commission calculation, it is not
surprising that the main uncertainties about the model relate to the assumptions made about how the
convergence mechanism will work. The model is calibrated to reproduce the Commission numbers as
nearly as possible. The team have been unable to do this precisely but the errors are tiny in relation to
the total budget. There is always a risk in this type of approach of a form of ‘selection bias’ whereby
amendments to the model are only accepted if they bring the results closer to the reference model.
The team acknowledge that it is possible that they may have made incorrect assumptions which by
coincidence balance one another out and generate similar figures to the Commission’s. But they do
not consider this to be a high risk, which seems a reasonable assessment.
104. The capability for undertaking sensitivity analysis is built into the Pillar 1 model and a
worksheet in the model makes direct comparison to the Commission’s results. The model has been
regularly used to simulate the effects of different convergence mechanisms and specifically to allow
the exploration of the effect of alternative assumptions about the size of the overall budget and the
parameters of the convergence mechanism (specifically the intervention point, reduction point,
premium (or uplift), minimum payment and reduction cap). Simulations using these parameters have
been run during the course of this review and found to produce intuitive results.
105. The model benefits from a user-friendly front end for simulations and a clear lay-out of results
with comparisons against the baseline scenario. There is scope to make the text box explanations
clearer, to simplify some of the formulae (see Annex 3) and to set out in a user guide the process for
running scenarios and recording their outputs. The team have also suggested that it may be useful to
undertake some sensitivity analysis of the main parameters in order to document formally how they
respond and in the process confirm the robustness of the model. This seems a good idea.
27
CAP Finance models - Pillar 2
106. For the Pillar 2 model, the modelling team identified the main assumptions underpinning the
model as: the accuracy of the baseline data and the accuracy of the method for applying the
Commission’s funding formulae for ‘objective’ allocation. Assumptions also had to be made about the
year-by-year profile of cuts to rural development programmes. In extending the model to the
devolved administration level, it was further assumed that the likely allocation method had been
correctly assessed and that the data used was on the same basis as the data used by the Commission
at member state level.
107. As with the Pillar 1 model, the main uncertainties relate to whether the Commission’s
calculations have been replicated exactly. A further complication with the Pillar 2 model is that the
Commission did not make a firm proposal on Pillar 2, so it is harder to establish a reference scenario.
However, in similar fashion to the Pillar 1 model, allocation scenarios produced under the Pillar 2
model have been benchmarked against the Commission’s own calculations as a test of accuracy and
robustness.
108. The capability for undertaking sensitivity analysis is built into the Pillar 2 model and a
worksheet in the model makes direct comparison to the Commission’s results. The model has been
regularly used to simulate the effects of different allocation keys and specifically to allow the
exploration of the effect of alternative assumptions about the size of the overall budget, the size of
the Pillar 1 transfer, the parameters of the allocation mechanism (historic shares on different bases
versus ‘objective’ shares based on different criteria) and co-financing rates. Simulations using these
variables and parameters have been run during the course of this review and found to produce
intuitive results.
109. The model benefits from a user-friendly front end for simulations and a clear lay-out of results
with comparisons against the baseline scenario. There is scope to make the text box explanations
clearer, to simplify some of the formulae (see Annex 3) and to set out in a user guide the process for
running scenarios and recording their outputs. As with the Pillar 1 model, the team have also
suggested that it may be useful to undertake further sensitivity analysis in order to document more
formally how the model responds. While some of this has been documented in the reports already
produced for Pillar 2, a formal test of the sensitivity of the models may also be useful to check there
are no unexpected results.
28
Linkages to other models
110. Interaction between the four models was considered in the review as an area of potential
concern. Problems can arise if errors or uncertainties in one model interact with those in another
model to which the results are linked. Compounding of errors across models could in principle lead to
some very large inaccuracies in model predictions or forecasts. Conversely, if models are not linked
when they should be then independent modelling could lead to inconsistent results.
111. The CAP Finance Pillar 1 and Pillar 2 models are for the moment distinct from the two RDPE
finance models and run by separate teams. There is a link between the two CAP Finance models in
that they both should run off the same overall EU budget outcome and the Pillar 2 model allows for
the modelling of a transfer from Pillar 1 to Pillar 2 funding in any member state. These linkages are
not formally modelled and it is left to the modelling team to ensure that the two models are run off
consistent assumptions about the overall EU budget and the budget for direct payments.
112. Outputs from the Pillar 2 model are used alongside outputs generated by RDPE EU budget
model in scoping the possible implications of future budgetary outcomes. Moving into the next
programme period, the outcomes from the Pillar 2 model should in principle impact both on the EU
funding available both for Natural England’s budget and the overall shape of the total RDPE Budget.
Again, these linkages are not formally modelled.
113. Finally, the FFM model outputs for Natural England’s budget feeds into the RPDE EU Budget
model and, in combination with budgets for DEFRA and the other delivery bodies, helps determine
unallocated EU funding and contributes to decisions about the profile and composition of future
spend. Again the RPDE model is not formally linked to the FFM outcomes and it is up to the user to
ensure that they are kept aligned
114. The potential for problems to arise as a result of these interdependencies was discussed
during the course of the review. However, the linkages are not strong and the risks of significant
adverse interaction effects are not judged to be large. Maintaining alignment between the Pillar 1 and
Pillar 2 models as assumptions change about the overall EU budget has not proved to be a problem,
and the variants of the Pillar 2 allocation which allowed transfers from Pillar 1 have not been central.
The outcome of the EU budget negotiations on Pillar 2, and the remaining fine detail to be settled for
Pillar 1, should be known with certainty before decisions need to be made on the future funding of
RDPE for the next programme period, thereby reducing the risks attaching to projections of available
financing necessary for management of the RDPE budget. Maintaining alignment between the FFM
model and the RDPE EU Budget Model has also not proved to be a problem.
115. It seems reasonable to conclude that the additional complexity that would be introduced by
any attempt to make a formal link between any of the four models would not be justified.
Nevertheless, it will be important to ensure that procedures are in place, and recorded in the
documentation and governance framework, to make sure that the models are kept aligned. This is
particularly important between the FFM and RDPE models, but there is a similar need for consistency
between the two CAP Finance models. Any interaction between the CAP Finance models and the
RDPE models is minor and not judged to be a source for concern.
29
116. One potential linkage that might need to be established is to link the Pillar 1 model up with a
similar model that looks at mechanisms for allocating Pillar 1 within the UK. There could also be scope
for linking the total payments under Pillar 1 and Pillar 2 models more formally to the future
programmes under which payments will be made to farmers. This could be helpful in understanding
the distributional implications within the UK of future changes to CAP Budgets
30
Key risks
117. As part of the review, model owners were asked about the key risks to the models. In no case
did the discussions identify any serious concern about the risk of errors in the coding of the model.
Modellers, users, reviewers and policy customers all expressed a high degree of confidence in the
soundness of the models, based on their experience of using them to date, scrutiny through developer
testing and internal review, and a track record of success in managing DEFRA budgets or mimicking
Commission calculations.
118. In view of recent problems discovered in DfT modelling, the modelling teams were also asked
explicitly about any risk of confusion between real and nominal variables in the modelling work. In the
case of the RDPE budget models, everything is done in nominal prices and the modellers have a high
degree of confidence that inflation is not an issue for the modelling.
119. For the CAP Finance models, the initial development work was undertaken mainly in current
price terms, but as the EU budget negotiations approached the end-game, the debate tended to focus
on headline EU budget figures denominated in 2011 constant prices. The CAP Finance models each
contain a routine (in the enigmatically named Croatia_calc tab) for deriving a series of current price
figures from the headline EU budget numbers in real terms, thus allowing all subsequent calculations
to be carried out in nominal prices. A ‘switch’ in the model allows final results to be displayed in real
or nominal terms as preferred. This feature of the models appears to work correctly, but would
benefit from clearer documentation.
120. In presenting results, the CAP Finance modelling team generally present figures in real terms
and make this clear to policy customers. Occasionally figures have to be presented in both nominal
and real terms at the same time, and in these circumstances, the team is careful to spell out clearly
the difference between the two sets of numbers and how they should be used.
121. Asked about other key risks or potential weaknesses in the models, model owners responded:
FFM – Manual inputs by NE; insufficient policy insight into assumptions; lack of understanding of
assumptions underpinning forecasts; training and skills of use; user manual.
RDPE EU Budget Model – Lack of understanding of the purpose and application of the tool; user
manual; training and skills to operate/implement; single operator; little understanding of importance
of the tool outside immediate team.
CAP Finance models – Pillar 1. The main weakness is the uncertainty over assumptions made by the
Commission in its convergence calculations. A consequence of this is some imprecision in the
calculation of ceilings for some Member States. Care is however taken to caveat the results
accordingly.
CAP Finance models – Pillar 2. Again, the main weakness is the uncertainty over assumptions made by
the Commission in calculation of allocation keys and the lack of a reference scenario. Model results
have been caveated accordingly.
Future development work should address these risks where they affect future use of the models.
31
ANNEX 1: TERMS OF REFERENCE
Review of DEFRA’s RDPE Natural England Future Funding Model, RDPE EU budget model and CAP
finance models
Commissioned by: Ulrike Hotopp, Director Analysis and Chief Economist, DEFRA
Reviewer: Warwick Economics
Terms of Reference
Objectives
- Review the following group of DEFRA spreadsheet models:
o RDPE Natural England Future Funding Model, RDPE EU budget model, CAP finance
models
Review the data used including that sourced from Natural England
Review the formulae of the models (verification)
Review the representativeness of the models (validation)
Review the version control and governance of the models
Review the associated documentation and guidance
Review the communication of assumptions and uncertainties of the models’ outputs
Review the use of the model outputs for informing policy thinking and development Review how the models interlink
Methodology
- Investigate each of the models and determine the approach used including their structure and
the appropriateness of the use of spreadsheets.
- Assess the governance and procedures for updating the models and also the data used.
- Determine the level and appropriateness of any training and guidance provided for users of
the models along with the suitability of available documentation.
- Investigate the rigour of the sensitivity analysis and if none has taken place, undertake such
analysis to determine the robustness of the results and any implications this may have on any
of the recommendations.
- Investigate the impact of the models’ results upon policy development and any final
recommendations, including dependencies between models and their combined output.
- DEFRA model owners will support the reviewer.
Meetings
32
- The reviewer will hold meetings with DEFRA model owners and customers and, for the RDPE
funding model, relevant Natural England staff. The number of meetings will depend on the
progress of the review but the sequence is expected to include:
o Commence meeting with DEFRA analysts and model owners
o Initial review and discussion with DEFRA analysts and model owners
o Discussion with Natural England staff where appropriate
o Discussion of preliminary findings with DEFRA analysts
o Discussion with wider group of DEFRA staff prior to final report
o Presentation and discussion of final report
Output
A short factual report stating:
Steps used in the assessment
Findings
recommendations
Timescale2
Week beginning 4 February: inception meeting
1st March: interim report
15th March: draft final report
29th March: final report
Above dates regarding meetings and the delivery of reports can be discussed but the draft final report
will need to be delivered no later than the middle of March.
30 January 2013
2 Original timetable was revised, with DEFRA’s agreement, in email exchanges on 15 February, 25-26 February
and 19 March 2013.
33
ANNEX 2: LIST OF MEETINGS HELD
Date Meeting Attendees 4-Feb-13 Inception meeting Alastair Johnson
12-Feb-13 RDPE models introduction Simon Raper, Simon West, Alastair Johnson
13-Feb-13 CAP models introduction Matt Molloy, Alastair Johnson
21-Feb-13 CAP Pillar 1 model Matt Molloy, Neville de Souza
27-Feb-13 CAP Pillar 2 model Matt Molloy, Neville de Souza
5-Mar-13 RDPE models review Simon Raper, Simon West, Nike Kojakovic
5-Mar-13 CAP Pillar 2 model Matt Molloy, Neville de Souza
5-Mar-13 Overview of models Ulrike Hotopp
8-Mar-13 CAP models review Matt Adey, Neville de Souza, Matt Molloy, Alastair Johnson
12-Mar-13 Phone meeting with Natural England Alan Drewer
13-Mar-13 Phone meeting with DEFRA reviewer Grant Davies
14-Mar-13 Phone meeting with DEFRA reviewer Mark Morley-Fletcher
14-Mar-13 Discussion of interim findings Alastair Johnston, Melissa Boulter, Matt Adey, Neville de Souza, Simon Raper, Nike Kojakovic
26-Mar-13 Phone meeting with HMT Brendan Bayley
30-May-13 Phone meeting with DEFRA Alastair Johnson
21-Jun-13 Presentation to modelling workshop DEFRA invitees
34
ANNEX 3: DETAILED COMMENTS ON THE FOUR MODELS
Future Funding Model
General
A useful guide to the methodology of the FFM (2013/14) has been prepared. The inclusion of a more
strategic overview of the model and a flow chart or logic map would help new users understand the
model better.
The methodology note relates however to a later version of the model than that provided to the
reviewer.
In-spreadsheet documentation is partial and incomplete in some tabs, and in need of update or
missing altogether in others.
It is good to see that the model includes an Updates log, but there are some remaining issues on
version control, for example notes that relate to earlier versions, and some coding changes that relate
to use of the model for simulation.
Making clear which cells use entered data and which are derived by formula would facilitate
understanding of the model and its structure
The detailed and helpful internal review by Mark Morley Fletcher highlights the main issues with the
model. His findings should be fully considered and implemented as appropriate in the next update of
the model.
Tab 1: Suggested allocations 12_13
The ownership of this tab needs to be clarified.
There is an odd use of SUM functions in this tab
The unexplained introduction of a 0.98 factor in the formulae in Row 94 is apparently to take account
of unspent commitment, but this should be treated in the same was as other tabs using the “Factoring
to Convert to Spend” tab.
The difference between the relatively simple calculation of New Business in this tab and the much
more complicated calculations in Tabs 12-16 (below) is not clear. Apparently it is being modelled
more fully in the update of the model.
Tab 3: Introduction
Notes are still in draft and incomplete and need to be rewritten and updated. Reference to “uptake
stats” does not relate to any tab header.
35
Tab 4: Spend to date
Data source is not given – reference is made to off-spreadsheet calculations.
Tab 5: New hls historic
No textbox explanation/descriptor in this worksheet.
How is this table updated and audited? Long list of data to be entered, with a risk of input error.
SUMIF (or similar) function could replace the formulae in cells calculating annual totals(B3:B7) to
reduce the risk of calculation error.
Tab 6: Existing classic commitments
Source “AAG002” should be explained.
Numbers for Habitat scheme are stylised assumptions – this should be explained.
Understand this table has been superseded in later versions.
Tab 7: Existing ES Commitments
No textbox explanation/descriptor in this worksheet. Should explain the basis – deriving figures on a
Resource basis from reported figures on a cash basis and then allocating across years.
In its current format, the worksheet will require a lot of manual manipulation to produce the required
totals, with partial entries over multiple columns. Coding these using more powerful Excel functions
would reduce the risk of error.
Apparently, this table has been superseded in later versions with data sourced on a Resource basis –
this will simplify the spreadsheet and address some of the issues.
Tab 8: Existing ES Commitments GO259
Columns are differently ordered here than in previous tabs – potentially confusing and/or risk of error.
Adoption of a common structure would help.
Summing across schemes is hard-coded using simple sums rather than Lookup or conditional sum
functions – current method runs risk of error (eg in distinguishing Revenue from Capital).
Tab 9: Existing UELS
Tab would benefit from inclusion of a textbox to explain its purpose.
As with previous tabs, spreading agreements over years is done manually, with attendant risks.
Again, adoption of a common structure would simplify and facilitate understanding of the model.
Source of entered data not documented.
Tab 10: Existing Comms Axis 1 and 3
36
Heading in Cell A1 refers to “Axes 1,3 and 5” – this is a hangover from an earlier version of the
spreadsheet and the heading should be corrected.
Tab 12: New Business Assumptions
In rows 7-31, some rows have entered data, some are blank, some are calculated and some rows seem
to be entered data. This should be put on a consistent basis, or the different approaches distinguished
and explained more clearly.
The scaling factor was introduced in an earlier version of the model to allow ad hoc simulations but
this is no longer used and has been removed from the later version. Amendments to the model for
simulation purposes should be clearly documented and it is good practice to gather variables and
parameters used in scenario analysis at a single point in the workbook.
Tab 13: Already live at 1 April
The source of data and purpose of this tab should be clearly explained.
It would be helpful if row headings and numbering exactly matched the previous tab.
Tab 14: Variables to convert uptake to £
The textbox refers to an “in-year profiling section” that no longer appears in the spreadsheet – it
relates to an earlier version.
Tab 15: New converted to value by FY
Again, manually spreading across Financial Years introduces complexity, risk of error. Re-code using
more powerful Excel functions.
The error in Row 85, identified in Mark Morley-Fletcher’s review needs to be corrected.
Given that this is a complicated calculation, it may be better to split this worksheet into two tabs, one
to convert to value, the other to convert to Financial Year.
Tab 16: Total value by UK Year
The heading at Row 22 should make clear that this applies to new ELS etc commitments – the same
applies to Tab 18.
This contains a number of redundant columns.
Some values entered as zero, when they could/should be coded.
Tab 18: Total RES by UK Year
The heading at Row 22 should make clear that this applies to new ELS etc commitments – the same
applies to Tab 16.
Tab 19: Headlines
37
The text box should make clear what use is made of this tab, understood to be primarily for Natural
England’s benefit.
There may be some concern about cumulating, as this tab does, on past “Unallocated” – what checks
are made to make sure that the underlying figures are sound?
Tab 23: %age to State Aid
Source of data – not clear why not all entered (some is copied from other cells)
38
RDPE EU Budget Model
General
Being newer, this model is less well documented and less well established in terms of governance and
wider understanding in the department
It is an example of a model developed for one purpose and extended to do something different, with
some attached. However, it is recognised that the extension improved the existing model and offered
a better way to manage the RDPE EU Budget.
The model is used as the basis for discretionary adjustments on funding decisions designed to make
best use of available EU budget funds given the forecast path of exchange rates. Attempts to model
the discretionary adjustments more systematically have been abandoned. This is probably sensible,
given the nature of the spending decision, but it makes it harder to judge precisely the impact of the
model in financial decision making.
There is some confusion over terminology for this model – variously referred to as the RDPE EU
Budget Model or the Exchange Rate Tool. The latter relates to its original purpose, but the title “RDPE
EU Budget Model” better captures its role in the management of EU funding of RDPE payments.
Tab 3: RDPE EU Budget Model
Column N (adjustments for accruals/prepayments) mixes formulae and entered data
It is unclear why Columns T-W are needed to convert data back from euros to sterling when original
payments are made in sterling –the raw data in sterling would be available from RPA and remove a
risk of error/approximation. However, RPA have indicated that additional effort would be required to
produce sterling data on this basis and it is not believed that any approximation introduced would be
significant.
Forecasts for 2013/14 and 2014/15 could be done separately using exchange rate forecasts for each
year (understood to be planned for next version of the model).
An expected 2015/16 prepayment of £60m is built into the model in Column P, but this assumption is
deeply embedded in the model – in a formula at cell O60, generating the result at cell R65. This would
be difficult for a new user to pick up and should be coded more clearly, and documented.
Comment in Cell Q17 (“Opposite entry made to reflect £49.893m adjustment”) is hard to understand
and does not seem to be reflected in the spreadsheet.
39
CAP Finance Models – Pillar 1
General
Generally, the CAP Finance models are more elegantly programmed using more powerful Excel
functions than the RDPE models, thus avoiding the need for a lot of manual re-coding, with the
attendant risks, in updating the model. On the other hand, the use of nested IF statements and other
conditional logic introduces more complexity and makes cell formulae harder to unpack. There is
potential to simplify in places.
In part, the use of conditional logic is because the model was built as a simulation tool, and designed
accordingly.
There is an issue about lack of documentation and the absence of an updates log for the model.
There are examples of multiple entry of the same data, leading to a risk of input error or failure to
update consistently
Tab 1: Read Me
The introductory text should make clear up-front that model is in euros in current prices, with data
translated to and from real (2011) prices according to a stylised assumption of 2 percent inflation at
the EU level in euro terms. Note that this assumption, while standard in EU budget discussion, may
not be appropriate for 2011-2013.
Tab indicates latest update but does not log sequential changes to the model
Tab should indicate if any off-model data sources are used.
Tab 3: Data
Not clear why Column D entered as data in this tab when most other data is drawn from other tabs in
the spreadsheet
Similarly, why is Column G data entered independently when this data is used elsewhere in this tab (eg
D91:D119?)
Tab 4: Croatia_Calc
This is a key table in moving from the headline numbers for the EU budget in real terms to the year by
year figures in current prices relevant for the convergence calculation. However , the title is
misleading and the logic of the calculations is hard to follow without explanation – better
documentation of this tab is needed.
Some of the formulae could be simplified, for example cell K5 could be expressed much more simply
as K3 + (K3*QuickLookUp!$B$3) – (K4-K2) rather than the complex nested IF statement used.
(This simplification stems from recognising that the general formula still applies when some of the
parameters are set to zero rather than requiring a switch and associated IF statement).
40
Tab 5: MFF Ceilings – Current
More notation on the purpose of this tab and reasons for difference to other measures would be
useful.
In particular, more explanation for advancing the figures from Annex II by one year in this Tab should
be provided.
Tab 9: Convergence Calculation – Linear
No background is given on the purpose of this tab or the logic behind the calculations. “Levels” would
be a better descriptor than “linear”.
Tab 10: DP Phasing
The formula given in the textbox for calculating annual growth rate is incorrect, though it is coded
correctly.
Tab should explain more clearly how these numbers differ from Annex VIII data (ie make the comment
on cell B2 more prominent).
Tab 13: DEFRA vs Cion
Version given to reviewer shows significant differences – needs to be explained that the comparison is
with the original Commission figures and parameter changes since that time will lead to discrepancies
vis a vis the Commission baseline, not necessarily due to differences in the underlying model.
Tab 14: DEFRA’s baseline
Not clear why data re-entered here for Commission’s baseline
This is not the data used to underpin the charts – would have been better to link the charts to data on
this page rather than embed the data in the charts tab.
41
CAP Finance Models – Pillar 2
General
Generally, the CAP Finance models are more elegantly programmed using more powerful Excel
functions than the RDPE models, thus avoiding the need for a lot of manual re-coding, with the
attendant risks, in updating the model. On the other hand, the use of nested IF statements and other
conditional logic introduces more complexity and makes cell formulae harder to unpack. There is
potential to simplify in places.
In part, the use of conditional logic is because the model was built as a simulation tool, and designed
accordingly.
There is an issue about lack of documentation and the absence of an updates log for the model.
There are examples of multiple entry of the same data, leading to a risk of input error or failure to
update consistently
Data links helpful but some lead to an intermediate page rather than the data itself.
Tab 1: Read Me
The introductory text should make clear up-front that model is in euros in current prices, with data
translated to and from real (2011) prices according to a stylised assumption of 2 percent inflation at
the EU level in euro terms. Note that this assumption, while standard in EU budget discussions, may
not be appropriate for 2011-2013.
The tab indicates the latest update (though mistakenly indicates as Feb 2012 not Feb 2013) but does
not log sequential changes to the model.
Links are included to off-model data sources, but they do not all link through to the precise data page.
Tab 2: FoPOct11_Comp
No source given for this enigmatically titled tab.
Tab 3: Croatia_calc
This is a key table in moving from the headline numbers for the EU budget in real terms to the year by
year figures in current prices relevant for the convergence calculation. However, the title is misleading
and the logic of the calculations is hard to follow without explanation – better documentation of this
tab is needed.
Tab 4: Quick Look Up
This is a useful and well-presented ‘front end’ to the model enabling a number of shocks to be
simulated, with switches pointing to different variants of the allocation mechanism and displaying
results for different countries.
42
There is some overlap between the box representing possible ‘shocks’ and the box representing
different ‘switches’. The operation of the ‘objective’ allocation mechanism is divided between the two
sources, creating a possible source of error for an inexperienced user. It should be made clear that the
choices in cells B36-B38 only apply when cell B5 is set to ‘Objective’. To an inexperienced user it is
unclear if and when the choices in the “Switch” box over-ride those in the “Shocks” box or vice versa.
The operation of the ‘tunnel’ could also be more clearly explained.
Tab 5: Funding
This is a powerful table but it is difficult to keep track of the thirteen blocks that make up the table,
and there may be a case for using more columns or dividing the table into separate tabs.
The textbox comment on the first block (“2014-2020 rolls the same shares forward “) is misleading in
that it implies that 2007-13 average shares have been rolled forward, whereas it is 2013 shares.
In the second block of data, it is not clear why data pre- and post-VM are entered several times (and
again in Tab 6: Allocation Keys).
It is not clear why an IF statement is needed for Budget changes equal to zero or not equal to zero.
The formula would work for both zero and non-zero budget changes. This would allow the removal of
an unnecessary switch in the Quick Lookup tab.
Tab 7: Obj_all_calc
The lack of documentation makes this tab difficult to follow, particularly the tunnel calculation. The
formulae in this tab have not been validated/verified in this review. However, simulations undertaken
using the switches and shocks on the Quick Look up page appear to produce results in line with
expectations.
Tab 8: Data
Data sources need to be documented here.
GDP and Labour Productivity transformations need to be explained – it is not clear why the inverse
should be divided by three. It is understood that this adjustment was found to bring the results closer
to the Commission’s calculation but this assumption should be checked and explained.
Data for pre and post-VM are again entered here. This should be the basic source, from which other
tabs draw.
Tab 13: DP_Ceilings_Select
This tab does not offer an option to link to European Council budget decision in Feb 2013. There could
be a link here to the Pillar 1 model, and/or the need to keep the Pillar 1 and Pillar 2 models fully
aligned .
Tab 16: 2014 DP Transfer
The formula for deflating 2014 to 2011 is incorrect – the factor applied should be (1/1.02)3 not (0.98)3.