sally edwards and simon gault

5/19/2018 Sally Edwards and Simon Gault

1/32Model Development Unit

Strategy Directorate

1

Developing & MaintainingDynamic Micro-Simulation

Models at DWP

Sally Edwards

Simon Gault

ESRC 2ndApril 2009


2/322 Model Development UnitStrategy Directorate

Developing Models at DWP

1.Overview

2.Objectives

3.Development Standards4.Simplifying Complexity

5.Maintenance Protocol

6.What worked well and Lessons Learned



Introductions

Simon Gault

Statistician, with a strong background in Economics

Extensive experience in developing models

Manages the Forecasting Simulation models team

Sally Edwards

Systems / Business Analyst

Predominantly worked in the private sector on large scale IT systems

Manage the Genesis Simulation Model

Members of a the Model Development Unit(MDU) which hasresponsibility for Micro-Simulation Models at DWP

MDU consists of 20+ people



Micro-Simulation Models

Dynamic Micro-Simulation Models, based on our standard

architecture, Genesis:

Pensim2private & state pension income

InformIntegrated Forecasting model for working agebenefit claimants

Individual Benefit Forecastingfor RP, IB, DLA/AA

Employment Model, currently being validated

Static Micro-Simulation Models:

Policy Simulation Model(PSM)

Employment and Hours Model(EHM)



Key factors taken into Consideration during

Development

Key factors considered when we designed our first dynamic micro-simulationmodel, Pensim2 - also valid for our other models:

Simplify the complexity of the model as much as possible, using a modularapproach and allowing a simple prototype model to be developed quickly

Several developers work on each model at the same time

Large staff turnover

Models are intended for long term use (e.g. Pensim2 anticipated to have15-20 year life) so most of the people using/maintaining the models arenot involved in their development

Clear documentation, processes, training and user guides are essential

Excel and SAS are the preferred tools, as both are well understood andextensively used among the Analysts at DWP




1.Overview

2.Objectives for Pensim2 (first DMS model)

3.Development Standards4. Simplifying Complexity




7/327 Model Development Unit


Original Objectives for Pensim2

Pensim2 Feasibility Study document 2002) Objectives

1. Flexibility 8. Efficiency (in use of memory)

2. Robustness 9. Timeliness

3. Transparency 10. Done with own expertise4. User-friendliness 11. Done with own resources

5. Speed 12. Ease of handover

6. Reliable Output 13. Ease of maintenance

7. Availability 14. Independence of base data

(i.e. desktop model)




Decision to Split the Model

Impossible to satisfy all the objectives in a single model solution sodecision was made to split the model into two separate parts,satisfying most of the objectives:





Impossible to satisfy all the objectives in a single model solution sodecision was made to split the model into two separate parts,

satisfying most of the objectives:

Excel based front end Pensim2

User friendly, transparentcode in a standard format

Easy to understand and flexible, enabling analysts to changeparameters & structure of model without understanding theunderlying code.

Easy to maintain and handover

Allowed independence of base data





Impossible to satisfy all the objectives in a single model solution sodecision was made to split the model into two separate parts,

satisfying most of the objectives: :

Excel based front end Pensim2

User friendly, transparentcode in a standard format

Easy to understand and flexible, enabling analysts to changeparameters & structure of model without understanding theunderlying code.

Easy to maintain and handover

Allowed independence of base data

SAS code generator - Genesis

Developed with our own expertiseandown resources

Robustgenerator, producingreliable outputs




Basic structure of Genesis

Dynamic Micro-Simulation Models at DWP

GENESISModel Engine

generates SAS model

and runs the simulation

Simulation

Modelparameter

sheets (Excel)

SASBase

Data

SASpost-

simulation

Data

- Standard Excel sheetsused to define the model

- SAS chosen for the model engine and the output data,

as SAS is the standard data analysis tool used

SAS Static

code





1.Overview

2.Objectives





13/32

13 Model Development UnitStrategy Directorate

Development Standards

Two separate development teams, with different

development standards adopted

Genesis Model Engine

Developed by IT staff Used standard IT project development procedures

Pensim2 Analytical team

Economists and Statisticians

Less structured approach to development


14/32


Genesis Model Engine Development Standards

Highly structured approach to the development

Project Management Protocol based on Prince2 methodology, although notall aspects were needed

Documentation 2 types Development documentationproduced at each stage of project, that

was signed off and used as input to the next stage Key Projectdocumentation (*)- maintained and kept up-to-date with

each new release

Development Processessimple process diagrams to provide clearguidelines, particularly useful for new staff

Detailed Requirements explaining the full functionality of the modelengine. Document translates Economist language into IT language(*)


15/32


Genesis Model Engine Development Standards

Design Diagrams and documentation used in development stage only

Change Control processchanges estimated and impacted before beingaccepted/rejected/on hold (*)

Problem Log processproblems recorded, prioritised, fixed/removedfrom the scope/user error (*)

Validation Process - detailed Test Plan based on Requirements document,used to ensure that each requirement was fully satisfied

Test Packensured existing functionality continued to work when changeswere made to the model engine (*)

User Guidecontains details of functionality written in simple format (*)

Trainingmaterials, inc presentations and training text for self-study (*)


16/32


Pensim2 Model Development

Pensim2 Analytical development followed a less structured

approach

Generally this was appropriate, as detailed requirements were notpossible to establish up-front

Effort was predominantly spent on regression analysis

Analysis results reviewed to determine the key components toinclude and the most appropriate sources for the assumptions

Using the standard Genesis templates, the Pensim2 model wasmore easy to develop, rather than if a traditional DMS model hadbeen built


17/32



1.Overview

2.Objectives





18/32


Key Aim of Genesis Structure - Simplify Complexity

No Hard Coding Rule which allows complete flexibility for:

Structure of the simulation, i.e. order of events processed

Data structures, i.e. data tables, data relationships and variables

Parameters, e.g. indexes, rates, dates

Regression Equations

Selection filters

Static SAS code, if required (e.g. addition of new cases to a model)

Ability to add / remove modules easily

Each model is defined in simple Excel spreadsheets, using standard templates,that users can easily understand and modify


19/32


Key Aim of Genesis Structure - Simplify Complexity

Data Access routines

Enable easy access to data so user does not have to write complex code

Default data access will process the current object in the current simulationyear

Facility to process a variable belonging to a related object, e.g. partner PAR;Relationships are not hard-coded and new relationships can easily be

defined

Data manipulation routines provide commonly required data facilities,e.g. maximum value during past 10 years = MA10

E.g.MA10_PAR_pa_salaryreturns the maximum value in the past 10 yearsof

the salaryvariable for the individualspartner,taken from thepa table

Keep Equations simple

Equations, filters & process flows are defined in a simple, standard format,that is unambiguous. Hence the models are easy to maintain


20/32



1.Overview

2.Objectives

3.Development Standards4. Simplifying Complexity




21/32


Change Management

Change Management Process is followed for all models

Every change goes through the formal Change Control procedure, i.e.requirements documented, impact assessed, agreed by users & signed off

Change Registerlisting all changes, each with a separate reference number

For each change to the model, a separate Change Controlform is produced

The model code contains the Change ref number, with a comment

Some changes are repeated regularly, e.g. assumptions, so clear

documentation is helpful for the repeat change

Questions arise after a change has been implemented - these can be easilyanswered by referencing the Change Control documentation


22/32


Problem Management

Problem Management Process is followed for all models

Everyproblem/error discovered in a model goes through the formalProblem Log procedure, i.e. requirements documented, impact is assessed,agreed by users & signed off

Problem Log Register, listing all problems identified, including those that

turn out to be User Errors or misunderstandings; PL ref number assigned

For each problem recorded, a separate Problem Logform is produced

Users are notified of significant problems that affect the model results

Many of the problems are minor and are picked up by the developmentteam these are recorded formally, even if they do not impact the results


23/32


User Model

Our models fall into 2 categories: Maintained & owned by the Model Development Unit Developed by Model Development Unit; handed over to User

team, who then own and maintain the model

Models owned by the Model Development Unit have a User Groupand/or a Steering Group consisting of representatives from eachteam using the model

The purpose of these groups is to agree the content, priority and

timing of changes made to the model


24/32


User Group Purpose

Specifically for Pensim2, our largest model:

Change Requests and Problem Logs are reviewed by the UserGroup, where appropriate

A key user sponsor is assigned to each change. The sponsor

ensures that the change is accurately specified & will sign-offtesting

The User Group are responsible for signing off new versions ofthe model before they are released

The users also have a separate User Forum where they presentanalysis carried out on the model outputs and explain how theresults have been used


25/32


Release Management

Again, specifically for Pensim2:

New Releases are implemented approximately every 6 to 9 months

Each Release includes one or two key components, e.g. Private PensionReform, and a bundle of smaller changes and problem fixes

The timing and content of each Release are agreed with the User Group

When a new Release is built: Each change/fix is added to the model one at a time Hence each change is reviewed and signed off separately Enables us to fully understand and agree the effect of each change.

Time consuming but gives confidence in the results

Clear audit trail

All previous versions of the model remain available


26/32



1.Overview

2.Objectives

3.Development standards4.Simplifying Complexity

5.Maintenance protocol



27/32


Original Objectives Satisfied




3. Transparency 10.Done with own expertise4. User-friendliness 11. Done with own resources

5.Speed 12. Ease of handover


7. Availability 14.Independence of base datai.e. desktop model)


28/32


Objectives Partially Satisfied







7. Availability 14.Independence of base data

i.e. desktop model)


29/32


Objectives that proved more tricky


1. Flexibility 8. Efficiency in use of memory)





7. Availability 14.Independence of base datai.e. desktop model)


30/32


What went well

Generic framework has proved successful and easy to use

Genesis architecture has been remarkably robust

Modules have been reused/copied from one model to another

Quicker development of new dynamic micro-simulation models,compared with a traditional build

Automated documentation facility and error checking producedusing VBA, listing where variables are used and providing an

overview of each module, to ensure consistency

Lessons learnt from Pensim2 development were used whendeveloping subsequent models


31/32


Lessons Learned

Key lessons learnt from Pensim2 development:

Tried to make the model too complicated particularly the analytical side

We should have developed a set of simple modules for a Phase 1 delivery,then added complexity to the more important modules for Phase 2

The structure of some of our modules was more complicated than the datawould support, e.g. Labour Market process

Insufficient documentation produced by the Analytical team during thedevelopment (e.g. Base Data Creation)

Genesis Engine code is difficult to understand and hence modify, althoughit is robust and does not require much modification

Some Genesis Engine functionality was over-ambitious and subsequentlydropped (e.g. Alignment of polychotomous variables)


32/32

32

Any questions?

sally edwards and simon gault

Documents

model development unitmdu

dlaaaemployment model

development standards4

people5192018 sally

dwp5192018 sally edwards

simple prototype model

models arenot

maintenance protocol6