solvency ii model validation - lloyd's of london/media/files/the-market/operating-at...model...
TRANSCRIPT
Solvency II model validation workshop 5
12 & 13 June 2012
© Lloyd’s
Agenda
Introduction
Five ways to improve your validation
Dry Run findings for major risk types
Lloyd’s review under BAU
Break-out – table discussions
Next steps
© Lloyd’s
Introduction
© Lloyd’s
Model validation workstream
solvency II syndicate timeline - key dates for 2012
January february march april may june july august september october November
IMSCRInternal Model
SCR
MVModel
Validation
31 May
Model Validation
Guidance
Data Audit Reviews
5 October
Validation Report
30 April
2013 SCR
Guidance
19 July
Draft SCR via LCR
(incl projected TPs)
20 September
Final SCR via LCR
(incl projected TPs)
8 & 11 May
Workshop
15 June
Data
Audit
Report13 & 18 April
Workshop
12 & 13
JuneWorkshop
Validation walkthrough follow up
& CAT MODEL REVIEW WORK
Workshop/Briefing Agent Deliverable - Dry Run / ProjectLloyd's PublicationKey shaded boxes Thematic ReviewAgent Deliverable - BAU
18 January
Cat/External models
Briefing
10 February
Draft Data Audit
Report Guidance
30 March
Final Data Audit
Report Guidance
31 July (PROV'L)
Standard Formula
SCR Re-run @ 31/12/11
You are here.
Please read the Model
Validation Guidance.
© Lloyd’s
five ways to improve your validation
© Lloyd’s
Top 5 ways to improve your validation
1. Evaluate your risk coverage
2. Rank the risks
3. Link to the syndicate experience
4. Clarify the steps in the validation cycle
5. Structure the tests
© Lloyd’s
Risk coverage & non/partially modelled risks
Solvency II requires all material risks to be covered by the model.
Easier said than done.
Tsunamis, floods, latent claims, tail risk not captured in the history,
dependencies between prior years…how well are these captured by your
model?
Begin with a qualitative assessment of what’s in and what’s out.
Data may be lacking and methods may be approximate – but the risk still
needs to be captured.
© Lloyd’s
Rank the risks
What are the biggest threats to syndicate solvency?
The most common reasons for threats to the ICA or RBC since 2003:
Reserve deteriorations on multiple years of account in a casualty portfolio
Severe natural catastrophes with significant losses from non-modelled
perils
At a more granular level, risk ranking:
Demonstrates an understanding of the risks
Allocates validation resource more effectively
• Rank risks using business definitions rather than statistical/actuarial ones.
© Lloyd’s
Clarify the steps in the validation cycle
Recall the four steps to the validation cycle:
1. Apply the validation test
2. Analyse the results
3. Escalate test results (if required)
4. Implement changes to the internal model (if required)
Common findings from the Dry Run on steps 2 and 3:
No clear statement of the result
Lack of justification or assessment of limitations
Lack of clarity around escalation process
Test results will rarely be clear-cut – they need to be explained.
© Lloyd’s
Link to the syndicate experience
The claims history is never sufficient but always necessary.
The model is an abstract exercise if there is no link to the past.
Use expert judgement to explain the impact of data limitations.
GAAP vs. economic basis
Best estimate vs. conservative reserving
Business no longer written, etc.
Consider tail risk not reflected in the experience – don’t just exclude poor
results.
The history also serves as a reference point for management.
© Lloyd’s
Structure the tests
Many agents have relied primarily on Sensitivity testing – Type I (“ST1”).
Across the board up/down movement in parameter type (e.g. CoVs)
Outcome: “Pass” if change in SCR is within a pre-defined range
Limitations:
It’s deterministic – so there’s no sense of the uncertainty in the input
Sensitivity within a pre-defined range doesn’t mean it’s the right value
ST1 has its advantages but won’t be sufficient on its own.
Validation of more material risks will need to build on ST1 with other tests.
© Lloyd’s
Validation tests: suggested hierarchy In
cre
asin
g r
isk m
ate
rialit
y
Reverse stress testing
P&L attribution
Stress & scenario Testing
Sensitivity testing (Type 2)
Testing against experience
Sensitivity testing (Type I)
Qualitative validation
© Lloyd’s
Dry Run findings for major risk types
© Lloyd’s
Reserving risk
© Lloyd’s
The validation elephant
Multi-year reserve deteriorations have been one of the leading threats to
ICA/RBC at Lloyd’s.
Does your reserve risk methodology reflect the risk of multi-year
deteriorations?
First, a brief detour down bootstrap lane…
© Lloyd’s
The bootstrap: last step in the algorithm*
Simulation No: XXXX
Simulated
AY/DY 1 2 3 4 5 Reserves
2007
2008 85 85
2009 282 35 317
2010 301 487 70 858 Result:
2011 211 341 403 66 1021 thin tailed distributions
TOTAL 2281
...etc.
Bootstrap AY/DY distribution parameters
Distributions sampled from independently
* England, P.D. (2001). Addendum to 'Analytic and bootstrap estimates of prediction errors in claims reserving.'
© Lloyd’s
Are statistical methods enough?
Reasons why most stochastic reserving methods (not just the bootstrap)
assume independence between accident years:
Mathematical tractability
Realism
The few that do will still require you to take a view on the calendar year
“trend”.
Back to validation…How material is the assumption of independence?
Two validation tools for dependencies (CEIOPS DOC 48/09 (5.245)):
Expert judgement of causal relationships
Statistical analysis
© Lloyd’s
Elephant sightings
The support from expert judgement for dependencies between accident years
will be very strong for some CoB (mostly long-tailed casualty):
Similar or same risks
Underwriting cycle, inflation, legal environment, etc.
Quantitative evidence…discussed in previous workshops.
Calendar Residuals
-2.0
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
2.5
2002 2003 2004 2005 2006 2007 2008 2009
Calendar Year
© Lloyd’s
Avoid getting stomped on
Your validation process should include:
Criteria for the existence of accident year dependencies
An assessment of their materiality
An approach for incorporating them in the internal model (where required)
The above need not (should not ?) rely on complex statistical methods.
Lloyd’s encourages agents to build on work done in the best estimate
reserving process:
Actual vs. best estimate comparisons
Knowledge of large claim deteriorations
Legal and other risks…
© Lloyd’s
dependencies
© Lloyd’s
So many correlations, so few dependencies
Why are the following often seen in the same model?
Hundreds of correlations
Diversification benefits exceeding 40%
How would your validation process respond?
“Dependencies are very hard to estimate and validate.” CEIOPS DOC
48/09 (5.233)
“Wounds from correlation matrices are still sore.” Nassim Nicholas
Taleb
© Lloyd’s
Seeing the forest through the trees
Broadly speaking…two ways of validating dependencies:
Both types of validation have their uses and limitations – neither is a magic
bullet
Validation
component Bottom-up Top-down
Risk distributions Pairs of marginals Aggregate
Inputs/Outputs Inputs Outputs
Sample metrics Correlations Joint exceedance
probabilities
Key validation tool Sensitivity testing Scenarios
Premium risk example:
Marine Hull & Cargo
Sensitivity test
correlation between
Hull & Cargo; observe
impact on model
output
Scenario based on
high loss ratios for key
accounts; compare to
model output for Hull &
Cargo combined
© Lloyd’s
Distressed…or just sensitive?
Most agents have emphasised “bottom-up” validation based on sensitivity
testing of correlations. Limitations of this approach:
Tests the sensitivities – not the level of dependency itself
Alternative dependency structures not evaluated
Ignores other drivers – e.g. inflation
In top-down validation, management takes a view of the material
“dependencies” that could impair profits or bankrupt the syndicate, e.g.
Multi-year reserve deterioration on US casualty
Several severe cats in the current YoA
(Very) poor underwriting results on several classes
Combinations of the above, etc.
These are then compared to model outputs.
© Lloyd’s
How to diversify away tail risk
Many agents did not undertake this high level validation of outputs.
Granular risk distributions
+ non-tail dependent dependency structures
= diversification in the tail
This will be a key area of interest for Lloyd’s (and the FSA) in BAU.
This works both ways. If you have “dependencies” in your model that have
limited/no justification, take them out. You’re just slowing down your model.
© Lloyd’s
Other risk types
© Lloyd’s
Summary of issues on other risk types
See the draft Validation Guidance for more detailed discussion.
See Appendix I for sample validation tests.
Risk type / area Key issues / remarks
Premium ex. Cat
• Perils covered (e.g. non-modelled cat)
• Validation of data adjustments
• The underwriting cycle
Catastrophe • Draft LMA guidance – awaiting comment from FSA
• Non/partially modelled perils remains a key issue
Market • Obtaining comfort on ESG
Credit • Materiality of different drivers
Operational • Comprehensive and realistic scenarios
One year risk • Consistency with ultimate
© Lloyd’s
BAU 2012
© Lloyd’s
Lloyd’s “Lines of defence”
Recap from May IMSCR workshop:
Line Description Tests Outcome
1 LCR data validation • Defective values
• Unintended input errors
LCR that agent can “sign-
off” on
2 LCR benchmarking
• Risk vs. reward and
other comparisons
• By SCR/main risks
Identification of areas of
focus for Line 4
3 SCR document review
• Methodology &
assumption descriptions
• Review of required
quantitative information
Completed SCR document
review template
4 Model walkthrough
• Focus on issues
identified in Lines 1-3
• Similar format to Dry
Run walkthroughs
Revisions to model if
required
5 Validation report review • Review against VR
guidance requirements
Completed/updated VR
review template
© Lloyd’s
Line 2: LCR benchmarking
A tool for generating questions – not a pass/fail test or a Lloyd’s “Standard
Formula”.
DRAFT example: risk vs. reward. Consistent with S2 concept of market
valuation.
(*) Illustrative applications of risk-reward. Not for LCR review.
Plus other approaches: reserving risk vs. best estimate reserves, etc.
Risk type Risk metric Reward
Market (*) Standard deviation of
returns
Expected return above
risk-free rate
Reinsurance (*) Probability capital falls
below £XXm Net premium
Premium Ultimate 99.5th VaR Profit loading in premium
Reserving Ultimate 99.5th VaR Risk margin
Capital requirement One year 99.5th VaR Expected one year profit
© Lloyd’s
© Lloyd’s
© Lloyd’s
Line 4: Model walkthroughs
The feedback from the model walkthroughs identified “material” issues for
most agents.
Agents are not expected to re-submit before 30 June (unless specifically
requested to do so by Lloyd’s).
Lloyd’s does expect agents to make progress on the issues identified – and
will follow up in the next round of walkthroughs.
Also expect questions on issues identified in Lines 1-3.
© Lloyd’s
Line 5: Validation report review
Update for October 2012 submission: A statement confirming that the SCR is
“not materially mis-stated” is not – repeat, NOT – required.
A statement that the model is appropriate WILL be required
Lloyd’s will re-visit when Level 3 text finalised
This update does not alter the validation requirements.
© Lloyd’s
table discussions
© Lloyd’s
Suggested topics
The proposed Lloyd’s LCR review structure:
What weight should be given to the different “lines of defense”?
Is the burden on syndicates reasonable?
Benchmarking:
What are the appropriate metrics?
How should they be used?
The most challenging area of validation for your syndicate:
The underwriting cycle
Dependencies between prior years
Non-modelled perils
Other
The draft Model Validation guidance: comments, suggestions, questions.
© Lloyd’s
next steps
© Lloyd’s
What happens next?
Slides will be available on lloyds.com after the 13th.
Draft Model Validation guidance – if you didn’t read it over the long weekend,
please do so and provide feedback before 15 June.
LCR and SCR documentation is due19 July (on initial SBF).
And…a mix of Olympics and SCR reviews.