delivering large post-trade initiatives: quality assurance and key challenges
DESCRIPTION
Difficulties in coping with functional complexity experienced by the project team: • Aligning tests correctly in batches that correspond to events in the daily life cycle, including multiple days • Accelerating the daily life cycle to speed up test execution and to cover Settlement Failure processes in testing within a reasonable timeframe • Accurately simulating and controlling test inputs from upstream systems and outputs to downstream systems • Management of complex Reference Data setup • Involvement of complex components such as Risk Management and Collateral Management systems • Test cycle turnaround time, client certification and test automation Keynote Speakers: Alyona Lamash FRM Head of Risk Management Practice, Exactpro Systems Mark Ryland Financial Markets Technology Consultant, Independent Consulting, United KingdomTRANSCRIPT
How Highly Effective Quality Assurance Techniques Can Make A Positive Difference In Delivering Large Post Trade Systems Initiatives
Post Trade Forum, Vienna 11 September 2014
Exactpro Company & Experience
Exactpro is:
2
• A specialist firm focused on functional and non functional testing of wholesale financial products systems
• An independent company incorporated in
2009 now employing over 240 specialists
Our locations in Russia:
• A US company with four QA & development centres in Russia and sales support in the UK
• Mostly engaged in trading platforms for exchanges and brokers, but also significant post trade experience
Introducing Your Speakers
3
Mark Ryland Strategic Adviser to Exactpro
• Financial Consultancy and QA of complex Risk Management solutions, including calculation algorithms
• BA for Collateral & Risk Management for investment banking;
• QA Delivery Manager for large Post Trade projects
Alyona Lamash Head of Exactpro Risk Management Practice
• IT Manager/CTO • Banking & Market Infrastructure• Trading and Post Trade• Insight and experience of QA
Talk Content
PART A
Challenges that Affect Large Projects - in General
How Great QA Techniques Can Help – Generically
PART B
Challenges that Affect Post Trade Initiatives – Specifically
QA Techniques that Help Post Trade Initiatives - Specifically
4
Testing Making a Positive Contribution
to the Whole
Responding to Issues/ Unknowns
Detailed Functional Understanding
Achieving & then Maintaining Test Automation
Speed of Test Execution
Clarity & Accuracy of Test Results and Defects Reporting
5
Testing a Large System Poses Challenges PART A:
Full Test Coverage
How Can Effective QA Be Part of The Solution, Not Part of The Problem
• GENERIC– Highly capable people with the right behaviours – Efficient and effective test process– Right use of technology – Independent & objective measurement of progress – RESULTS IN GREAT TEAM DYNAMICS ACROSS THE WHOLE PROJECT TEAM
6
Make a Positive Contribution to the Overall Team
7
Conception Initiation Development Test Implement
Project Sponsor
Project Manager
Business Analyst
Architect/Designer
Development Team
QA Team
Production Team
Make a Positive Contribution to the Overall Team
8
Make a Positive Contribution to the Overall Team
9
Pace
mak
er(s
)
1. High Quality Staff
The Elements of Highly Effective QA - 1
10
3. Energy, Enthusiasm &
Proactivity
2. Deep Domain
Knowledge
4. Creativity & Flexibility in Problem
Solving
Highly capable people with the right behaviours
8. Test Execution & Reporting
1. High Quality Staff
3. Energy, Enthusiasm &
Proactivity
5. Planning & Delivering Ability
2. Deep Domain
Knowledge
4. Creativity & Flexibility in Problem
Solving
The Elements of Highly Effective QA - 2
11
7. Defect Reporting & Evidence
Highly capable people with the
right behaviours
Efficient and effective test process
6. Test Case Quality
6. Test Case Quality
8. Test Execution & Reporting
10. Range of Excellent Automation Tools
9. Detailed Management of Test Environment
The Elements of Highly Effective QA - 3
12
7. Defect Reporting & Evidence
Right use of technology 1. High
Quality Staff
3. Energy, Enthusiasm &
Proactivity
2. Deep Domain
Knowledge
4. Creativity & Flexibility in Problem
Solving
Highly capable people with the right behaviours
5. Planning & Delivering Ability
Efficient and effective test process
Some of our Dynamic & Talented Team
13
Independence & Objectivity
14
1. Understanding of Requirements1. Understanding of Requirements
2. Delivery of Solution2. Delivery of Solution
3A. Independent Measurement by
QA Team
3A. Independent Measurement by
QA Team
3B. Independent Measurement by High Quality QA Team3B. Independent Measurement by High Quality QA Team
Developer Designer Business
Analyst Business
Manager End
Users
Team Dynamics When the QA Team is Excellent
15
DEVELOPERSDEVELOPERS
Try harder to get it right in the first placeGet rapid/timely feedback on what’s wrongWork harder to solve defects to keep the project on track
Try harder to get it right in the first placeGet rapid/timely feedback on what’s wrongWork harder to solve defects to keep the project on track
ANALYSTS AND DESIGNERS
ANALYSTS AND DESIGNERS
Have clear problem statements to work from –
with detailed supporting evidence
Have clear problem statements to work from –
with detailed supporting evidence
SPONSORS & PROJECT
MANAGEMENT
SPONSORS & PROJECT
MANAGEMENT
Have (justified) confidence in the system
Fewer problems are going to be found when Live
Project will cost less
Have (justified) confidence in the system
Fewer problems are going to be found when Live
Project will cost less
16
Features of Complex Post Trade Infrastructures PART B:
17
Features of Complex Post Trade Infrastructures
18
Collateral Upload
Risk Run
Margin Call
Risk Run
Trade InputTrade Input
Settlement Run
Trade Date
Settlement Date
Simulate DeliveryFailure
Trade InputTrade Input
Day 1
Examples of Test Scenarios (Before We Get to Specific Problems)
Penalties BUY INCash
Settlement
Risk Run
Trade Input
Day 2
Risk Run
Corporate Action
Trade Input
Day 3
Risk Run
Settlement Run
Trade InputTrade Input
Collateral Deficit
Delivery Default
Corporate Action
Delivery DefaultDelivery Default
6 Resulting Challenges From a QA Perspective
1) Testing of complex multistep scenarios2) Limited test availability of downstream and upstream
systems3) Most of endpoints are accessible via API or File exchange
(no GUI)4) Reference Data setup or migration5) Complexity of Risk Calculation Algorithms6) Regression Cycle for a substantial number of test
scenarios
19
Solving: 1) Testing of complex multistep scenarios
20
Solving: 1) Testing of complex multistep scenarios
21
Solving: 1) Testing of complex multistep scenarios
22
Solving: 2) Limited test availability of downstream and upstream systems
23
Solving: 2) Limited test availability of downstream and upstream systems
24
Solving: 3) Most of endpoints are accessible via API or File exchange (no GUI)
25
REQUEST RESPONSE
Solving: 4) Reference Data setup or migration
26
Reference Data required for Testing
Data Management Tools
Production Data Migration Tasks
Solving: 5) Complexity of Risk Calculation Algorithms
27
Test Model
Solving: 6) Regression Cycle for a substantial number of test scenarios
28
Test Library(100K Test cases)
Test Tools
Automated Test Scripts
Controlled Test Environment
Automated Regression Run(“The Big Button”)
A Holistic Integrated Automated Test Solution
29
Conclusion
30
Gre
at Q
A Te
am &
To
ols
for P
ost T
rade
Questions & Answers
Thank You
31