Download - Making Production Like Test Environments
www.ranger4.com
DevOpstastic
WebcastMaking Production Like Test
Environments
Helen Beal Nick Blamey
www.ranger4.com
DevOpstastic
2
www.ranger4.com
DevOpstastic
3
Traditional Testing: Build prototype, then try to break
the car after assembly.
www.ranger4.com
DevOpstastic
End to EndTest
Most Issues are found by QA / Load Testers prior to going live.
Defects / Issues Found
. . .
Pre-Service Virtualisation
Production
www.ranger4.com
DevOpstastic
5
Google: “How to build a supercar + McLaren”
Shift Left: Find Defects Earlier: Final End to End Test = Fine TuningSaves Cost, Isolate defects pre-production
www.ranger4.com
DevOpstastic
Most issues are found and fixed within a Sprint.
Defects / Issues Found
. . .End to End Test becomes an Assurance exercise rather than Testing
End to EndTest
Post-Service Virtualisation
Production
www.ranger4.com
DevOpstastic
7
Challenges
Agile: new demands on
testing
Mobile / Consumerism: Higher project
throughput (compressed
SDLC)
Complex ecosystem
of IT Delivery
Multiple stakeholders
Faster identification
of defects
Escalating cost of quality
www.ranger4.com
DevOpstastic
Presentation layer
Integrations, data and business logic
Only one-tenth of an iceberg is above water
Traditional UI testing:
Automated Integration Testing and Service Virtualization:
The majority of risk in modern systems is seldom tested as it is unseen.
www.ranger4.com
DevOpstastic
WebSphere – Example Technology Landscape
FINANCE(CICS)
MANUFACTURING(TIBCO BUISNESS WORKS)
CRM(SOFTWARE AG IS)
TRANSPORT(SAP PI)
XML/HTTP
CICS TG
REST
ABAP
SI Bus
wmBroker
JDBC
RMI
MQ
IBMINTEGRATIONBUS
WEBSPHEREAPPLICATIONSERVER
HTTP
SALES(ORACLE)
HR(WEBSPHERE)
DFDL/TCP
www.ranger4.com
DevOpstastic
Value Statement
Expected Business Case
Reduce software development costs and increase xxxxxxxx capability to deliver projects on time, budget and to the business requirements
Cumulative Savings of £3.84m over 5 years
Average annual benefit
£768k p.a.
The Impact of Testing Earlier Cost reduction by fixing Defects earlier in the software development lifecycle - based on XXXXXXs 15,692 defects from Nov 1st 2013 to Oct 31st 2014. Defects found in the later stages of the lifecycle cause substantially more rework, additional cost and delivery delays compared with finding these defects earlier
Integration Test
Big Bang
AcT Test OATSystem Test
£3,200
£7,680
£97
Reduction in cost of
fixing defects
Earlier testing and Risk mitigation
£97 - Integration Test £250 - System Test£3,200 - AcT Test £7,680 - OATSource: SEI*
£250 *Source: The Software Engineering Institute (SEI) is an independent body http://www.sei.cmu.edu/
www.ranger4.com
DevOpstastic
Sys Tests
Unit Tests
UAT
OAT
Performance
Quality Gate
The Old WorldBig Bang
• Multiple defects introduced at once• More unknowns – greater risk• Expensive!
Requirements
www.ranger4.com
DevOpstastic
Requirements
Unit Tests
Performance
UAT
OAT
The New WorldBig Bang
Integration Tests
Sys Tests
• Accelerated testing• Reduced costs• Lowered risk
www.ranger4.com
DevOpstastic
Concept: Create and Sending of Payloads
Payload
www.ranger4.com
DevOpstastic
Concept: Formating of Payloads
FormatterPayload
Schema?
www.ranger4.com
DevOpstastic
Concept: Use of Schema for Fortmatting
SchemaFormatterPayload SOAP XML
SWIFT COBOL CopybookDTD XSDWSDL HL7EDI IATAText MIMEByte Array Java ObjectsOAG SAP BAP/RFCFIX .Net ObjectsTIBCO ActiveEnterprisewebMethods IB and IS DocumentsCustom…
www.ranger4.com
DevOpstastic
Concept: Use of Test Data
Test DataSchemaFormatterPayload
www.ranger4.com
DevOpstastic
Concept: Use of Transport protocols
TransportTest DataSchemaFormatterPayload JMS webMethods
HTTP/S FTPNDM Oracle/BEAJDBC Sonic MQTCP/UDP PL/SQLFlat Files Shell commands
TIBCO EMS TIBCO iProcessTIBCO RendezvousIBM WebSphere MQCustom Transports…
www.ranger4.com
DevOpstastic
This perspective of RIT shows a logical representation of all available services, servers, DBs, etc. Environments can be
build be mapping the logical representation to different physical
resources.
Model the system
www.ranger4.com
DevOpstastic
Recorded messages. Each color shows a different transport
(MCE, SOA, Saclus DB)
Structure of the selected message.
Format is independent of the message format
(here XML)
Record and re-use live data
www.ranger4.com
DevOpstastic
Testcase created out of the recorded messages. In the case, the test iterates over test date
stored in a CSV file, which is easy to maintain and which could be
reused in other tests/stubs
Hierarchical structure of tests, testdata sets and stubs, grouped by the
different services
Build : Virtual Services, Tests & Performance Tests
www.ranger4.com
DevOpstastic
Execution of a test suite, one of the included tests
failed, one succeeded.
Execute tests, virtual services & performance Tests
www.ranger4.com
DevOpstastic
Report of the test results of a test suite. The links can be used to
jump to a detailed description of a single test. This reports is also
accessible as web page.
Understand the results
www.ranger4.com
DevOpstastic
Library of all deployed stubs. These can be started and stopped with this dashboard.
Deploy for Enterprise re-use of Virtual Services
www.ranger4.com
DevOpstastic
“RBS Markets deploys IBM Rational for virtual testing”Speaking at an IBM Innovate 2013 event in London, Stephen King, Head of Middleware at RBS Markets, said:
We selected IBM Rational Integration Test and piloted a payment compliance application which had to be delivered quickly to prevent the bank facing £5m fines.
Using Rational and the new methodology, the developer team was able to deliver the application in six weeks.
Following the success of the payment application, King said RBS rolled out the methodology and toolset across all the projects the team was working on.
Prior to deploying Rational and the software development methodology, King said regression testing used to take four weeks. It now takes a couple of hours, he said, saving the bank £40,000 in project costs.
Computer Weekly – 7th November 2013 - http://www.computerweekly.com/news/2240208647/RBS-Markets-deploys-IBM-Rational-for-virtual-testing
Key metrics quoted in King’s presentation included:
• 95% increase in project delivery • 99.64% reduction in production incidents • Cost reduction of 63%• Time to market reduced by 40%• 4 week regression tests reduced to 1 hour• Potential fines of 5 million pounds avoided
www.ranger4.com
DevOpstastic
Intangible Benefits
• RTVS will deliver a Common standard for creation of all stubs. A standardized simple and efficient approach to creation of stubs will ensure:
– 100% coverage of all opportunities to create stubs and hence deliver a complete integration testing solution to improve quality delivery
– Stubs are created without need for Development or programming effort – Stubs are re-usable by development and test areas– The process of creation and enhancement of stubs will reduce reliance on scarce
and expensive Subject Matter Experts and Resources e.g. WebSphere MQ + assorted Middleware, SOA Experts
• IBM Rational Test Workbench enables more testing without availability of test environments::
– Some systems are not available the whole time they need to test, therefore, Green Hat VIE can extend test window
– Customer’s 3rd party provided test environment is unavailable / expensive
Centre of Excellence for Stubbing: Creating Testing Assets
www.ranger4.com
DevOpstastic
Intangible BenefitsStart of day testing – • Customer can perform Smoke tests with Rational Integration Tester to ensure that all applications and their components are available and working before starting either the Functional or Performance Testing Process
• This smoke test on completion can then advise Integration / Functional and Performance test teams on what is and isn’t working.
Benefits: • Reduce costs of Managed Test Service: Less Managed Test Service wastage / Consumption based infrastructure charges.
• Data / Batch files are correct prior to load and functional test execution
Smoke Tests to Prove the Environment
www.ranger4.com
DevOpstastic
Continuous Testing Approach
• Test every build• Automatically build, deploy applications to
environments • Leverage virtualization – Use service
virtualization to emulate 3rd party connections
• Automatically provision environments – based on patterns and templates
• Once deployed – Automatically run tests – functional / integration / etc.
www.ranger4.com
DevOpstastic
• System hasn’t been built yet• Components haven’t been delivered yet• Business hasn’t decided which component to buy
yet e.g. AML, Fraud Detection, single sign on, ESB• Two speed IT• Complex Environment with escalating Costs• Data Contention• 3rd party ownership / external stakeholder• Testing window issues
Bottlenecks to Continuous Testing
IBM Continuous Testing is an enabler to solve each of these challenges
www.ranger4.com
DevOpstastic
Be DevOpstastic