light speed with test data management

Light Speed with Test Data ManagementEfficiency Out of Chaos

Kellyn Pot’Vin-Gorman | Technical Intelligence Manager

2© 2016 Delphix Corporation

Kellyn Pot’Vin-GormanTechnical Intelligence Manager, Delphix• Multi-platform DBA, (Oracle, MSSQL, MySQL,

Sybase, Postgres…..)• Oracle ACE Director, (Alumni)• Oak Table Network• APEX Women in Technology Award, CTA 2014• STEM education with Raspberry Pi and Python• Board of Director, RMOUG• Training Days Conference Director• Author, blogger, (http://dbakevlar.com)

This session is focused on the database and application environment. No actual database or vendor platform special

knowledge is required to gain value from the session, although I can answer questions on most platforms due to my

multi-platform technical background.

© 2016 Delphix Corporation 4

AgendaStory Time1What is Test Data Management2Agile Testing, Speed is Everything3Data Masking4Code Control5


Story Time

• Large company

• 4TB transactional database, (small by today’s standards)

• Financial data, aggregated to other financial systems.

• Agile development released, (most often) directly to production

• Archaic development, (1/3 of size of production) and a rarely used test environment.


After 4-6 Weeks of Research…

• Over 40% data corruption to main transactional system that feeds into datamarts.

• Even higher percentage of corruption impact in marts due to poor agile development practices and percentage of highly volatile source data.

I now have to reveal my findings to senior management as the new Lead DBA…


“Accuracy is over-rated…”


https://www.tutorialspoint.com/software_testing_dictionary/test_data_management.htm

Test Data management is very critical during the test life cycle. The amount of data that is generated is [often] enormous to test the required changes.

Over 80% of organizations stated that RECEIVING or REFRESHING the data to perform tests was the largest consumer of testing time, (over 90%) leaving actual work to consume less than 10% of the overall testing scenario.

Test Data Management


What Is Test Data Management?• Tools that assist in producing data sets for testing.

• Not only produce, but do so in an interval that is able to match the testing cycle.

• An ability to quickly isolate and deliver test FAILURES for development to investigate.

• Ability to identify code changes by versions

• To deliver in a timely manner, many TDM applications create synthetic or subsets of data that aren’t a full copy of the production experience.

• Don’t take copy data management and security into consideration.


As important as development environments are, providing proper

testing that’s able to handle the speed of your companies

deployments are a bigger hurdle.


Cloning- Can’t Be Done Effectively

http://www.informationweek.com/pdf_whitepapers/approved/1345732672_back_to_basics.pdf


What is the “Right Size”?

These are not the same data or the same challenges the

developer and tester will face once it gets to production!

http://www.informationweek.com/pdf_whitepapers/approved/1345732672_back_to_basics.pdf


▶▶▶

Virtualize and Deployed ▶ ▶▶

Virtualization Eliminates 90%+ of Storage Issues

Storage Pool for Delphix

QADEV PATCH TEST

PRODUCTIONDatabase/App Tier

1 TB

1 TB

600GBRead From Production

Each Virtual Database takes up around 5-10Gb upon creation, (dependent upon parameters)

TEST

Read AND Write

14© 2016 Delphix Corporationhttp://www.idtheftcenter.org/images/breach/ITRCBreachReport_2016.pdf

Data Security

Almost 30 Million!


Confidential data

Production

Non-Production

Exposure


Confidential data

Production

Non-Production

Exposure

Encryption

Masking

Solution


As 80% of data in a company are copies, then 80% of data won’t be subject to security like a production environment. Securing this data is not just a priority, but in many cases, subject to legal ramifications, (i.e. PCI/PII)

• Masking Requirements• Masking shouldn’t be reversible• The masked data should be representative of the original data type to ensure

performance is consistent.• Referential Integrity should be maintained as part of the masking process.

Masking should be a simple, repeatable process with a user interface that ensures it is simple.

Masking in the Picture


Do I Have to Mask Data? Nah….

Type of Data Year Passed RulingData Masking in the EU

2014 ARTICLE 29 DATA PROTECTION

HIPAA 1996 Health Insurance Portability and Accountability Act

PCI 2016, (Updated)

Payment Card Industry Standards

PII Personably Identifiable Information

SOX 2002 Sarbanes-Oxley Act

https://secure.edps.europa.eu/EDPSWEB/edps/site/mySite/Art29



http://www.hhs.gov/hipaa/for-professionals/security/laws-regulations/

http://www.hhs.gov/hipaa/for-professionals/security/laws-regulations/

https://www.pcisecuritystandards.org/

http://www.gsa.gov/portal/content/104256

http://www.soxlaw.com/


Ability to Deliver and Mask Data for Testing- FAST

DB

APP

DB

APPDB

APP

DB

APP

DB

APP

PRODUCTION DB

Mask PCI/PII and then virtualize

Develop Test Deploy

Refresh and Start Over


Source Control

“A component of software configuration management, version control, also known as revision control or source control, is the management of changes to documents, computer programs, large web sites, and other collections of information.”


Branching and Bookmarking• The ability to mark each iteration of development with a

bookmark

• Simplify to lock and deliver while testing a consistent image via a virtual database, (VDB)

• If a test goes wrong, the ability to “bookmark”, (and subsequent snapshot) to deliver to development to address.


Environments

Snapshot

Previous Snapshots

Virtual to Physical

Timeflow Slider

Actions


How This All Comes Together…

• Virtualization is the key to fast, efficient and FULL copies of production environments for agile and automated testing for agile shops.

• Data masking that can be done once, easily maintained with a repeatable process via a strong discovery and implementation as part of the virtualization process secures the 80% of data that is outside the control of production.

• Virtualized environments that are built with development and testing in an Agile or DevOps environments makes it simple to accomplish what may see impossible and do so at light speed.

Kellyn Pot’Vin-GormanTechnical Intelligence [email protected]://dbakevlar.com

mailto:[email protected]

http://dbakevlar.com/