![Page 1: File System Test Results Analysis - SNIA · 2019. 12. 21. · File System Test Results Analysis AridayBalderas (ariday@mx1.ibm.com) Drew Olson (drolson@us.ibm.com) Julian Cachua (cachuaju@mx1.ibm.com)](https://reader031.vdocuments.site/reader031/viewer/2022011915/5fcf622414f16078a92f58a5/html5/thumbnails/1.jpg)
2018 Storage Developer Conference. © IBM. All Rights Reserved. 1
File System Test Results Analysis
Ariday Balderas ([email protected])Drew Olson ([email protected])
Julian Cachua ([email protected])
IBM
![Page 2: File System Test Results Analysis - SNIA · 2019. 12. 21. · File System Test Results Analysis AridayBalderas (ariday@mx1.ibm.com) Drew Olson (drolson@us.ibm.com) Julian Cachua (cachuaju@mx1.ibm.com)](https://reader031.vdocuments.site/reader031/viewer/2022011915/5fcf622414f16078a92f58a5/html5/thumbnails/2.jpg)
2018 Storage Developer Conference. © IBM. All Rights Reserved. 2
Filesystem CI Development Cycle
Multiple functional teams Specialized areas Complex scenarios Tight release schedules
Test Automation is the preferred approach to assure quality.
![Page 3: File System Test Results Analysis - SNIA · 2019. 12. 21. · File System Test Results Analysis AridayBalderas (ariday@mx1.ibm.com) Drew Olson (drolson@us.ibm.com) Julian Cachua (cachuaju@mx1.ibm.com)](https://reader031.vdocuments.site/reader031/viewer/2022011915/5fcf622414f16078a92f58a5/html5/thumbnails/3.jpg)
2018 Storage Developer Conference. © IBM. All Rights Reserved. 3
Complicated Scenario
End up with thousands of validation scripts.
Execution time increases. Infrastructure issues
probability raises. Review automation results
requires more human intervention. If distributed, manual test analysis
can result in inefficiencies.
![Page 4: File System Test Results Analysis - SNIA · 2019. 12. 21. · File System Test Results Analysis AridayBalderas (ariday@mx1.ibm.com) Drew Olson (drolson@us.ibm.com) Julian Cachua (cachuaju@mx1.ibm.com)](https://reader031.vdocuments.site/reader031/viewer/2022011915/5fcf622414f16078a92f58a5/html5/thumbnails/4.jpg)
2018 Storage Developer Conference. © IBM. All Rights Reserved. 4
Implications
Test Automation value decreases. False positives raise. CI quality report becomes misleading. Human effort to clarify status. Test teams need to grow to
accommodate test analysis.
![Page 5: File System Test Results Analysis - SNIA · 2019. 12. 21. · File System Test Results Analysis AridayBalderas (ariday@mx1.ibm.com) Drew Olson (drolson@us.ibm.com) Julian Cachua (cachuaju@mx1.ibm.com)](https://reader031.vdocuments.site/reader031/viewer/2022011915/5fcf622414f16078a92f58a5/html5/thumbnails/5.jpg)
2018 Storage Developer Conference. © IBM. All Rights Reserved. 5
Test Analyzer Library
CI focused framework to help us analyze test results and provide verdicts with the least possible human intervention.
![Page 6: File System Test Results Analysis - SNIA · 2019. 12. 21. · File System Test Results Analysis AridayBalderas (ariday@mx1.ibm.com) Drew Olson (drolson@us.ibm.com) Julian Cachua (cachuaju@mx1.ibm.com)](https://reader031.vdocuments.site/reader031/viewer/2022011915/5fcf622414f16078a92f58a5/html5/thumbnails/6.jpg)
2018 Storage Developer Conference. © IBM. All Rights Reserved. 6
Questions on Failure Analysis
What• Culprit• Test hierarchy• Test Suite / Test Case
When• First time?• Last success?• Version
Where• Cluster• Node• Re-creatable
How• Did it happen?• Installation type?• Often?
![Page 7: File System Test Results Analysis - SNIA · 2019. 12. 21. · File System Test Results Analysis AridayBalderas (ariday@mx1.ibm.com) Drew Olson (drolson@us.ibm.com) Julian Cachua (cachuaju@mx1.ibm.com)](https://reader031.vdocuments.site/reader031/viewer/2022011915/5fcf622414f16078a92f58a5/html5/thumbnails/7.jpg)
2018 Storage Developer Conference. © IBM. All Rights Reserved. 7
Highlights
Data Schema Test Execution Data Extraction* Rule Validation* Verdicts Actions
# bin/testanalyzer init
# bin/testanalyzer run
# *bin/testanalyzer post
![Page 8: File System Test Results Analysis - SNIA · 2019. 12. 21. · File System Test Results Analysis AridayBalderas (ariday@mx1.ibm.com) Drew Olson (drolson@us.ibm.com) Julian Cachua (cachuaju@mx1.ibm.com)](https://reader031.vdocuments.site/reader031/viewer/2022011915/5fcf622414f16078a92f58a5/html5/thumbnails/8.jpg)
2018 Storage Developer Conference. © IBM. All Rights Reserved. 8
Analyzing Test within the Pipeline
Start Bucket
Preconditions
Preconditions
Preconditions
Preconditions
Preconditions
Test
Test
Test
Test
Test
Cleanup
Cleanup
Cleanup
Cleanup
CleanupTest
Analysis
![Page 9: File System Test Results Analysis - SNIA · 2019. 12. 21. · File System Test Results Analysis AridayBalderas (ariday@mx1.ibm.com) Drew Olson (drolson@us.ibm.com) Julian Cachua (cachuaju@mx1.ibm.com)](https://reader031.vdocuments.site/reader031/viewer/2022011915/5fcf622414f16078a92f58a5/html5/thumbnails/9.jpg)
2018 Storage Developer Conference. © IBM. All Rights Reserved. 9
Data Schema
![Page 10: File System Test Results Analysis - SNIA · 2019. 12. 21. · File System Test Results Analysis AridayBalderas (ariday@mx1.ibm.com) Drew Olson (drolson@us.ibm.com) Julian Cachua (cachuaju@mx1.ibm.com)](https://reader031.vdocuments.site/reader031/viewer/2022011915/5fcf622414f16078a92f58a5/html5/thumbnails/10.jpg)
2018 Storage Developer Conference. © IBM. All Rights Reserved. 10
Extraction examples
Demo Queries
![Page 11: File System Test Results Analysis - SNIA · 2019. 12. 21. · File System Test Results Analysis AridayBalderas (ariday@mx1.ibm.com) Drew Olson (drolson@us.ibm.com) Julian Cachua (cachuaju@mx1.ibm.com)](https://reader031.vdocuments.site/reader031/viewer/2022011915/5fcf622414f16078a92f58a5/html5/thumbnails/11.jpg)
2018 Storage Developer Conference. © IBM. All Rights Reserved. 11
Rules
Common behavior: Return code validation Test Case step Regex where applies Dates System information
Stable suites 3 successful execution
![Page 12: File System Test Results Analysis - SNIA · 2019. 12. 21. · File System Test Results Analysis AridayBalderas (ariday@mx1.ibm.com) Drew Olson (drolson@us.ibm.com) Julian Cachua (cachuaju@mx1.ibm.com)](https://reader031.vdocuments.site/reader031/viewer/2022011915/5fcf622414f16078a92f58a5/html5/thumbnails/12.jpg)
2018 Storage Developer Conference. © IBM. All Rights Reserved. 12
Defects
Walkthrough Demo
![Page 13: File System Test Results Analysis - SNIA · 2019. 12. 21. · File System Test Results Analysis AridayBalderas (ariday@mx1.ibm.com) Drew Olson (drolson@us.ibm.com) Julian Cachua (cachuaju@mx1.ibm.com)](https://reader031.vdocuments.site/reader031/viewer/2022011915/5fcf622414f16078a92f58a5/html5/thumbnails/13.jpg)
2018 Storage Developer Conference. © IBM. All Rights Reserved. 13
Summary
Test Analysis becomes a demanding task on CI environments.
A framework to analyze and provide verdicts helps on agile adoption.
Rules need to be defined according to product and conditions.
![Page 14: File System Test Results Analysis - SNIA · 2019. 12. 21. · File System Test Results Analysis AridayBalderas (ariday@mx1.ibm.com) Drew Olson (drolson@us.ibm.com) Julian Cachua (cachuaju@mx1.ibm.com)](https://reader031.vdocuments.site/reader031/viewer/2022011915/5fcf622414f16078a92f58a5/html5/thumbnails/14.jpg)
2018 Storage Developer Conference. © IBM. All Rights Reserved. 14
Q&A