machine learning for test automation - anztb...test automation uses properties of controls to find...
TRANSCRIPT
![Page 1: Machine Learning for Test Automation - ANZTB...Test automation uses properties of controls to find and steer them 2. Controls on a web-page change as they are developed Therefore test](https://reader034.vdocuments.site/reader034/viewer/2022052001/601329eaf2024f11c646f1da/html5/thumbnails/1.jpg)
Machine Learning for Test AutomationSam Fernando - May, 2017
![Page 2: Machine Learning for Test Automation - ANZTB...Test automation uses properties of controls to find and steer them 2. Controls on a web-page change as they are developed Therefore test](https://reader034.vdocuments.site/reader034/viewer/2022052001/601329eaf2024f11c646f1da/html5/thumbnails/2.jpg)
Agenda
• What is machine learning?
• Application to test automation
• Recommendations
![Page 3: Machine Learning for Test Automation - ANZTB...Test automation uses properties of controls to find and steer them 2. Controls on a web-page change as they are developed Therefore test](https://reader034.vdocuments.site/reader034/viewer/2022052001/601329eaf2024f11c646f1da/html5/thumbnails/3.jpg)
What is Machine Learning?
What is artificial intelligence?
What is human intelligence?
Is this a smart phone?(does it have intelligence)
![Page 4: Machine Learning for Test Automation - ANZTB...Test automation uses properties of controls to find and steer them 2. Controls on a web-page change as they are developed Therefore test](https://reader034.vdocuments.site/reader034/viewer/2022052001/601329eaf2024f11c646f1da/html5/thumbnails/4.jpg)
Machine Learning 101
AI and Machine learning
• An intelligent agent is “any device that perceives its environment and takes actions that maximize its chance of success at some goal”
• Machine Learning is the training of a model from data that generalizes a decision against a performance measure
• ‘Learning’: Getting better at a task over time (i.e. takes into account new data points)
![Page 5: Machine Learning for Test Automation - ANZTB...Test automation uses properties of controls to find and steer them 2. Controls on a web-page change as they are developed Therefore test](https://reader034.vdocuments.site/reader034/viewer/2022052001/601329eaf2024f11c646f1da/html5/thumbnails/5.jpg)
Machine Learning 101
Supervised learning
• Learn from test data
• Make predictions
• Data is labelled
Examples
• Email spam filter
• Image recognition
![Page 6: Machine Learning for Test Automation - ANZTB...Test automation uses properties of controls to find and steer them 2. Controls on a web-page change as they are developed Therefore test](https://reader034.vdocuments.site/reader034/viewer/2022052001/601329eaf2024f11c646f1da/html5/thumbnails/6.jpg)
Machine Learning 101
Unsupervised learning
• Discover structure within the data
• Data is unlabelled
Examples
• Google news
• Social network analysis
![Page 7: Machine Learning for Test Automation - ANZTB...Test automation uses properties of controls to find and steer them 2. Controls on a web-page change as they are developed Therefore test](https://reader034.vdocuments.site/reader034/viewer/2022052001/601329eaf2024f11c646f1da/html5/thumbnails/7.jpg)
What we have been working on
How could machine learning help with testing?
- Natural language processing: turn requirements/stories into tests?
- Create new test data based on defects found?
- Automation maintenance: Recognise controls even when they change
![Page 8: Machine Learning for Test Automation - ANZTB...Test automation uses properties of controls to find and steer them 2. Controls on a web-page change as they are developed Therefore test](https://reader034.vdocuments.site/reader034/viewer/2022052001/601329eaf2024f11c646f1da/html5/thumbnails/8.jpg)
What’s the problem?
• When a control – say an edit box on a web-page – changes, we, as humans, can see what control our test script it meant to steer.But our scripts cannot.
• This leads to false negatives:• Lost execution time
• Increase analysis time
• Increased maintenance time
![Page 9: Machine Learning for Test Automation - ANZTB...Test automation uses properties of controls to find and steer them 2. Controls on a web-page change as they are developed Therefore test](https://reader034.vdocuments.site/reader034/viewer/2022052001/601329eaf2024f11c646f1da/html5/thumbnails/9.jpg)
What’s the problem?
1. Test automation uses properties of controls to find and steer them
2. Controls on a web-page change as they are developed
Therefore test automation scripts won’t be able to find the controls
Machine learning solution:
• How can we get our automation script to learn what the right control is even when it has changed in some respects?
![Page 10: Machine Learning for Test Automation - ANZTB...Test automation uses properties of controls to find and steer them 2. Controls on a web-page change as they are developed Therefore test](https://reader034.vdocuments.site/reader034/viewer/2022052001/601329eaf2024f11c646f1da/html5/thumbnails/10.jpg)
Machine learning process
• The web-archive stores the html of each iteration of web-sites
• Use selenium to scrape the html elements, and their properties, for historic version of the same web-site
Example:
https://www.kiwibank.co.nz/ CSV file
![Page 11: Machine Learning for Test Automation - ANZTB...Test automation uses properties of controls to find and steer them 2. Controls on a web-page change as they are developed Therefore test](https://reader034.vdocuments.site/reader034/viewer/2022052001/601329eaf2024f11c646f1da/html5/thumbnails/11.jpg)
Step 1: Gather Data
• The web-archive stores the html of each iteration of web-sites
• Use selenium to scrape the html elements, and their properties, for historic version of the same web-site
Example:
https://www.kiwibank.co.nz/ CSV file
![Page 12: Machine Learning for Test Automation - ANZTB...Test automation uses properties of controls to find and steer them 2. Controls on a web-page change as they are developed Therefore test](https://reader034.vdocuments.site/reader034/viewer/2022052001/601329eaf2024f11c646f1da/html5/thumbnails/12.jpg)
Step 1: Gather Data
Simplify the problem to just input elements
Demo
![Page 13: Machine Learning for Test Automation - ANZTB...Test automation uses properties of controls to find and steer them 2. Controls on a web-page change as they are developed Therefore test](https://reader034.vdocuments.site/reader034/viewer/2022052001/601329eaf2024f11c646f1da/html5/thumbnails/13.jpg)
Step 2: Choose an algorithm
• We need to pick a model which represents the patterns in the data
• The algorithm will then optimise the model’s parameters to suit our data
• Would normally pick multiple models, and find the best one
We will use supervised learning: linear regression
- Label which controls are matches to other controls
- Look at each property independently
![Page 14: Machine Learning for Test Automation - ANZTB...Test automation uses properties of controls to find and steer them 2. Controls on a web-page change as they are developed Therefore test](https://reader034.vdocuments.site/reader034/viewer/2022052001/601329eaf2024f11c646f1da/html5/thumbnails/14.jpg)
Step 3: Pre-process the Data
• Use supervised learning:
Label which controls – across different web site iterations - are in fact the same as each other
• Machine learning algorithms only deal with numbers not strings
Compare how similar each property of a control is to the same property of the other controls, assigning a number between 0 and 1
![Page 15: Machine Learning for Test Automation - ANZTB...Test automation uses properties of controls to find and steer them 2. Controls on a web-page change as they are developed Therefore test](https://reader034.vdocuments.site/reader034/viewer/2022052001/601329eaf2024f11c646f1da/html5/thumbnails/15.jpg)
Step 4: Train the Classifier
• The algorithm, Linear Regression, learns how important each property is in determining whether or not a control is a match
Demo
![Page 16: Machine Learning for Test Automation - ANZTB...Test automation uses properties of controls to find and steer them 2. Controls on a web-page change as they are developed Therefore test](https://reader034.vdocuments.site/reader034/viewer/2022052001/601329eaf2024f11c646f1da/html5/thumbnails/16.jpg)
Step 6: Keep learning
• When the classifier is used, feedback if it picked the correct control
• That becomes a new piece of data which can be fed into the algorithm, which refines the parameters of the classifier’s model
![Page 17: Machine Learning for Test Automation - ANZTB...Test automation uses properties of controls to find and steer them 2. Controls on a web-page change as they are developed Therefore test](https://reader034.vdocuments.site/reader034/viewer/2022052001/601329eaf2024f11c646f1da/html5/thumbnails/17.jpg)
Step 5: Make Predictions
• The classifier can now be ran against a web-page of controls, comparing each control to the one our test script is looking for
• It returns how likely it is that each control is a match
• The tool, here Tosca, will execute the test step on that control
Demo
![Page 18: Machine Learning for Test Automation - ANZTB...Test automation uses properties of controls to find and steer them 2. Controls on a web-page change as they are developed Therefore test](https://reader034.vdocuments.site/reader034/viewer/2022052001/601329eaf2024f11c646f1da/html5/thumbnails/18.jpg)
Recommendations and Limitations
• Lack of generalisation• How one control is changed, by one developer, for one web-site, could be
totally different from another
• Even with more data, better pre-processing (i.e. ‘submit’ ~ ‘login’), and non-linear relationships between inputs, it is unlikely that a general pattern exists.
• Effort would be better spent on a pure rules-based best match solution: filtering, cases, weighted similarity
• Using automation best practices would resolve most solvable robustness issues: repeatability, timing, recovery
![Page 19: Machine Learning for Test Automation - ANZTB...Test automation uses properties of controls to find and steer them 2. Controls on a web-page change as they are developed Therefore test](https://reader034.vdocuments.site/reader034/viewer/2022052001/601329eaf2024f11c646f1da/html5/thumbnails/19.jpg)
Questions and Discussion