testing machine learning, learning machine testing. eurostar 2016 rik marselis
TRANSCRIPT
1980
2007 2012 20122008
Workgroup-member
20142009
Rik Marselis@rikmarselis
Who uses a robot today?
You will use multiple within just a few years!!
And please realise: these are robots too!
Chatbot Self-driving mini-bus
WePod: self-driving bus
This minibus is allowed on the road now in Wageningen (the Netherlands) andwas tested by mySogeti-colleaguesDragan, Hugo andRemco.
Robots, ok, but in less than 10 years????
Well;
Did you understand this
trafficsign 10 years ago?
That’s how fast technology
progresses!!
What is Robotics?
• Internet of Things (IoT)
• Big data analytics
• Cloud technology
• Machine learning
• Artificial Intelligence
Robotics
Definition: A robot is a machine that gathers information about its environment by input of sensors and based on this input changes its behavior. Combined with machine learning and artificial intelligence the robot’s reactions over time get more and more adequate. The use of Internet of Things, Bid Data Analytics and Cloud technology make a robot versatile. Robots may appear in a physical shape (e.g. a tin man) or non-physical (e.g. chatbot).
Traditional: Robots for dull & dangerous work
New possibilities, just 2 examples
Exo-skeleton
Healthcare disabled peopleTransport drone delivery
New risks
The bull doesn’t likethe manure robot andconstantly pushes it off its track. (to much frustration of the farmer who needs to cometo rescue the robot ;-)
New angles of quality, aimed at roboticsMechanical
Electrical
Information Processing
Machine Intelligence
Business Impact
Social Impact
e.g. Ethics
e.g. Embodiment
e.g. Empathy
Part 1:
Testing machine learning
Machine Learning in the 80’s
Machine learning today: demonstrating
Machine learning today: demonstrating
Learning by creating rules
How can we find the rules that self-drivingcars must use?
Ask you!
moralmachine.mit.edu
Some
results
Machine learning or machine training?
• A learning machine consists of a complex network
of algorithms
• The learning machine has very many “dials and
switches”
• Making sure the machine learns the right things is
just like ordinary training (well, in a different way…)
Testing Machine Intelligence
• Human: “Dear robot please get me a softdrink”
• Robot: “Dear human here is your cola”
• Is this the correct output?
• No!
In Sweden I want:
Testing Machine Intelligence
• Human: “Dear robot please get me a softdrink”
• Robot: “Dear human it’s Saturday
evening now, so here is your beer”
• Is this the correct output?
• After studying your behavior,
the robot knows your patterns
Testing traditional IT vs. machine intelligence
Traditional (example):
• 1 + 1 = ?
• There’s only one correct output
With Machine Intelligence (example):
• Prepare a meal
• When is the meal “good enough”?
• Is it always the same meal?
2
Two sides of testing machine intelligence
• Testing the output
This we know from traditional testing,
but it’s hard to predict outputs, the
result may be different every time.
• Testing the input
If the machine gets the right input
it will learn and do the right things.
To control the results of AI:
control the inputs…
Helpful chatbots?
• There’s a very thin line between a robot that is not helpful,
a robot that is helpful and a robot that’s too helpful
Helpful chatbots?
The uncanny valley:
What is the role of the tester?
• The tester must see whether the intelligent
machine will maintain the intended behavior in live
operation
• In a way the tester is the psychologist of the
intelligent machine
• A tester must ask even more questions than
before !!
Note: I picked up the idea of the tester being the psychologist of robots from James Bach.
It’s no longer about right or wrong
• Testing intelligent machines is not an easy check
whether the result is right or wrong
• It becomes an appraisal of a specific response in a
certain situation with a certain amount of
knowledge for a certain audience
• If these parameters change the right behavior will
also change
After finishing the test…
• The machine continues to learn and adapt…
• What about confidence?
• The test must result in the confidence that the
machine will learn and adapt in the right way…
Part 2:
Learning machine testing
Using a robot for dull or dangerous testing
• Mechanical tests use robot-arm
Using a robot for intelligent testing
• Functional tests use machine intelligence to
– create test cases and
– execute test cases
– and (important!!) to judge the results
Using a robot for testing physical objects
Using a robot for intelligent testing
• Non-functional tests use machine intelligence
For example intelligent performance testing
(not just simple transactions but a normal variety)
People and Robots testing together
• Teamwork of person & robot
• Each does what it is best at
– Robot repetitive work
– Human creative work
Machine learning: unsupervised & supervised
Deep learning:
Robot learns at various
layers. Lower layers
are unsupervised.
Highest layer(s) is (are)
supervised.
Machine Intelligence generates test data
• Based on Big Data Analytics of live data
• This is so-called “weak intelligence”
The machine is only intelligent in a limited field
Machine Intelligence generates test cases
• Evolutionary algorithms
• Brute force testing
The machine is able to generate and execute
many many more test cases than a human tester
Conclusion
Testing machine learning
• Testing inputs is very important because the output
is not predictable
• The objective of testing is to gain confidence that
the machine will learn and develop itself in a right
way
Learning machine testing
• Test automation will go to the next level
• Use machine intelligence (e.g. big data analytics)
to create your optimal test data
• Use machine intelligence to create many more test
cases, execute them and analyze results
Conclusion…
As a tester better prepare yourself for:
• Testing of Robots
and
• Testing with Robots
Thanks for attending my talk.
Any questions ???????????
Rik Marselis, Sogeti Nederland