dr. shankar ramamoorthy customer success...

©2015 HARMAN INTERNATIONAL INDUSTRIES, INCORPORATED 1

NEXT GEN -COGNITIVE ROBOTIC PROCESS TEST AUTOMATION FRAMEWORK-( NGCRPTAF)

6 D MACHINE LEARNING MODEL FOR SOFTWARE TESTING

DR. SHANKAR RAMAMOORTHY

CUSTOMER SUCCESS OFFICER


1. What we do

2. How we do it

3. Where we are going

AGENDA


LIFESTYLE AUDIO PROFESSIONAL SOLUTIONS

CONNECTEDSERVICESCONNECTED CAR

Navigation, Multimedia, Connectivity, Telematics, Safety & Security Solutions

Premium Branded Audio Products and Sound Management Softwarefor Car, Home and on the Go

Audio, Lighting, Video Switching and Automation for Enterprise and Entertainment

Cloud, Mobility and Analytics Solutions with OTA Updates for Car, Mobile and Enterprises

1. What we do

TECHNOLOGIES FOR A CONNECTED WORLD


CONNECTED CAR LIFESTYLE AUDIO PROFESSIONAL SOLUTIONS CONNECTED

SERVICES

1. WHAT WE DO

WE ARE A HOUSE OF BRANDS


Technologies to enable the connected car, enterprise and lifestyle

2.0 HOW WE DO IT

A PASSIONATE PURSUIT OF INNOVATION

Signal Processing Automation & Controls

Live Performance

OTA, Safety & Security Cloud & Analytics Sustainability

Connectivity Solutions

Experience Design


HARMAN Brand Ambassadors and partnerships help us connect

DEMI LOVATO

DAMIAN LILLARD

LINKIN PARK

QUINCY JONESDNCE (JOE JONAS) LANG LANG AR RAHMAN

STEPHEN CURRY

2.0 HOW WE DO IT

WE ARE INFUSED IN CULTURE

ALEX RODRIGUEX JEROME BOATENG RAPHAEL VARANE MARIANO RIVERA DAN PATRICK

OFFICIAL SOUND OF THE NBAAN OFFICIAL AUDIO PARTNER OF THE GRAMMY AWARDS®

SUNNERY JAMES & RYAN MARCIANO

IVAN DORN MAITRE GIMS TINIE TEMPAH


SAMSUNG AND HARMAN SYNERGIES

Scale Networks R+D Brands Talent

SUHD TV

HOME THEATER

SPEAKERS

SMART HOME/ IOTDISPLAY

VIDEO

AUDIO CONTROL

IN THE HOME

AT VENUES

Scale Channels R&D Brands Talent

HEADPHONES

MOBILE

W EARABLE

S E R V I C E D E L I V E R Y P L A T F O R M

HUD, UX/UI

DISPLAY

A I

5 GELECTRIFICATION

S E C U R I T Y D I G I T A L CLUSTER

S

A D A SB I G D A T AA N A L Y T I C S

CONNECTIVITY

S O U N D M A N A G E M E N

T

C L O U D

O T A U P D A T E S

IN THE CAR

ON THE GOFUTURE COCKPIT

L I G H T I N GSMART APPLIANCES

R E A R S E A T E N T E R T A I N M E N

T

3.0 WHERE WE ARE GOING

©2015 HARMAN INTERNATIONAL INDUSTRIES, INCORPORATED

INDUSTRY RECOGNITIONLEADERSHIP ACROSS KEY TECHNOLOGY & PLATFORM

Recognized by Microsoft for Accelerating Innovation across Connected Car, Connected Home and Connected Enterprise

Largest Android Device Commercialization PartnerScaling Partner for IOT and Automotive Initiatives Approved Google Android Device Certifier

Unique partnership to bring High Performance Computing and System Integration services to accelerate connected ecosystem development

Partnership to deliver analytics and prognostics for IoT ecosystem across automotive and enterprise

Ranked as leaders in high growth areas in Analytics, Mobile and Software Platform Development

KEY ECOSYSTEMSCALING PARTNER

8


AND COMPETING WITH HOLLYWOOD ON COGNITIVE


AI VS ML VS DL AI vs. Machine Learning vs. Deep Learning

AI and machine learning are often used interchangeably, especially in the realm of big data. But these aren’t the same thing, and it is important to understand how these can be applied differently.

Artificial intelligence is a broader concept than machine learning, which addresses the use of computers to mimic the cognitive functions of humans. When machines carry out tasks based on algorithms in an “intelligent” manner, that is AI. Machine learning is a subset of AI and focuses on the ability of machines to receive a set of data and learn for themselves, changing algorithms as they learn more about the information they are processing.

Training computers to think like humans is achieved partly through the use of neural networks. Neural networks are a series of algorithms modeled after the human brain. Just as the brain can recognize patterns and help us categorize and classify information, neural networks do the same for computers. The brain is constantly trying to make sense of the information it is processing, and to do this, it labels and assigns items to categories. When we encounter something new, we try to compare it to a known item to help us understand and make sense of it. Neural networks do the same for computers.

Extract meaning from complicated data

Detect trends and identify patterns too complex for humans to notice

Learn by example

Speed advantages

10


DL AND TESTING- DATA IS AT THE HEART OF MATTER

Deep learning goes yet another level deeper and can be considered a subset of machine learning.

The concept of deep learning is sometimes just referred to as "deep neural networks," referring to the many layers involved. A neural network may only have a single layer of data, while a deep neural network has two or more. The layers can be seen as a nested hierarchy of related concepts or decision trees.

Deep learning networks need to see large quantities of items in order to be trained. Instead of being programmed with the edges that define items, the systems learn from exposure to millions of data points. An early example of this is the Google Brain learning to recognize cats after being shown over ten million images. Deep learning networks do not need to be programmed with the criteria that define items; they are able to identify edges through being exposed to large amounts of data.

Whether you are using an algorithm, artificial intelligence, or machine learning, one thing is certain: if the data being used is flawed, then the insights and information extracted will be flawed.

Algorithms can be as flawed as the humans they replace — and the more data they use, the more opportunities arise for those flaws to emerge.” Decisions need to be based off clean and meaningful data.

11


AI AND TESTING

Reviewing specifications tell us what a program should do and how it should work. I.’s pattern matching helps us eliminate unneeded “too close’ test cases by seeing which ones are too similar. As we mentioned earlier, this may mean focusing on boundary value analysis (edge cases, literally), emphasizing state transition, or ensuring all-pairs testing.

Exploration testing session logs, via pattern recognition of verbose logging, seek activity patterns of specific warnings tracking to specific user actions, modules, forms, etc.

Known product issues, once analyzed, can have A.I. cluster similar bugs through pattern recognition, suggesting likely duplicates. Bugs from automated test cases can be auto-run on previous builds to find the causal build to help pinpoint root cause code changes.

Discussions with knowledgeable personnel (product owners, developers or Marketing, etc.) may determine code danger areas. White box-driven test design targets the actual revised code, hunting for specific code level problems. Factors may include the coder, change date, functions referenced, or specific non-standard notations. A.I. pattern-matching techniques help pinpoint applicable code based on your search parameters.

End user analysis applies to two different areas. The first is studying the frequency of specific user feedback words to help the most popular concerns bubble to the top of a list for further research. The second is end user usage analysis, where logfile statistics (based on A.I. pattern searching) show how much time each type of user spends in different program areas on different actions. Early focus on these heatmapped areas concentrate attention where the most user time is spent.

12


This is how cognition works.


The making of an expert.


But even experts face challengesin our current environment.


Watson scales expertise to expand what’s possible.


Watson is creating a new partnershipbetween people and computers

that enhances, scales and accelerateshuman expertise.


6D MACHINE LEARNING FRAMEWORK

D1Testing Approach

D4 Learning Techniques in Testing

D6 Elements Learned in Testing

D5Learning Property in Testing

Testing General ActivityD2

D3Testing Level


6D MACHINE LEARNING – DIMENSION/SUB DIMENSION

BACK


ML TESTING STRATEGY

Dimension Sub Dimension Value OutcomeTasks

D1- Testing Approach Refer subdimension in classification chart –slide 19

D2- Testing General Activity Refer subdimension in classification chart -slide -19

D3- Testing Level Refer subdimension in classification chart slide 19

D4- Learning Techniques in Testing • Data visualization tool for identifying defects• correlating them across applications and driving RCA• conducting ML based CAPA ( Corrective Action Preventive Action)• clustering test cases to groups based on NLP algorithms,• Defect predictions based on application complexity, degree of change in

applications,

D5- Learning Property in testing . • Prediction , prevention and automation of self learning algorithms• missing test coverage against requirement• identification of dead test cases, for changed or redundant requirement• un attended execution• identify hotspots and automatically execute test cases• application stability


DIMENSION D1: TESTING APPROACH

According to the older testing terminology, one can define the testing approach dimension with three possible values: black-box, white-box, and greybox .

Based on the black-box approach, testing can be performed using the external description of a software system such as the software specification.

In a white-box approach, the internal properties of a software system like source code can be used for testing purposes.

The grey-box approach is a combination of the two, which considers both internal and external properties at the same time.

BACK


DIMENSION D2: TESTING GENERAL ACTIVITY

In the second dimension, the standard testing process life-cycle inspired us to define the a) test planning, b) test case management, and c) debugging sub-dimensions.

With respect to the testing process life-cycle, It is found out that these three phases are critical and that automation can effectively assist them in order to reduce the cost and time of the whole testing process. The possible activities that can be automated based on ML in a, b, and c are presented in Table I in next slide.

In the test planning sub-dimension, testing cost estimation can help test managers to predict testing process cost and time and provide good testing plan to manage the testing process efficiently.

Test case management includes several tasks such as test case prioritization, which intends to prioritize the test input space in terms of test case effectiveness; test case design, which intends to generate high quality test cases; test case refinement, which intends to map the current specification of a software system to the existing test cases in order to reuse the available test cases; and test case evaluation which intends to measure the quality of the generated test cases.

In the Debugging sub-dimension, fault localization can help to find the exact location of the program that is defected.

In addition, bug prioritization intends to prioritize the revealed faults based on their severities; later test engineer can focus on more critical faults accordingly. Fault prediction can assist test engineers in the debugging stage, in the sense that potential faults for a given program are predicted.

BACK


TABLE-1-TESTING GENERAL ACTIVITY

TESTING GENERAL ACTIVITY DIMENSION

Sub-dimension Value Outcomes Tasks

A: Test Planning Testing cost estimation

B: Test Case Management

Test case prioritizationTest case designTest case refinementTest case evaluation

C: Debugging Fault localizationBug prioritization Fault prediction


DIMENSION D3-TESTING LEVEL

Software development process includes a range of stages form ”requirement analysis” to ”implementation” .

The development process goes further even after the implementation stage and concentrates on software maintenance. In Dimension 3, six types of testing levels have been identified to classify existing work: acceptance testing, system testing, integration testing, module testing, unit testing, and regression testing, which refer to requirement analysis, architectural design, subsystem design, detailed design, implementation, and maintenance in the software development stages, respectively.

BACK


DIMENSION D4-LEARNING TECHNIQUES IN TESTING

In the machine learning area, various types of learning methods have been introduced and each of them has its own specific characteristics.

Decision Trees (DT)

Artificial Neural Networks (ANN),

Genetic Algorithms (GA),

Bayesian Learning (BL),

Instance Base Learning (IBL),

Clustering,

Hybrid methods, which can be combination of several other learning methods.

BACK


DIMENSION D5-LEARNING PROPERTY IN TESTING

Each learning method has its own specific characteristics. This dimension is devoted to evaluating learning methods properties in various aspects.

To do so, three sub-dimensions; Training data properties, Supervision, Time generalization are defined.

The Training data properties evaluate the properties of a data set. The gathered data for learning process can be small or large in terms of its quantity. Data can be noisy or accurate in terms of errors that might exist in a data set.

In addition, learning can be supervised or unsupervised; therefore, the Supervision sub-dimension is defined. For a given target function the time generalization sub-dimension exposes how its generalization is; which can be either eager (at learning phase) or lazy (at classification phase).

In terms of online and offline learning, also time generalization shows how a learning system updates its approximation of the target function after the initial training data are used.

The last sub-dimension, automation degree, is responsible for addressing the learning system in terms of the degree of automation. The designed system can be fully or partially automatic.

BACK


DIMENSION - D6- ELEMENTS LEARNED IN TESTING

In each learning system for software testing automation, various types of data can be used to build the target function. The training data can be collected in different stages of the software testing process or software development life-cycle.

The learning elements could be

software metrics

software specification

CFG (control flow graph)

call graph,

test case,

execution data,

failure reports,

coverage data. BACK


TEN COMMANDMENTS FOR TESTERS

1. I am the tester. I am proud of it. I own the product, I can somersault it.

2. I am proud to be a tester. I can shift left, shift right, Shift up. Though I am shot by the customer , I am a proud postman.

3. I am the customer advocate , I am at the center of Universe.

4. I test because I love, I test not because I don’t know to code

5. I test because I love, I have arrived, I can work with machines and work with them in tandem, harmoniously

6. I am all along the software delivery chain, you can see me from concept to finish. Right from

7. I am the shiva, the destroyer, I destroy the product and help the brahma ( creator) in co creating it too.

8. I don’t have a religion , manual or automation, I am learning it all, Try me out, give me time, I can help and will help

9. I am a generalizing specialist

10. I am the king. I am the context imperial king. At the stroke of midnight, when the whole IT industry is seeing job losses, No one can touch me , as I have embraced the future , I have seen the future, ………….Testers Zindabad..

28


REFERENCES

ML Based software testing by Dr Noorian, Bagheri, Weichang Du,

29

dr. shankar ramamoorthy customer success...

Documents