industry-academia collaborations in software engineering research: experience from canada and turkey

1Dr. Vahid Garousi

Industry-academia collaborations in software engineering research:

Experience from Canada and Turkey

Dr. Vahid GarousiAssociate Professor of Software Engineering

Department of Computer EngineeringHacettepe University, Ankara, Turkey

[email protected]://web.cs.hacettepe.edu.tr/~vahid

vgarousi

2Dr. Vahid Garousi

Outline

Background Of the Principal Investigator (PI), and his research teams in the past

Success stories of industry-academia collaborations Success stories in Canada Success stories in Turkey Challenges, best practices, and collaboration models

3Dr. Vahid Garousi

About the Speaker Work experience:

Hacettepe University, Turkey, Feb. 2015- University of Calgary, Canada, 2006-2014 Middle East Technical University, Turkey, 2013 Founder, Maral Software Consulting Corporation, 2008-

Education: PhD, Carleton University, Canada, 2006 MSc, University of Waterloo, Canada, 2003 BSc, Sharif University of Technology, Iran, 2000

Research expertise: Software Engineering Software Testing and Quality Assurance (QA) Software Maintenance Agile methodologies “Action research”, i.e., industry-academia collaborations

Cultural background and citizenship: Dual citizenship: Canadian-Iranian, now living in Turkey

4Dr. Vahid Garousi

Research in University of Calgary, Canada2006-2014• Established and led the Software Quality Engineering

Research Group (SoftQual)• www.softqual.ucalgary.ca

• Alumni 1 Post-doctoral fellow 3 PhD students 9 MSc students 30+ undergraduate students

• Almost all our projects were applied R&D projects in collaboration with the industry.

• More than 10 industrial partners such as IBM• More than $1.3 Million CND in funding in 6 years

• Output: • More than 30 journal papers and 45 conference papers• Several software tools• One start-up firm and one commercialized software

product

5Dr. Vahid Garousi

6Dr. Vahid Garousi

Hacettepe University Software Engineering (HUSE)

2 faculty members 20+ graduate students 10+ industry partners

7Dr. Vahid Garousi

Researcher's profile: International collaborations

8Dr. Vahid Garousi

Outline

Background Of the speaker, his research team and expertise


9Dr. Vahid Garousi

Collaboration model Our research approach: “Action Research”

T. Gorschek, C. Wohlin, P. Carre, and S. Larsson, "A Model for Technology Transfer in Practice," IEEE Software, vol. 23, pp. 88-95, 2006

10Dr. Vahid Garousi

Success stories in Canada Project 1

Industry partner: Pason Systems Corporation, Calgary, Canada Challenge: The company could not systematically decide “when to

automate testing” and “what (test cases) to automate”

Project 2 Industry partner: NovAtel Inc., Calgary, Canada Challenge: Huge amounts of effort had been spent for developing and

maintaining software documentation

Project 3 Industry partner: MR Control Systems International,

Calgary, Canada Challenge: The cost of manual testing was very high. Need for test

automation

Project 4 Industry partner: Alberta Energy Resources Conservation Board

(ERCB), Calgary, Canada Challenge: The cost of manual “environment configuration” testing

was very high. Need for test automation

11Dr. Vahid Garousi

Selected Project #1 Industry partner: Pason Systems Corporation, Calgary, Canada.

Developing software and hardware for oil rig reservoirs Challenge: The company could not decide “when to automate testing”

and “what to automate” Systems under test (SUT’s):

12Dr. Vahid Garousi

Selected Project #1 Solution: Decision support for balancing manual software

testing versus test automation

Test-case Design

Test Scripting

Test Execution

Test Evaluation

Pass

Fail

Test Suites (set of test

cases)

Test Results

Bug (Defect) Reports

Scripted test suites

“Exercise” (test)

Manual test suites

Automated test suites (e.g., Junit)

AM

Criteria-based (Systematic)

Human knowledge-based

(Exploratory)

Computer(Automated)

Human(Manual)

M

OR/AND

AM

System Under Test (SUT)

AM

AM

Activity

Data/ Entity

Legend

Test-Result

Reporting

13Dr. Vahid Garousi



UC: use-case

Goal: maximize

Test-case Design

Test Scripting

Test Execution

Test Evaluation

Pass

Fail

Test Suites (set of test

cases)

Test Results

Bug (Defect) Reports

Scripted test suites

“Exercise” (test)

Manual test suites

Automated test suites (e.g., Junit)

AM

Criteria-based (Systematic)

Human knowledge-based

(Exploratory)

Computer(Automated)

Human(Manual)

M

OR/AND

AM

System Under Test (SUT)

AM

AM

Activity

Data/ Entity

Legend

Test-Result

Reporting

14Dr. Vahid Garousi



15Dr. Vahid Garousi


testing versus test automation More details in the following papers:

Invited paper

16Dr. Vahid Garousi

Outline



17Dr. Vahid Garousi

Success stories in Turkey Multiple projects (since 2015)

Industry partner: HAVELSAN (Hava Elektronik Sanayi, Aviation Electronics Industries), 2015-now

1. Test automation for several test groups2. Assessing and improving an in-house test automation framework for

test group Q3. Establishing a systematic, effective and efficient GQM-based

measurement program for the testing directorate4. Assessment and improvement of test process maturity using TMMI

and TPI-Next 5. Bi-directional knowledge transfer from/to international venues and

organizations in the aviation industry, e.g., NASA, DoD

2015 Industry partner: Turkish Aerospace Industries, Inc. (TAI) Project topic: A tool for automated inspection of software design

documents

2013 Industry partner: Department for Turks Abroad and Relative Societies

(Yurtdışı Türkler ve Akraba Topluluklar Başkanlığı) Challenge: The performance (response time) of the web application

became poor and non-responsive when many users used the system in high user loads, response time~50 seconds!

18Dr. Vahid Garousi

Company background An example product: Helicopter simulators

19Dr. Vahid Garousi

Collaborations with HAVELSAN Multiple projects

Industry partner: HAVELSAN (HAVa ELektronik SANayi, Aviation Electronics Industries), 2015-now

1. Test automation for test group X (an overview next…)2. Test automation for test group Y3. Test automation for test group Z4. Assessing and improving an in-house test automation framework for test group Q5. Establishing a systematic, effective and efficient GQM-based measurement program

for the testing directorate6. Assessment and improvement of test process maturity using TMMI and TPI-Next 7. Bi-directional knowledge transfer from/to international venues and organizations in

the aviation industry, e.g., NASA, DoD

We even wrote a methodological paper on how we came up with the above list of projects…

20Dr. Vahid Garousi

Collaborations with HAVELSAN Context: Visual GUI testing of tactical map systems

(not the real SUT, but a good representative taken from internet)

21Dr. Vahid Garousi

Collaborations with HAVELSAN Context: Visual GUI testing of tactical map systems Two test tools were chosen:

Sikuli jAutomate

Goal: To compare Sikuli and JAutomate in order to determine a suitable tool for GUI

testing in the project GQM was used:

How do features of the two tools compare to each other (static comparison)? How do the two tools compare when they are used to develop test scripts and

test SUTs (dynamic comparison)?How do the two tools compare in terms of quality of the Record and Replay features?How do the two tools compare in terms of robustness and repeatability?How do the two tools compare in terms of fault tolerance?How do the two tools compare in terms of test development effort? (some overview next…)How do the two tools compare in terms of test maintenance effort?How do the two tools compare in terms of fault detection effectiveness?

22Dr. Vahid Garousi

Collaborations with HAVELSAN Several papers have been written so far…

23Dr. Vahid Garousi

Outline



24Dr. Vahid Garousi

Challenges and best practices Synthesized the findings of 33 primary studies

And 30 other studies…

25Dr. Vahid Garousi

Challenges and best practices

Challenges Best practices (success patterns)1. Lack of research relevance2. Research method related3. Lack of training, experience, and

skills4. Lack or drop of interest /

commitment5. Mismatch between industry and

academia6. Communication-related issues7. Human and organizational factors8. Management-related issues9. Resource-related issues10. Contractual, and privacy concerns

1. Knowledge management (communication, terminology, transfer, training and skills)

2. Ensure engagement and manage commitment3. Consider and understand industry's needs,

challenges, goals and problems4. Ensure giving explicit industry benefits and solve the

right problem5. Have mutual respect, understanding and

appreciation6. Be Agile7. Work in (as) a team and involving the "right"

practitioners8. Consider and manage risks and limitations9. Researcher's on-site presence and access10. Follow a proper research/data collection method11. Manage funding/recruiting/partnerships and

contracting privacy12. Understand the context, constraints and language13. Efficient research project management14. Conduct measurement/ assessment15. Test pilot solutions before using them in industry16. Provide tool support for solutions

26Dr. Vahid Garousi

Collaboration models Our research approach: “Action Research”

T. Gorschek, C. Wohlin, P. Carre, and S. Larsson, "A Model for Technology Transfer in Practice," IEEE Software, vol. 23, pp. 88-95, 2006

27Dr. Vahid Garousi

End of the talk



Questions and answers

28Dr. Vahid Garousi

End of the talk

29Dr. Vahid Garousi


Industry partner: Pason Systems Corporation, Calgary, Canada Challenge: The company could not decide “when to automate testing”

and “what to automate”





automation




30Dr. Vahid Garousi

Success stories in Canada Industry partner:

NovAtel Inc., Calgary, Canada, a global leader in GPS and GNSS software and systems (embedded software with C/C++ code inside)

31Dr. Vahid Garousi


NovAtel Inc., Calgary, Canada The company had CMMI Level-3 certification

Challenge: Huge amounts of effort (cost $$$) were spent for developing and

maintaining software documentation in the last several yearsReminder: Software maintenance typically consumes 40 to 80% of

software project costs.

Solution: A systematic approach for optimizing cost of software documentation was

developed and deployed in the industrial context

Impact : Based on quantitative measurements, the approach optimized cost of

developing and maintaining software documentation.

32Dr. Vahid Garousi

How did we systematically optimize cost of software documentation?

SW Architect (Requirements Engineer,

Business analyst…)

Reviews

Change the doc and create the Next Version

Draft

Comprehension Activity

<<may lead to a change in the doc>>

Developer(down the stream)

Approver (Team lead, etc.)

For requirements and design documents

Revision

Creating 1st

Draft First Draft

Activity

Artifact/Document

Working Draft

ApprovedVersion Use doc to develop

artifact/product at hand(e.g., design, code, etc.)

Maintainer(debugger, developer)

Artifact at hand

Bug reports

Feature Requests

SW maintenance (super) activity

SW maintenance (super) activity

Activity 1...

Activity n...

E.g., do the actual changes to fix the bug at hand

Next Version Draft

LegendLegend

E.g., fault localization, root-cause analysis, impact analysis

Doc-related Activity incurring costs

Activity using documentation

(benefits)

0a

0b, 0c, ...

1a2a

33Dr. Vahid Garousi

• Initial cost• Number of minutes• Number of words in the file

• Maintenance cost • The above• + Amount of change from

each version to the next• …

• Benefit (usage) during development cycle• Benefit (usage) after the development:

during maintenance• Which sections of a certain

documentation are used?• How much help a certain

documentation artifact has made for a task at hand?

Benefit MetricsCost Metrics

Requirements Design Implementation Test CasesBug/maintenance

Reports

Artifacts... ... ... ... ...

34Dr. Vahid Garousi

How can we systematically optimize cost of software documentation?

35Dr. Vahid Garousi

Economics (how we applied it in our context)

35

CostCost)(BenefitROI

Min=-1 when benefit=0

No extra benefit…when benefit=cost

-1 0

Worth having a given artifact (investing in it)

Does not worth having it

Benefit>costBenefit<cost

0

5

10

15

20

25

30

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29N

umbe

r of h

ours

Month #

Hypothetical example for Cost Benefit of a Requirements Document

Cost

Benefit

In above example, ROI=-0.60

36Dr. Vahid Garousi

How can we systematically optimize cost of software documentation?

Recommendations were given to the team to:

First Draft

Reviews/Revisions

Version Approved

Next Version Draft

Usage during development

Usage during maintenance

Purpose: for development of down-the-stream artifacts

Purpose: for comprehension purposes for Bug fixing or Feature adding

COST

BENEFIT

Artifact retired/archived

E.g., the team may decide it is no longer worth it or needed to keep this artifact updated

minimize

maximize

37Dr. Vahid Garousi


NovAtel Inc., Calgary, Canada Solution:

A systematic approach for optimizing cost of software documentation was developed and released to the industrial context.

More details in the following articles:

38Dr. Vahid Garousi

Collaboration models There are similar models, such as the: “design

science” Also used in other disciplines

39Dr. Vahid Garousi

Collaboration models Also, called “design science”

https://biologytodesign.wordpress.com/2011/09/12/how-to-read-scientific-papers/

40Dr. Vahid Garousi

Collaborations with HAVELSAN Goal: To compare Sikuli and JAutomate in order to determine a

suitable tool for GUI testing in the project Two SUTs:

Script Development Effort (SDE) versus # of test steps

41Dr. Vahid Garousi

Why should industry and academia collaborate? If done

Collaboration between industry and academia supports improvement and innovation in industry …

and helps to ensure industrial relevance in academic research. If not done: The lack of collaboration between industry and academia hurts

both sides: Practitioners will not learn what researchers have already discovered that might be useful

to them While researchers will have fewer insights on the problems that are important to

practitioners Unfortunately, many software engineering practitioners and researchers are

not talking to each other (adequately). Various reasons have been discussed by researchers and practitioners for

such a lack of motivation for collaborations between industry and academia, such as:

Each side having different objectives, industrial problems lacking scientific novelty or challenges, and the low applicability and scalability problems of the solutions developed in academia

42Dr. Vahid Garousi

A typical simplified process for industry-academia collaborations

Inception (formation) phase: approaching and topic selection

Initiator

Initiator

Joint discussions

Operational phase: controlling and monitoring projects

Transition phase: technology/knowledge transfer and impact

Joint efforts

Joint efforts

Another upcoming collaborative proj ect?

Project/case under study

studystudy

Project/case under studyResearch

literature

review

Research literature

contribute positively impact

Planning

Joint efforts

Project plan (objectives, etc.)

define

Motivationsinfluence influenceinfluence influence

Success criteria

Measures of success and satisfaction

use d to measure

define

use d to measure

Challenges

Success factors (patterns)

Anti-patterns

Outcomes

Cross-cutting

influence

influence influence

influence

43Dr. Vahid Garousi



1. Test automation for test group X2. Test automation for test group Y3. Test automation for test group Z4. Assessing and improving an in-house test automation framework for test group Q5. Establishing a systematic, effective and efficient GQM-based measurement

program for the testing directorate6. Assessment and improvement of test process maturity using TMMI and TPI-Next 7. Bi-directional knowledge transfer from/to international venues and organizations

in the aviation industry, e.g., NASA, DoD

2015 Industry partner: Turkish Aerospace Industries, Inc. (TAI) Project topic: A tool for automated inspection of software design documents

2013 Industry partner: Department for Turks Abroad and Relative Societies (Yurtdışı

Türkler ve Akraba Topluluklar Başkanlığı) Challenge: The performance (response time) of the web application became poor

and non-responsive when many users used the system in high user loads, response time~50 seconds!

44Dr. Vahid Garousi

Joint work with Turkish Aerospace Industries, Inc. (TAI) Industry partner:

Turkish Aerospace Industries, Inc. (TAI) Challenge:

Manual inspection of software design documents is time consuming (expensive) and error prone

Manual checklist…

45Dr. Vahid Garousi

Joint work with Turkish Aerospace Industries, Inc. (TAI) Industry partner:

Turkish Aerospace Industries, Inc. (TAI)

Challenge: Manual inspection of software design documents is time consuming (expensive)

and error prone

Need: A tool for automated inspection of software design documents

Solution: A tool for automated inspection of software design documents

Impact : Based on quantitative and qualitative measurements, the tool reduced the

inspection cost of design documents and increased inspection effectiveness (ability to detect defects)

46Dr. Vahid Garousi

Joint work with Turkish Aerospace Industries, Inc. (TAI) A tool for automated inspection of software design documents

AutoInspect

input toSoftware Design Document

Inspection Engineer

Design inspection rules

inspects interacts with

Defects found during inspection

Design verification report

uses

analyzes

47Dr. Vahid Garousi

Joint work with Turkish Aerospace Industries, Inc. (TAI)

A tool for automated inspection of software design documents

48Dr. Vahid Garousi


Measuring the Efficiency and Effectiveness of the approach

Design document

name

Size metrics Efficiency (inspection effort in hours) Effectiveness (# of defects found)

Number of pages

Number of GUI

screensManual Automate

dImprovement % Manual Automate

dImprovement %

EYTS 52 12 8 4 50% 6 8 33%EFAB 109 31 29 17 41% 100 129 29%TAS 321 34 51 27 47% 13 16 23%

49Dr. Vahid Garousi


Paper…

50Dr. Vahid Garousi



1. Test automation for test group X2. Test automation for test group Y3. Test automation for test group Z4. Assessing and improving an in-house test automation framework for test group Q5. Establishing a systematic, effective and efficient GQM-based measurement

program for the testing directorate6. Assessment and improvement of test process maturity using TMMI and TPI-Next 7. Bi-directional knowledge transfer from/to international venues and organizations

in the aviation industry, e.g., NASA, DoD

2015 Industry partner: Turkish Aerospace Industries, Inc. (TAI) Project topic: A tool for automated inspection of software design documents

2013 Industry partner: Department for Turks Abroad and Relative Societies (Yurtdışı

Türkler ve Akraba Topluluklar Başkanlığı) Project 1: Performance engineering of a major web application used by

thousands of users world-wide Project 2: Usability assessment (testing) of the web app

51Dr. Vahid Garousi

Another project

Industry partner: Department for Turks Abroad and Relative Societies (Yurtdışı Türkler ve

Akraba Topluluklar Başkanlığı)

System under test (SUT): A major .Net-based web application was developed Serving over 50,000 Industry partners from all over the world each year, who would

submit applications and documents Staff members review and score the applications using this system

52Dr. Vahid Garousi

Another project

Challenge: Project 1-The performance (response time) of the web application became

poor and non-responsive when many users used the system in high user loads, response time~50 seconds!

Project 2-Usability of the web app needed systematic assessment (testing)

Led to these projects: Project 1-Systematic performance engineering of the web app:

measurements, testing and improvement Project 2-Usability assessment (testing) of the web app based on these

standards: ISO 9241-151 - Ergonomics of human-system interaction — Guidance on WWW

user interfaces ISO 9241-20 ICT Accessibility guidelines Kamu Kurumları İnternet Sitesi (KAKIS ) Önerileri Rehberi

53Dr. Vahid Garousi

Software Performance Engineering

Challenge: Need for systematic software performance testing and detection of performance

problems (e.g., bottlenecks)

Solution:

54Dr. Vahid Garousi

Challenge: Need for systematic software performance testing and detection of performance

problems (e.g., bottlenecks)

Solution: systematically used various performance testing and engineering tools in the Visual Studio environment

Concurrency and Contentions

Creating additional database Indices


55Dr. Vahid Garousi

Solution: performance testing and engineering Measured and reduce wait times in the DBMS


56Dr. Vahid Garousi

Challenge: Need for systematic software performance testing and

detection of performance problems (e.g., bottlenecks)

Solution: We systematically applied various performance testing

and engineering tools

Outcome and impact: Thanks to systematic software performance testing … The performance (response time) of the web application

improved significantly when many users use the system


57Dr. Vahid Garousi

Approach: Objective usability assessment and testing of the web app based on these standards: ISO 9241-151 - Ergonomics of human-system interaction — Guidance on

WWW user interfaces ISO 9241-20 ICT Accessibility guidelines Kamu Kurumları İnternet Sitesi (KAKIS ) Önerileri Rehberi

Output: A comprehensive report

Impact: Major improvement of the web app usability as reported by the stakeholders

Usability assessment (testing) of the web app

58Dr. Vahid Garousi

What to do to ensure success

…

59Dr. Vahid Garousi

Challenges, best practices, and collaboration models

Our research approach: “Action Research” Example…

Need (challenge) Solution ImpactFurther reading (resulting papers)

Lack of a systematic approach to decide what test cases to automate in software testing

A systematic approach, based on optimization and on system dynamics, was developed and released to the industrial context.

Based on quantitative measurements, the approach improved the cost effectiveness of software testing activities.

[Amannejad, Garousi, et al. 2014][Sahaf, Garousi, et al. 2014]

60Dr. Vahid Garousi

BACK UP

61Dr. Vahid Garousi

Success stories of industry-academia collaborations in Software Engineering:

Experience from Canada and Turkey

Yazılım mühendisliğinde üniversite-sanayi işbirliği başarı hikayeleri: Kanada ve Türkiye'den tecrübe ve kanıtlar

Doç. Dr. Vahid GarousiAssociate Professor of Software Engineering

Hacettepe University Software Engineering Research Group (HUSE)Department of Computer EngineeringHacettepe University, Ankara, Turkey

Senior Software Consultant and Managing PartnerMaral Software Engineering Consulting Corporation, Calgary, Canada

[email protected]://web.cs.hacettepe.edu.tr/~vahid

vgarousi

62Dr. Vahid Garousi

Researcher's publication record More than 38 journal papers in ‘high-profile’ journals More than 45 conference papers in top IEEE/ACM conferences

63Dr. Vahid Garousi

Five levels of different closeness between academia and industry

Level 1: Not in Touch

Source:

64Dr. Vahid Garousi


Level 2: Hearsay

65Dr. Vahid Garousi


Level 3: Sales Pitch

66Dr. Vahid Garousi


Level 4: Offline

67Dr. Vahid Garousi


Level 5: One Team (true collaboration)

1

2

3

68Dr. Vahid Garousi

About the Speaker Education:

PhD (Carleton University, Canada), 2006 MSc (University of Waterloo, Canada), 2003 BSc (Sharif University of Technology, Tehran, Iran), 2000 High school, Sadi High school, Tabriz, Iran, 1996

Work experience: Hacettepe University, Ankara, Türkiye, since Feb. 2015 Atılım University, Ankara, Türkiye, 2014 Middle East Technical University, Ankara, Türkiye, 2013 University of Calgary, Canada, 2006-2013

Research expertise: Software Engineering Software Testing and Quality Assurance (QA) Software Requirements and Software Maintenance “Action research”, i.e., industry-academia collaborations

Cultural and ethnic background: Canadian citizen, with Azerbaijani background (from Iran), now

living in Turkey

69Dr. Vahid Garousi

Challenges, best practices, and collaboration models

Exploratory case studies are particularly useful To systematically understand “what is going on” in a

given industrial context and to identify challenges and issues to be addressed

Improving case studies are also useful Systematic means to improve a certain aspect of the

studied phenomenon Essentially, we need to conduct precise “before” and

“after” analyses Example:

70Dr. Vahid Garousi

Challenges, best practices, and collaboration models Source: "A Model for Technology Transfer in Practice", IEEE Software, 2006

71Dr. Vahid Garousi

Brainstorming

Discussing the software engineering opportunities and mutual interest areas with the Industry partner…

What type of software engineering challenges do you have? Software requirements? Need for more software test automation? Having hard time deciding “when to automate

testing” and “what to automate”? …

72Dr. Vahid Garousi

,

73Dr. Vahid Garousi

Survey of software testing practices in Turkey

74Dr. Vahid Garousi

Survey of software testing practices in Turkey

Test Türleri/Seviyeleri (soru 1)

75Dr. Vahid Garousi








automation




76Dr. Vahid Garousi

Selected Project #2 Industry partner: MR Control Systems International, Calgary, Canada.

Developing software for energy and process control (SCADA) SCADA: Supervisory Control and Data Acquisition Challenge: The cost of manual testing was very high. Need for test

automation System under test (SUT): named Rocket Had been tested manually during development, for 5 years

77Dr. Vahid Garousi

Selected Project #2 Solution: Development and evaluation of automated unit and

functional testing infrastructure We proceeded with automated black-box Unit Testing (BBUT)

But it has challenges:• If we apply the equivalence classing, we will get 19,683 test cases for

only this function block. Bad news ;( • Challenge 1: Coding of test cases (in NUnit): Too much effort• Challenge 2: Coupling of test cases to test input data • Challenge 3: Manual generation of expected outputs (test oracle)

78Dr. Vahid Garousi


functional testing infrastructure• One possible solution → Automated generation of NUnit test

code• We implemented a test tool

79Dr. Vahid Garousi

Selected Project #2 Reminder: Test code is automatically generated, saving

many hours

80Dr. Vahid Garousi

Return On Investment (ROI) of Test Automation•Cost and benefit drivers in the project:

Cost Drivers Benefit DriversCD 1: Development of the AutoBBUT tool

BD 1: Time saving and test repeatability – No need any more to conduct manual regression testing

CD 2: Maintenance of automated unit test suite

BD 2: Time saving - No need to manually develop the test code and some of the test oracles (expected outputs)

CD 3: Manual coding of some of the expected outputs (test oracles) in test code

BD 3: Detection of faults undetected in manual testing iterations

81Dr. Vahid Garousi

In-depth Analysis of the ROI of Test Automation•Careful measurements were conducted•Summary: We could get a time saving of:

ROI = -120 hours (AutoBBUT’s development time)-3 hours (test code inspection and completion)+87 hours (initial development of manual test suite, if it was to be done)+87*6 hours (test code maintenance, since the system evolved 6 times)ROI = 486 hours = 60 man-days

•More details in: our article in the IEEE International Conference on Software Testing, Verification and Validation (ICST), Industry Track, April 2012

82Dr. Vahid Garousi








automation




83Dr. Vahid Garousi

Selected Project #3 Industry partner: Alberta Energy Resources Conservation Board (ERCB), Calgary,

Canada. Developing enterprise software for public and internal use for monitoring energy resources

Challenge: The cost of manual “environment configuration” testing was very high. Need for test automation

System under test (SUT): The organization’s projects are developed on a

backbone of four staging environments: development, test, acceptance, and production

Issue: Staging Environment Instability All in-house development teams share the

staging servers. Software applications have to be deployed on

several staging environments during different milestones of the release lifecycle.

Changes to server configurations made by one team

have the potential to impact other teams. This situation happened very frequently in the

organization under study.

84Dr. Vahid Garousi

Selected Project #3 Challenge:

Issue: Staging Environment Instability The cost of manual “environment

configuration” testing was very high. Need for test automation

As a result: Teams had to constantly and manually

verify server configurations to ensure that all different environments were set-up correctly for their applications.

The entire situation was almost ad-hoc, caused last-minute surprises, and chaos in many occasions.

“It worked for me yesterday, but I don’t know why it doesn't work today!”

85Dr. Vahid Garousi

Selected Project #3 Challenge:

Issue: Staging Environment Instability The cost of manual “environment configuration” testing was very high. Need for test automation

Solution: Automated environment configuration testing

Automated test cases (using NUnit) were developed in ten groups (configurations they were trying to validate):

(1) folder permissions, (2) database stored procedures permissions, (3) services availability, (4) COM+ installation and access, (5) universal data links, (6) web configuration files, (7) IIS settings, (8) network and machine user membership, (9) global assembly cache, (10) database security.

86Dr. Vahid Garousi

Selected Project #3 Solution: Automated environment configuration testing An example of the XML test case inputs and expected output

87Dr. Vahid Garousi

Selected Project #3 Solution: automated build verification testing (BVT) Impact of the solution:

88Dr. Vahid Garousi

Selected Project #3 Solution: automated build verification testing

Details in:

89Dr. Vahid Garousi

Success stories in Turkey

Project 1 Industry partner: Yurtdışı Türkler ve Akraba Topluluklar Başkanlığı

(Department for Turks Abroad and Relative Societies) Challenge: The performance (response time) of the web application

became poor and non-responsive when many users used the system in high user loads, response time~50 seconds!

Multiple projects Industry partner: HAVELSAN (Hava Elektronik Sanayi, Aviation

Electornics Industries) Test automation for test group X Test automation for test group Y Test automation for test group Z Assessing and improving an in-house test automation framework for

test group Q Establishing a systematic, effective and efficient GQM-based

measurement program for the testing directorate Assessment and improvement of test process maturity using TMMI

[30] and TPI-Next [31] Bi-directional knowledge transfer from/to international venues and

organizations in the aviation industry

90Dr. Vahid Garousi

Research in Atilim UniversityJan. 2014-Feb. 2015

• Established the System and Software Quality Engineering Research Group (SySoQual) in Atilim Uni.

• se.atilim.edu.tr/sysoqual• Students:

4 MSc students 10+ undergraduate students

• Provided several consulting R&D projects• Was involved in one ITEA2• Some of industry partners:

91Dr. Vahid Garousi

Researcher's publication record Systematic Literature Review (SLR) and Systematic Mapping (SM)

papers, like “survey” papers

92Dr. Vahid Garousi

Researcher's publication record What is in a Systematic Literature Review (SLR) and Systematic

Mapping (SM) paper? Let’s review this one…

93Dr. Vahid Garousi

94Dr. Vahid Garousi

The classification scheme developed and used in our study

95Dr. Vahid Garousi

The classification scheme developed and used in our study

96Dr. Vahid Garousi

Researcher's publication record Systematic Literature Review

(SLR) and Systematic Mapping (SM) papers, like “survey” papers

97Dr. Vahid Garousi

Researcher's leadership record Involved as an organizing or program committee member in many top

international conference, such as ICST, ICSP, CSEE&T and the Turkish National Software Engineering Conference.

98Dr. Vahid Garousi

A reminder of Software Engineering Software engineering is the study and an application of

engineering to the design, development, testing and maintenance of software.

Software Engineering is not just programming (development)!

Many people ask me “what language do you program in?” !

Project 1 Industry partner: Pason Systems Corporation,

Calgary, Canada Challenge: The company could not systematically

decide “when to automate testing” and “what test cases to automate”

Project 2 Industry partner: NovAtel Inc., Calgary, Canada Challenge: Huge amounts of effort had been spent

for developing and maintaining software documentation

99Dr. Vahid Garousi

Selected Project #1 A decision support tool was developed for balancing manual software testing

versus test automation

100Dr. Vahid Garousi

Company background









automation





Selected Project #2 Industry partner: MR Control Systems International, Calgary, Canada.

Developing software for energy and process control (SCADA) SCADA: Supervisory Control and Data Acquisition Challenge: The cost of manual testing was very high. Need for test

automation System under test (SUT): named Rocket Had been tested manually during development, for 5 years



functional testing infrastructure We proceeded with automated black-box Unit Testing (BBUT)

But it has challenges:• If we apply the equivalence classing, we will get 19,683 test cases for

only this function block. Bad news ;( • Challenge 1: Coding of test cases (in NUnit): Too much effort• Challenge 2: Coupling of test cases to test input data • Challenge 3: Manual generation of expected outputs (test oracle)



functional testing infrastructure• One possible solution → Automated generation of NUnit test

code• We implemented a test tool


Selected Project #2 Reminder: Test code is automatically generated, saving

many hours


Return On Investment (ROI) of Test Automation•Cost and benefit drivers in the project:

Cost Drivers Benefit DriversCD 1: Development of the AutoBBUT tool

BD 1: Time saving and test repeatability – No need any more to conduct manual regression testing

CD 2: Maintenance of automated unit test suite

BD 2: Time saving - No need to manually develop the test code and some of the test oracles (expected outputs)

CD 3: Manual coding of some of the expected outputs (test oracles) in test code

BD 3: Detection of faults undetected in manual testing iterations


In-depth Analysis of the ROI of Test Automation•Careful measurements were conducted•Summary: We could get a time saving of:

ROI = -120 hours (AutoBBUT’s development time)-3 hours (test code inspection and completion)+87 hours (initial development of manual test suite, if it was to be done)+87*6 hours (test code maintenance, since the system evolved 6 times)ROI = 486 hours = 60 man-days


Resulting papers


What we do (offer): Software Engineering Software engineering is the study and an application of

engineering to the specification, design, development, testing and maintenance of software systems.

Software Engineering is not just programming (development)


Data collection techniques

Analysis of documentation usage logs using an automated tool…

1-Apr-12

1-Jan-12

1-Oct-11

1-Jul-11

1-Apr-11

1-Jan-11

1-Oct-10

1-Jul-10

1-Apr-10

1-Jan-10

1-Oct-09

1-Jul-09

1-Apr-09

1-Jan-09

1-Oct-08

1-Jul-08

1-Apr-08

1-Jan-08

1-Oct-07

1-Jul-07

D15673D13850D13680D13613D13612D13212D12974D12810D12694D12224D12218D11562D11551D11471D11452D11376D11308D11162D11064D09633

Access Time

Doc

No.

Distribution of Design Documents Accesses over Timeline (including all revisions)

File server

Automated tool


Hacettepe University Software Engineering (HUSE) Research areas:

Software Engineering, in general Software testing Software requirements, software

maintenance and software documentation

Empirical studies and experimentation in software engineering

Search-based Software Engineering (SBSE)

Software engineering education research: testing education in particular

Software process improvement Software metrics Process maturity, attributes and

performance Healthcare IT and process analytics