are you covered? new tooling for quality assurance · rdi expands into new territory –quality...

© 2018 IBM Corporation© 2018 IBM Corporation

Are you covered?

New tooling for Quality Assurance

Edmund Reinhardt – IBM i

Application Development Tooling

[email protected]

© 2018 IBM Corporation

2

Agenda

• What is code coverage

• Code Coverage while developing

• Demo with RPGUNIT and Code Coverage and Debug

• Code Coverage in DevOps pipeline (integrated with build)

• Integration with Test Automation products


The Vision

• IBM i developers are able to confidently change their code

– Because their code is tested 100%

– They can achieve this through Test Driven Development

– They can prove this coverage level (this session)

• Detect untested code

• Detect unreachable code

– They can refactor the code confidently to make it readable, well-

structured and taking advantage of the latest technology.

– As a result the platform is not known for old, hard to modify code,

• Rather as the most reliable, cost-effective platform for business


Code Coverage while developing

• Expected to do more with less

• Cannot afford long cycle times to test

• Cannot afford to let a bug go through because of a missed path

• Need to work smarter

• Solution

– Use technology to help you

• Develop productively using an IDE with rich language understanding

• Test thoroughly – automatically detect untested branches in code


5

RDi expands into new territory – Quality Assurance

• RDi Line level Code Coverage Analysis Capability

• You can see exactly which lines were covered and not.

• These results can be accumulated see the coverage of the total suite of tests as well as speeding up the analysis of

subsequent test runs.

• Can determine effectiveness of tests, and guide improvements

• Can help locate dead code


6

Launching Code Coverage

• Code coverage can be launched on any program or service program that can be debugged – independent of language

• Batch program analysis supported as of RDi 9.1

• Interactive and SEP supported added in RDi 9.1.1 (December 2014)


7

Code Coverage Report

• After running code coverage, a report is shown as an editor.

• You can drill down through programs, modules and

procedures and see the coverage statistics for each


8

Coverage annotated in the editor

• Drilling down from the report,

• editor will be opened on the related member

• green and red annotations showing which lines covered.


9

PDF Report

• The results can shown via PDF reports so they are

available for those who are not in the Eclipse client, i.e.

quality assurance, management etc.

• Note: this functionality was missing in 9.6 until 9.6.0.5


How to generate PDF reports


11

Launch Options

Allow code coverage analysis of programs that touch data in

production libraries

Use *LIST if executable code in /COPY, availability depends on

DBGVIEW compile option.


12

Customize how the program is invoked


13

Customize the Code Coverage Analysis

Can track the coverage at the higher level of

procedures or programs

Option to generate PDF reports from the initial launch.

Can always export it later.

Associated metadataCould be used track which

testcase covered which lines.


14

Code Coverage History view operations on reports

Coverage History can be exported

and imported into other

workspaces

Can generate workspace or

comparison reports and

export them in HTML or PDF

14


15

Compare 2 runs to see improvement in coverage

• Select 2 runs in the Code Coverage Results view and Compare –

see % improvement for different procedures


16

Service Entry Points

• A special kind of entry breakpoint that can be set directly from Remote

System Explorer

• Debugger is automatically started when first line of program is executed

• Supports batch, interactive, and Web service programs


17

Use Service Entry Breakpoints with Code Coverage

• In SEP view, toggle to Code Coverage mode from Debug mode

• Set the SEP breakpoint just as you would for debugging

• Invoke the application in whatever manner you normally would

Switch to Code

Coverage mode

Optionally, customize the

launch configuration


18

RDi 9.1.1 COBOL Code Coverage Enhancement

• The coverage analysis is now reported by paragraph as

well as by line


New Code Coverage Features in RDi 9.5

• Much, much, much faster! Typically 20x or better improvement

– The architecture was completely redone to streamline code coverage to a

minimum of interaction between the RDi client and the IBM i server

– Solution is ready for real world scale of programs and tests

• Ability to use shared results, so coverage results of test cases can be computed

once and reused for the team

• Ability to merge code coverage results from multiple runs

– Compute the combined coverage and show exactly which lines are missing

– Exactly which modules/files/procedures do not meet specified thresholds

19


20

The results of multiple test runs can be merged

• The bucket of manual or automatic tests are all run

• The coverage results are all analyzed and merged

into a single baseline


21

Holes in the Testbucket can now be analyzed

• Procedures and files below a set threshold

are highlighted in red and need more testingClick on Column Headings To Sort


22

Results can be given meaningful names


One advantage of the new (9.6.0.5) Web View of coverage


Code Coverage now supports SonarQube integration!


25

Results can be exported to a shared location

Export to:

• Your PC

• IFS

• Shared drive


26

Shared result can be viewed & analyzed from any RDi client


Some may be color-blind and so the annotations can be customized

to show different symbols as well!

27

Customize coverage annotations


28

Code Coverage – Filtered File Report

• On any Code Coverage Result, the Open Filtered… action is ava

• Specify a filter string and the report will show code coverage results for lines that match

the filter string


Integration with RPGUNIT• Follow https://irpgunit.sourceforge.io/help/

https://irpgunit.sourceforge.io/help/


Check Code Coverage of Automated Test

Set a service breakpoint on our service program modules

Set "Code

Coverage"

mode


Oh oh … our tests don't test all our code

Only 86% of our code is tested!

Drill down into the code. We haven't tested the "found = *off" line


Introduction to IBM i Code Coverage Command


33

Imagine…

• Having code coverage on an individual invocation

is interesting

• But for transforming your quality, things have to

be automated

• What if you could check the coverage of your

regular test runs?

• And this all ran on the host with no dependency

on any client software

• And it was FREE!


Continuous Integration / Continuous Delivery


Command Line Code Coverage

• Freely available with Rational Development Studio (5770-WDS)

option 60 on the IBM i

IBM i MRM MRI

V7R3 PTF 5770WDS SI65229 2924 PTF 5770WDS SI64655

V7R2 PTF 5770WDS SI65228 2924 PTF 5770WDS SI64544

Option Charge

Base No

Option 60 Workstation Tools No

ILE Compilers Yes

Heritage Compilers Yes

Application Development Toolset Yes


36

How command line code coverage is structured

IBM i

QSYS

Application PGM/SRVPGM

Under analysis

CODECOV CMD

IFS

Analysis files (CCZIP)

In directory

PC

RDi

Result Directory

Views

Net share or copy

CcApiExtAPI

Your Tools!


37

Requirements on job environment

• ILE Programs and service programs for instrumentation must be compiled with one of the

following debug options:

• DBGVIEW(*SOURCE)

• DBGVIEW(*LIST)

• DBGVIEW(*ALL)

• Main command: CODECOV


Features of Headless Code Coverage Tools

• Start a code coverage session against a run command• Add programs and service programs into the instrumentation list• Module filter support

• A list of modules • (M1 M2 M3)

• All modules • (*ALL)

• EXCEPT keyword: • (*EXCEPT M1 M2) - All modules except M1 and M2

• Support both *SOURCE and *LIST views• Support LINE and PROCEDURE level coverage• Save code coverage result in designated location

• When output directory is specified, save result as a generated file under the target directory• When output stream file is specified, save result in the designated file


Prompt screens for CODECOV command


Prompt screens for CODECOV command (cont.)


Include one program into the MODULES list

• Enter one program into the Modules list• Enter “+” into the second “+ for more values” field


Include another service program into the MODULES list


Specify a list of modules

• Enter the first module into the Module field and then enter “+” into the first “+ for more values”


Specify a list of modules (cont.)

Enter other modules


Code Coverage Mandatory Parameters

• Two parameters are mandatory:• Command to run: The full command line that will run in code coverage instrumentation mode

• Command length restriction: 6000• At least one entry in the MODULES list

• Include at least one program or service program for code coverage instrumentation

• Does not have to be the main program


Code Coverage View

• Three code coverage view configurations are supported:• *SOURCE• *LIST• *DFT

• Programs and service programs included in instrumentation must be compiled with one of the following debug options:

• DBGVIEW(*SOURCE)• DBGVIEW(*LIST)• DBGVIEW(*ALL)

• *SOURCE:• Use the *SOURCE view for source file display. The programs must be compiled with

DBGVIEW(*SOURCE) or DBGVIEW(*ALL)• *LIST:

• Use the *LIST view for source file display. The programs must be compiled with DBGVIEW(*LIST) or DBGVIEW(*ALL)

• *DFT: The default behavior• Use the *SOURCE view when the program is compiled with DBGVIEW(*SOURCE) or DBGVIEW(*ALL)• Use the *LIST view when the program is compiled with DBGVIEW(*LIST)


Code Coverage Level

• *LINE: Line level

• Include line-by-line statistics

• The default level

• *PROC: Procedure level

• Only capture the information whether a procedure entry point is hit or not

• No line-by-line statistics

• Run faster, but capture less information


Output options

• Two parameters to control output options:• Output directory and Output stream file • They cannot be specified at the same time• Error message is issued if they are specified at the same time:

• Parameters OUTDIR and OUTSTMF cannot be used at the same time• Output directory:

• Code coverage result file is saved to this directory. • Use generated name for code coverage result file

• Generated result name format:• ProgramName_TimeStamp.cczip

• The default output directory is *CURRENT.• Output stream file:

• Fully qualified IFS file name for code coverage result file• If the base name does not have extension

• Extension .cczip is appended to the base name• If the base name has an extension, it must be .cczip, otherwise an error is issued:

• The extension for the result name … is not .cczip.


Code Coverage Result File

• Code coverage result file has .cczip extension• Internally it is a ZIP file• An XML entry (ccdata) contains code coverage statistics• Source files are stored under the “src” folder inside the .cczip file


Code Coverage Test Identifier

• An identifier to set on the current code coverage session• Not used for any feature right now• Business Partners can use it• IBM ADDI uses it to identify tests and optimizes the minimal set of tests

to get the desired shelving


Requirements: Viewing Code Coverage Data

• CCZIP results generated by CODECOV can be viewed on RDi v9.5.1.2 or later

• Install RDi 9.5.1.2 or laterNOTE: If you don’t have access to the above location, install as follows:

• Go to: http://ibm.biz/rdi_download• Click on:

• Select Window (Automated web install using IBM Installation Manager) • Follow the steps in Automated procedure for full instructions.

http://ibm.biz/rdi_download

http://www-01.ibm.com/support/docview.wss?uid=swg21667734#AUTO


Viewing Coverage Results in RDi

• Assume CODECOV generated into /home/REINHARD/CodeCoverage


RDi –- Code Coverage report and Code Coverage Results view


RDi –- Code Coverage Annotation in source editor


Limitations

• It is currently not possible to generate HTML/PDF reports from

the .cczip results from CODECOV

– PDF report support delivered with 9.6.0.5

• CODECOV only works with ILE languages


57

Vision

• Can raise the maturity of your software development process

– Get the code coverage on your regular build/test on the host

– Detect drops in coverage on each build

– Mandate a minimum threshold for your tests

– Integrate with automated test solution

– Capture analysis with only dependence on a command shipped with

IBM i

– Analyze the results with the full power of RDi


Part 3 – Vendors lining up to support this

• Freshe Legacy

• XaTester from Xact

• Original Software

• ILEditor by Liam

• Integrate Code Coverage with their automated testing

• Include Code Coverage statistics with their reporting


ILEditor

• Save command,

modules as a

named test

• Invoke test

• See Coverage

result display in

the editor


Example TestBench by Original

• Existing tools can be used by

wrapping calls with the CODECOV

command.

• In this case, the CMD would simply

be

CALL PGM(TB_OEFIL/OEORDRR

PARM(‘DEMO’).

• TestBench would then ‘wrap’ the

program call inside CODECOV and

handle specifying all the necessary

MODULE parameters for ‘child’

programs etc.


TestBench Results – Test Run Details


TestBench Results – Program Calls


TestBench Results – Database Effects Summary


TestBench Results – Database Effects Detail


TestBench Results – Database Effects Update Details


TestBench Results – Data Queue Activity


TestDrive Results – Showing Capture and Inclusion of Code

Coverage Results – Future support (not available today)

Selecting the htmlfolder opens the folder in File Explorer so the HTML file can be launched.


Fresche - X-Analysis and Code Coverage

• The regression testing capabilities in X-Analysis enables the generation of Code Coverage reports

through the creation and configuration of Test Processes.

– Test Process defines the setup, the program to be executed, the generation of a Test Result, Test

Result report and teardown after the test has completed. The Test Process can be run on

demand or submitted to batch.

– Test Result report provides a detailed report of any data area or table that differs from the original

Test Result.

– X-Analysis will also detect problems associated with the object that can be used to alert the

developer of the problem. The results of Problem Analysis are also available via an API to

condition the promotion of the object into production.

• Some of the Code Coverage integrations that are planned for the next release are:

– Setup & Configuration – Enhanced control of which Programs/Modules are included in the Code

Coverage report.

– Measurement - Code Coverage percentages will be tracked over time in order to track trends

– Application Management - Problem Analysis will provide the ability to identify code coverage

conditions that require attention before an object should be moved into production.


ARCAD Verifier Integration

• ARCAD Verifier is a very capable tool that not only automates batch and interactive tests, but also

detects and validates changes in the user interface, database and spools

• By the end of the year they are planning to release integration with IBM’s CODECOV

https://arcadsoftware.com/products/arcad-verifier-regression-testing/


Conclusion

• Code coverage gives you concrete measurement of your test

effectiveness

• At development time, use RDi to launch and view code coverage

• At build time, use the command line CODECOV to test at scale

and the use RDi to view results

• As a command line it is easy to integrate CODECOV in your own

scripts or in automated test tooling


Where to learn about Code Coverage

• Learn More about Code Coverage

– Charlie Guarino Debugging and Code Coverage 1/3



– Code Coverage Product Documentation

– Code Coverage article on DeveloperWorks

– IBM Application Delivery Intelligence based on Code Coverage available

for z

• Test Automation

– RPGUnit integration for RDi – free, applicable for batch programs only

– ARCAD Verifier product description

– Original Software –TestDrive

– Xact XaTester

71

https://www.youtube.com/watch?v=roesIrpiIAs&list=PLPXmBSxtNwSlcVtSqt8Ks8kvx35Et4wLN&index=8

https://www.youtube.com/watch?v=KWm7sXzqvI8&index=10&t=0s&list=PLPXmBSxtNwSlcVtSqt8Ks8kvx35Et4wLN

https://www.youtube.com/watch?v=7qjIfeBBhKU&list=PLPXmBSxtNwSlcVtSqt8Ks8kvx35Et4wLN&index=10

http://www-01.ibm.com/support/knowledgecenter/SSAE4W_9.5.0/com.ibm.debug.pdt.codecoverage.i.doc/topics/tcc_run_about.html?cp=SSAE4W_9.5.0

http://www.ibm.com/developerworks/library/d-code-coverage-tools-rational-developer-i/

https://www.youtube.com/watch?v=1eLMurepPPg

http://www.tools400.de/English/Freeware/RDi_Plug-Ins/rdi_plug-ins.html

http://arcadsoftware.com/products/arcad-verifier-regression-testing/

http://origsoft.com/solutions/software-test-automation/

https://www.xact.dk/en/services/software/unit-test-for-mainframe/


For More Information:

Some Links You Need Twitter #Hashtags

IBM i Home Page:

www.ibm.com/systems/i

IBM 30th Anniversary

http://ibmi30.mybluemix.net/

IBM Systems Magazine IBM i Edition:

http://ibmsystemsmag.com/ibmi/

Support Life Cycle:

https://www-01.ibm.com/software/support/ibmi/lifecycle/

License Topics:

https://www-01.ibm.com/support/docview.wss?uid=nas8N1022087

@IBMSystems

@COMMONug

@IBMChampions

@IBMSystemsISVs

@IBMiMag

@ITJungleNews

@SAPonIBMi

@SiDforIBMi

#IBMi30

#PowerSystems

#IBMi

#IBMAIX

#POWER8

#LinuxonPower

#OpenPOWER

#HANAonPower

#ITinfrastructure

#OpenSource

#HybridCloud

#BigData

http://www.ibm.com/systems/i

http://ibmi30.mybluemix.net/

http://ibmsystemsmag.com/ibmi/

https://www-01.ibm.com/software/support/ibmi/lifecycle/

https://www-01.ibm.com/support/docview.wss?uid=nas8N1022087

https://twitter.com/IBMSystems

https://twitter.com/COMMONug

https://twitter.com/IBMChampions

https://twitter.com/IBMSystemsISVs

https://twitter.com/IBMimag

https://twitter.com/ITJungleNews

https://twitter.com/SAPonIBMi

https://twitter.com/SiDforIBMi


For More Information:

Blogs

IBM Blogs:

• IBM Systems Magazine You and i (Steve Will)

• IBM Systems Magazine i-Can (Dawn May)

• IBM Systems Magazine: Open your i (Jesse Gorzinski)

• IBM DB2 for i (Mike Cain)

• IBM DB2 Web Query for i (Doug Mack)

IBM Champion’s Blogs:

• IBM Systems Magazine: iDevelop (Jon Paris and Susan Gantner)

• IBM Systems Magazine: iTalk with Tuoy

http://ibmsystemsmag.com/blogs/you-and-i/

http://ibmsystemsmag.com/blogs/i-can/

http://ibmsystemsmag.com/blogs/open-your-i/

http://db2fori.blogspot.co.uk/

http://db2webqueryi.blogspot.co.uk/

http://ibmsystemsmag.com/blogs/idevelop/

http://ibmsystesmag.com/ibmi/trends/italk-with-tuohy/

http://www.ibmsystemsmag.com/Blogs/you-and-i/

http://www.ibmsystemsmag.com/Blogs/you-and-i/

http://ibmsystemsmag.com/Blogs/i-Can/

http://www.ibmsystemsmag.com/Blogs/Open-Your-i/

http://db2fori.blogspot.com/

http://db2webqueryi.blogspot.com/

http://www.ibmsystemsmag.com/Blogs/iDevelop/

http://ibmsystemsmag.com/ibmi/trends/iTALK-WITH-TUOHY/

http://ibmsystemsmag.com/blogs/i-can/

http://ibmsystemsmag.com/blogs/open-your-i/

http://db2fori.blogspot.co.uk/

http://db2webqueryi.blogspot.co.uk/

http://ibmsystemsmag.com/blogs/idevelop/

http://ibmsystemsmag.com/ibmi/trends/italk-with-tuohy/

are you covered? new tooling for quality assurance · rdi expands into new territory –quality...

Documents