automatically measuring code coverage in distributed environment

1 / Platform Development Process

In-house limited

Automatically Measuring Code Coverage in Distributed EnvironmentLi Fei PengPlatform QA Part, NSC2009- 08-10

2 / Platform Development Process

In-house limited

Content

• Introduction• Tools for code coverage, lcov and

bullseye• Executing command remotely, jsch• A tool for automatically measuring code

coverage• Troubles in this work• Improvement

3 / Platform Development Process

In-house limited

Introduction

• A complicated application which runs in a distributed environment, CUBRID

• Code coverage need to be measured automatically

• Set up environment, update and compile source code

• Run test cases• Process code coverage result and upload to QA

website• Linux (support SSH by default)

4 / Platform Development Process

In-house limited

……

Introduction

SVN Server

Machines Running TC

QA Website

Control Machine

5 / Platform Development Process

In-house limited

Tools for code coverage

• Lcov– Line coverage– Function coverage– Free – Compile: -fprofile-arcs -ftest-coverage

• Bullseye– Function coverage– Condition/Decision coverage– Proprietary– User command cov01 to enable/disable coverage

build

6 / Platform Development Process

In-house limited

Tools for code coverage - lcov

• a graphical front-end for gcov

• collect gcov data for multiple source files

• create HTML pages containing the source code annotated with coverage information

• add overview pages for easy navigation within the file structure

7 / Platform Development Process

In-house limited

Tools for code coverage - bullseye

• Function coverage gives you a quick overview and condition/decision coverage gives you high precision

• Works with everything you can write in C++ and C, including system-level and kernel mode

• Short learning curve • Include or exclude any portion

of your project code • Run-time source code included,

for custom environments • Simple licensing. • Merge results from distributed

testing • Integration with Microsoft Visual

Studio

8 / Platform Development Process

In-house limited

Jsch

• JSch is a pure Java implementation of SSH2.• JSch allows you to connect to an sshd server

and use port forwarding, X11 forwarding, file transfer, etc., and you can integrate its functionality into your own Java programs.

• Ant provides sshexec and scp tasks by using Jsch

• We use two functions: – Remote exec: ls, lcov, cubrid server start– File transfer: scp, sftp

9 / Platform Development Process

In-house limited

Jsch

• We use two functions: – Remote exec:

• lcov• cubrid server start• cov01• make install• …

– File transfer: • scp• sftp

10 / Platform Development Process

In-house limited

A tool for automatically measuring code coverage

• Configure environment in each machines– Install lcov, bullseye, subversion– Set up environment variables, add the bin directory of lcov and

bullseye to PATH– Checkout the source code, save password

• Runing process– Update source code– Set up environment, compile and Install in local machine– Compress the source code and upload the source code to other

machines– Set up enviroment, uncompress, compile and install in each remote

machine– Run test cases– For lcov, collect code coverage data– Copy the code coverage data from other machines to control machine– Generate the visual result of code coverage in control machine– Upload the result to QA website

11 / Platform Development Process

In-house limited

A tool for automatically measuring code coverage

• SSHClient.java– Encapsulate the exec and file transfer

functions in this class– SSHClient.exec(String host, String user,

String password, String command)– SSHClient.scpFrom(String host, String user,

String password, string rfile, String lfile)

12 / Platform Development Process

In-house limited

A tool for automatically measuring code coverage

• CovUtil.java– Main()– Update and install:

• updateAndInstallCubrid()– updateAndInstallCubridLocal– updateAndInstallCubridRemote

– Code coverage• collectCCAndUpload()

– collect bullseye data or lcov data– analyze code coverage data(genhtml, no analyze for

bullseye)– Upload the result to QA website

13 / Platform Development Process

In-house limited

A tool for automatically measuring code coverage

• UsageUsage: java

com.nhncorp.cubridqa.codecoverage.CovUtil <mode> <properties file>

mode: install, sa, cc

0 20 * * 5 java -jar ~/cubrid_cov/cubrid_cov.jar install ~/cubrid_cov/cov.properties0 20 * * 6 java -jar ~/cubrid_cov/cubrid_cov.jar sa ~/cubrid_cov/cov.properties0 23 * * 6 java -jar ~/cubrid_cov/cubrid_cov.jar cc ~/cubrid_cov/cov.properties

java -jar ~/cubrid_cov/cubrid_cov.jar <mode> <properties file>

14 / Platform Development Process

In-house limited

A tool for automatically measuring code coverage

• Configuration file– The production installation directory– The directories of related tools– QA website– Remote machines– Other parameters:

• removeCubrid• Bullseys or lcov• Exec locally or remotely• Etc.

15 / Platform Development Process

In-house limited

A tool for automatically measuring code coverage

• cubridBuildDir=/home/xdbms/build/RB-8.2.0• cubridHome=/home/xdbms/CUBRID• lcovHome=/home/xdbms/opt/lcov-1.7• svnPath=/home/xdbms/opt/subversion/bin/svn• removeCubrid=true• debug=true• enable64bit=false• uploadHost=59.108.93.74• uploadUser=qahome• uploadPassword=password• uploadPath=/home/qahome/tomcat-6.0.18/webapps/qaresult/qaresultfile/cubrid/code coverage• sqlDir=/home/xdbms/workspace/qa_repository/scenario/sql• execLocal=true• execRemote=true• svnUpdate=true• bullseyeHome=/home/xdbms/opt/bullseye• enableBullseye=true• #remote.1=59.108.93.81,xdbms,password,/home/xdbms/build/RB-8.2.0,/home/xdbms/CUBRID,/home/xdbms/opt/lcov-1.7,/home/xdbms/opt/

subversion/bin/svn• #remote.2=59.108.93.67,xdbms,password,/home/xdbms/build/RB-8.2.0,/home/xdbms/CUBRID,/home/xdbms/opt/lcov-1.7,/home/xdbms/opt/

subversion/bin/svn• #remote.3=59.108.93.68,xdbms,password,/home/xdbms/build/RB-8.2.0,/home/xdbms/CUBRID,/home/xdbms/opt/lcov-1.7,/home/xdbms/opt/

subversion/bin/svn• #remote.4=59.108.93.69,xdbms,password,/home/xdbms/build/RB-8.2.0,/home/xdbms/CUBRID,/home/xdbms/opt/lcov-1.7,/home/xdbms/opt/

subversion/bin/svn• #remote.5=59.108.93.70,xdbms,password,/home/xdbms/build/RB-8.2.0,/home/xdbms/CUBRID,/home/xdbms/opt/lcov-1.7,/home/xdbms/opt/

subversion/bin/svn• #remote.6=59.108.93.71,xdbms,password,/home/xdbms/build/RB-8.2.0,/home/xdbms/CUBRID,/home/xdbms/opt/lcov-1.7,/home/xdbms/opt/

subversion/bin/svn• remote.7=127.0.0.1,xdbms,password,/home/xdbms/build/RB-8.2.0,/home/xdbms/CUBRID,/home/xdbms/opt/lcov-1.7,/home/xdbms/opt/

subversion/bin/svn

16 / Platform Development Process

In-house limited

Troubles in this work

• Environement variables: execute . /etc/profile . ~/.bash_profile before executing each command

• Bullseye can not enabled with lcov• In order to successfully merge two coverage

files, the source files used to build them must have the same timestamps

• There is some problems in utility lcov of v1.7: v1.6 is ok

• An error will be thrown when using genhtml: modifying the source code of genhtml

17 / Platform Development Process

In-house limited

Improvement

• Analyzing the data generated by bullseye

• Output the visual result and upload them to QA website

• Code reorganization

18 / Platform Development Process

In-house limited