automatically measuring code coverage in distributed environment

1 / Platform Development Process

In-house limited

Automatically Measuring Code Coverage in Distributed EnvironmentLi Fei PengPlatform QA Part, NSC2009- 08-10


In-house limited

Content

• Introduction• Tools for code coverage, lcov and

bullseye• Executing command remotely, jsch• A tool for automatically measuring code

coverage• Troubles in this work• Improvement


In-house limited

Introduction

• A complicated application which runs in a distributed environment, CUBRID

• Code coverage need to be measured automatically

• Set up environment, update and compile source code

• Run test cases• Process code coverage result and upload to QA

website• Linux (support SSH by default)


In-house limited

……

Introduction

SVN Server

Machines Running TC

QA Website

Control Machine


In-house limited

Tools for code coverage

• Lcov– Line coverage– Function coverage– Free – Compile: -fprofile-arcs -ftest-coverage

• Bullseye– Function coverage– Condition/Decision coverage– Proprietary– User command cov01 to enable/disable coverage

build


In-house limited

Tools for code coverage - lcov

• a graphical front-end for gcov

• collect gcov data for multiple source files

• create HTML pages containing the source code annotated with coverage information

• add overview pages for easy navigation within the file structure


In-house limited

Tools for code coverage - bullseye

• Function coverage gives you a quick overview and condition/decision coverage gives you high precision

• Works with everything you can write in C++ and C, including system-level and kernel mode

• Short learning curve • Include or exclude any portion

of your project code • Run-time source code included,

for custom environments • Simple licensing. • Merge results from distributed

testing • Integration with Microsoft Visual

Studio


In-house limited

Jsch

• JSch is a pure Java implementation of SSH2.• JSch allows you to connect to an sshd server

and use port forwarding, X11 forwarding, file transfer, etc., and you can integrate its functionality into your own Java programs.

• Ant provides sshexec and scp tasks by using Jsch

• We use two functions: – Remote exec: ls, lcov, cubrid server start– File transfer: scp, sftp


In-house limited

Jsch

• We use two functions: – Remote exec:

• lcov• cubrid server start• cov01• make install• …

– File transfer: • scp• sftp


In-house limited

A tool for automatically measuring code coverage

• Configure environment in each machines– Install lcov, bullseye, subversion– Set up environment variables, add the bin directory of lcov and

bullseye to PATH– Checkout the source code, save password

• Runing process– Update source code– Set up environment, compile and Install in local machine– Compress the source code and upload the source code to other

machines– Set up enviroment, uncompress, compile and install in each remote

machine– Run test cases– For lcov, collect code coverage data– Copy the code coverage data from other machines to control machine– Generate the visual result of code coverage in control machine– Upload the result to QA website


In-house limited


• SSHClient.java– Encapsulate the exec and file transfer

functions in this class– SSHClient.exec(String host, String user,

String password, String command)– SSHClient.scpFrom(String host, String user,

String password, string rfile, String lfile)


In-house limited


• CovUtil.java– Main()– Update and install:

• updateAndInstallCubrid()– updateAndInstallCubridLocal– updateAndInstallCubridRemote

– Code coverage• collectCCAndUpload()

– collect bullseye data or lcov data– analyze code coverage data(genhtml, no analyze for

bullseye)– Upload the result to QA website


In-house limited


• UsageUsage: java

com.nhncorp.cubridqa.codecoverage.CovUtil <mode> <properties file>

mode: install, sa, cc

0 20 * * 5 java -jar ~/cubrid_cov/cubrid_cov.jar install ~/cubrid_cov/cov.properties0 20 * * 6 java -jar ~/cubrid_cov/cubrid_cov.jar sa ~/cubrid_cov/cov.properties0 23 * * 6 java -jar ~/cubrid_cov/cubrid_cov.jar cc ~/cubrid_cov/cov.properties

java -jar ~/cubrid_cov/cubrid_cov.jar <mode> <properties file>


In-house limited


• Configuration file– The production installation directory– The directories of related tools– QA website– Remote machines– Other parameters:

• removeCubrid• Bullseys or lcov• Exec locally or remotely• Etc.


In-house limited


• cubridBuildDir=/home/xdbms/build/RB-8.2.0• cubridHome=/home/xdbms/CUBRID• lcovHome=/home/xdbms/opt/lcov-1.7• svnPath=/home/xdbms/opt/subversion/bin/svn• removeCubrid=true• debug=true• enable64bit=false• uploadHost=59.108.93.74• uploadUser=qahome• uploadPassword=password• uploadPath=/home/qahome/tomcat-6.0.18/webapps/qaresult/qaresultfile/cubrid/code coverage• sqlDir=/home/xdbms/workspace/qa_repository/scenario/sql• execLocal=true• execRemote=true• svnUpdate=true• bullseyeHome=/home/xdbms/opt/bullseye• enableBullseye=true• #remote.1=59.108.93.81,xdbms,password,/home/xdbms/build/RB-8.2.0,/home/xdbms/CUBRID,/home/xdbms/opt/lcov-1.7,/home/xdbms/opt/

subversion/bin/svn• #remote.2=59.108.93.67,xdbms,password,/home/xdbms/build/RB-8.2.0,/home/xdbms/CUBRID,/home/xdbms/opt/lcov-1.7,/home/xdbms/opt/





subversion/bin/svn• remote.7=127.0.0.1,xdbms,password,/home/xdbms/build/RB-8.2.0,/home/xdbms/CUBRID,/home/xdbms/opt/lcov-1.7,/home/xdbms/opt/

subversion/bin/svn


In-house limited

Troubles in this work

• Environement variables: execute . /etc/profile . ~/.bash_profile before executing each command

• Bullseye can not enabled with lcov• In order to successfully merge two coverage

files, the source files used to build them must have the same timestamps

• There is some problems in utility lcov of v1.7: v1.6 is ok

• An error will be thrown when using genhtml: modifying the source code of genhtml


In-house limited

Improvement

• Analyzing the data generated by bullseye

• Output the visual result and upload them to QA website

• Code reorganization


In-house limited

automatically measuring code coverage in distributed environment

Documents