an update on automatic regression testing at diamond
DESCRIPTION
An Update on Automatic Regression Testing at Diamond. Contents. Motivation Update on the Testing Framework at Diamond Hardware Regression Test EPICS Base Regression Test. Introduction. - PowerPoint PPT PresentationTRANSCRIPT
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICS
An Update on Automatic Regression Testing at Diamond
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICSContents
1. Motivation2. Update on the Testing Framework at
Diamond3. Hardware Regression Test4. EPICS Base Regression Test
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICSIntroduction• Diamond is now operating for ~4500 hours per year
which will increase to ~5500 hours per year over the next two years
• Shutdown time is being reduced, and so are becoming increasingly busy– 2010 and 2011 see installation “Mini Beta Optics” in two
long straight sections necessitate replacing pairs of SR girders
• All this results in less time to test new software releases
• Drive to improve operational reliability– The Control System has not had significant number of
faults– Don’t want to change this
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICSThe Grand Objective• Test all 200 support modules
– Test modules and EPICS DB templates
• Test device/drivers that interface to hardware– Hardware test environment
• Test the versions of EPICS base we are running – Issues of changing EPICS base and /Tool chain/OS/
version
• Test software against a tool chain, build and run environment – We want to ensure that operational system use the same
environment
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICSGood Software Engineering Practice
• Four major parts to software engineering– Design: Should come first!
– Code: We are good at this.
– Documentation: Doxygen etc.
– Testing: Often manual and hard to repeat.
• Testing is not an optional extra– Can be carried out by the Software Developer
– Or can be carried out by the Software User
– Or can be automated
• Tests must be properly repeated – At least before every release and preferably more often
• Tests are easier to repeat if they are completely automatic
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICSTest Framework• A software framework that aids the creation of
automatic test suites for modules.• Based on PyUnit, the Python unit testing library.• Test reports conform to the TAP protocol and/or
JUnit XML format.• Monitoring of EPICS database records for coverage
reporting.• Hardware and simulation targets can use same test
suite.
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICSThe Run Script• Searches a directory tree for modules with automatic test
support.• Runs tests suites it finds, logging results and output.• Can run multiple tests in parallel subject to resource
constraints.• Level of diagnostic output can be specified.• Tests run can be restricted to single module, single target,
single case.• “dls-run-tests -h” for command help.• The run script is designed to operate from the ‘support’
directory of the work tree.
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICSDirectory Structure Changes
etc
tests.py
<moduleName>
test
support
makeDocumentation
makeIocs
iocs
new directories
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICSDevice Simulations• Based on existing DLS practice.
• Written in Python.
• Currently capable of simulating any device that communicates through serial or IP connection.
• Support for instrumentation to allow protocol coverage reporting.
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICSFramework Class Diagram
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICSExample – The Test Suitefrom pkg_resources import requirerequire('dls.autotestframework')from dls.autotestframework import *class Fw102TestSuite(TestSuite): def createTests(self): # Define the targets for this test suite Target("simulation", self, iocDirectory="example", iocBootCmd="bin/linux-x86/stfw102Ex_sim.boot", epicsDbFiles="db/fw102Ex.db", simDevices=[SimDevice("controller1", 9016)], guiCmds=['edm -m "P=FGZ73762,M=:WHEEL1" -eolc -x data/fw102.edl']) Target("hardware", self, iocDirectory="example", iocBootCmd="bin/linux-x86/stfw102Ex.boot", epicsDbFiles="db/fw102Ex.db", guiCmds=['edm -m "P=FGZ73762,M=:WHEEL1" -eolc -x data/fw102.edl']) # The tests CaseLocalIncrementSwitch(self) CaseLocalDecrementSwitch(self) CasePowerOffOn(self) CasePvIncrement(self) CasePvDecrement(self) CasePvMultipleIncrement(self) CasePvMultipleDecrement(self) CasePvSetPosition(self) CasePvTriggerMode(self) CasePvPolling(self)
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICSExample – Intermediate Classclass Fw102Case(TestCase):
def curDevicePos(self): ''' Get the current wheel position from the device simulation ''' result = 0 self.command("controller1", "getpos") args = self.recvResponse("controller1", "pos", 1) if args is not None: result = int(args[0]) return result
def verifyPosition(self, intended): ''' Verify that the wheel is in the intended position ''' if self.simulationDevicePresent("controller1"): self.verify(self.curDevicePos(), intended) self.verifyPv("FGZ73762:WHEEL1:POSITION_RBV", intended) self.verifyPv("FGZ73762:WHEEL1:POSITION", intended) self.verifyPv("FGZ73762:WHEEL1:INPOS", 1)
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICSExample – A Test Caseclass CaseLocalIncrementSwitch(Fw102Case): def runTest(self): '''The local increment switch.''' if self.simulationDevicePresent("controller1"): # Check the current position of the wheel before = self.initialPosition() # Take the wheel round twice for i in range(12): # Now advance the wheel using the local button self.command("controller1", "incr") after = before + 1 if after > 6: after = 1 # Check the final position of the wheel self.sleep(2) self.diagnostic("Before=%d, after=%d" % (before, after), 1) self.verifyPosition(after) before = after
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICSExample - Test Report[1] 1..10 [fgz73762@pc0054 diamondtest]$ ./runtests.py -t simulation -f default.config -i -g -b -p 3 -l tests.log -q -m fw102
[1] ok 1 - The local increment switch.
[1] ok 2 - The local decrement switch.
[1] ok 3 - Power off and on.
[1] ok 4 - The PV increment command.
[1] ok 5 - The PV decrement command.
[1] ok 6 - Fast increment sync correction.
[1] ok 7 - Fast decrement sync correction.
[1] ok 8 - Movement directly to a position.
[1] ok 9 - Control of the trigger mode.
[1] ok 10 - Control of the polling mechanism.
[1] # ==============================
[1] # Passed 10/10 tests, 100.00% okay, in 260.57s
[1] #
[1] # ==============================
[1] # Sim device controller1 coverage report:
[1] # setpos: ok
[1] # settrigger: ok
[1] # getpos: ok
[1] # gettrigger: ok
[1] #
[1] # ==============================
[1] # EPICS database coverage report:
[1] # FGZ73762:WHEEL1:TRIGGER(mbbo): ok
[1] # FGZ73762:WHEEL1:TRIGGER_RBV(mbbi): ok
[1] # FGZ73762:WHEEL1:RESTART(ai): not touched
[1] # FGZ73762:WHEEL1:INPOS(calc): ok
[1] # FGZ73762:WHEEL1:REINIT1(fanout): ok
[1] # FGZ73762:WHEEL1:POSITION(mbbo): ok
[1] # FGZ73762:WHEEL1:COMMSFAIL(bi): ok
[1] # FGZ73762:WHEEL1:POSITION_RBV(longin): ok
[1] # FGZ73762:WHEEL1:CALCCOMMS(calcout): ok
[1] # FGZ73762:WHEEL1:DISABLEPOLL(bo): ok
[1] # FGZ73762:WHEEL1:STEPBK(calcout): ok
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICSHudson• Use Hudson framework
– For scheduling and reporting of test
• A Continuous Integration Engine– Checkout from repository – Does some actions
• Build, Post-build actions, (execute test script)
– Determine Status
• Invoked by period checks on SVN repository• Status by web and emails to developers
• Success (all actions successful)• Unstable (some tests failed)• Failed (compile error)
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICS
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICSHardware Regression Test• Test IOC consists of:
– Prodex OMS-58 8 axis servo control card• A voltage to quadrature encoder conversion card
– Hytec 8001 DIO and 8515 Octal Serial– EPICS IOC test specific db
• Stimulation IOC:– 8005 – digital I/0 for limit switches, home switches– 8401 – ADC input card to monitor OMS-58 servo demand
output– 8402 – DAC to drive the encoder simulation– VxWorks application implemented by cut down drivers for
above cards and a simulation of some ‘realish’ hardware to control
• Test harness:– Wiring to join Test IOC to Stimulation IOC.
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICSHardware Test Framework
Test Unit (EPICS IOC)
Hardware under test
Switch
Terminal Server
Stimulation Unit
(vxWorks stimulations)
Linux control PC
Test suite control,
redirection, etc
Power strip (IP)
Hudson
Processor Processor
Serial
Digital & Analog I/O
Test harness
Network
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICSEPICS Base Regression Test Suite (1)• Came out of the Codeathon at Diamond
– Nick Rees, Jon Thompson and Matt Pearson
• Objective – To generate automated testing of the EPICS Base project
code on a diverse range of architectures, both workstation and real-time OSs.
– These will be run nightly at Diamond on head of trunk and results published on web site.
– Provide as service to the collaboration
• This is work in progress. Only started it last week.
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICSEPICS Base Regression Test Suite (2)• There are established EPICS tests, but these are
run manually as part of releases• Wrap the established EPICS tests in Python to
make callable from test framework. • Run test on
– EPICS Base, C Unit tests– Report on test coverage for each EPICS Base test– IOCs based tests
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICSEPICS Base Types of tests (1)• C unit tests based on epicUnitTest.h functions.
– These are mostly used to test libCom functions at this point
– Generate TAP output that is interpreted by the test harness
– Propose to add new make targets to describe how the tests are run on embedded architectures (give a download binary and a command line function to run)
– Make generates a test specification that is used by the test framework to run the tests
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICSEPICS Base Types of tests (2)• IOC based system tests based on Diamond module
test framework– Python framework that loads IOC under test and a
stimulation system (if required)– Can get, set and monitor PV’s, and/or control or monitor
the IOC command line– Tests reside in a new etc/test directory in the application
tree– Have a wide range of verification commands. – Many of the EPICS “soft-test” suite tests were converted
to this framework during the Codeathon
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICSEPICS Base Platforms
• Plan is to run this for popular development platforms and target architectures.
• What about historic development platforms, Solaris, and latest platforms say Fedora XX or Ubuntu XX.YY and other target architectures ?
Linu
x R
HE
L 4
/ G
CC
XX
X
Liln
ux R
HE
L5 /
G
CC
4.1
.2
Win
dow
s X
P/
MS
VC
Win
dow
s C
ygw
in
GC
C ?
??
VxW
orks
5.6
GC
C
3.4.
6
RT
EM
S ?
?? G
CC
??
?
Mac
OS
X G
CC
???
Linu
x G
CC
4.3
.2
Glib
c 2.
7
Linu
x G
CC
4.3
.2
Glib
c 2.
6.1
68030 MVME 167 x xPPC 7457 MVME5500 x x xx86 x x x x xARM xXilinx Virtex 2 PPC 405 x
Target OS / Toolchain and Library
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICSEPICS Base Test Results• Results will be published as a web page containing:
– Results of ‘make runtests’ on the target architectures– Results of running the ‘soft-tests’ on the target
architectures– Code coverage report for the Linux host.
• URL to be announced
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICSEPICS Base Test Results Web Interface
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICSEPICS Base Test Results Web Interface
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICS EPICS Base Test Results Web Interface
03/06/10Mark Heron
An Update on Automatic Regression Testing at Diamond
EPICSAcknowledgement
• This is being lead by Jon Thomson at Diamond
• It’s the work of many of the Controls Group at Diamond