establishing an environment for continuous integration and...

FSWFSW--09, November 200909, November 2009

Establishing an Environment for Continuous Establishing an Environment for Continuous Integration and Test of Flight SoftwareIntegration and Test of Flight Software

Robert Klar, Christopher Mangels, and Randal HarmonRobert Klar, Christopher Mangels, and Randal Harmon

Southwest Research InstituteSouthwest Research Institute


Integration and TestIntegration and Test

�� Integration and Test is an important part of the Integration and Test is an important part of the development process and represents a significant development process and represents a significant investment in terms of cost and timeinvestment in terms of cost and time�� In recent organizational estimates, our software integration andIn recent organizational estimates, our software integration and

testing activities have accounted for more the 40% of the total testing activities have accounted for more the 40% of the total planned effortplanned effort

�� Testing tends to be done late in the project lifecycle, limitingTesting tends to be done late in the project lifecycle, limiting its its effectivenesseffectiveness

�� Setting up a test environment for a flight subsystem can be Setting up a test environment for a flight subsystem can be challengingchallenging

�� Ground Support Equipment (GSE) often is more complex Ground Support Equipment (GSE) often is more complex than the flight subsystem itselfthan the flight subsystem itself

�� Latent errors in flight software can be extremely costlyLatent errors in flight software can be extremely costly


Improving Integration and TestImproving Integration and Test

�� Why put emphasis on improving integration and test?Why put emphasis on improving integration and test?�� Since integration and testing activities represents a large partSince integration and testing activities represents a large part of of

the overall planned activities, improving Return On Investment the overall planned activities, improving Return On Investment here has a big effect.here has a big effect.

�� W. Edwards Deming put forth the notion that W. Edwards Deming put forth the notion that ““You cannot test You cannot test quality into a system.quality into a system.””�� This is true. However, by analyzing and understanding our testinThis is true. However, by analyzing and understanding our testing g

processes, we are able to collect information that can be used tprocesses, we are able to collect information that can be used to o improve our design and implementation processes. This can improve our design and implementation processes. This can improve quality.improve quality.

�� Planning for integration and test should be an important Planning for integration and test should be an important part of the software development plan.part of the software development plan.�� Espouse the principle of Espouse the principle of ““Design for TestDesign for Test””�� Hardware/Software resource planning is important (to avoid Hardware/Software resource planning is important (to avoid

Marching Army costs)Marching Army costs)


Continuous Integration and TestContinuous Integration and Test

�� Continuous IntegrationContinuous Integration�� Idea that emerged in Extreme Programming community [1]Idea that emerged in Extreme Programming community [1]�� With multiple developers, components for a particular software With multiple developers, components for a particular software

item are being developed or modified in parallel. Over time, thitem are being developed or modified in parallel. Over time, this is may result in a very difficult integration (may result in a very difficult integration (““Integration HellIntegration Hell”” [3]).[3]).

�� To avoid this problem, integration is done more frequently.To avoid this problem, integration is done more frequently.�� A plethora of tools exist today for automating continuous A plethora of tools exist today for automating continuous

integration.integration.�� Bitten, Bitten, BuildBotBuildBot, , CruiseControlCruiseControl, , TinderBoxTinderBox, etc., etc.�� Because there are so many choices, it is important to find Because there are so many choices, it is important to find

one that works well for you.one that works well for you.

�� Continuous TestContinuous Test�� Including testing with each integration is a good practiceIncluding testing with each integration is a good practice

�� ““Test early and oftenTest early and often””


Establishing an EnvironmentEstablishing an Environment

�� Desirable Characteristics for a CI&T EnvironmentDesirable Characteristics for a CI&T Environment�� Simple. The system needs to be simple to deploy and use. If Simple. The system needs to be simple to deploy and use. If

not, it will not provide any advantage to improving the I&T not, it will not provide any advantage to improving the I&T process.process.

�� Reusable. We have many small independent software Reusable. We have many small independent software development efforts. We do not want to development efforts. We do not want to ““reinvent the wheelreinvent the wheel”” with with each one.each one.

�� Cost effective. Since we are involved in many small programs, Cost effective. Since we are involved in many small programs, expensive tools are difficult to procure. Open source tools andexpensive tools are difficult to procure. Open source tools andlowlow--cost commercial tools are attractive for this reason. Tools cost commercial tools are attractive for this reason. Tools with recurring license costs can be problematic.with recurring license costs can be problematic.

�� Maintainability. Since space programs need support for many Maintainability. Since space programs need support for many years, we need to make choices to guard against obsolescence.years, we need to make choices to guard against obsolescence.

�� Flexible. Science Instruments and Spacecraft Bus Subsystems Flexible. Science Instruments and Spacecraft Bus Subsystems have different needs. We need to accommodate both. have different needs. We need to accommodate both.


Establishing an EnvironmentEstablishing an Environment

�� Plan for testing Plan for testing –– do not let it evolve!do not let it evolve!�� Resources required for testing are available at different pointsResources required for testing are available at different points in in

the schedule. This is true of most programs.the schedule. This is true of most programs.

�� It is desirable to maintain consistency at each level of integraIt is desirable to maintain consistency at each level of integration tion to avoid reworkto avoid rework

�� Testing progress needs to be measurableTesting progress needs to be measurable

�� Number of test cases totalNumber of test cases total

�� Number of test cases completed successfullyNumber of test cases completed successfully

�� Need to be able to quickly adapt when problems are foundNeed to be able to quickly adapt when problems are found

�� Consider backup options for test environmentsConsider backup options for test environments

�� Setup a regular schedule for integration and test.Setup a regular schedule for integration and test.�� NightlyNightly and and OnOn--demanddemand were good choices for us.were good choices for us.


CI&T Case 1: Instrument (JunoCI&T Case 1: Instrument (Juno--UVS)UVS)

Detector Electronics

Detector Assembly

OAP Mirror

Grating

Entrance Baffle

Fail-Safe Door

Detector Door

Scan Mirror & Assembly

Slit

Aperture Door

Pump-out Tube

Ta Radiation Shield

UVS is an Ultraviolet Spectrometer for the Juno Mission (JPL). The spinning Juno spacecraft will perform science observations for about one year around 2017 from thirty highly elliptical 11-day polar orbits around Jupiter. UVS main observations will be performed +/- 3 hours of perijove each orbit.



Host Workstation

Telemetry and Command System

with Python Interpreter(GSEOS)

Keil51 Environment

UVSFlight

Software

TCPSocket

SerialI/F Sim

Sim Input

Host Workstation


with Python Interpreter(GSEOS) UVS Engineering

C&DH Board

Terminal Server

(ser2net)

RS485 to RS422

Converter

TCPSocket

Serial Serial

UVSFlight

Software

Sim Input

Host Workstation


with Python Interpreter(GSEOS) UVS Engineering

or Flight Model

UVSFlight

SoftwareTCPSocket

SerialSpacecraft Emulator

(SCE)UVS Detector

Electronics

High-speedSerial

SignalGenerator

Analog

�� Plan for integration Plan for integration in stages.in stages.�� Using the same Using the same

Host environment, Host environment, allows for allows for progression and progression and regressionregression



�� Build environment is simpleBuild environment is simple�� C language and makeC language and make

�� Our CI&T environment should also be simpleOur CI&T environment should also be simple�� Evaluated CI tools such as Evaluated CI tools such as CruiseControlCruiseControl and Antand Ant

�� Not a good fitNot a good fit�� Instead chose to base CI on Python and Instead chose to base CI on Python and croncron, and CVS, and CVS�� Integration and test by Integration and test by ““Nightly ScriptNightly Script””

�� Rebuilds software and starts execution of Test ScriptsRebuilds software and starts execution of Test Scripts�� Runs until stoppedRuns until stopped�� Runs Test Scripts in sequence and out of nominal sequenceRuns Test Scripts in sequence and out of nominal sequence



2009/11/03 03:59:26:TM2009/11/03 03:59:26:TM--HK: 669760: Command accepted: Control Pixel StimulatorHK: 669760: Command accepted: Control Pixel Stimulator

2009/11/03 03:59:26:TM2009/11/03 03:59:26:TM--HK: 669760: Command executedHK: 669760: Command executed

2009/11/03 03:59:26:TM2009/11/03 03:59:26:TM--HK: 669760: Detector STIM activatedHK: 669760: Detector STIM activated

2009/11/03 03:59:26:TSTOK: : 010: all commands executed2009/11/03 03:59:26:TSTOK: : 010: all commands executed

2009/11/03 03:59:26:TC : : Control Pixel 2009/11/03 03:59:26:TC : : Control Pixel Stimulator(StimStimulator(Stim Off): 0xb3000000Off): 0xb3000000

2009/11/03 03:59:34:TM2009/11/03 03:59:34:TM--HK: 669768: Command accepted: Control Pixel StimulatorHK: 669768: Command accepted: Control Pixel Stimulator

2009/11/03 03:59:34:TM2009/11/03 03:59:34:TM--HK: 669768: Command executedHK: 669768: Command executed

2009/11/03 03:59:34:TM2009/11/03 03:59:34:TM--HK: 669768: Detector STIM deactivatedHK: 669768: Detector STIM deactivated

2009/11/03 03:59:34:TSTOK: : 020: all commands executed2009/11/03 03:59:34:TSTOK: : 020: all commands executed

2009/11/03 03:59:34:TFAIL: 669768: 030: No STIM counts availa2009/11/03 03:59:34:TFAIL: 669768: 030: No STIM counts available cannot test event count rates.ble cannot test event count rates.

2009/11/03 03:59:39:TSEQ : : 2009/11/03 03:59:39:TSEQ : : >>>:============================================================>>>:====================================================================================

2009/11/03 03:59:39:TSEQ : : >>>: Test Series Summary: 2009/11/03 03:59:39:TSEQ : : >>>: Test Series Summary: nightly_testsnightly_tests has been executing for 92 minuteshas been executing for 92 minutes

2009/11/03 03:59:39:TSEQ : : >>>: Summary results:2009/11/03 03:59:39:TSEQ : : >>>: Summary results:

2009/11/03 03:59:39:TSEQ : : >>>: 56 Failures and 2009/11/03 03:59:39:TSEQ : : >>>: 56 Failures and 12 Warnings in 33 Tests12 Warnings in 33 Tests

2009/11/03 03:59:39:TSEQ : : >>>: 82.50% tests complete2009/11/03 03:59:39:TSEQ : : >>>: 82.50% tests complete (33 of 40)(33 of 40)

2009/11/03 03:59:40:TSEQ : : 2009/11/03 03:59:40:TSEQ : : >>>:============================================================>>>:====================================================================================

2009/11/03 03:59:40:TSEQ : : 2009/11/03 03:59:40:TSEQ : : >>>:============================================================>>>:====================================================================================

2009/11/03 03:59:40:REMRK: : Attempting to start test s2009/11/03 03:59:40:REMRK: : Attempting to start test sequence: 0160_checkmemequence: 0160_checkmem

2009/11/03 03:59:40:REMRK: : Test sequence started: 0162009/11/03 03:59:40:REMRK: : Test sequence started: 0160_checkmem0_checkmem

� Log files from Nightly Script are analyzed in the morning


CI&T Case 2: Central Instrument Data CI&T Case 2: Central Instrument Data Processor (MMSProcessor (MMS--CIDP)CIDP)

�� Magnetospheric MultiScale (MMS) MissionMagnetospheric MultiScale (MMS) Mission�� Constellation of 4 identically spacecraft in Constellation of 4 identically spacecraft in

variably spaced tetrahedron (1 km to several variably spaced tetrahedron (1 km to several RREE))

�� Ground contacts must be multiplexed in time Ground contacts must be multiplexed in time in order to retrieve data from all 4 spacecraft in order to retrieve data from all 4 spacecraft each dayeach day

http://stp.gsfc.nasa.gov/missions/mms/mms.htm

�� Overall Objectives: To discover the Overall Objectives: To discover the detailed physics of the reconnection detailed physics of the reconnection process including its controlling factors, process including its controlling factors, its spatial distribution, and its temporal its spatial distribution, and its temporal behavior.behavior.


CI&T Case 2: MMSCI&T Case 2: MMS--CIDPCIDP

�� Layered Software Architecture benefits testingLayered Software Architecture benefits testing�� Architecture can be verified in layerArchitecture can be verified in layer

�� ““Nightly ScriptNightly Script”” development is indevelopment is in--progressprogress�� also based on Pythonalso based on Python


CI&T Case 2: MMSCI&T Case 2: MMS--CIDPCIDP

�� Plan for integration in Plan for integration in stages.stages.�� Using the same Host Using the same Host

environment, allows for environment, allows for progression and regressionprogression and regression

�� Need to configuration Need to configuration manage test environment manage test environment software as well as the software as well as the Flight Software!Flight Software!

�� We use CVSWe use CVS

Host Workstation (Linux)

Telemetry and

Command System

with Python

Interpreter

(GSEOS)

SpaceBel SPARC Sim.

CIDP

FSWTCP

Socket

SpW

I/F

Sim

Sim Input

C2I Worksation

(Linux)

Host Workstation

(Linux)

Telemetry and

Command System

with Python

Interpreter

(GSEOS)

TCP

Socket

CIDP to Inst. I/f

Card (PCI)

CIDP

FSW

Input

Sim /

C2I

I/F

Loopback

Engineering or

Flight

CIDPHost Workstation

(Linux)

Telemetry and

Command System

with Python

Interpreter

(GSEOS)

CIDP

FSW

S/C

Simulator

Instrument

Simulator

TCP

Socket

TCP

Socket

(IGSE

Prot.)

SpW

Serial

SerialSerial


Challenges AheadChallenges Ahead

�� Automated Continuous Testing generates large volumes of test datAutomated Continuous Testing generates large volumes of test dataa

�� Data analysis tools are necessary to make use of the test resultData analysis tools are necessary to make use of the test results s in a timely mannerin a timely manner

�� Currently test development significantly lags code developmentCurrently test development significantly lags code development

�� ““How can I test it if I have not built it yet?How can I test it if I have not built it yet?””

�� Looking for applications of TestLooking for applications of Test--Driven Development [2]Driven Development [2]

�� Test Code is developed before a feature is implementedTest Code is developed before a feature is implemented

�� Test is run to verify a FAILTest is run to verify a FAIL

�� Feature is implemented and test run againFeature is implemented and test run again

�� Currently good quality simulation environments are expensiveCurrently good quality simulation environments are expensive

�� Adaptations of public domain emulators (Adaptations of public domain emulators (BochsBochs, QEMU, etc.) , QEMU, etc.) provide promise to reducing costs here making complete virtual provide promise to reducing costs here making complete virtual software environments more affordablesoftware environments more affordable

�� Want to better integrate with Observatory I&TWant to better integrate with Observatory I&T


Questions?Questions?


ReferencesReferences

1.1. Martin Fowler. Martin Fowler. hhttp://www.martinfowler.com/articles/continuousIntegration.htmlttp://www.martinfowler.com/articles/continuousIntegration.html. .

2.2. Kent Beck.Kent Beck. TestTest--Driven Development: By ExampleDriven Development: By Example. Addison. Addison--Wesley. 2002.Wesley. 2002.

3.3. Ward Cunningham.Ward Cunningham.http://c2.com/cgi/wiki?IntegrationHellhttp://c2.com/cgi/wiki?IntegrationHell. .

establishing an environment for continuous integration and...

Documents