presentation at chep 2006

The LCG SPI project in LCG Phase II

CHEP’06, Mumbai, India

Feb. 14, 2006

Andreas Pfeiffer -- for the SPI team

[email protected]

14-Feb-2006 CHEP'06, Mumbai, India -- [email protected] 2

Overview

• Introduction and Motivation• Services of SPI

• Savannah service• External software and distribution services• Configuration and Build Infrastructure• Software testing and QA services• Documentation service

• Outlook and future plans• Summary


The SPI project in the LCG Applications Area

• Goal of the project • Provide to the development projects of the LCG Application Area

- Basic environment for physics SW development

- General scientific libraries and class libraries

- Software development tools

- Documentation tools and document templates

- Compiler expertise

• Have consistency and homogeneity in the development of the different packages of the LCG Application Area

- Configuration management

- Central software repositories

- Common policies (naming, building, etc.)


Software Process and Infrastructure

• Work guided and prioritized by Architects Forum• Providing services to the projects (and experiments)• Channelling the requests from projects (and experiments)

• Providing the following services:• Savannah service

- issue and task tracker

• Software services - central installations of external software, software distribution

• Software development service - configuration mgt, pre- and post-build service, tools for testing

and QA

• Web and Documentation


Savannah Service

• Web portal for LCG software projects• Single entry point for all projects

- Uniform “look and feel” for all projects

• Totally web based

• Used for various functionalities• Bug/issue tracking• Task management• Download area

• Customized from GNU savannah project • In collaboration with main developer


0

200

400

600

800

1000

1200

1400

1600

aug'02 feb'03 aug'03 feb'04 aug'04 feb'05 aug'05 feb'06

Adoption by projects

1575 users

0

50

100

150

200


176 projects

0

2000

4000

6000

8000

10000

12000

14000


500

1000

1500

2000

2500

3000


2770 tasks13500 bugs


Functional enhancements

• Project specific 'cookbooks' • based on the tracker engine• Inherits the tracker functionality while providing a friendly

book-like item display.

• Export facility • Helps extracting data to insert into reports (new releases,

statistics, project monitoring, etc)

• Voting system • Gives registered users the possibility to prioritize the way

items are handled

• Additional browsing criteria• Selection of items modified/unmodified since a certain date

• Extended transition mechanism• Automatic update of multiple fields on specific value

transitions


External Software Services

• Install software needed by LCG projects and experiments• Open source and public domain software

- Libraries and tools

• Standard structure of directories (central AFS repository)• <packageName>/<version>/<platform>/(lib, src, bin, …)

• Build and dependency information now stored in XML files with scripts to build (semi-)automatically • Very easy to build a new version or new platform• Easy to convert to formats for other tools (e.g. pacman)

• Experiments and projects decide what is installed• Through “Architects Forum”


External software service -- example


Software Distribution Service

• Existing solution since 2002• Script to download and install tar-balls of binaries for

software of LCG AA projects with all dependencies• Limited to binaries of platforms build at CERN• Dependency information from configuration management

and build tool (scram)

• Effort started to download source tarballs and (re-) build as needed• Simplifies porting to new platforms (or development work)• Re-using build and dependency information from XML files• Will be extended to download/install binaries

• Work ongoing to fully automate this service


Configuration and Build Infrastructure

• LCG Librarian• Maintaining configuration information centrally for build

systems in use- XML files + converters to scram and cmt

• Coordinating releases and pre-releases

• Librarians integration group• Discuss with experiments’ librarians/integrators additional

needs on external packages• Bi-weekly meeting (LIM)

• Developed simple tools to automate builds of LCG AA software• Three phases: pre-build, build (on all platforms in parallel),

post-build

• Work ongoing to improve automation• Provide tools for release managers in projects to do this


Software Testing Service

• Providing Test frameworks, support, policies and documentation• CppUnit, Oval• Qmtest• valgrind• On different compilers/platforms

• Running the tests is an integral part of the development in the projects• Unit tests, integration tests and regression tests

• Also available for experiments• Initial setup• User support


Quality Assurance Service

• Central QA checklist on each release of LCG AA projects• Code coverage• Providing statistics on tests and from savannah as a

coherent view

• Automated procedure to generate reports• At every release• Published on the SPI web site

• Evaluation and usage of external tools• E.g. RuleChecker, Logiscope


QA example (ROOT project)


QA example (POOL project)


Documentation Service

• Document LCG Policies• Homogenize the projects “look and feel”• CVS and release directory structures• Software testing policies

• Code documentation• Through SPI web site and wiki pages• Tools used: doxygen and LXR• Run for every release of LCG AA projects plus some

external packages (CLHEP, ROOT, HepMC)- Procedure is now semi-automated

• Internal procedures documented• “explicit knowledge” rather than “reading scripts”


Web and Documentation

• Maintain and improve existing web pages• Update internal and external documentation (wiki pages)• SPI and LCG Workbooks

• Automate content wherever possible• Create more “active/dynamic” web pages

- E.g.: status of external tools, configuration info- New: pages to do pre- and post- build procedures

• Update and clarify existing policies• To be reviewed by AF

• Update web pages for external packages• Responsibles, status, … for each package


Challenges for LCG phase II

• Servicing LHC experiments more than AA projects• Discussion with experiments to (re-)define scope

- What is needed, what can be improved

• Participation from people in experiments• “On-the-spot” help with requests from experiments• SPI provides a “frame” to do complex work

- Guidelines, AFS space, web space for docs, …

• Coping with strong (50%) reduction of resources • Optimization opportunities • Further optimization and automatization of procedures


Summary

• SPI provides a large set of stable services that fully use the current resources• Savannah, External Software, Testing Frameworks, Software

Distribution, Build and Release, QA Activities

• SPI work is guided by the Architects Forum• AF represents the users and steers our priorities

• In Phase II we will concentrate more on the experiments’ needs• Who see LCG AA project software as “external”

• Ongoing discussion with AF and experiments• To redefine the scope of the project

- Resources are significantly reduced


“Thank you!” to the SPI team(s)

• 2005• Alberto Aimar, Johanne Benard, Ian McLaren,

Yannick Patois, Yves Perrin, Andreas Pfeiffer, Eric Poinsignon

• 2006• Full time:

Ian McLaren, Yves Perrin, Andreas Pfeiffer• Part time:

Bertrand Bellenot, Manuel Gallas, Stefan Roiser


Additional Information

• SPI project web:• http://spi.cern.ch

• SPI project wiki pages:• https://twiki.cern.ch/twiki/bin/view/SPI/WebHome

• LCG Applications Area web:• http://lcgapp.cern.ch

• LCG web:• http://cern.ch/lcg

presentation at chep 2006

Documents

lcg projects

software of lcg aa projects

software testing service

development projects

external software services

software process

postbuild service

builds of lcg aa software