Tools and software process for the FLP prototype

B. von Haller

9. June 2015

CERN

Overview

▶Scope ▶FLP prototype▶O2

▶Software development ▶Software quality ▶Software deployment ▶Continuous Integration

▶Specific to FLP proto :

B. von Haller | CWG11 | 09.06.2015 2

▶Operating Systems▶SLC6, CC7, Ubuntu, Mac

▶Compilers▶Gcc, llvm, icc

▶ Languages▶Low level : C▶Middle level : C++11, stl, boost libraries, ROOT6▶Open to higher level languages (e.g. python)

▶Expose interfaces using generators such as SWIG

▶Web : PHP or Python with Django. To be decided

Ecosystem

B. von Haller | CWG11 | 09.06.2015 3

OnlyLimited to these 2

Software development

▶Tools▶Version control system: Git with GitLab/GitHub▶Software build system: CMake ▶Issue tracking system: JIRA▶Code formatter: clang-format

▶Procedures▶Workflows A la GitHub (pull request, see next

slides)

B. von Haller | CWG11 | 09.06.2015 4

GitLab to be gradually

introduced

Software development GitHub development workflow

B. von Haller | CWG11 | 09.06.2015 5

▶ Collaborators have a full remote clone of the main repo, called a fork▶ Full rights on their fork, none on the main repo▶ Anything in the master branch is always deployable.▶ Owners of the main repo (librarians) comment and accept/reject the

pull requests▶ Open source clone of GitHub: GitLabs

Software developmentGitHub/Gitlab

B. von Haller | CWG11 | 09.06.2015 6

▶Advantages of GitHub/GitLab and the related development workflow▶Collaborative and intuitive web interface▶Code always reviewed and commented before

being accepted▶Code can be checked automatically by tools before

human reviews▶Feature-rich api

Software development

▶ Procedures (continued)▶ Workflows A la GitHub (pull request, see next slides)▶ Coding conventions, naming, formatting -> CWG2▶ Code access policy Open (license still under discussion) -> CWG2

▶ Repositories ▶ Neither monolithic nor excessively fragmented▶ Group components that share same dependencies and/or similar functionalities

▶ Packages▶ Does not always match with repositories (also more flexible)▶ Release pace or common dependent packages to take into account

▶ Dependency handling for developers▶ External : original + patch -> recompile (tool to be decided) ; packages▶ Internal : packages (avoid recompiling what does not need to) ▶ CernVM-FS

B. von Haller | CWG11 | 09.06.2015 7

Software quality

▶Tools▶Static analysis -> Clang Static Analyzer▶Dynamic analysis -> Valgrind▶Tool for code reviews -> GitLab▶Code conventions and formatting enforcer ->

clang-format

B. von Haller | CWG11 | 09.06.2015 8

Not initially

Software quality

▶Testing▶Unit tests, usually at the function level

▶CTest and boost.test

▶Functional tests ▶Integration tests, software modules are combined

and tested as a group. Full chain. ▶Requires the code to be designed for it and to provide

mock systems at the interfaces▶Environment must be controlled

B. von Haller | CWG11 | 09.06.2015 9

Software deployment

▶Packaging▶CPack or fpm (to be decided, evaluation ongoing)

▶Dependency manager for users▶Packages using native system offered by OS▶CernVM-FS

B. von Haller | CWG11 | 09.06.2015 10

Continuous Integration (CI)

▶Actually wider than pure CI▶ Inter connects development, quality and deployment

▶ Jenkins▶Extensible, open source CI server▶Hundreds plugins, wide industry support▶Can do many type of tasks (e.g. build, test, release)▶On scheduled time or following a trigger (e.g. pull

request)▶API to build our own tools in front or make other tools

collaborate (e.g. JIRA)

B. von Haller | CWG11 | 09.06.2015 11

Continuous Integration

▶Master hosted in IT for ALICE on a VM▶Use VMs with Docker on Openstack▶Authentication using CERN SSO and egroups▶Mesos deployment containers to deal with

non-homogeneous tasks and load▶Integration with JIRA and GitHub/GitLab▶Time scale : infrastructure ready this summer

Jenkins for O2

B. von Haller | CWG11 | 09.06.2015 12

Conclusion

▶Tools have been chosen and procedures defined to ensure ▶Code quality, readability and maintainability▶Products delivery, stability and performance

▶The FLP prototype development can start in July directly within this frame

B. von Haller | CWG11 | 09.06.2015 13