jenkins pipeline @ scale. building automation frameworks for systems integration

Jenkins Pipeline @ ScaleBuilding Automation Frameworks for Systems Integration

CloudBees | Jenkins Automotive and Embedded DayBöblingen, April 05, 2017

Oleg Nenashev, CloudBees

About me. Hardware/Embedded

@oleg_nenashev

oleg-nenashev LibreCoresSt. Petersburg Polytechnic University

About me. Jenkins project

@oleg_nenashev

oleg-nenashev

Oleg’s “Hall of Shame”(c)

• Jenkins Core• Remoting• Plugins

http://bit.ly/nenashev-automotive-day-2016

ØHow/why do we build frameworks for Embedded?ØWhat has changed in Pipeline since 2016?ØHow to effectively use and develop Pipeline?ØLessons learnt

Agenda

• Pipeline 101• Pipeline Generic Features

• Blue Ocean• Declarative Pipeline

• Automation Infrastructure• Agents, Tools, Hardware Peripherals

• Added-value Pipeline features in CloudBees Jenkins Enterprise

What is NOT in the talk?

Ask the Experts,Jenkins Online Meetup

Birds of Feather,Ask the Experts

What is a typical Embedded Product?

`Hardware

External IPFirmware / Embedded

External SW

Development tools / EDAs

Demo software Demo hardware

Developer kits

“Main product”

Documentation

Embedded area. Welcome to the automation hell!

• Almost everything needs to be automated

• Multiple technologies and tools

• Integration testing is important

• Unstable hardware & networks

• Legacy stuff is everywhere

• …

OK, Automotive…

Source: http://www.popsautoelectric.com/auto-electric/

• System of systems• High level of integration• Components by 3rd

parties• Many standards• High quality requirements

1. The most popular CI automation server in the world

2. It’s a framework => flexible• Swiss Army Knife of Automation

3. It’s open source => customizable4. Commercial support is available5. Useful plugins

Why Jenkins?

Система

Plugins Configs

Infra-structure

SamplesDemos

How do we buildCI Systems?

Infra-structure

• Agents (FKA ”slaves”)• Tools• Hardware peripherals• Computing grids

Approach: Using “classic” Job types:• Freestyle, Matrix, JobDSL, …

Classic Approaches

Keywords:• Script steps to run the logic• Custom Tools Plugin to manage tools• Throttle Concurrent Builds and Lockable

Resources to dispatch access to hardware• Naginator Plugin to restart builds on-demand• Parameterized Trigger / Copy artifacts to

parallelize tests and to collect results

JUC2015/London

https://jenkins.io/solutions/embedded/

Classic Approaches

Spaghetti Automation

• Barely traceable dependencies

• Complex build flow• Complex

configurations • Duplication

• Expensive maintenance

ü Store build definitions with the projectü Track changes together with the projectü Tests together with the project

Configuration as Code

Configuration as Code in Jenkins

Job ConfigurationJenkins system configuration

Jenkins System Configuration… as Code

External Tools

Jenkins CLI and REST

python-jenkins

jenkins-client (java)

Configuration Management

systems

Ansible, Chef

Docker

Solutions within Jenkins

project

Groovy Boot Hooks

Scriptler Plugin

SCM Sync Configuration

UnstableJust examples…

Configuration as Code for jobs

Job DSL

Job Builder Plugin

Jenkins PipelinePipeline is offered by default in Jenkins 2.0

Groovy

Pipeline is a…Ø Groovy DSL for automatic flow description *

• Alternatives – Job DSL, Jenkins Job BuilderØ Job type in Jenkins (Pipeline Job)

• Job definitions can be stored in SCM• Automatic job creation from repositories and organizations

Ø Ecosystem for automation as code• ~20 dedicated plugins• Integration in many other plugins• Suggested by default in Jenkins 2

// Run on a node having the “my_fpga” labelnode("linux && ml509") { git url:"http://github.com/myorg/myproject.git"sh "make all"

} http://bit.ly/pipeline-tutorial

Why Jenkins Pipeline?

• Robustness - master restarts & network

• Parallelization across multiple nodes:• parallel()• node()

• Ability to store the shared logic in standalone Pipeline libraries

• Security

• Two modes: Scripted and Declarative

• Visualization: Stage View, Blue Ocean

Система

Plugins Configs

Infra-structure

SamplesDemos

Automation Framework Vision

Infra-structure

Infrastructure as Code

Configuration as Code

Documentation as Code

Pipeline Libraries

Pipelines

Developer Experience

• Encapsulation• Simple UI• Most of the code is in

GitHub• No need in classic UI

for 90% of users• Notifications to

email/GitHub

Jenkins Pipeline in Embedded2016

http://bit.ly/nenashev-automotive-day-2016

2016 – Summary

• Jenkins can be used to run tests on hardware in embedded and automotive projects

• Jenkins Pipeline is helpful for test runs• Pipeline lacks some critical features

• Management of resources.• Integration with Throttle Concurrent Builds plugin• Use-case: limiting the license utilization in automation

infrastructures (JENKINS-31801)

• Better support of tools.• Common build flow – dozens of tools• Integration with Custom Tools Plugin seems to be the best option,

especially in the case of multiple tool versions (JENKINS-30680)

• Pipeline package manager (JENKINS-34186)• Lack of plugins for EDA tools in Jenkins, code duplications• A common approach is to keep the shared "functions" in libraries• Pipeline Global Library and Pipeline Remote Loader can be used

2016 – Pipeline improvement areas

From my talk at CloudBees |

Jenkins Automotive and Embedded Day

• Management of resources.• Integration with Throttle Concurrent Builds plugin• Use-case: limiting the license utilization in automation

infrastructures (JENKINS-31801)

• Better support of tools.• Common build flow – dozens of tools• Integration with Custom Tools Plugin seems to be the best option,

especially in the case of multiple tool versions (JENKINS-30680)

• Pipeline package manager (JENKINS-34186)• Lack of plugins for EDA tools in Jenkins, code duplications• A common approach is to keep the shared "functions" in libraries• Pipeline Global Library and Pipeline Remote Loader can be used

2016 – Pipeline improvement areas

From my talk at CloudBees |

Jenkins Automotive and Embedded Day

Management of resources• Lockable Resources Plugin – since 2016• Throttle Concurrent Builds – coming soon• Pipeline Milestones

PARTIAL - Better support of tools

• withEnv() for environment variables• tool() step in Pipeline was improved

• Tool reference in the Job• Section in Declarative Pipeline

• Custom tools is not fully integrated• Follow JENKINS-30680• ETA – Q2, 2017

DONE – Pipeline Libs (Pipeline Package Manager)

• Early 2016 • Global Pipeline Library• Pipeline Remote Loader Plugin

• Late 2016• Pipeline Shared Libraries• Support of @Grab – Pipeline libs

can pull common Groovy libs

Library Structure:

• Pipeline Shared Libraries• Libs are located in SCM• Versioning• Definition for scopes:▸ Jenkins system-wide▸Folder▸ Job

Pipeline Library Definition [1/2]

Pipeline Library Definition [2/2]

Using Pipeline Libraries

• Implicit Loading• Via Jenkins Global or Folder configuration

• Explicit Loading• @Library

Using Pipeline Libraries

com.mycorp.libs.make

Jenkinsfile

makeProject()

makeProject() implementation - global variable

com.mycorp.libs.coverity

com.mycorp.libs.cppunit

Globally-defined library, automatic

update

@Library() definition

Early adoption of any framework

http://devopsreactions.tumblr.com/post/150674592321/petting-your-incomprehensible-piece-of-code

• It is working!• In several pet projects…

Lessons Learnt.What does NOT work well?

• Development Tools• Non-Success Path handling• Declarative Pipeline• Manual release management

Pipeline Developer Experience

• No Debugger• No IDE• No built-in test framework• Limited static analysis features

•Pipeline DSL != Groovy

How do I work?

Intellij IDEAFilesystem SCM Plugin (with patch)

Disclaimer: There may be better ways

• Syntax file• Built-in documentation

Jenkins Test Instance

Source Code (local .git repos)

• Production System Snapshot

• Custom Pipeline Library Settings

• Repositories for libraries• Jenkinsfile repos

Getting the Syntax File

Filesystem SCM Plugin

• Pending pull request:• https://github.com/jenkinsci/filesystem_scm-plugin/pull/2

Library definition in the test instance

Pipeline Testing

• No standard test framework offered by the Jenkins project• External project: https://github.com/lesfurets/JenkinsPipelineUnit

“mvn verify” has been invoked…

And the execution passed…

For the source Jenkinsfile…

Error Handling

• IMHO: Pipeline is designed for Success Path scenarios

When it “Just works”

When it does not

Error Handling. Top-levelScripted Pipeline Declarative Pipeline

• Global try/catch()

When it does not…

Error Handling

When it “Just works”

When it does not

The interesting part…Failover, repeat

Error Handling. Hardware failover

• Pipeline allows re-running unreliable stages on other nodes.

System failure(script returns 253)

Node #1 Node #2. . .

for (def board : boards) {echo "trying board " + board;try {node(board) {checkout scmsh ‘./bin/run.sh’ // Call passed => DONEbreak;

}} catch (Exception ex) {if (ex.message.contains

("exit code 253")) {// Fatal errorfail("Test run failed")

Try all possible boards:

Error handling. Hardware failover

for (def board : boards) {echo "trying board " + board;try {node(board) {checkout scmsh ‘./bin/run.sh’ // Call passed => DONEbreak;

}} catch (Exception ex) {if (ex.message.contains

("exit code 255")) {// Fatal errorfail("Test run failed")

• Hard to extract error cause• Special handling is required for

every case• Cannot just use existing parsers

• Build Failure Analyzer• Log Parser• Warnings

• Try-catch blocks pollute the code• Hard to test

Context retention

• No way to propagate any data• Solution - @NonCPS steps within the library steps

Error handling. Hardware failover

Declarative Pipeline

Encapsulation via Libraries

Simple use-case

• Exception try/catch• timeout() wrapper• Proper failover• Logic in @NonCPS

Declarative Pipeline

• It is great for simple use-cases• Publishers & Notifiers

• We do not need a framework for simple use-cases

• Lack of features• “How do I pass option for the Docker Agent”• “How do I declare a resource requirement for Throttling?”• …

• Scripted Pipeline is a recommended fallback, but there is no automatic migration

Declarative Pipeline. Recommendations

• Use Declarative Pipeline with caution• Complex agent definitions – use Scripted Pipeline• Complex throttling – use Scripted Pipeline• Post-build steps – work well• Tool definitions – work well• …

Pipeline vs. Application Lifecycle

BUT:• Non-CD process with manual promotion• Dependent products• Snowflake development stages

Solutions

• Input() step + milestone() for synchronization• Multiple Pipelines

• Release() step is coming soon• Release Plugin

Pipeline 1 Pipeline 2Run Selector

Staging Artifact Storage

External ALM tool

CD for the Component 1

CI/CD for the Component 2

Lessons learnt• No area-specific issues so far• High barrier to entry• It worth investment

What I need? 2017

• Development Tools for Pipeline• IDE, Debugger, Test Tools

• Community Pipeline Libraries• Common Embedded cases without plugins

• Promotion engine for Pipeline• Ideally: Merge of standard Promoted Builds and Release plugins

• Plugins for area-specific tools

Takeaways

• Jenkins Pipeline is a powerful engine for highly-customized Jenkins instances

• Jenkins Pipeline can be effectively used in Embedded/Automotive use-cases

• Barrier to entry is still high

More info

• Embedded solutions page:• https://jenkins.io/solutions/embedded/

• Links to the articles and videos• Some categorization

Embedded Solutions Page

• Jenkins project:• Website: https://jenkins.io• Embedded Solutions page: https://jenkins.io/solutions/embedded/

• Pipeline:• Tutorial: https://github.com/jenkinsci/workflow-plugin/blob/master/TUTORIAL.md• Compatibility: https://github.com/jenkinsci/pipeline-plugin/blob/master/COMPATIBILITY.md• Examples: https://github.com/jenkinsci/pipeline-examples

• CloudBees website:• Website: https://www.cloudbees.com• CloudBees Jenkins Platform: https://www.cloudbees.com/products/cloudbees-jenkins-platform

Useful Links

Thank you!

Contacts:

E-mail: onenashev@cloudbees.com

GitHub: oleg-nenashev

Twitter: @oleg_nenashev

Software at the speed of ideas

THANK YOU!www.cloudbees.com

jenkins pipeline @ scale. building automation frameworks for systems integration

Software

devops ccsa checkpoint · ci/cd: introduction to ci/cd...

hacking jenkins! - pass the salt conference•pipeline...

ci & cd pipeline with aws ecs & jenkins -...

how to do continuous delivery with jenkins pipeline ... how...

snowcamp io grenoble 2017 - bâtissons ensemble un pipeline...

ci/cd pipeline-as-code with jenkins and docker

juc europe 2015: scaling of jenkins pipeline creation and...

pipeline: continuous delivery as code in jenkins 2.0

scale your jenkins pipeline - fosdem...scale your jenkins...

pipeline-as-code mit jenkins pipeline - doag.org · pdf...

pipeline as code - new feature in jenkins 2

scaling your jenkins ci pipeline

deployment pipeline mit maven, chef und jenkins

continuous integration (ci) pipeline with git, jenkins...

automated deployment pipeline using jenkins, puppet,...

efficient performance test automation - opitmizing the...

jenkins imperative pipeline vs declarative pipeline

pipeline based deployments on jenkins

continuous delivery pipeline with docker and jenkins

sd devops meet-up - jenkins 2.0 and pipeline-as-code