martin d'anjou oleg nenashev alexandru somai mentors student · google summer of code 2016 @...
TRANSCRIPT
![Page 1: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/1.jpg)
Mentors StudentOleg Nenashev Alexandru SomaiMartin d'Anjou
External Workspace Manager Plugin
1
Google Summer of Code 2016 @ Jenkins Project
25 August 2016
![Page 2: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/2.jpg)
About myself
alexsomai
@alex_somai
2
● GSoC Student at Jenkins project● Major in Software Engineering at
Babes-Bolyai University of Cluj-Napoca, Romania
![Page 3: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/3.jpg)
Background● >2 years experience in Software Development● Coding in Java, Groovy, JavaScript● Interested in Spring, Hibernate, Web Services● Build tools and CI: Maven, Jenkins
3
alexsomai
@alex_somai
![Page 4: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/4.jpg)
The problem● Difficult to reuse the same workspace for multiple jobs● e.g. Running parallel testing across nodes
4
![Page 5: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/5.jpg)
Possible solutions● Stash/unstash pre-made artifacts (pipeline only)● Copy Artifacts plugin● Advanced job setting
○ custom workspace (freestyle)○ ws (pipeline)
5
![Page 6: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/6.jpg)
About the project● Jenkins plugin● Focus on Pipeline jobs● Share and reuse workspaces across multiple jobs
○ No need to copy, archive or move files
6
![Page 7: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/7.jpg)
7
Concept
![Page 8: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/8.jpg)
8
Infrastructure
![Page 9: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/9.jpg)
Scope of work
9
External Workspace Manager Plugin
Workspace Cleanup Plugin
Run Selector Plugin
Job Restrictions Plugin
Pipeline examples
Acceptance Test Harness
NEW!
NEW!
Pipeline support
Minor: default restriction adjustments
Inherited from Copy Artifact, serious rework@Symbol support
![Page 10: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/10.jpg)
Basic features● Disk configuration
○ Jenkins global config○ Node config
● Workspace reuse in the same Pipeline job● Workspace reuse in multiple Pipeline jobs
○ Uses the Run Selector Plugin● Works on both Unix and Windows systems
10
![Page 11: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/11.jpg)
11
Jenkins global config
Disk 1
Disk 2
![Page 12: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/12.jpg)
12
Jenkins global configNode 1 config
![Page 13: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/13.jpg)
13
Node 2 configJenkins global config
![Page 14: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/14.jpg)
Example 1. Usage in a single jobdef extWorkspace = exwsAllocate 'diskpool1'node('linux') { exws(extWorkspace) {
checkout scmsh 'mvn clean install -DskipTests'
} }node('test') { exws(extWorkspace) { sh 'mvn test'
}}
14
![Page 15: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/15.jpg)
15
Example 2. Upstream/Downstream
Allocate
Upstream job
Build
Trigger
MetadataAllocate from
upstream
Downstream job
Test
Select triggering build
Data
![Page 16: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/16.jpg)
def extWorkspace = exwsAllocate 'diskpool1'
node('linux') {
exws(extWorkspace) {
checkout scm
sh 'mvn clean install -DskipTests'
}
}
build 'downstream-job'
16
Example 2. Upstream job● No changes in External Workspace commands!
![Page 17: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/17.jpg)
17
Example 2. Downstream job
def run = selectRun job: 'upstream-job',
selector: triggering()
def extWorkspace = exwsAllocate selectedRun: run
node('test') {
exws(extWorkspace) {
sh 'mvn test'
}
}
● Run Selector Plugin picks the upstream build
![Page 18: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/18.jpg)
Demo part 1
18
![Page 19: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/19.jpg)
19
Advanced features● Workspace cleanup
○ Uses Workspace Cleanup Plugin● Provide custom workspace path
○ By default /mountPoint/pathOnDisk/$JOB_NAME/$BUILD_NUMBER
○ Override■ Jenkins global config, for each Disk Pool■ In the Pipeline script
![Page 20: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/20.jpg)
20
Example 3. Workspace cleanupdef extWorkspace = exwsAllocate 'diskpool1'node('linux') {
exws(extWorkspace) {try {
checkout scmsh 'mvn clean install'
} catch(e) {currentBuild.result = 'FAILURE'
} finally {step ([$class: 'WsCleanup', cleanWhenFailure: false])
}}
}
● Workspace Cleanup Plugin version 0.30
![Page 21: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/21.jpg)
21
Example 4. Custom workspace pathdef customPath =
"${env.JOB_NAME}/${PR_NUMBER}/${env.BUILD_NUMBER}"
def extWorkspace = exwsAllocate diskPoolId: 'diskpool1', path: customPath
node('linux') {exws(extWorkspace) {
checkout scmsh 'mvn clean install'
}}
![Page 22: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/22.jpg)
Demo part 2
22
![Page 23: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/23.jpg)
More features● Disk Pool restrictions
○ Restrict a Disk Pool to be allocated only for specific jobs○ Integrates Job Restrictions Plugin
● Flexible Disk allocation strategies○ Select the disk with the most usable space○ Select the disk with the fastest write/read speed
23
![Page 24: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/24.jpg)
24
Example 5. Disk Pool Restrictions
![Page 25: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/25.jpg)
25
Example 6. Disk allocation strategy
def extWorkspace = exwsAllocate diskPoolId: 'diskpool1', strategy: fastestReadSpeed(estimatedWorkspaceSize: 100)
node('linux') {exws(extWorkspace) {
checkout scmsh 'mvn clean install'
}}
![Page 26: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/26.jpg)
Demo part 3
26
![Page 27: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/27.jpg)
Completed work summary - Phase 2
● Decouple BuildSelector extension point from Copy Artifact to Run Selector Plugin○ https://github.com/jenkinsci/run-selector-plugin
● Make Workspace Cleanup Plugin compatible with Pipeline○ See PR #27
● Custom workspace path feature
27
Phase 2 EPIC JENKINS-35971 Planned Achieved
![Page 28: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/28.jpg)
Completed work summary - Phase 2
● Disk Pool restrictions● Extension points
○ DiskInfoProvider & DiskAllocationStrategy● Unit tests● Acceptance tests #172● Jenkinsfile - build on ci.jenkins.io● Stable release● Details on https://alexsomai.github.io/gsoc-2016/
28
Planned Achieved
![Page 29: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/29.jpg)
Stable release
● Stable version has been released!
● Usage guidelines on the plugin’s README page● Feedback is welcome on Jira and Gitter
29
![Page 30: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/30.jpg)
Community interaction● Set up public meetings (May to August 2016)
○ https://jenkins.io/hangout● Status updates on dev mailing list● Discussions in IRC and Gitter
○ https://gitter.im/jenkinsci/external-workspace-manager-plugin
● Blog posts about plugin releases○ https://jenkins.io/blog/2016/06/30/ewm-alpha-version/○ https://jenkins.io/blog/2016/08/09/ewm-beta-version/○ https://jenkins.io/blog/2016/08/22/ewm-stable-release/
30
![Page 31: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/31.jpg)
Future work
● WIP: Workspace browsing feature ○ See PR #37
31
Phase 3 EPIC JENKINS-37543
![Page 32: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/32.jpg)
Future work● Integrate fingerprints
● Integration with external disk providers JENKINS-36072 ○ At least one Extension point in the plugin○ Candidates: Amazon EBS, Google Cloud Storage 32
![Page 33: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/33.jpg)
Link summary● External Workspace Manager repository
○ https://github.com/jenkinsci/external-workspace-manager-plugin
● Run Selector repository○ https://github.com/jenkinsci/run-selector-plugin
● Gitter○ https://gitter.im/jenkinsci/external-workspace-manager-plugin
● Work product page○ https://alexsomai.github.io/gsoc-2016/
● Blog posts○ https://jenkins.io/blog/2016/06/30/ewm-alpha-version/○ https://jenkins.io/blog/2016/08/09/ewm-beta-version/○ https://jenkins.io/blog/2016/08/22/ewm-stable-release/
● Phase 2 & 3 EPICs JENKINS-35971 JENKINS-37543 33
![Page 34: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/34.jpg)
Join me at Jenkins World● Day 2: Wednesday, September 14
●
● Community booth○ Demo Thursday 12:45 - 13:30
34
![Page 35: Martin d'Anjou Oleg Nenashev Alexandru Somai Mentors Student · Google Summer of Code 2016 @ Jenkins Project ... NEW! NEW! Pipeline support Minor: default restriction adjustments](https://reader035.vdocuments.site/reader035/viewer/2022070711/5ecade651362536a27461679/html5/thumbnails/35.jpg)
Q&A
35https://gitter.im/jenkinsci/external-workspace-manager-plugin