Continuous Integration and Delivery at Shapeways

Matt Boyle NordicAPIs Platform SummitStockholm, Sweden25 October 2016

What is Shapeways?The worlds largest 3d printing service and marketplace

Marketplace, service hub, and community hosted at Shapeways.com

REST API provided at api.shapeways.com

Why CI/CD?

Idea

Exploration

Definition

Execution

Release

Evaluation

Consider the SDLC

…except this one

All stages represent active, valuable work by software team members.

Releasing Software Starts Easy

Everything Working?

Releasing Software Gets Harder

Git pull Build FrontEnd?

Manage Symlinks?

Bounce Memcache

?

Restart Apache?

Releasing Software Gets REALLY hardBuild Artifact

Run tests

Deploy to multiple

datacenters

Intra-DC deployment to

servers

Manage artifacts on

serverRotate through load balancer

Generate documentation

Restart services

Detect regressions

Monitor for Breakages Deployments like this take time and

resources

If you don’t automate, you’ll spend as much time running deployments as you do writing code

The Shapeways CI Story

In The Beginning…

60 minute deployments No API documentation Frustrating Manual Testing

Baby Steps

Continuous Integration with Jenkins

Integration + Functional Testing with

Selenium

New tests required to ship features

Pro’ing up - JenkinsActive-Active Jenkins instances – one per datacenter

Use Jenkins API for remote execution of deployments and test jobs

Flow-based deployment using Jenkins Pipeline DDL

Pro’ing up - Coyote FrameworkIn-house FOSS testing framework for integration and functional testing

Thoughtfully developed testing infrastructure to ensure consistent, reliable, and maintainable tests

Two-level API testing: contract enforcement and functional flow

Pro’ing up - ACMEIn-house developed test management application

Schedule test runs on-demand for developers

Test analysis and intelligence

Pro’ing up - ChatOps via HubotGitHub-developed extensible chat bot written in CoffeeScript

Manage deployments, development/QA infrastructure, permission management

Acts as interface between our different tools: jenkins, coyote, acme, slack

Looking Ahead – Staged RolloutsPresent deployment is fast but requires us to be all-in on new code

Ability to use infrastructure for new feature A/B testing rather than user groups

Increased deployment time, but faster recovery should something go wrong

Looking Ahead - Swagger for all APIs, int+extEasier documentation generation

Client generation/update for multiple languages

Source of truth for service-based applications – contract enforcement, test generation

Looking Ahead - Improved Testing ToolingGenerate testing profiles based on changes made in codebase

Code coverage for functional testing via URL mapping

Performance and capacity measurement based on previous test runs and environments