devops for dishwashers - yow! conferences · + full service consultancy - chips to cloud + iot,...
TRANSCRIPT
DevOps for DishwashersBringing grown-up practices to the Internet of Things
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Christopher Biggs, Accelerando Consulting
@unixbigot @accelerando_au
1 / 133
Who am I?
Christopher Biggs — @unixbigot — @accelerando_au
▷ Brisbane, Australia
▷ Former developer, architect, development manager
▷ Founder, Accelerando Consulting
▷ Full service consultancy - chips to cloud
▷ IoT, DevOps, Big Data
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017 2 / 133
Why Devops?
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Problems
3 / 133
Why Dishwashers?
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Problems
4 / 133
"Software is eating the world"
-- Mark Andreesen
Wall St Journal, Six years ago last month.
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Problems
5 / 133
Interlude: What do I mean by "DevOps"?
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017 6 / 133
DevOps is NOT THIS
DevOps is not a thing you do,
it's the way you do things.
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Problems DevOps?
7 / 133
DevOps is NOT THIS
Empower everyone
to maximise value.
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Problems DevOps?
8 / 133
"When every Thing is connected, Everything is connected"-- Me
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017 9 / 133
Welcome to the Internet of Things
pop. 10 Trillion
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape
10 / 133
1936 (Information Pandemic Year Zero)
10-7 device/person
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape
11 / 133
Mainframe era
10-4 device/person
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape
12 / 133
Minicomputer era
10-2 device/person
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape
13 / 133
Desktop era
100 device/person
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape
14 / 133
Mobile era
100.5 devices/person
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape
15 / 133
Cloud era [YOU ARE HERE]
~101 devices/person
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape
16 / 133
Internet of things
103 devices/person
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape
17 / 133
Solve the next problem, not thelast one
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape Challenges
18 / 133
Beware of false analogies and straight line trends
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape Challenges
19 / 133
Observe, Orient, Decide, Act
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape Challenges
20 / 133
"Bad people will break your stuff"
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape ChallengesLandscape Challenges Risks
Do you want to know more? "The Internet of Scary Things" christopher.biggs.id.au/talk
21 / 133
Everything is awful
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape ChallengesLandscape Challenges Risks
22 / 133
Everything is awful
and the awful is on recurl http://my-dishwasher/../../../../../../../../../../../../etc/shadow
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape ChallengesLandscape Challenges Risks
23 / 133
It's not rocket science
No really, I mean actual rockets.
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape Challenges
24 / 133
Desiderata
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
Agenda Landscape ChallengesLandscape Challenges Risks Desiderata
Select appropriate tools and platforms
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape Challenges
26 / 133
Comprehensive identity management
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape Challenges
27 / 133
Automate for developer and user convenience
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape Challenges
28 / 133
Testing and testability kept front-of-mind
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape Challenges
29 / 133
Train, and Audit, and keep doing both
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape Challenges
30 / 133
Monitor and react (automatically)
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape Challenges
31 / 133
Platforms
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017 32 / 133
People are more expensive than circuits
(Sorry, robots)
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape Challenges Solutions Platforms
33 / 133
Hardware is DevOps too
(Robots, I hope this makes it up to you :)
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape Challenges Solutions Platforms
34 / 133
Open and well supported
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape Challenges Solutions Platforms
35 / 133
Case study: ARM v7 and Debian Linux
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape Challenges Solutions Platforms
36 / 133
Meet the #3 top-sellingcomputer of all time
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape Challenges Solutions Platforms
37 / 133
Artisanal free-range small-batch Linux?
No.
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape Challenges Solutions Platforms
38 / 133
Without the Internet, it's just a Thing.
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape Challenges Solutions Platforms
39 / 133
"Is there anybody out there?"-- Pink Floyd
(also, my lighting controller)
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape Challenges Solutions Platforms
40 / 133
"Put the robot back in the ocean, kid."-- Oceanographers, everwhere
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape Challenges Solutions Platforms
41 / 133
Management is not a dirty word
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Agenda Landscape Challenges Solutions Platforms
42 / 133
Development
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017 43 / 133
Aside: Go Serverless
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev
44 / 133
Nice languages are portable, memory-safe and asynchronous.
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev
45 / 133
Case study: Javascript
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev
46 / 133
Case Study - Go
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
Landscape Challenges Solutions Platforms Dev
Naughty languages are like a tightrope over a pit of spikes.
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev
48 / 133
'The wi password is "abc123';cat /etc/passwd#" '
Say no to shell scripts.
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev
49 / 133
Use, and reuse, a framework.
Yours, mine, Google, Amazon, Microsoft, Apple, whatever.
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev
50 / 133
AWS IoT
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev
51 / 133
Choose your own framework adventure
▷ Amazon IoT and Greengrass
▷ Google IoT
▷ Azure IoT
▷ Open Connectivity Foundation IoTivity
▷ Resin.io
▷ Mongoose-OS
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev
52 / 133
Containment is complexity management
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev
53 / 133
You can run Docker on a $6.95 linux computer
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev
54 / 133
Use your CI to produce docker images as artifacts
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev
55 / 133
Testing
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017 56 / 133
Total Infrastructure Awareness
replicate your whole ecosystem
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA
57 / 133
No snow ake servers.
A dev team should have access to disposable instances of everything
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA
58 / 133
DevOps is a disaster, every day
and that's good
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA
59 / 133
Painful testing practices beget painfully bad testing
provide easy test data
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA
60 / 133
Quick xes are good
and cheap
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA
61 / 133
Listen to that annoying hipster tech blogger
At least four eyeballs per line.
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA
62 / 133
Test-before-merge
Never* "break the build"
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA
* well, almost never
63 / 133
Fail fast
Unit tests rst, followed by slower end-to-end tests
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA
64 / 133
Every pair of eyeballs costs $$$
(and no, you can't save $-½ by poking one out)
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA
65 / 133
Do not poke customers with sticks (either)
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA
(Wherein Christopher does math to "prove" a point)
66 / 133
A modern embedded system is faster than a Cray 1
But that's still reaaally slow
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA
67 / 133
Cross-platform CI pipelines
Option zero: cross-platform languages are win
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA
68 / 133
Cross-platform CI pipelines
Option one: Emulate
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA
69 / 133
Cross-platform CI pipelines
Option two: Enrol embedded systems in your CI
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA
70 / 133
Case Study: My Pipelines
Minimal requirement: one x86 and one ARM server (eg Raspberry Pi)
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA
71 / 133
Case Study: My Pipelines
Stage 1: common policy checks
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA
72 / 133
Case Study: My Pipelines
Stage 2: compile and package
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA
73 / 133
Case Study: My Pipelines
Stage 3: Testing (on target arch)
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA
74 / 133
Case Study: My Pipelines
Stage 4: Deploy to container registry
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA
75 / 133
Case Study: My Pipelines
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA
76 / 133
Case Study: My Pipelines# note the pi build job can be run on x86 because Go is awesomebuild for pi: stage: build script: - make installdeps image contents ARCH=pi artifacts: paths: - GPIOpower - GPIOpower_docker_pi.tar.gz - GPIOpower_contents_pi.tar.gz## Run tests on RasPi#test on pi: stage: test tags: - pi script: - make test
deploy to staging: stage: deploy environment: name: staging only: - masterscript:
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
Landscape Challenges Solutions Platforms Dev QA
Package separate lab and eld artifacts
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA
78 / 133
Defeat laziness by making it easier to do the right thing
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA
79 / 133
Regression tests, longitudinal tests
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA
80 / 133
Dashboards as "live tests"
Containerise your BI stack and write dashboards alongsidecode
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA
81 / 133
Dashboards learn "green" state and alert on red
Obligatory reference to Machine Learning
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA
Do you want to know more? "Continuous Dashboarding" christopher.biggs.id.au/talk
82 / 133
Deployment
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017 83 / 133
Orchestrate: Never doanything by hand.
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment
84 / 133
Build a provisioning work ow
Customise a clean OS (via ethernet or emulation)
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment
85 / 133
Robo-con gure the target system from a provisioning system
Then save a lesystem image
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment
86 / 133
How do you create a provisioning system?
Turtles all the way down!
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment
Do you want to know more? github.com/unixbigot/kevin
87 / 133
Case study - My orchestration scripts
1. Create a read only recovery partition
2. Install SaltStack orchestration minion (now switch protocols)
3. Set timezone, locale, etc.
4. Change default passwords
5. Configure network
6. Provision message bus clients
7. Install language runtimes (nodejs, java etc.) if needed
8. Configure VPN client
9. Fetch initial application containers
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
Landscape Challenges Solutions Platforms Dev QA Deployment
Zoom level 1: Device roles# top.sls
'G@roles:controller': - match: compound - salt.syndic - mqtt.relay - mqtt.client - net.hostapd_bridge - net.gpsd - fleetvalid.station-console 'G@roles:sensor': - match: compound - os.initramfs - os.hostname - net.aws.cli - docker - fleetvalid.docker_auth - fleetvalid.service-advertiser 'G@roles:inroad_sensor' - match: compound - fleetvalid.inroad_sensor
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment
89 / 133
Zoom level 2: Device pro le# fleetvalid/inroad_sensor.sls
include: - fleetvalid.aggregator - fleetvalid.rfid - fleetvalid.radar - fleetvalid.vibration - fleetvalid.rfidpower
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment
90 / 133
Zoom level 3: Device component# fleetvalid/rfid.sls
{% set image = reg.path + '/roadcurtain/' + rfid + 'rfid-' + pillar.fleetvalid.arch + ':' + pillar.fleetvalid
fleetvalid-rfid-image: docker_image.present: - name: {{image}}
fleetvalid-rfid: docker_container.running: - image: {{image}} - links: - fleetvalid-aggregator:aggregator - fleetvalid-rfidpower:power - binds: - {{rfid_device}}:{{rfid_device}} - environment: - RCAGGREGATOR: aggregator:9091 - POWER_API: power:80 - privileged: True
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment
91 / 133
Zoom level 4: System con gurationgpsd: pkg.installed: - pkgs: - gpsd - chrony
gpsd-device-configuration: file.replace: - name: /etc/default/gpsd - pattern: ^DEVICES=.* - repl: DEVICES="{{pillar.gps.device}}" - append_if_not_found: True
gpsd-running: service.running: - name: gpsd - enable: True
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment
92 / 133
Hey, that all sounds a bit like PaaS
Yeah, it does.
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment
93 / 133
Amazon AWS Greengrass
IoT PaaS built on AWS IOT + AWS Lambda
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment
94 / 133
Resin.io
IoT PaaS with Linux and Docker
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment
Do you want to know more? https://resin.io/
"The Internet of Scary Things" christopher.biggs.id.au/talk
95 / 133
Apache MyNewt
Embedded component-based OS for wireless sensors
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment
96 / 133
Mongoose-OS
Multiplatform embedded OS with cloud integration and remote upgrade
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment
Do you want to know more? http://mongoose-os.com/
"IoT in two Minutes" christopher.biggs.id.au/talk "Javascript Rules My Life (CampJS 2017)" christopher.biggs.id.au/talk
97 / 133
Automate PKI enrolment
IoT Makes PKI Easy.
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment
-er
98 / 133
Your "secure key distribution channel" is a cardboard box
1. Build and sign a root certificate
2. Upload root cert to the SaltStack master
3. Create minion certificate
4. Install minion certificate on minion
5. Upload a copy to the master
6. All automatically
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment
99 / 133
Containerised version control
Use your docker registry the way it was intended
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment
100 / 133
Use Salt "grains" to de ne which container to use(i.e. live, staging, dev or other)
fleetvalid-rfid-image: docker_image.present: - name: ((image)) - force: True
fleetvalid-rfid: docker_container.running: - image: ((image)) - links: - fleetvalid-aggregator:aggregator - fleetvalid-rfidpower:power - binds: - ((rfid_device)):((rfid_device)) - environment: - RCAGGREGATOR: aggregator:9091 - POWER_API: power:80
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment
101 / 133
Maintenance
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017 102 / 133
Self-care
FLASH memory longevity tweaks
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance
103 / 133
Boot to ramdisk
Sanity check, then proceed to target environment
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance
104 / 133
Recovery mode
(ab)use DHCP
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance
105 / 133
Liveness monitoring
If a device goes silent, notify the site custodian
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance
106 / 133
Sickness monitoring
Use an audible or visual attention signal (think smoke-alarms)
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance
107 / 133
While you were sleeping
Intermittent connections
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance
108 / 133
Kill or Cure
Feature/component disable
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance
109 / 133
Kill or Cure
The Cassini Solution
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance
110 / 133
Monitoring (platform data)
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017 111 / 133
Heartbeats
SaltStack's presence monitor
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring
112 / 133
Flatliners
Detect missing devices
(i.e. known to saltstack but not connected)
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring
113 / 133
Health stats
SaltStack beacons - cpu/memory, often
Full process list, less often
Network stats
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring
114 / 133
Pour it all into a data lake
And pretend to be a Bond Villain
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring
Do you want to know more? "Continuous Dashboarding" christopher.biggs.id.au/talk
115 / 133
Case study: Log pooling for a building safety startup
Ram disk on local ARM devices
Streaming to cloud with Filebeat
Processing with Logstash
Set a storage budget and expire to meet the budget
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring
116 / 133
Measurement (application data)
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017 117 / 133
Use orchestration message bus
SaltStack message bus is the fast, lightweight ZeroMQ
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement
118 / 133
Can you use your orchestration bus for application events?
Yes, with care
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement
119 / 133
Extend orchestration system with custom modules
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement
120 / 133
Record as much as you can, digest later
Shove all your client data in ElasticSearch
Purge oldest indexes until CFO stops whinging
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement
121 / 133
Case study: Saltstack plus ELK
Bridge orchestration bus to application message bus
Engine module at top level master (or intermediate)
Gateway messages to elasticsearch, via logstash
Want MQTT? You already built a PKI to deploy it in 2 minutes
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement
122 / 133
Case study: MQTT plus ELK
"Rapids Rivers Ponds"
MQTT brokers at each site
Broker in the cloud federates with on-site brokers
Logstash MQTT plugin subscribes to all events
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement
Do you want to know more? "Implementing Microservice Architectures" Fred George, YOW 2014
123 / 133
Visualisation
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017 124 / 133
Real time status
Liveness, resources, environment
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement Visualisation
125 / 133
Measure your KPIs
Whatever makes you money, count it
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement Visualisation
126 / 133
Measure your KPIs
Set high and low water marks, alert on them
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement Visualisation
127 / 133
Measure your KPIs
Pay: Elastic and other vendors have commercial alert engines
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement Visualisation
128 / 133
Measure your KPIs
Free: Node-RED makes a good FOSS alerting engine
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement Visualisation
Do you want to know more? "Continuous Dashboarding" christopher.biggs.id.au/talk
129 / 133
Longitudinal comparisons
View long-term trends in KPIs
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement Visualisation
130 / 133
Longitudinal comparisons
Pay attention to device longevity, wear, etc.
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Platforms Dev QA Deployment Maintenance Monitoring Measurement Visualisation
131 / 133
Summary▷ Lots of devices, too many to administer by hand
▷ Swimming in a soup of malware and bad actors
▷ Choose tools that support quality
▷ Pipelines for automated build/test/stage
▷ (Ab)use traditional cloud management tools for IoT Fleet
▷ Message bus all the things
▷ Big data now, play later
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWC d S 2017
Landscape Challenges Solutions Coda Summary
Resources, QuestionsMy SaltStack rules for IoT - github.com/unixbigot/kevin
Related talks - http://christopher.biggs.id.au/#talks
Me - Christopher Biggs
Twitter: @unixbigot
Email: [email protected]
Slides, and getting my advice: http://christopher.biggs.id.au/
Accelerando Consulting - IoT, DevOps, Big Data - https://accelerando.com.au/
Christopher Biggs — DevOps for Dishwashers @unixbigot @accelerando_au
#YOWConnected Sep 2017
Landscape Challenges Solutions Coda Summary Resources
133 / 133