andrea corna - openinfraday.it · andrea corna tommaso sardelli dashboard as code openinfra...
TRANSCRIPT
![Page 1: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/1.jpg)
![Page 2: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/2.jpg)
Andrea Corna Tommaso Sardelli
Dashboard as Code
OpenInfra Days@Milan, 02/10/2019
![Page 3: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/3.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Bottega52Boutique Consulting
● Specialized
● Quality
● Application Lifecycle
3
![Page 4: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/4.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Life @ Bottega52Our structure
● A lot of customers with different needs
● Different use cases
● But we are a small team
● People first (a.k.a., “bottegai”)
4
![Page 5: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/5.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Our ApproachAutomation
● Infrastructure (Terraform, Ansible)
● Deployments (CI/CD, AWX)
● Monitoring (Kubernetes, Prometheus)
● Dashboards?
5
![Page 6: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/6.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Why dashboardsOur system at a glance
● Visibility
● Aggregations
● Quick Overview
● Proactive
6
![Page 7: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/7.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
How we like our dashboards
7
Dev-Ops Approach
● Configurable
● Extensible
● Reusable
![Page 8: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/8.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Evolution of dashboard automation
8
![Page 9: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/9.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Australopithecus
9
Create and edit dashboards manually
![Page 10: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/10.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Australopithecus
10
Create and edit dashboards manually
● Configurable ✘
● Extensible ✘
● Reusable ✘
![Page 11: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/11.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Homo Habilis
11
Create dashboards manually then export them as JSON
![Page 12: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/12.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Homo Habilis
12
Create dashboards manually then export them as JSON
● Configurable ✔
● Extensible ✘
● Reusable ✘
(40K SLOC for single client dashboards)
![Page 13: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/13.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Homo Erectus
13
Don’t reinvent the wheel (yet)
![Page 14: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/14.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Grafana Cloud
14
![Page 15: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/15.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
N/A Problem
15
![Page 16: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/16.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Variables
16
sum(container_memory_working_set_bytes{ id=“/“, kubernetes_io_hostname=~”^$Node$”}) /sum(machine_memory_bytes{ kubernetes_io_hostname=~”^$Node$”})
![Page 17: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/17.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Homo Erectus
17
Don’t reinvent the wheel (yet)
● Configurable ✘
● Extensible ✘
● Reusable ✔
![Page 18: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/18.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Variables
18
Dashboards and alerts should not be opinionated about labels - this should be
configuration.
![Page 19: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/19.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Homo Sapiens (Sapiens)
19
Dashboard as Code
● Configurable ✔
● Extensible ✔
● Reusable ✔
![Page 20: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/20.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Enters Jsonnet
20
A data templating language for app and tool developers.
A “simple” extension of JSON.
![Page 21: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/21.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Jsonnet - Objects
21
{ person1: { name: ‘Alice’, welcome: ‘Hello ‘ + self.name + ‘!’, }, person2: self.person1 { name: ‘Bob’ },}
![Page 22: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/22.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Jsonnet - Objects
22
{ "person1": { "name": "Alice", "welcome": "Hello Alice!" }, "person2": { "name": "Bob", "welcome": "Hello Bob!" }}
![Page 23: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/23.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Jsonnet - Variables
23
local conference_name = ‘openinfraday’;local conference_year = 2019;
{ conference: conference_name, edition: conference_year,}
![Page 24: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/24.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Jsonnet - Variables
24
{ “conference”: “openinfraday”, “edition”: 2019,}
![Page 25: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/25.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Jsonnet - Functions
25
local my_function(x, y=10) = x + y;
{ result: my_function(2),}
![Page 26: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/26.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Jsonnet - Functions
26
{ “result”: 12,}
![Page 27: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/27.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Jsonnet - Hidden fields and overriding
27
config+:: { versions+:: { grafana: '6.3.3', },}
![Page 28: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/28.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Grafonnet
28
dashboard.new( 'JVM', schemaVersion=16, tags=['java'],).addTemplate( grafana.template.datasource( 'PROMETHEUS_DS', 'Prometheus', hide='label', ))
![Page 29: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/29.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Grafonnet
29
.addPanel( singlestat.new( 'uptime', format='s', datasource='Prometheus', span=2, ) .addTarget( prometheus.target( 'time() - process_start_time_seconds{env="$env", job="$job", instance="$instance"}',) )
![Page 30: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/30.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Demo
30
![Page 31: Andrea Corna - openinfraday.it · Andrea Corna Tommaso Sardelli Dashboard as Code OpenInfra Days@Milan, 02/10/2019. ... Dev-Ops Approach Configurable Extensible Reusable. ... Meetup](https://reader034.vdocuments.site/reader034/viewer/2022042313/5edd1d7fad6a402d666815e2/html5/thumbnails/31.jpg)
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
Bottega52 © 2018 | www.bottega52.it | Confidential DocumentBottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967 Bottega52 © 2018 | www.bottega52.it | Confidential Document
Bottega52 SRL| Piazza della Vittoria 47, 26900 Lodi (LO) | P.IVA/CF 08848340967
Digital Innovation ArtisansBig Data - IoT - Real Time Analytics - Data Liberation
@bottega52
Apache Kafka Meetup MilanoBottega52 co-organize with Confluent Inc. the Milano Apache Kafka Meetup.
Subscribe to the next event by clicking here! Or just scan the QR code.
/bottega52
/company/bottega52