db engineering - from antiquated to engineer
TRANSCRIPT
![Page 1: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/1.jpg)
from antiquated to engineer
AllYourBase, November 2015Laine Campbell, DB Architect/[email protected] @lainevcampbell
September, 2015
![Page 2: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/2.jpg)
engineering, not administration
2
![Page 3: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/3.jpg)
yesterday’s DBA
3
gatekeeper
master builder
superhero
siloed
specialized
![Page 4: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/4.jpg)
engineering
4
quantitative
interdisciplinary
results focused
repeatable and code-driven
![Page 5: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/5.jpg)
database engineering
5
shared services for:
cross-functional development teams
operations engineering
systems engineering
![Page 6: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/6.jpg)
database engineering
6
provides expertise to other team’s processes to facilitate all work (for example, SRE teams)
understands and teaches data access and storage nuances to ensure all performance metrics can be met
anchors teams with expertise for troubleshooting, recovery and other tasks requiring depth, not breadth
![Page 7: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/7.jpg)
7
paradigm shifts
![Page 8: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/8.jpg)
polyglot persistence
8
relational is not the end of the line
data exists as part of a pipeline
function dictates form
we cannot predict all uses
![Page 9: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/9.jpg)
![Page 10: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/10.jpg)
virtualization and cloud
10
forces horizontal scaling
forces designing for resilience
elasticity drives new data storage
management by API
enables continuous deployment
![Page 11: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/11.jpg)
infrastructure as code
11
forces standardization
enables immutability
forces us to learn to code
we start building platforms
changes become deployments
![Page 12: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/12.jpg)
continuous delivery
© 2014 Pythian 12
brings us to the source code control paradigm
we must be teachers, not gatekeepers
testing and compliance become top priorities
![Page 13: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/13.jpg)
devops cultures
13
lean manufacturing defines our workflows
theory of constraints (we are a constraint)
tighter feedback loops require organizational shifts
experimentation and controlled failure shift architecture and process design
integration drives empathy
![Page 14: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/14.jpg)
database engineers:O.G. devops
14
DEV OPS
DBE
shared goals, tools and processes
![Page 15: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/15.jpg)
15
![Page 16: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/16.jpg)
db engineer’s manifesto
16
it’s all about the mission
protect the data
eliminate waste
data-driven decision making
databases are not special
eliminate the barriers between software and ops
![Page 17: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/17.jpg)
thecraft
17
![Page 18: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/18.jpg)
maslow’s hierarchy of needs
18
physiological
safety
love and belonging
esteem
self-actualization
with loving credit and glory to Charity F. Majors
![Page 19: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/19.jpg)
physiological
19
physiological does it run and service your application?
is it stable?
can data be backed up and restored?
can basic levels of service be monitored?
![Page 20: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/20.jpg)
safety
20
physiological
safety
do I have n+1 available copies?
is failover built, documented and tested?
is my data distributed to multiple physical locations?
can I detect, and recover from corruption?
is access managed via roles, separation of duties and appropriate authorization?
is appropriate data encrypted in transport and at rest?
![Page 21: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/21.jpg)
love and belonging
21
physiological are database changes part of integration and deployment processes?
is continuous testing done with proper data sets?
is infrastructure managed via configuration management, automation and orchestration?
is all database interaction done via scripted, repeatable processes?
is consistency of configuration monitored and managed?
safety
love and belonging
![Page 22: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/22.jpg)
esteem
22
physiological
safety
love and belonging
esteem
are all relevant metrics being captured at appropriate granularity?
can problems be detected preemptively?
are capacity correlations understand and monitored?
are developers trained and knowledgeable on the datastore?
can the system automatically remediate frequent and simple faults?
![Page 23: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/23.jpg)
self-actualization
23
physiological
safety
love and belonging
esteem
self-actualization
are you using the right datastore for the right job?
are your developers in tight feedback loops with the datastore?
is experimentation and data driven decision making supported by the datastore?
is the mission fully supported?
![Page 24: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/24.jpg)
a day in the life...
24
selecting datastores to add to production platform catalogs
dbms and feature education to software, systems and operations engineers
integrating with company-wide services (reporting, monitoring, config mgmt etc…)
![Page 25: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/25.jpg)
a day in the life...
25
validating acceptable configurations
testing and benchmarking new versions, features and configurations
documenting and sharing standards
![Page 26: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/26.jpg)
deploying infrastructure
26
configuration management definition
orchestration enablement
change and deployment test automation
building self-service tools for the team
![Page 27: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/27.jpg)
deploying software
27
agile participation with engineers
using version control for schemas and metadata
more flexible data model approaches
teach your engineering teams how to assess risk, performance and impact
support test planning, building and execution
![Page 28: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/28.jpg)
a day in the life...
28
attending scrums, grooming and planning
committing deployment scripts to the codebase
building, running and automating test cases
monitoring commits for key areas that need your attention
![Page 29: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/29.jpg)
a day in the life… (cont)
29
providing new or modified recipes for CM and automation
pairing with, and teaching engineers, iterating on the schema
![Page 30: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/30.jpg)
the data must flow
30
data brokers
pub/sub systems
real-time analytics
batch processing
cache population, use, and flushing
search stores
![Page 31: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/31.jpg)
backup and recovery
31
this has not changed, we live and die by the safety of our data
borrow ideas of continuous deployment, for continuous recovery testing
build backup and recovery into every possible process
![Page 32: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/32.jpg)
a day in the life...reviewing current workloads and tuning
managing escalations on DB issues
continuous improvement
writing, testing and performing change plans as part of the deployment process
32
![Page 33: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/33.jpg)
33
how?
![Page 34: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/34.jpg)
software engineering
34
bring DBEs into your agile workflows
learn the software versioning system
learn the testing frameworks
study the language, the framework, the drivers and the ORMs
teach, teach teach!
![Page 35: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/35.jpg)
systems engineering
35
design your data pipelines together
collaborate on data standards together
learn the entire stack to moderate proficiency
understand the entire flow
teach, teach, teach!
![Page 36: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/36.jpg)
operations engineering
36
build your scripting chops
learn the build and deploy frameworks
learn the configuration management, automation and orchestration layers and patterns
learn operational visibility layers and patterns
plan, execute and learn from failures
![Page 37: DB Engineering - From Antiquated to Engineer](https://reader033.vdocuments.site/reader033/viewer/2022051705/5885490d1a28abb5368b65c7/html5/thumbnails/37.jpg)
further deep dives
37
understand the statistics and math around distributions, anomaly detection and correlation
write and push code!
answer the customer service phones
dive into your network layers
teach everyone about the data