xtreme deployment

51
/ Xtreme Deployment develop in production; push to dev Tuesday, October 2, 12

Upload: theo-schlossnagle

Post on 16-May-2015

1.403 views

Category:

Documents


1 download

DESCRIPTION

User generated data is an old problem. Systems and network telemetry, page analytics and application state combine to form an ever growing mountain of data collected by today's tools. Collecting and storing this data requires more than just a single application, having no single point where the user touches the system and gets an answer makes debugging a nightmare and reproducing the error intractable. Distributed systems require a clear perspective on production systems and access to data in real time to have any hope of solving complex problems related to state, all while not impacting user experience. We will explain the problem, the pains and how we solved them. Develop in production; push code to development.

TRANSCRIPT

Page 1: Xtreme Deployment

/

Xtreme Deployment

develop in production; push to dev

Tuesday, October 2, 12

Page 2: Xtreme Deployment

Who are we?

Brian ClapperCTO of CirconusWhipping boy of the modern SaaS stack.

Theo SchlossnagleCEO of CirconusRepeat denier of reality.

Tuesday, October 2, 12

Page 3: Xtreme Deployment

How the world used to look (less xtreme)

Architectures driven by user interaction:(e.g. web page load ➠ web page response)

Tuesday, October 2, 12

Page 4: Xtreme Deployment

How the world used to look (less xtreme)

isolateduser A’s page load doesn’t effect user B.

latentuser A’s affect shows up “later” in user B’s view.

Tuesday, October 2, 12

Page 5: Xtreme Deployment

How the world used to look (less xtreme)

Problem reproducibility is achievable(even if sometimes painful)

Tuesday, October 2, 12

Page 6: Xtreme Deployment

How the world looks now (more xtreme)

Architectures driven byhigh volume data and users

Tuesday, October 2, 12

Page 7: Xtreme Deployment

How the world looks now (more xtreme)

every piece of data entering the architecture can effect a user’s subsequent experience.

Tuesday, October 2, 12

Page 8: Xtreme Deployment

How the world looks now (more xtreme)

sometimesthe data must effect a user’ssubsequent experience.

Tuesday, October 2, 12

Page 9: Xtreme Deployment

How the world looks now (more xtreme)

Social systems cause indirect impactvia user action eroding isolation.

Tuesday, October 2, 12

Page 10: Xtreme Deployment

How the world looks now (more xtreme)

A more responsive web is erodingusers’ acceptance of latent state.

Tuesday, October 2, 12

Page 11: Xtreme Deployment

How the world looks now (more xtreme)

Highly distributed systems(due to complexity and scalability) meansno single world state

Tuesday, October 2, 12

Page 12: Xtreme Deployment

How the world looks now (more xtreme)

Highly distributed systems(due to complexity and scalability) meansdifficult repeatability of state and failures

Tuesday, October 2, 12

Page 13: Xtreme Deployment

How the world looks now (more xtreme)

Highly distributed systems(due to complexity and scalability) meanstreacherous debugability

Tuesday, October 2, 12

Page 14: Xtreme Deployment

Despair all ye’ who debug here

I studied distributed systems long enoughto understand “the suck” you do not wish to have in your lives.

Tuesday, October 2, 12

Page 15: Xtreme Deployment

Despair all ye’ who debug here

Expedient debugging requires a quick realization of a precipitating state of failure.

Tuesday, October 2, 12

Page 16: Xtreme Deployment

Despair all ye’ who debug here

In distributed systems,post-mortem techniques are less useful because recreating possible global state is amind-bending (or brain-damaging) exercise.

Tuesday, October 2, 12

Page 17: Xtreme Deployment

Despair all ye’ who debug here

Applying older techniquesto small controlled distributed systemsis still feasible,but complex, asynchronous architecturescan be nearly impossible.

Tuesday, October 2, 12

Page 18: Xtreme Deployment

Some context about us

Circonus ingests telemetry dataacross an entire business

Tuesday, October 2, 12

Page 19: Xtreme Deployment

Some context about us

alerting • correlation • visualization • analytics

Tuesday, October 2, 12

Page 20: Xtreme Deployment

How we see the world today

High velocity non-user-generated data inflow.

Tuesday, October 2, 12

Page 21: Xtreme Deployment

How we see the world today

Users expect sub-second reflection ofall input data and drawn-conclusions.

Tuesday, October 2, 12

Page 22: Xtreme Deployment

How we see the world today

Users expect extremely high availability(100%, you can only disappoint)

Tuesday, October 2, 12

Page 23: Xtreme Deployment

How we see the world today

I was surprised howimpatient and demanding users are...

until I used the tool...and felt exactly the same way.

Tuesday, October 2, 12

Page 24: Xtreme Deployment

Circonus architecture

Voltron

Tuesday, October 2, 12

Page 25: Xtreme Deployment

Circonus architecture

Voltron• distributed data collection (via brokers)

Tuesday, October 2, 12

Page 26: Xtreme Deployment

Circonus architecture

Voltron• distributed data collection (via brokers)

• distributed data ingestion (via aggregators)

Tuesday, October 2, 12

Page 27: Xtreme Deployment

Circonus architecture

Voltron• distributed data collection (via brokers)

• distributed data ingestion (via aggregators)

• distributed data distribution! (via message queueing)

Tuesday, October 2, 12

Page 28: Xtreme Deployment

Circonus architecture

Voltron• distributed data collection (via brokers)

• distributed data ingestion (via aggregators)

• distributed data distribution! (via message queueing)

• distributed data storage (via snowth)

Tuesday, October 2, 12

Page 29: Xtreme Deployment

Circonus architecture

Voltron• distributed data collection (via brokers)

• distributed data ingestion (via aggregators)

• distributed data distribution! (via message queueing)

• distributed data storage (via snowth)

• distributed streaming fault detection (via muppet love)

Tuesday, October 2, 12

Page 30: Xtreme Deployment

Circonus architecture

Voltron• distributed data collection (via brokers)

• distributed data ingestion (via aggregators)

• distributed data distribution! (via message queueing)

• distributed data storage (via snowth)

• distributed streaming fault detection (via muppet love)

• ... distributed nightmare.

Tuesday, October 2, 12

Page 31: Xtreme Deployment

Despair actualized

I shall not memorialize this in slides.

Tuesday, October 2, 12

Page 32: Xtreme Deployment

Despair actualized

I shall only cry in front of you now.

Tuesday, October 2, 12

Page 33: Xtreme Deployment

Hope (keys to the city)

• Leveraging• stream mirroring and• idempotent behavior.

Tuesday, October 2, 12

Page 34: Xtreme Deployment

Hope

Allows for the recklessness that we all appreciate in a no-consequences development environment...

Tuesday, October 2, 12

Page 35: Xtreme Deployment

Hope

withproduction data,production volume, andproduction “personality.”

Tuesday, October 2, 12

Page 36: Xtreme Deployment

Hope actualized

Development In Production

Tuesday, October 2, 12

Page 37: Xtreme Deployment

Hope actualized

Ex #1: Dual storage backends

Tuesday, October 2, 12

Page 38: Xtreme Deployment

EXPLORATION

PostgreSQLTuesday, October 2, 12

Page 39: Xtreme Deployment

EXPLORATION

SnowthTuesday, October 2, 12

Page 40: Xtreme Deployment

EXPLORATION

{Safety & Happiness

Tuesday, October 2, 12

Page 41: Xtreme Deployment

EXPLORATION

Safety & HappinessTuesday, October 2, 12

Page 42: Xtreme Deployment

Hope actualized

Ex #2: Dual streaming analysis tools

Tuesday, October 2, 12

Page 43: Xtreme Deployment

Hell

Tuesday, October 2, 12

Page 44: Xtreme Deployment

Hell

Tuesday, October 2, 12

Page 45: Xtreme Deployment

Hell

Tuesday, October 2, 12

Page 46: Xtreme Deployment

Hell

Tuesday, October 2, 12

Page 47: Xtreme Deployment

Hell

Tuesday, October 2, 12

Page 48: Xtreme Deployment

Hell

Tuesday, October 2, 12

Page 49: Xtreme Deployment

Hope actualized

Ex #3: Dual messages queueing systems

Tuesday, October 2, 12

Page 50: Xtreme Deployment

The Future

This is the foundation ofreal responsive designgoing forward.

Tuesday, October 2, 12

Page 51: Xtreme Deployment

Thank You

Questions?

Tuesday, October 2, 12