Emerging Storage Trends

for Containers

@openebsDocker Bangalore Meetup #24@kiranmova@CloudByteInc

Happy Docker!Stateless Containers

Ephemeral Storage from local disks.

Rebuilding is Easy

5 million software downloads and 650,000 registered users

Containers in Production!Scale-up

Long-running

Highly Networked Micro-services

Downtime costs money

Rough tides...Sydney storms on Sunday knocked out the operations of numerous Amazon Web Services customers.

The ferocious storms that hit NSW, left AWS clients including Domino's Pizza, Foxtel, The Iconic, Stan and Domain without websites or key systems for hours.

http://www.afr.com/technology/cloud-computing/amazon-web-services-storm-outages-serve-as-a-warning-of-cloud-risk-to-businesses-20160606-gpcugk

Containers - Dev Vs Prod

https://twitter.com/mfdii/status/697532387240996864

Docker in Real-worldClusters Everywhere!

Network and Storage - Can’t be local!

Containers - Dev Vs Prod

https://twitter.com/mfdii/status/697532387240996864

Container - Where is my Data?Flocker / ClusterHQ

Back in 2014 with first DockerCon “ClusterHQ were the first to say that container data management was essential for containers…” - Mark Davis CEO ClusterHQ

Long live Data Persistence

https://clusterhq.com/2016/07/17/dockercon-ecosystem-clusterhq/

"Containers have moved beyond ephemeral storage to persistent storage to hold data and protect your applications."- 451 Research

Mesosphere Declares ‘Container 2.0,’ the Stateful Era

Kubernetes Version 1.3 include additional resources for supporting stateful containers

Docker Adoption by Storage Vendors

Scott Sinclair, a storage analyst at Enterprise Strategy Group, said 2016

has ushered in a "significant adoption cycle" for Docker.

"We are starting to see more and more organizations taking a look at how

they can use containers," he said. "There's tremendous excitement by

vendors, too. It's rare for me to talk with a storage vendor that doesn't

have, or is planning to have, an offering for Docker."

http://searchstorage.techtarget.com/news/450303805/Vendors-target-storage-for-containers-with-DevOps-in-mind

"Containers have moved beyond ephemeral storage to persistent storage to hold data and protect your applications."- 451 Research

Mesosphere Declares ‘Container 2.0,’ the Stateful Era

Kubernetes Version 1.3 include additional resources for supporting stateful containers

Emerging Storage

Technologiesfor Containers

@openebs

Connecting Storage to Containers

Orchestrating Storage for Containers

Optimizing Underlying Storage Technology for Containers

Connect - Volume PluginsConnecting Storage

The advent of Volume Plugins

External SAN/NAS

Pre-configured and auto-provision

https://clusterhq.com/2015/12/09/difference-docker-volumes-flocker-volumes/

Connect - Volume PluginsConnecting Storage

The advent of Volume Plugins

External SAN/NAS

Pre-configured and auto-provision

Connect - Volume PluginsConnecting Storage

The advent of Volume Plugins

External SAN/NAS

Pre-configured and auto-provision

Connect - Volume Plugins

https://clusterhq.com/2015/06/17/flocker-1-0/

Connecting Storage

The advent of Volume Plugins

External SAN/NAS

Pre-configured and auto-provision

Flocker

Connect - Volume Plugins

http://rancher.com/introducing-convoy-a-docker-volume-driver-for-backup-and-recovery-of-persistent-data/

Connecting Storage

The advent of Volume Plugins

External SAN/NAS

Pre-configured and auto-provision

Flocker

Convoy

Connect - Volume PluginsNetApp

Rook (Ceph)

Hedvig

RedHat

HP

AWS

GPE

EMC

Connect - Volume PluginsConnecting Storage

EMC - libStorage

Connect - Orchestration LayerThe right place!

Volumes outlive the containers. They are configured at the pod.

Adoption on the rise.

Slowdown of convoy, flocker, possibly docker

Growth vs Infrastructure30% increase in Docker adoption in one year.

The number of containers running in production quintuples (= 5x) 9 months after initial deployment.

June 2016 https://blog.newrelic.com/2016/06/16/docker-container-infographic/

Containers Demand Robust Infra!

Capacity!

Data Protection

Elastic deployment (scale-out/distributed)

Low latency predictable performance

Faster and simpler provisioning and upgrades

Hybrid Clouds https://static-ssl.businessinsider.com/image/51560a4869bedd425f000003-960-720/pakistani-men-in-islamabad-try-to-right-a-truck-transporting-wheat-straw.jpg

Trend#1 Elastic DeploymentStorage Orchestration

Hedvig

NetApp/SolidFire

Ceph

CoreOS Torus

Docker Infinit.sh

Rancher LongHorn/OpenEBS

PortWorx

Distributed Storage (No SPOFs)

ScaleOut

Share-Nothing

Trend#2 Ease of Access Storage Orchestration

CoreOS Torus

Docker Infinit.sh

Rancher LongHorn/OpenEBS

PortWorx

No Networking Provisioning Overheads (Single Volume/NBD)

Uses the docker networks (overlay) and containers

Seamless Integration into Orchestration Layers

Trend #3 Hyper ConvergedStorage Orchestration

CoreOS Torus

Docker Infinit.sh

Rancher LongHorn/OpenEBS

(3) Hyper Converged. Developer Centric.

Trend #4 Hybrid CloudsStorage Orchestration

Deploy Storage Clusters along side K8s in GCP or AWS.

Container in GCP talking to volumes in AWS.

DP and DR

Rancher LongHorn/OpenEBS

S3 Snapshots

Move data across clouds

EBS Compatible. Eliminate need for new drivers

Trend#6 Containerized StorageStorage

Orchestration

Ease of Upgrades / Isolation

Rancher LongHorn/OpenEBS

DevOps Delivered

Trend#5 Tiered StorageStorage Technology

Higher Capacity, Faster and Cheaper$$ flash.

Rancher LongHorn/OpenEBS

NVMe Flash Caching

2TB Flash with 20TB Volumes

Watch out..DataHub

Cloud Federated Infrastructures

OpenSource - Sooner or Later..

OpenEBSOrchestration(K8s)

Remote Storage

Local Storage

OpenEBS Storage Hosts

OpenEBS VSMs

iSCSI/TCMU/NBD(data)

OpenEBS Maya Masters

Clients - Containers(Docker) / VMs

EBS Client

HTTPS (EBS API(manage)

https://github.com/openebs

www.openebs.io

Hiring Gopher Warriors!!

Referenceshttps://www.joyent.com/blog/persistent-storage-patterns

http://forums.theregister.co.uk/forum/1/2016/07/18/docker_and_storage_solving_the_problem_of_data_persistence/

http://thenewstack.io/emc-pushes-alternative-dockers-persistent-storage-volumes/

https://www.sdxcentral.com/articles/news/emc-connects-storage-containers/2016/06/

https://kafka.apache.org/uses

http://www.dbta.com/DBTA-Downloads/WhitePapers/MOVING-TO-A-MODERN-DATA-ARCHITECTURE-6743.aspx

https://docs.opensvc.com/agent.service.container.docker.enterprise_architecture.html

https://developer.ibm.com/urbancode/2016/03/21/docker-deployments-for-the-enterprise/

https://blog.docker.com/2016/06/docker-enterprise-hpe/

http://searchstorage.techtarget.com/feature/Hot-data-storage-technology-trends-for-2017

Thank You@kiranmova

@openebs

https://github.com/openebs/openebs