to vms on cloud providers economics of using local … storage.pdfeconomics of using local storage...

Economics of using Local Storage Attached to VMs on Cloud ProvidersPavel Snagovsky, Software engineer, Quantum Corp

Pavel Snagovsky, KubeCon 2017

Agenda

● Storage options to support Kubernetes workloads on public clouds

● Storage Orchestrators● Example of utilizing storage orchestrator backed by “local

storage” with demonstrated benefits● Storage beyond persistent volumes● Conclusions and Q&A


Persistent storage in public cloud

Persistent block store offerings are falling short to meet the needs of modern cloud native applications

• Slow provisioning and failover times (more on this later)• Expensive• Zone locality limitations• Proprietary lifecycle management• Mediocre and inconsistent performance


Local storage

GCE and AWS offer instances with local storage● Pros

○ High Performance○ Transactional and streaming IO options○ Inexpensive ○ Consistent IO performance

● Cons○ No dynamic provisioning (AWS)○ Data and durability tied to the node


Harnessing local storage

Local storage delivers solid performance at low cost.Problem:

■ Durability■ Lifecycle Management■ Locality tied to instance

By solving these problems we can capitalize on the benefits of cloud instances local storage. Reduce the bill and get the remainder of compute resources hosting it for free.


REX-Ray

Storage solutions

● Rook (backed by ceph)● PortWorx● REX-Ray● Minio● - CSI *● GlusterFS● - StorageOS *● ?

etc.


Harnessing the benefits with Rook

Rook is…● Leverages Kubernetes features,

Operator, CRDs, StorageClass, PV● Backed by Ceph● Fully automated lifecycle

management of underlying storage● Self healing and monitoring● Block, file and object store

https://rook.io Rook High level functionality diagram


Performance

● Storage backed by instance local disks is significantly more performant (particularly for random reads) than persistent volume offerings of public cloud providers.

● Write performance depends on the type of replication and impacted by network throughput.


Performance

Using local disks provides very consistent IO performance.● IO completion latency demonstrates

solid lead for storage backed by local disk

● Persistent volumes IO latencies are less consistent with some operations taking few milliseconds


Compatibility

Storage orchestrator such as Rook abstracts storage layer making the solution highly portable for Kubernetes users.

■ No vendor lock-in■ Multi provider environments■ Testing on any environment■ Avoiding the dependence on provider specific functionality

Kubernetes administrator would have to setup a storage cluster, define StorageClass, the user managed resources can remain the same for any environment.


Pod failover comparison

Typical pattern of dynamically provisioning pv-ssd/EBS volumes for a pod result in a pod failover times of 1-5 minutes due to:

● Device attach/detach penalties● API calls to trigger attach/detach● Locality constraints and resource dependencies● Stack volumes

Storage orchestrators have none of these penalties and constraints. From the moment scheduler decided to reprovision the pod it takes milliseconds to a second for a pod to start with mounted volume.


Costs

WIP• The cost of storage devices attached to the instance is minimal, disk is

cheap, you are mostly paying for Compute resources that are available for utilization within your Kubernetes cluster.

• Slicing the bigger pool of storage will allow more efficient utilization.• Reduce costs by self hosting some of cloud provider services


Testing environments

WIP:GCE and AWS resource to use.

Kubernetes and storage setup details.


Other storage options and use cases

● S3/GS - get it off your bill. More flexibility on using custom domains, path, self managed auth etc. when you use storage orchestrator.

● NFS/shared filesystem can be provided by your storage orchestrator reducing the cost.

● Multiple clusters and/or storage pools to meet different performance requirements and isolation

● Use persistent volumes from your cloud provider as a backend devices of your storage orchestrator

● Save by switching to in-cluster services over the provider equivalents, such as databases - $ helm install --name my-release stable/mysql


Conclusion

Storage orchestrators backed by Cloud provider ‘local’ disks deliver better cloud native experience in terms of

● Compatibility● Performance● Provisioning and failover times● Cost

Additional specific storage requirements can be met by adding more storage clusters or utilizing cloud provider persistent storage offerings.Rook on!


Questions

Pavel [email protected]: @pahaHiring, email me if interested.

FIO test results and related files

RE: https://github.com/paha/rook-aws

to vms on cloud providers economics of using local … storage.pdfeconomics of using local storage...

Documents