enabling microservices frameworks to solve business problems
TRANSCRIPT
Ken Owens
Enabling Microservices Frameworks to Solve Business Problems
CTO, Cisco Intercloud Services
Problem & Opportunity
Rapid innovation in computing and application development services
No single service is optimal for all solutions
Customers want to run multiple services in a single cluster and run multiple clusters in Intercloud environment
...to maximize utilization ...to share data between services
Technologies matrix* Service Product Cloud/Virtualization CIS/AWS/Metacloud/UCS… Provisioning Open Stack/Terraform Automation Ansible
Clustering & Resource Management Mesos, Marathon, Docker Load Balancing Avi Networks ETL & Data Shaping Log Data Gathering Logstash Metrics Gathering CollectD, Avi Networks Messaging Kafka, Solace Data Storing (Batch) HDFS Data Storing (OLTP/Real-time) Cassandra Data Storing (Indexing) Elastic search Data Processing Apache Spark Visualization Zoomdata*Subset example
How does this work? Can deploy in many environments Get orchestrated by Openstack, Ansible (scripts), Cloudbreak
True Hybrid Cloud deployment: CIS, AWS, UCS, vSphere, other
CIS/AWS/Metacloud/vSphere/UCS…
VM5VM1 VM2 VM3 VM4
Terraform
REST API REST API
Scripted provisioning
Direct provisioning
Policy, Auto-scaling
How does this work? Microservices managed and scaled separately Microservices managed by Mesos in a single platform
Microservices architecture for Mesos frameworks and other components
CIS/AWS/Metacloud/vSphere/UCS…
VM5VM1 VM2 VM3 VM4
Terraform
Spark Executor N
Spark Executor 1
Spark Scheduler
Kafka Broker N
Kafka Broker 1
Kafka Scheduler
Docker DockerHAPROXYMicroservices …
REST API REST API
Scripted provisioning
Direct provisioning
Policy, Auto-scaling
What have we done so far?Working with partners on adopting and co-developing services
Partners Co-development Partners
Shipped (Devops SDLC)
ProjectManagement
ContinuousIntegration
SourceControl
ContinuousDeployment
ApplicationOrchestration
DeveloperTools
ServiceAssurance
JiraRallyGitHub
ConsulInfluxDBElasticsearchApache SparkZoomDataLogStash
IssueManagement
MesosConsulKubernetesKafka
Terraform
DroneGitLabShipped CICD
GitHubGitLab
Build PacksEclipseVagrantDocker Compose
JiraRallyGitHub
Collaboration
Cisco Spark
Networking Enhancements &Containers as a first-class citizen of existing infrastructure. Co-operate with bare metal and IaaS workloads. Enable Multicast, IPv6 and IP-per-container.
Seamlessly integrate. Allow SP’s and Enterprise to re-use existing knowledge, skills and infrastructure by using internet-proven networking techniques.
Allow the network to unify the platforms. Use the best tool for the job and expose a platform-agnostic UX to the user. Ie; K8’s workload consuming mesos service.
Define policy in your application, let the network enforce that policy across the multi-platform stack. Both in software / cloud or hardware if available.
Riak Service: Architecture
Riak Service: Highly Scalable
E-commerce Application with Varying Traffic
Zoomdata Service: Mesos + Kubernetes
Mesos Slave
Mesos Master
Mesos SlaveMesos Slave
Zoomdata web app
MongodbSpark Worker
Spark Executor
Spark Executor
Proxy(haproxy, nginx)
KubernetesMongoService/RC
KubernetesSpark-ProxyService/RC
Spark-Proxy
Zoomdata web app
Zoomdata web app
Kubernetes Framework
KubernetesZoomdataSchedulerService/RC
Zoomdata Scheduler
ProxyGen Script
User
● Every component (Zoomdata App, MongoDB, Spark-Proxy, Scheduler) must be started in independent K8s Pod and there must be exactly one MongoDB, Spark-Proxy and Scheduler Pods meanwhile Zoomdata App can be scaled with help Kubernetes Replication Controller.
● There must be defined Kubernetes Service for MongoDB, Spark-Proxy, Scheduler as they will be used in Zoomdata’s App Pod. Every docker container will have env variables for every present Service injected automatically.
Lambda Reference Architecture
Monitoring / Analytics Cluster (local, Texas-3)
Global Monitoring / Analytics Cluster (global, Texas-1)
Monitoring / Analytics Cluster (local, Ams. -1 )
Monitoring / Analytics Cluster (local, Lon.-1)
Local components and deployment is the same as global, just smaller
Real-time and batch processing (Lambda), anomaly detection, visualization
SSL
Kafka
SSL
SSL
MQTT
TBA
© 2015 Mesosphere, Inc.
Mesosphere Infinity
Events
Ubiquitous data streams from connected devices
Feeds
Kafka
Analytics
Spark
Storage
Cassandra
Reactive App
Akka
Ingest millions of events per second
Real-time and batch process data
Distributed & highly scalable database
Scalable, resilient, data driven applications
Powered by Datacenter Operating System (DCOS)
Sensors
Devices
Clients
© 2015 Mesosphere, Inc.
© 2015 Mesosphere, Inc.
IoE PoCs with Drones
Drone storage Charging and Fog Node
Data processing for further elaboration and presentation
Local Pilot is present To comply safety regulation
Flight Schedule and planning is coordinated by the local fog Node
Issues and solutions presented
1
2
3
4
5
6
Optimized and reduced Data secure transmissionto the cloud is performed by the ground station Fog Node
Fog Enabled Precision Agriculture
Data can be pre-parsedOn drones (micro fog node) and on the ground station Fog Node
Hierarchical approach
Citrus
Grapes
Olives
Peaches
Regional Fog Node
Interface towards Cloud services
Interface towards Traffic Management systems
© 2015 Mesosphere, Inc.
© 2015 Mesosphere, Inc.
Demo
Next stepsContinue partnerships and co-devlopment efforts with industry leaders to deliver innovation that enable business to accelerate their objectives
Continue applying new developed technology to real use cases and PoC with customers and partners
Continue R&D activities to stay closely aligned with direction the industry aligns
Deliver the platform for the Internet of Everything