data microservices in the cloud + 日本語コメント
TRANSCRIPT
JSUG 勉強会〜 SpringOne Platform 2016 報告会 - Takuya Saeki
• 所属三井情報株式会社( MKI : Mitsui Knowledge Industry )
R&D 部• 興味クラウドネイティブみたいなこと全般
佐伯 拓哉( Takuya Saeki )
自己紹介
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Data Microservices in the Cloud
By Mark Pollack
HTTP
JMS
Kafka
File
HDFS
Cassandra
HAWQ
JDBC
?
Real Time Analytics
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Spring XD Streams
Container Containergpfdist
Cassandrajms
http
ZooKeeper
Message Broker
XD Admin
stream1 = http | cassandrastream2 = jms | gpfdist
On Metal/VMs
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Spring XD Limitations• How to scale up/down instances at runtime?• How to upgrade/downgrade module instances at runtime?• How to specify resources unique to each module, e.g.
memory?• Container architecture lead to parent/child class loader issues• Too many libraries in root class path
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Refactoring to a Microservice Architecture
「コンテナに組み込まれた複数モジュール」から「単体で実行されるスタンドアロンアプリ」 へ
「独自ランタイム」 から「プラットフォーム委任」 へ
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Data Microservices• Stand-alone, production grade applications focused on data
processing• Communicating with ‘lightweight mechanisms’ – messaging
middleware• ‘Event Driven’ - Microservices“ 一つのことを行い、またそれをうまくやるプログラムを書け。”“ 協調して動くプログラムを書け。”“ 標準入出力(テキスト・ストリーム)を扱うプログラムを書け。標準入出力は普遍的インターフェースなのだ。”
$ cat book.txt | tr ' ' '\ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Application Types• Long lived Stream applications
• Spring Cloud Stream • Short lived Task Applications
• Spring Cloud Task
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Spring Cloud Stream• Spring Boot based event-driven microservice framework• Opinionated primitives for streaming applications
• Persistent Publish/Subscribe semantics• Consumer Groups• Partitioning
• Pluggable messaging middleware bindings• Programming model focused on input/output objects
• Adaptable to different event processing APIs
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Spring Cloud Stream Applications
Cassandra
java –jar cassandrasink-1.0.0.RELEASE.jar --ingestQuery=<some cql> --spring.cassandra.keyspace=tweetdata --spring.cloud.stream.bindings .input.destination=dataDest
http
java –jar twittersource-1.0.0.RELEASE.jar --consumerKey=XYZ --consumerSecret=ABC --spring.cloud.stream.bindings .output.destination=dataDest
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Stream orchestration in Spring Cloud Data Flow
ingest = twitterstream | cassandraStream Definition
|Spring Cloud Stream ApplicationsMap DSL names to maven/docker artifacts
dataflow:>stream create --name ingest --definition “twitterstream | cassandra” --deploy
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Features: Persistent Messaging
HTTP log
s1.http DLQ
Message Broker
• Production Ready• RabbitMQ• Kafka
• Experimental • JMS• Google PubSub
• Planned • Kinesis
s1 = http | log
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Features: Named Destinations
HTTP
JMS
S3
myInputDestination
s1 = http > :myInputDestination
s2 = jms > :myInputDestination
s3 = aws-s3 > :myInputDestination
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Features: Consumer Groups
HTTP
s1.http
HDFS
s1 = http | hdfs
s2 = :s1.http > counter
COUNTER
group: s1 group: s2
HDFS
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Simple Real Time Analyticstweets = twitterstream | hdfs
analytics = :ingest.twitterstream > field-value-counter --fieldName=lang
HTTP
s1.http
HDFS COUNTER
Data Flow ServerREST API
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Spring Cloud Stream Programming Model
@EnableBinding(Processor.class)public class TransformProcessor {
@StreamListener(“input”) @SendTo(“output”) public String transform(String s) { return s.toUpperCase(); }}
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Spring Cloud Stream Programming Model
@EnableBinding(Processor.class)public class TransformProcessor {
@StreamListener @Output(“output”) public Flux<String> transform(@Input(“input”) Flux<String> input) { return input.map(s -> s.toUpperCase()); }}
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Spring Cloud Stream Programming Model
@EnableBinding(Processor.class)public class TransformProcessor {
@StreamListener @Output(“output”) public Flux<WordCount> countWords(@Input("output") Flux<String> words) { return words.window(ofSeconds(5), ofSeconds(1)) .flatMap(window -> window.groupBy(word -> word) .flatMap(group -> group.reduce(0, (counter, word) -> counter + 1) .map(count -> new WordCount(group.key(), count)))); }}
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Platform Runtimes
Docker Swarm Apache YARN
Apache Mesos + Marathon
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Spring Cloud Data Flow Deployment Platforms
Data Flow ServerREST API
Deployer SPI
SCDF FloSCDF Shell
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Spring Cloud Data Flow Streams
gpfdist
cassandrajms
http
stream1 = http | cassandrastream2 = jms | gpfdist
Message Broker
Data Flow Server DBPlatform Runtime
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Deployment: Partitioning and Instance Count
http
http
work
work
work
hdfs
hdfs
hdfs
hdfs
Load Balancer
stream create s1 --definition “http | work | hdfs”
stream deploy s1 --propertiesFile ingest.properties
app.http.count=2app.work.count=3app.hdfs.count=4app.http.producer.partitionKeyExpression=payload.id
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Deployment: Resource Management
http
http
work
work
work
app.work.spring.cloud.deployer.cloudfoundry.memory=2048
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Spring Cloud Task• Spring Boot based framework for short lived processes
• Auto-configuration provides a task repository and pluggable data source
• Result of each process persists beyond the life of the task for future reporting
• Tasks can be any arbitrary short lived code• Well integrated with Spring Batch
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Task Orchestration in Spring Cloud Data Flow>task create jdbc2hdfs –sql=‘select * from table’>task launch jdbc2hdfs
jdbc2hdfs
Data Flow Server DB Task NameStart TimeEnd TimeExit CodeExit MessageLast Updated TimeParameters
task-event
Message Broker
job-execution-eventsstep-execution-
eventsitem-read-eventsitem-process-eventsitem-write-eventsskip-events
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Spring Cloud Data Flow Tasks
spark
Data Flow Server DB
http | task-launcher
sqoop
Message Broker
task-event
Platform Runtime
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Spring Cloud Task Programming Model@SpringBootApplication@EnableTaskpublic class ExampleApplication {
@Bean public CommandLineRunner commandLineRunner() { return strings -> System.out.println("Executed at :" + new SimpleDateFormat().format(new Date())); }
public static void main(String[] args) { SpringApplication.run(ExampleApplication.class, args); }}
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Provided Applications
• ~60 stream and task apps• https://github.com/spring-cloud/spring-cloud-stream-app-starters• https://github.com/spring-cloud/spring-cloud-task-app-starters/
• Customize provided apps - http://start-scs.cfapps.io/ • Create new stream/task apps - http://start.spring.io/• Easy import of provided apps/tasks
• dataflow> app import --uri http://bit.ly/1-0-2-GA-stream-applications-kafka-maven
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
UI : Dashboard with Designer
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
XD to SCDF - Terminology
XD-Admin Data Flow Server(local, CF, YARN, k8s, Mesos)
XD-Container N/A
Modules Applications
Admin UI Dashboard
Message Bus Binders
Job Task
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
DEMO
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Upcoming features• Some ‘porting’ from XD
• Batch Job DSL + Designer• Role based access
• Looking forward• Spring Cloud Sleuth• JavaDSL• In-place application version upgrades with Spinnaker• Application Groups• Polyglot• Expanded analytics with Redis and Python/R ecosystem• More provided apps/tasks
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Related Talks• Building Resilient and Evolutionary Data Microservices – Tuesday
2:00pm• Cloud Native Java – Tuesday 2:00pm• Task Madness - Modern On Demand Processing – Tuesday 2:40pm• Spinnaker – Land of a 1000 Builds – Tuesday 5:00pm• Spring and Big Data – Tuesday 5:00pm• Migrating from Spring XD to Spring Cloud Data Flow – Thursday
10:10am• Orchestrate All the Things! with Spring Cloud Data Flow – Thursday
11:10am• Cloud Native Streaming and Event-Driven Microservices – Wednesday
4:20pm
Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:
http:/ /c reativecommons .org/ l icenses/by-nc/3.0/
Get Started…• http://cloud.spring.io/spring-cloud-dataflow/• http://cloud.spring.io/spring-cloud-stream/• http://cloud.spring.io/spring-cloud-task/• https://github.com/spring-cloud/spring-cloud-deployer