aws lambdaによるデータ処理理の⾃自動化とコモディティ化
Post on 13-Aug-2015
2.552 Views
Preview:
TRANSCRIPT
- 1. AWS Lambda @ 2015.08.01 Yuta Imai Solutions Architect, Amazon Data Services Japan
- 2. 2 ( ) Solutions Architect
- 3. 3 AWS AWS Lambda AWS Lambda
- 4. 4
- 5. 5 () 11 1.US EAST (Virginia) 2.US WEST (N. California) 3.US WEST 2 (Oregon) 4.EU WEST (Ireland) 5.JAPAN (Tokyo) 6.South America (Sao Paulo) 7.ASP 1 (Singapore) 8.ASP 2 (Sydney) 9.GovCloud 10.BJS 1 (Beijing China) limited preview 11.EU (Frankfurt) 28(AZ) 50
- 6. 6 AWS 2014516 24 48 61 82 159 280 516 2008 2009 2010 2011 2012 2013 2014
- 7. 7 TECHNICAL & BUSINESS SUPPORT Account Management Support Professional Services Solutions Architects Training & Certification Security & Pricing Reports Partner Ecosystem AWS MARKETPLACE Backup Big Data & HPC Business Apps Databases Development Industry Solutions Security MANAGEMENT TOOLS Queuing Notifications Search Orchestration Email ENTERPRISE APPS Virtual Desktops Storage Gateway Sharing & Collaboration Email & Calendaring Directories HYBRID CLOUD MANAGEMENT Backups Deployment Direct Connect Identity Federation Integrated Management SECURITY & MANAGEMENT Virtual Private Networks Identity & Access Encryption Keys Configuration Monitoring Dedicated INFRASTRUCTURE SERVICES Regions Availability Zones Compute Storage Objects, Blocks, Files Databases SQL, NoSQL, Caching CDNNetworking PLATFORM SERVICES APP Mobile & Web Front-end Functions Identity Data Store Real-time DEVELOPMENT Containers Source Code Build Tools Deployment DevOps MOBILE Sync Identity Push Notifications Mobile Analytics Mobile Backend ANALYTICS Data Warehousing Hadoop Streaming Data Pipelines Machine Learning
- 8. 8 AWS Lambda Run your code in the cloud.
- 9. 9 AWS Lambda
- 10. 10
- 11. 11 Bring your own code Node.js, Java / /tmpread/write
- 12. 12 100
- 13.
- 14. 14 S3 AWS LambdaAmazon S3 Bucket 1 2 3
- 15. 15 DynamoDB AWS Lambda Amazon DynamoDB Table and Stream
- 16.
- 17. Web S3 Amazon S3 Consumer Game DB Game Servers Kinesis Amazon Redshift Amazon Elastic MapReduce DSV JSON
- 18. If..Then.. S3 Amazon S3 Consumer Game DB Game Servers Kinesis Amazon Redshift Amazon Elastic MapReduce DSV JSON
- 19.
- 20. Worker While true Supervisord
- 21. If..Then.. with AWS Lambda! S3 Amazon S3 Consumer Game DB Game Servers Kinesis Amazon Redshift Amazon Elastic MapReduce DSV JSON If..Then.. Lambda ( )
- 22. 22 If..Then.. with AWS Lambda! With your code!
- 23. AWS
- 24. 24 DataAnswers Collect Process Analyze Store
- 25. 25 Collect Process Analyze Store Data Collection and Storage Data Processing Event Processing Data Analysis
- 26. 26 S3KinesisDynamoDBAWS Lambda KCL AppsEMR EMR Redshi> Machine Learning Collect Process Analyze Store Data Collection and Storage Data Processing Event Processing Data Analysis
- 27. 27 S3KinesisDynamoDBAWS Lambda KCL AppsEMR EMR Redshi> Machine Learning Collect Process Analyze Store Data Collection and Storage Data Processing Event Processing Data Analysis AWS
- 28. 28 AWS Lambda as a pipeline glue Collect Process Analyze Store Data Collection and Storage Data Processing Event Processing Data Analysis
- 29. 29 : media, log les (sets of records) : records (eg: device stats) : DBRead/Write
- 30. 30 S3 Kinesis DynamoDB
- 31. S3 Event Notications Kinesis stream DynamoDB Streams AWS Lambda
- 32. S3 Event Notications Kinesis stream DynamoDB Streams AWS Lambda Consumer Consumer
- 33. AWS Lambda
- 34. 34 Expedia Pipeline http://blogs.aws.amazon.com/bigdata/post/Tx1R28PXR3NAO1I/How-Expedia- Implemented-Near-Real-time-Analysis-of-Interdependent-Datasets
- 35. 35 TempTracker AWS
- 36. 36 raspberry pi
- 37. 37 Python (boto) DynamoDBKinesis App Kinesis Lambda SNS TempTracker: IoT
- 38. 38
- 39. 39 Scalable & Responsive Big Data Interface with AWS Lambda at FireEye http://blogs.aws.amazon.com/bigdata/post/Tx3KH6BEUL2SGVA/Building-Scalable-and-Responsive-Big-Data-Interfaces- with-AWS-Lambda Lambda+S3 Web UI Lambda function
- 40. 40 API Gateway Slack Lambda ECS Lambda S3 Aurora Outgoing Webhook - cluster name - # of tasks - commands RunTasks - cluster name - # of tasks - commands as environment variables - output location Output STDOUT as file Spin up containers and run tasks Incoming Webhook - Read file from S3 and emit it to Slack Sysbench Slack+AWS API Gateway+AWS Lambda+Amazon ECS
- 41.
- 42. 42 AWS Lambda Amazon EMRAmazon ECS HadoopSpark
- 43. 43 AWS Lambda enables... AWS Lambda If Then Amazon DynamoDB Amazon Kinesis Amazon S3 Amazon EMR Amazon Redshift Amazon Machine Learning Amazon API Gateway Amazon ECS With your code!
top related