1. AWS 2014424

2. Who am I ? ( ) 1978 AWS 140 60 250 Amazon Web Services http://aws.typepad.com/aws_japan/ 10+ years web engineer in startups Director of V-cube (perl), 2001 - 2006 CTO of FlipClip (perl), 2006 - 2009 CTO of gumi (python), 2009 - 2012 @horiuchi horiyasu ! 3. Amazon 4. 5. AWS GovCloud EU US ITAR Region 10 Regions () | 26 Availability Zones() | 51 Edge Locations 6. 30AWS Support CertificationTrainingProfessional Services Technology Partners Consulting Partners AWS MarketplaceEcosystem Elastic Beanstalk for Java, Node.js, Python, Ruby, PHP and .Net OpsWorks CloudFormationContainers & Deployment (PaaS) Management & AdministrationIAM CloudWatchCloudTrail APIs and SDKsManagement ConsoleCloud HSM Command Line Interface Direct Connect Route 53VPC Networking Analytics Data PipelineRedshiftEMR Kinesis SWFSNS SQS CloudSearchSES AppStreamCloudFront Application Services WorkSpaces Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling 7. 8. 201470 70+ new features since Feb 2013 Regional expansion to US West (Oregon) Support for temporary credentials when loading data from Amazon S3 Regional expansion to EU West (Dublin) SOC1/2/3 Compliance certification Ability to UNLOAD encrypted files in parallel to Amazon S3 Regional expansion to Asia Pacific (Tokyo) Support for JDBC fetch size to enable extraction of large data sets over JDBC/ODBC Enable logging of UNLOAD statements New built-in function to compute the SHA1 hash of a value Added support for UTF-8 characters up to 4 bytes in size Ability to share snapshots between accounts to simplify manageability. Support for statement timeouts to automatically terminate queries that exceeded allotted execution time Added support for timezone conversion in SQL Added support for datetime values expressed in milliseconds since EPOCH to simplify ingestion Simplified ingestion by automatically detecting date and time formats. Added support for automatic query timeouts to workload management queues. Enabled the use of wildcards when assigning queries to workload management queues. New built-in function to enable customers to calculate the CRC32 checksum of a value Console improvements to show progress bars for backup and restore operations. Added the ability to support IAM at the resource level allowing tight control of who can take what actions on which resources. Obtained PCI compliance Added the ability to substitute a customer chosen character for invalid UTF-8 characters to simplify ingestion Allowed customers to store JSON data in VARCHAR columns and added built-in functions to enable data extraction Added support for POSIX regex expressions when using SIMILAR to in SQL queries Added Cursor support to enable extraction of large data sets over ODBC connections Built-in function to enable splitting a string using a supplied delimiter to make parsing values easier Added system tables to enable logging of database activity for auditing Regional expansion to Asia Pacific (Singapore, Sydney) Enable customers to control cluster encryption keys by using an on premises hardware security module (HSM) or Amazon CloudHSM Enable customers to receive alerts via SNS for informational or error-related events for cluster monitoring, management, configuration and security. Integration with Canal to enable streaming data ingestion Copy from an arbitrary SSH connection enabling direct copy from Amazon EMR, HDFS, or any other database that supports SSH access and script execution Enable distributing tables to all compute nodes to speed up queries, especially those involving star or snowflake schemas Logging of database logins, failed logins, SQL execution and data loads to S3 and integration with CloudTrails for control plane events Enabled caching of database blocks to speed up access to frequently queried data Increase cluster concurrency limits from 15 to 50 to enable higher concurrent query execution Optimizations to resize code that lead to 2-4x improvement in resize performance Approximate COUNT DISTINCT using HyperLogLog giving 10-20x performance improvements with less than 1% error Enable customers to continuously, automatically and incrementally back up data to a second AWS region for DR On track to obtain Fedramp certification Deliver Redshift on SSD instances enabling a lower-cost, high performance entry point IAM Redshift Elastic Map Reduce Data Pipeline Route 53CloudFront CloudFormation Elastic Load Balancer EC2 S3 EBS DynamoDB AppStream Kinesis 9. Internet scale throughput Consistent performance and unlimited storage DynamoDB: A Generation Beyond a NoSQL Key/Value Store 10. Internet scale throughput Global and local secondary indexes Item-level access control Consistent performance and unlimited storage DynamoDB: A Generation Beyond a NoSQL Key/Value Store 11. Deep mobile SDK integration Geospatial indexing library Local test tool Storage pricing reduction by up to 75% Throughput pricing reduced by up to 35% Batched writes In-console item updatesGovCloud (US) availability Parallel scan Transaction library Cross-region copy, export and import Internet scale throughput Global and local secondary indexes Item-level access control Consistent performance and unlimited storage DynamoDB: A Generation Beyond a NoSQL Key/Value Store 12. (R3) () - GiB$0.0138 (large$0.0140) - m2(GiB$0.0168)8 2014/04/10 http://bit.ly/1kvdG2V 13. AWS Elastic Beanstalk for Docker 2014/04/24 http://bit.ly/1f8Bd7z Meets Amazon Linux AMI 2014.03 Docker 14. 1 3 4 2 2006 42 15. 42 51% 36%65% $0.033 /GB/ S3 EMR 27%61% 28% RDS ElastiCache 34% M3 38% C3 30% EC2 M1M2C1CC2 10%40% 201441 16. AWS 17. A. 18. 1 19. http://bit.ly/1rnIX5P 20. CPU 21. GPU IO CR1M2CC2C1 HI1 HS1 CG1M1 22. GPU IO CR1M2CC2C1 HI1 CG1M1 G2M3 R3C3 I2 HS1 23. M3 - Intel Xeon E5-2670 (Sandy Bridge) SSD CPU vCPU (GiB) SSD (GB) ) m3.medium 1 3.75 1 x 4 $0.101 m3.large 2 7.5 1 x 32 $0.203 m3.xlarge 4 15 2 x 40 $0.405 m3.2xlarge 8 30 2 x 80 $0.810 24. C3 - CPU Intel Xeon E5-2670 v2 (Ivy Bridge) SSD (SR-IOV, VPC) CPUCPU vCPU (GiB) SSD (GB) ) c3.large 2 7 2 x 16 $0.128 c3.xlarge 4 14 2 x 40 $0.255 c3.2xlarge 8 28 2 x 80 $0.511 c3.4xlarge 16 55 2 x 160 $1.021 c3.8xlarge 32 108 2 x 320 $2.043 25. R3 - Intel Xeon E5-2670 v2 (Ivy Bridge) SSD (SR-IOV, VPC) GiB vCPU (GiB) SSD (GB) ) r3.large 2 15 1 x 32 $0.210 r3.xlarge 4 30.05 1 x 80 $0.420 r3.2xlarge 8 61 1 x 160 $0.840 r3.4xlarge 16 122 1 x 320 $1.680 r3.8xlarge 32 244 2 x 320 $3.360 26. I2 - Intel Xeon E5-2670 v2 (Ivy Bridge) SSD TRIM (SR-IOV, VPC) I/OIOPS i2.8xlarge365,000315,000 vCPU (GiB) SSD (GB) ) i2.xlarge 4 30.05 1 x 800 $1.051 i2.2xlarge 8 61 2 x 800 $2.101 i2.4xlarge 16 122 4 x 800 $4.202 i2.8xlarge 32 244 8 x 800 $8.404 27. IO 28. EBS IOPS IO(IOPS) IOPS10040001 - (StandardEBSIOPS100) IO 29. EBS EBS 500Mbps1000Mbps Net work Net work 30. 2 31. 32. Start Simple Write Events File on Device Periodically Upload to S3 Process into Redshift Point GUI Tool to Redshift 2014-01-24,nateware,e4df,login 2014-01-24,nateware,e4df,gamestart 2014-01-24,nateware,e4df,gameend 2014-01-25,nateware,a88c,login 2014-01-25,nateware,a88c,friendlist 2014-01-25,nateware,a88c,gamestart Profit! 33. Amazon Redshift 10 GigE (HPC) Ingestion Backup Restore SQL Clients/BI Tools 128GB RAM 16TB disk 16 cores Amazon S3/DynamoDB JDBC/ODBC 128GB RAM 16TB disk 16 coresCompute Node 128GB RAM 16TB disk 16 coresCompute Node 128GB RAM 16TB disk 16 coresCompute Node Leader Node - SQL - - ! - - Amazon S3 - Amazon DynamoDB ! 34. Tableau + Redshift 35. More Data Sources Also Collect Server Logs Periodically Upload to S3 Stuff into Redshift External Analytics Data Too External Analytics EC2 36. Dealing With Messy Data Different File Formats Device vs Apache vs CDN Cleanup with EMR Job Output to Clean Bucket Load into Redshift EC2 37. Direct From DynamoDB Integrate Game DB Load Directly into Redshift Redshift does Intelligent Merge Tracks Hash Keys, Columns EC2 38. Direct From DynamoDB Integrate Game DB Load Directly into Redshift Redshift does Intelligent Merge Tracks Hash Keys, Columns Or Stream into EMR EC2 39. 40. Back To Basics 2014-01-24,nateware,e4df,login 2014-01-24,nateware,e4df,gamestart 2014-01-24,nateware,e4df,gameend 2014-01-25,nateware,a88c,login 2014-01-25,nateware,a88c,friendlist 2014-01-25,nateware,a88c,gamestart 41. Back To Basics [Dubstep Remix] Always Batch Due to S3 EC2 42. Need Data Faster! Stream Data With Kinesis Multiple Writers and Readers Still Output to Redshift EC2 43. Lots of Ins and Outs Stream Data With Kinesis Multiple Writers and Readers Still Output to Redshift Stream to Spark on EMR Storm via Kinesis Spout Custom EC2 Workers EC2 EC2 44. Amazon Kinesis Data Sources App.4 ! [Machine Learning] ! ! ! A WS En dp oin t App.1 ! [Aggregate& De-Duplicate] Data Sources Data Sources Data Sources App.2 ! [Metric Extraction] S3 DynamoDB Redshift App.3 [Sliding Window Analysis] Data Sources Availability Zone Shard 1 Shard 2 Shard N Availability Zone Availability Zone 45. 46. ## : AWS - Yliharju, iOS 850 : AWS: EC2: S3: 4TB/ Glacier: Hadoop: http://aws.amazon.com/jp/solutions/case-studies/supercell/ 47. Clash of Clans Amazon Kinesis Redshift Clickstream archive EC2: In-game engagement trends dashboard Real-time clickstream processing app Kinesis: Real-time data stream of in-game activity Multiple Kinesis applications: Dashboards, analytics and storage Redshift: Business intelligence reporting and interactive queries S3 and Glacier: Data storage and long term archival In-game activity S3 Aggregate statistics Business-intelligence user Kinesis-enabled apps on EC2 48. S3EMRRedshift ~ ~ (), (), () http://amzn.to/1ih9n9M 49. 3 50. AWS 51. AWSGame 52. 190 53. 20,000 54. 55. ! 3DSWii U Miiverse http://aws.amazon.com/jp/solutions/case-studies/nintendo/ *3DS Wii U 56. PS4 http://www.slideshare.net/AmazonWebServices/killzones-servers-flexible-architecture-and- componentbased-design-mbl305-aws-reinvent-2013-28622640 : AWS AWS: EC2, S3, DynamoDB, Elastic Beanstalk, SQS, ELB. : DynamoDBElastic Beanstalk Multi-region, hybrid cloud (Sony Hosting + AWS) 57. : AWS -- , 80 GREE FP1 : AWS: EC2 Windows Server RDS for MySQL 5,000req/sec http://www.awsmicrosite.jp/game/case-studies/grani.html 58. iPhone http://www.awsmicrosite.jp/game/case-studies/appbankgames.html : AWS: EC2, ELB, DynamoDB, EMR, SQS : AWS -- 59. http://www.awsmicrosite.jp/game/case-studies/gmode.html : AWS: EC2, RDS, CloudFront, Route53, Storage Gateway, Direct Connect : AWS -- , 60. GREE Platform Award 2012 http://www.awsmicrosite.jp/game/case-studies/enish.html : AWS: EC2, EMR, CloudFront, RDS, S3, Redshift, ElastiCache : Amazon Redshift -- , 61. http://www.awsmicrosite.jp/game/case-studies/gnt.html : 80 AWS: 5 : AWS -- , 62. 12120020 http://aws.amazon.com/ko/solutions/case-studies/devsisters/ : IDC AWS: EC2, RDS, S3, CloudFront, EBS : AWS 63. Auto-scaling http://aws.amazon.com/ko/solutions/case-studies/devsisters/ 64. http://www.allthingsdistributed.com/2012/06/amazon-dynamodb-growth.html : AWS: DynamoDB, etc. : DynamoDB - Glen Arrowsmith, Systems Architect 65. Angry Birds: http://aws.amazon.com/solutions/case-studies/rovio/ : AWS: EC2, RDS, EMR, S3, CloudFront : The flexibility of the AWS Cloud allows us to do things that we otherwise couldnt do. - Mikko Peltola, Operations Lead 66. Tiny Village: http://aws.amazon.com/solutions/case-studies/tinyco/ : AWS: EC2, S3, CloudFront, EMR, RDS : AWS allows us to focus on our core competency--making games - Rob Kotredes, Head of Engineering 67. 380 http://aws.amazon.com/jp/solutions/case-studies/naughty-dog/ : IT 90% AWS: EC2, S3, CloudFront, EBS : AWS - Christophe Balestra, 68. AWS http://aws.amazon.com/solutions/case-studies/ubisoft/ : ! AWS: EC2, S3, EBS : AWS - Lenin Gali, Senior Director, Dev Operations/Analytic Services 69. 3D MMOFPS http://aws.amazon.com/jp/solutions/case-studies/cmune/ : AWS AWS: EC2, CloudFront : EC2 Dedicated Instances 70. Want to increase innovation? Lower the cost of failure ! Joi Ito 71. $25AWS ! http://bit.ly/awsvy-20140424 72. AWS Summit Tokyo 2014 2014717() 18() () () http://www.awssummittokyo.com/ -