amazon kinesis data streams - 開発者ガイド · amazon kinesis data streams 開発者ガイド...

Amazon Kinesis Data Streams


Amazon Kinesis Data Streams: Copyright 2018 Amazon Web Services, Inc. and/or its affiliates. All rights reserved.

Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any mannerthat is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks notowned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored byAmazon.


Table of ContentsAmazon Kinesis Data Streams ...................................................................................................... 1

Kinesis Data Streams ....................................................................................................... 1Kinesis Data Streams ................................................................................................. 2 ............................................................................................................................. 2 ....................................................................................................................... 2

......................................................................................................... 2 .................................................................................................................................. 3

Data Streams ............................................................................................................................. 5Kinesis Data Stream ......................................................................... 5 ............................................................................................................... 6 ............................................................................................................... 6

.......................................................................................................................... 7 .......................................................................................................................... 8

........................................................................................................................................ 8 .......................................................................................................................................... 8

API ........................................................................................................................ 9 .................................................................................................................. 9

.......................................................................................................................................... 10 ............................................................................................................................ 10

AWS .................................................................................................. 10 ............................................................................... 11 .......................................................................................................... 11

: ............................................................................... 11Kinesis Data Streams ............................................. 12 .......................................................................................................................... 12 1: ................................................................. 12 2: ......................................... 13 3: ................................................................. 16 4: ................................................................................................... 16

: CLI ..................................................................................... 17AWS CLI ......................................................................................... 17 ............................................................................ 19

: ............................................................................ 24 .......................................................................................................................... 24 1: ................................................................................... 25 2: IAM .......................................................................... 26 3: .............................................................. 29 4: ................................................................................ 30 5: ................................................................................ 33 6: () .............................................................. 36 7: ......................................................................................................... 37

.................................................................................................................... 39 ...................................................................................................................... 39

Kinesis Data Streams ........................................................................... 39 ....................................................................................................... 40

.................................................................................................................. 41 .................................................................................................................. 42 ............................................................................................. 43 ............................................................................................................... 43 ............................................................................................. 44

........................................................................................... 44 ................................................................................................................. 452 ................................................................................................. 46 .......................................................................................................... 47

iii


............................................................................................................... 49 ............................................................................................................... 49

....................................................................................................................... 50 ........................................................................................ 50 ....................................................................................................................... 50Kinesis Data Streams .......................................... 51AWS CLI .......................................................................... 51Kinesis Data Streams API ................................................... 52

......................................................................................................... 52CloudWatch ......................................................................... 52CloudWatch ................................................................... 62AWS CloudTrail Amazon Kinesis Data Streams API ...................... 63CloudWatch KCL ............................................................................. 66CloudWatch KPL .............................................................................. 74

......................................................................................................................... 78 .................................................................................................................... 79Kinesis Data Streams ..................................................................................... 80Kinesis Data Streams Amazon (ARN) ..................................................... 80Kinesis Data Streams ..................................................................................... 80

......................................................................................................... 82Kinesis Data Streams ............................................................... 82 ................................................ 83 ............................................................................ 84 KMS ...................................................................... 85 KMS .............................................. 86KMS .................................................................... 87

VPC .............................................................................. 87Kinesis Data Streams VPC ........................................... 87Kinesis Data Streams VPC .................................. 87 .............................................................................................. 88

....................................................................................... 88 ........................................................................................................ 90

KPL .............................................................................................................................. 90KPL ................................................................................................................... 91KPL .................................................................................................... 91KPL ................................................................................................ 92KPL .................................................................................................... 92Kinesis Producer Library Amazon Trust Services (ATS) ................................ 92KPL ........................................................................ 93KPL .................................................................................................... 93KPL .................................................................................... 95Kinesis data stream ....................................................................................... 96KPL ...................................................................................................................... 97 ................................................................................................. 98Kinesis Data Firehose KPL ............................................................................. 100

API ............................................................................................................................. 100 ............................................................................................. 101

................................................................................................................. 105 ........................................................................................................................ 105 ....................................................................... 106 ................................................................................................ 106 ......................................................................................................... 107 ..................................... 109 .................................................................... 110 CLI ............................................................................................... 113

........................................................................................................... 113 ..................................... 114

iv


KMS .............................................................. 115 ....................................................................................................................... 115

......................................................................................................... 115KPL ................................................................................... 116

................................................................................................... 117 .............................................................................................................. 118

Kinesis Client Library 1.x ....................................................................................... 118Kinesis Client Library 2.0 ....................................................................................... 133API ..................................................................................................................... 137

............................................................................ 141Kinesis Client Library 2.0 ....................................................................................... 142API ..................................................................................................................... 146AWS ................................................................................ 147

Kinesis Client Library 1.x 2.x ................................................................................ 148 ................................................................................................ 149 ............................................................................. 152 ............................................................................................................... 153Amazon Kinesis ................................................................................. 154 ......................................................................................................... 156 ................................................................................................... 156

........................................................................................................... 157Kinesis Kinesis Data Streams .............................................................................................................................. 157 .......... 157 ..................................... 158GetRecords Records ......................... 158 .......................................................................... 159 ............................................................................. 159 KMS .............................................................. 160

....................................................................................................................... 160 ..................................................................................................................... 160 ......................................................................................................... 161Kinesis Producer Library AWS Lambda ............................................................. 162 ................................................................................... 162 ......................................................................................................... 163 ............................................................................................................... 165 .................................................................... 166

............................................................................................................................ 168AWS ................................................................................................................................ 170

v

Amazon Kinesis Data Streams Kinesis Data Streams

Amazon Kinesis Data Streams Amazon Kinesis DataStreams Kinesis Data Streams Kinesis Data Streams Kinesis Client Library Amazon EC2 AWS Kinesis Data Streams Amazon Kinesis DataStreams

Kinesis Data Streams Kinesis Data FirehoseKinesis Kinesis Data Analytics Kinesis

AWS AWS AWS ?

Kinesis Data Streams (p. 1) Kinesis Data Streams (p. 2) (p. 2) Kinesis Data Streams (p. 2) (p. 5) Kinesis Data Streams (p. 7) Kinesis Data Streams (p. 8) Kinesis Data Streams (p. 8)

Kinesis Data Streams Kinesis Data Streams IT

Kinesis Data Streams

KinesisData Streams


1

streaming-data/kinesis/streams/kinesis/streams/big-data/streaming-data/


KinesisData Streams

Kinesis Data Streams Directed Acyclic Graphs (DAG) Kinesis Data Streams Kinesis Data Streams

Kinesis Data Streams Kinesis Data Streams MapReduce

Kinesis () 1 Kinesis Data Streams Kinesis Data Streams MapReduce Kinesis Data Streams

Kinesis Data Streams 2 Amazon DynamoDB 2 Amazon Simple Storage Service (Amazon S3) DynamoDB

Kinesis Client Library KinesisData Streams

Amazon EMR Kinesis Kinesis

Kinesis Data Streams Amazon Kinesis Data Streams

Kinesis Data Streams Kinesis Data Streams Kinesis Data Streams (Amazon EC2 Amazon Kinesis Data Firehose

2

emr-kinesis.html


) Amazon DynamoDBAmazon RedshiftAmazon S3 AWS

Kinesis Data Streams Kinesis Data StreamKinesis data stream (p. 4)Kinesis Data Streams (p. 4)

Kinesis data stream (p. 3) (p. 4) (p. 4) BLOB () Kinesis Data Streams BLOB BLOB 1 MB

24 IncreaseStreamRetentionPeriod 168 (7 ) DecreaseStreamRetentionPeriod 24 24 Amazon Kinesis Data Streams


Amazon Kinesis Data Streams Amazon Kinesis Data Streams Application (p. 4)

3

https://docs.aws.amazon.com/kinesis/latest/APIReference/API_IncreaseStreamRetentionPeriod.htmlhttps://docs.aws.amazon.com/kinesis/latest/APIReference/API_DecreaseStreamRetentionPeriod.htmlhttps://aws.amazon.com/kinesis/pricing/


Amazon Kinesis Data Streams ApplicationAmazon Kinesis Data Streams application EC2

2 Amazon Kinesis Data Streams (p. 117)

Kinesis Data Streams AWS 1

1 1 5 1 2 MB 1 1,000 1 1 MB ()

Kinesis Data Streams 256 Unicode MD5 128

client.putRecords client.putRecord KinesisData Streams

Note

Kinesis Client LibraryKinesis Client Library Kinesis Client Library Kinesis ClientLibrary Amazon DynamoDB 1

4

Amazon Kinesis Data Streams Data Streams

Kinesis Client Library 2 Amazon Kinesis Data Streams (p. 117)

Amazon Kinesis Data Streams application AWS Amazon DynamoDB Amazon CloudWatch

Amazon Kinesis Data Streams Kinesis Data Streams AWS KMS (p. 82)

Note

the section called KMS (p. 86)Note

AWS Key Management Service (AWS KMS) AWS Key Management Service


Kinesis Data Streams (p. 8)

(p. 7) (p. 8)

Kinesis Data Stream AWS UpdateShardCount API / Kinesis Data Streams

( KB ) (average_data_size_in_KB)

1 (records_per_second)

5

http:///kms/pricingAPI_UpdateShardCount.html


() Kinesis Data Streams (number_of_consumers)

KB (incoming_write_bandwidth_in_KB)average_data_size_in_KB records_per_second

KB (outgoing_read_bandwidth_in_KB)incoming_write_bandwidth_in_KB number_of_consumers

(number_of_shards)

number_of_shards = max(incoming_write_bandwidth_in_KB/1000, outgoing_read_bandwidth_in_KB/2000)

Kinesis Data Streams Kinesis Data Streams API AWSCommand Line Interface (AWS CLI)

1. AWS https://console.aws.amazon.com/kinesis Kinesis

2. 3. [] 4. [Kinesis ] [Kinesis

]

[Kinesis ] [] [Creating] [] [Active]

5. []

Kinesis Data Streams API

Kinesis Data Streams API (p. 39)

AWS CLI

AWS CLI create-stream

Kinesis Data Streams Kinesis Data Streams API AWS CLI

Note

6

https://console.aws.amazon.com/kinesiskinesis/create-stream.html


1. https://console.aws.amazon.com/kinesis/ Amazon Kinesis 2. 3. []

4. [] []

5. [] []

KMS Kinesis aws/kinesis AWS KMS AWS KMS AWS KMS thesection called KMS (p. 86)

6. [] []

7. [] [] the section calledCloudWatch (p. 52)

API API

AddTagsToStream DecreaseStreamRetentionPeriod DisableEnhancedMonitoring EnableEnhancedMonitoring IncreaseStreamRetentionPeriod RemoveTagsFromStream StartStreamEncryption StopStreamEncryption UpdateShardCount

AWS CLI AWS CLI Kinesis CLI

Kinesis Data Streams Amazon Kinesis Kinesis data stream (p. 8)

Important

Kinesis Data Streams (p. 49)

7

kinesis/API_AddTagsToStream.htmlAPI_DecreaseStreamRetentionPeriod.htmlAPI_DisableEnhancedMonitoring.htmlAPI_EnableEnhancedMonitoring.htmlAPI_IncreaseStreamRetentionPeriod.htmlAPI_RemoveTagsFromStream.htmlAPI_StartStreamEncryption.htmlAPI_StopStreamEncryption.htmlAPI_UpdateShardCount.htmlkinesis/index.html


BLOB

(p. 101) (Java )Kinesis DataStreams API PutRecords PutRecord put-record

Kinesis Data Streams Amazon Kinesis Data Streams application Kinesis

Amazon Simple Storage Service (Amazon S3)Amazon RedshiftAmazon Elasticsearch Service(Amazon ES)Splunk Kinesis Data Firehose Kinesis Data Firehose Amazon Kinesis Firehose Amazon Kinesis Data Streams (p. 117)

Amazon Machine Image (AMI) Amazon EC2 Auto Scaling Amazon EC2 Auto Scaling EC2 Auto Scaling EC2 AutoScaling CPU EC2 Amazon EC2 Auto Scaling

Kinesis Data Streams Amazon Kinesis Data Streams

1 1 1 MiB ()

1 1,000 5,000 1 5 GiB 500 AWS UpdateShardCount API

AWS () () 500 200

Base64 1 MiB

8

API_PutRecords.htmlAPI_PutRecord.htmlkinesis/put-record.htmlbasic-create.htmlhttps://docs.aws.amazon.com/autoscaling/ec2/userguide/API_UpdateShardCount.html

Amazon Kinesis Data Streams API

GetRecords 1 10 MiB 10,000GetRecords 1

1 5 10,000 10 MiB

GetRecords 2 MiB/GetRecords 10 MiB 5

API AWS API Kinesis Data Streams API API Amazon Kinesis API ReferenceAPI

API

1. AWS () 2. Kinesis Data Streams

9

API_GetRecords.htmlAPI_GetRecords.htmlhttps://docs.aws.amazon.com/kinesis/latest/APIReference/https://console.aws.amazon.com/support/v1#/case/create%3FissueType=service-limit-increase%26limitType=service-code-kinesis



Amazon Kinesis Data Streams Kinesis Data StreamsAmazon Kinesis Data Streams (p. 1)

Amazon Kinesis Data Streams (p. 10) : Amazon Kinesis Data Streams (p. 11) : AWS CLI Amazon Kinesis Data Streams (p. 17) : Kinesis Data Streams (p. 24)

Amazon Kinesis Data Streams Amazon Kinesis Data Streams

AWS (p. 10) (p. 11) (p. 11)

AWS (AWS) AWS AWS (Kinesis Data Streams )

AWS AWS

AWS

1. https://aws.amazon.com/ [AWS ]

Note

AWS AWS [Sign in to a different account ()] IAM [Sign-in using root account credentials ()] [ AWS ]

2.

10



Amazon Kinesis API Reference Kinesis Data Streams Java Amazon Kinesis Data Streams API AWS SDK for Java (p. 100) AWS SDK for Java Kinesis Data Streams API (p. 137) (p. 39)

GoJavaJavaScript.NETNode.jsPHPPython Ruby AWS SDK KinesisData Streams AWS SDK for Java Kinesis Data Streams GitHub

Kinesis Client Library (KCL) KCL Kinesis Data Streams JavaNode.js.NETPythonRuby (p. 117)

AWS Command Line InterfaceKinesis Data Streams AWS CLI AWS

KCL Java

Java 1.7 (Java SE 7 JDK) Java Oracle Java SE

Apache Commons (HTTP ) Jackson JSON

AWS SDK for Java Apache Commons Jackson SDK for Java Java 1.6 Kinesis Client Library Java 1.7

: Amazon Kinesis Data Streams

Amazon Kinesis Data Streams KinesisData Streams (p. 5) (p. 7) (p. 8) (Amazon Kinesis Data Streams (p. 1))

JavaScript N DynamoDB

11

https://docs.aws.amazon.com/kinesis/latest/APIReference/https://docs.aws.amazon.com/sdk-for-go/api/service/kinesis/developers/getting-started/java/sdkforbrowser/developers/getting-started/net/developers/getting-started/nodejs/developers/getting-started/php/https://github.com/boto/botodevelopers/getting-started/ruby/https://github.com/aws/aws-sdk-java/tree/master/src/sampleshttp://www.oracle.com/technetwork/java/javase/downloads/index.htmlhttp://www.oracle.com/technetwork/java/javase/downloads/index.htmlsdkforjava/


Kinesis Data Streams Kinesis Data Streams URL Kinesis data stream URL

10 N 3 2

AWS CloudFormation AWS CloudFormation AWS Amazon EC2 Amazon DynamoDB

Note

Kinesis Data Streams AWS AWS 3: (p. 16)


AWS (p. 5) (p. 7) (p. 8)

1: AWS AWS CloudFormation Kinesis data stream HTTPS

1. AWS CloudFormation 2. [] URL []

12

cloudformation/freecloudformation/home?region=us-east-1#/stacks/new?stackName=KinesisDataVisSampleApp&templateURL=https:%2F%2Fs3.amazonaws.com%2Fkinesis-demo-bucket%2Famazon-kinesis-data-visualization-sample%2Fkinesis-data-vis-sample-app.template

Amazon Kinesis Data Streams 2:

3. [Specify Details ()] t2.micro T2 VPC AWS VPC [InstanceType] m3.medium []

4. [] EC2 []

5. [Review ()] [I acknowledge that this template might cause AWS CloudFormation tocreate IAM resources ( AWS CloudFormation IAM )] [Create ()]

CREATE_IN_PROGRESS KinesisDataVisSample CREATE_COMPLETE

2:

Kinesis Data Stream (p. 13) (p. 14) (p. 15)

Kinesis Data Stream (p. 5) 1

AWS CloudFormation AWSCloudFormation CreateStream

1. [KinesisDataVisSample] 2. [Outputs ()] URL URL http://ec2-xx-xx-xx-

xx.compute-1.amazonaws.com3. 10

[Kinesis Data Streams Data VisualizationSample] 10 URL 1 2

13

https://github.com/awslabs/amazon-kinesis-data-visualization-sample/blob/master/src/main/static-content/cloudformation/kinesis-data-vis-sample-app.template#L74-80API_CreateStream.html


1. https://console.aws.amazon.com/kinesis Kinesis 2. ( KinesisDataVisSampleApp-KinesisStream-[randomString])

3. 4.

(p. 7) Kinesis data stream PutRecord

PutRecord

14

https://console.aws.amazon.com/kinesis/API_PutRecord.html


6 URL 2 URL HttpReferrerKinesisPutter 99132 Kinesis Data Streams 3 PutRecord pair.getResource HttpReferrerStreamWriter 85 92 6 URL

Kinesis Data Streams EC2 EC2 EC2 EC2

1. https://console.aws.amazon.com/ec2/) Amazon EC2 2. [] 3.

KinesisDataVisSample

4. []

(p. 8) Kinesis data stream GetShardIterator GetRecords

ID

() DynamoDB DynamoDB

2 URL N N 3 2 Kinesis Data Streams DynamoDB

Amazon DynamoDB

1. https://console.aws.amazon.com/dynamodb/ DynamoDB 2. [Tables ()] 3. 2

15

https://github.com/awslabs/amazon-kinesis-data-visualization-sample/blob/master/src/main/java/com/amazonaws/services/kinesis/samples/datavis/producer/HttpReferrerKinesisPutter.java#L99-132https://github.com/awslabs/amazon-kinesis-data-visualization-sample/blob/d7fbcb994caad606635414b01c4eeada15c04f0b/src/main/java/com/amazonaws/services/kinesis/samples/datavis/HttpReferrerStreamWriter.java#L85-92https://console.aws.amazon.com/ec2//API_GetShardIterator.html/API_GetRecords.htmlhttps://console.aws.amazon.com/dynamodb/


KinesisDataVisSampleApp-KCLDynamoDBTable-[randomString] KinesisDataVisSampleApp-CountsDynamoDBTable-[randomString]

N 4. Select the KinesisDataVisSampleApp-KCLDynamoDBTable-[randomString] 2

(leaseKey) (checkpoint) (leaseOwner)

5. Select the KinesisDataVisSampleApp-CountsDynamoDBTable-[randomString] (referrerCounts)

Kinesis (KCL)

Kinesis Client Library (KCL) KCL KCL

KCL KCL GetShardIterator IRecordProcessor KCL KCL KCL GetRecords IRecordProcessor

HttpReferrerCounterApplication 9298 KCL KCL

HttpReferrerCounterApplication 104108 IRecordProcessor KCL KCL

CountingRecordProcessor 186 203 IRecordProcessor N

3: 2 AWS AWS CloudFormation

1. https://console.aws.amazon.com/cloudformation AWS CloudFormation 2. 3. [][Delete Stack ()] 4. [Yes, Delete]

AWS CloudFormation [DELETE_IN_PROGRESS] AWS CloudFormation

4: GitHub

16

https://github.com/awslabs/amazon-kinesis-clienthttps://github.com/awslabs/amazon-kinesis-data-visualization-sample/blob/master/src/main/java/com/amazonaws/services/kinesis/samples/datavis/HttpReferrerCounterApplication.java#L92-98https://github.com/awslabs/amazon-kinesis-data-visualization-sample/blob/master/src/main/java/com/amazonaws/services/kinesis/samples/datavis/HttpReferrerCounterApplication.java#L104-108https://github.com/awslabs/amazon-kinesis-data-visualization-sample/blob/d7fbcb994caad606635414b01c4eeada15c04f0b/src/main/java/com/amazonaws/services/kinesis/samples/datavis/kcl/CountingRecordProcessor.java#L186-203https://console.aws.amazon.com/cloudformation/https://github.com/awslabs/amazon-kinesis-data-visualization-sample

Amazon Kinesis Data Streams : CLI

Kinesis Data Streams API Amazon Kinesis Data StreamsAPI AWS SDK for Java (p. 100)AWS SDK for Java Kinesis Data Streams API (p. 137) (p. 39)

SDK Kinesis Data Streams AWS SDKfor Java

: AWS CLI AmazonKinesis Data Streams

AWS Command Line Interface Amazon Kinesis DataStreams Kinesis Data Streams Kinesis data stream

For CLI access, you need an access key ID and secret access key. Use IAM user access keys instead ofAWS access keys. IAM lets you securely control access to AWS services andresources in your AWS account. For more information about creating access keys, see Understanding andGetting Your Security Credentials in the AWS General Reference.

IAM IAM

() Kinesis Data Streams

AWS CLI (p. 17) (p. 19)

AWS CLI AWS CLI Windows LinuxOS XUnix AWS CLI

Windows

1. AWS Command Line Interface Windows MSI

2. MSI 3.

Linux, macOS, or Unix

Python 2.6.5 AWS Command Line Interface

1. pip

17

developers/getting-started/java/developers/getting-started/java/https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.htmlhttps://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.htmlget-set-up-for-amazon-ec2.html#create-an-iam-user#ak_region#ak_regioninstalling.htmlinstalling.htmlhttps://pip.pypa.io/en/latest/installing.html

Amazon Kinesis Data Streams AWS CLI

curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"sudo python get-pip.py

2. pip AWS CLI

sudo pip install awscli

aws help

Kinesis Data Streams Kinesis Data Streams AWS CLI

aws kinesis help


AVAILABLE COMMANDS

o add-tags-to-stream

o create-stream

o delete-stream

o describe-stream

o get-records

o get-shard-iterator

o help

o list-streams

o list-tags-for-stream

o merge-shards

o put-record

o put-records

o remove-tags-from-stream

o split-shard

o wait

Amazon Kinesis API Kinesis DataStreams API create-stream CreateStream API

AWS CLI

18


AWS CLI aws configure AWS CLI AWS CLI 1

aws configureAWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLEAWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEYDefault region name [None]: us-west-2Default output format [None]: json

AWS CLI 4 AWS ID AWS

Note

AWS CLI AWS

JSONtexttable JSON

aws configure AWSCommand Line Interface AWS

AWS CLI Kinesis data stream Kinesis Data Streams (p. 2): AmazonKinesis Data Streams (p. 11)

Note

Kinesis Data Streams AWS Kinesis Data Streams AWS 4: (p. 23)

1: (p. 19) 2: (p. 21) 3: (p. 21) 4: (p. 23)

1: Foo

19

cli-chap-getting-set-up.html#cli-signupcli-chap-getting-set-up.html#cli-signuprande.htmlcli-chap-getting-started.html


aws kinesis create-stream --stream-name Foo --shard-count 1

--shard-count 1

aws kinesis describe-stream --stream-name Foo

{ "StreamDescription": { "StreamStatus": "CREATING", "StreamName": "Foo", "StreamARN": "arn:aws:kinesis:us-west-2:account-id:stream/Foo", "Shards": [] }}

CREATING

{ "StreamDescription": { "StreamStatus": "ACTIVE", "StreamName": "Foo", "StreamARN": "arn:aws:kinesis:us-west-2:account-id:stream/Foo", "Shards": [ { "ShardId": "shardId-000000000000", "HashKeyRange": { "EndingHashKey": "170141183460469231731687303715884105727", "StartingHashKey": "0" }, "SequenceNumberRange": { "StartingSequenceNumber": "49546986683135544286507457935754639466300920667981217794" } } ] }}

"StreamStatus": "ACTIVE" list-streams

aws kinesis list-streams

:

{ "StreamNames": [ "Foo" ]}

20


2: put-record "testdata"

aws kinesis put-record --stream-name Foo --partition-key 123 --data testdata

{ "ShardId": "shardId-000000000000", "SequenceNumber": "49546986683135544286507457936321625675700192471156785154"}

3: ( get-record ) get-shard-iterator

aws kinesis get-shard-iterator --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON --stream-name Foo

aws kinesis Kinesis Data Streams API GetShardIterator API ()

{ "ShardIterator": "AAAAAAAAAAHSywljv0zEgPX4NyKdZ5wryMzP9yALs8NeKbUjp1IxtZs1Sp+KEd9I6AJ9ZG4lNR1EMi+9Md/nHvtLyxpfhEzYvkTZ4D9DQVz/mBYWRO6OTZRKnW9gd+efGN2aHFdkH1rJl4BL9Wyrk+ghYG22D2T1Da2EyNSH1+LAbK33gQweTJADBdyMwlo5r6PqcP2dzhg="}

() get 300 get-shard-iterator

get-records Kinesis Data Streams API GetRecordsGetRecords

get-records ()

aws kinesis get-records --shard-iterator AAAAAAAAAAHSywljv0zEgPX4NyKdZ5wryMzP9yALs8NeKbUjp1IxtZs1Sp+KEd9I6AJ9ZG4lNR1EMi+9Md/nHvtLyxpfhEzYvkTZ4D9DQVz/mBYWRO6OTZRKnW9gd+efGN2aHFdkH1rJl4BL9Wyrk+ghYG22D2T1Da2EyNSH1+LAbK33gQweTJADBdyMwlo5r6PqcP2dzhg=

21

API_GetShardIterator.htmlAPI_GetRecords.html


bash Unix ()

SHARD_ITERATOR=$(aws kinesis get-shard-iterator --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON --stream-name Foo --query 'ShardIterator')

aws kinesis get-records --shard-iterator $SHARD_ITERATOR

PowerShell ()

aws kinesis get-records --shard-iterator ((aws kinesis get-shard-iterator --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON --stream-name Foo).split('"')[4])

get-records ()

{ "Records":[ { "Data":"dGVzdGRhdGE=", "PartitionKey":"123, "ApproximateArrivalTimestamp": 1.441215410867E9, "SequenceNumber":"49544985256907370027570885864065577703022652638596431874" } ], "MillisBehindLatest":24000, "NextShardIterator":"AAAAAAAAAAEDOW3ugseWPE4503kqN1yN1UaodY8unE0sYslMUmC6lX9hlig5+t4RtZM0/tALfiI4QGjunVgJvQsjxjh2aLyxaAaPr+LaoENQ7eVs4EdYXgKyThTZGPcca2fVXYJWL3yafv9dsDwsYVedI66dbMZFC8rPMWc797zxQkv4pSKvPOZvrUIudb8UkH3VMzx58Is="}

get-records ()

testdata put-record Base64 AWS CLI Kinesis Data Streams Base64 Base64 raw stdout Base64 (https://www.base64decode.org/ ) dGVzdGRhdGE= testdata AWS CLI (describe-stream list-streams)Kinesis (KCL) Base64 KCL Kinesis Client Library 1.x (p. 118)

get-records / NextShardIterator () get-records 300

22

https://www.base64decode.org/


get-records get-shard-iterator

MillisBehindLatest GetRecords () 24 168 (7 )

get-records NextShardIterator KCL

get-records ()

{ "Records": [], "NextShardIterator": "AAAAAAAAAAGCJ5jzQNjmdhO6B/YDIDE56jmZmrmMA/r1WjoHXC/kPJXc1rckt3TFL55dENfe5meNgdkyCRpUPGzJpMgYHaJ53C3nCAjQ6s7ZupjXeJGoUFs5oCuFwhP+Wul/EhyNeSs5DYXLSSC5XCapmCAYGFjYER69QSdQjxMmBPE/hiybFDi5qtkT6/PsZNz6kFoqtDk="}

4:

aws kinesis delete-stream --stream-name Foo

describe-stream

aws kinesis describe-stream --stream-name Foo

delete

{ "StreamDescription": { "StreamStatus": "DELETING", "StreamName": "Foo", "StreamARN": "arn:aws:kinesis:us-west-2:account-id:stream/Foo", "Shards": [] }}

describe-stream not found

A client error (ResourceNotFoundException) occurred when calling the DescribeStream operation: Stream Foo under account 112233445566 not found.

23

https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html

Amazon Kinesis Data Streams :

: Kinesis Data Streams


Important

Kinesis Data Streams AWS Kinesis Data Streams Amazon DynamoDB DynamoDB AWS 7: (p. 37)

2015 2 25 5 ()

(: Amazon Elastic Compute Cloud (Amazon EC2))

() Kinesis Data Streams AWS

(p. 24) 1: (p. 25) 2: IAM (p. 26) 3: (p. 29) 4: (p. 30) 5: (p. 33) 6: () (p. 36) 7: (p. 37)

: Kinesis Data Streams (p. 24)

Amazon Web Services Kinesis Data Streams (p. 2): Amazon KinesisData Streams (p. 11)

24

#ak_region#ak_region


:Amazon Kinesis Data Streams (p. 11):AWS CLI Amazon Kinesis Data Streams (p. 17)

AWS AWS

IAM IAM AWS IAM AWS AWS How Do I GetSecurity Credentials?(AWS General Reference)

IAM IAM

Java 7 Java Development Kit (JDK) Oracle Java SE

Eclipse Java IDE

AWS SDK for Java Eclipse IDE AWSToolkit for Eclipse

Kinesis Client Library (KCL) 1.2.1 GitHub Kinesis Client Library (Java)

1: (p. 25)

1: : Kinesis Data Streams (p. 24)

1. AWS https://console.aws.amazon.com/kinesis Kinesis

2. [] 3. 4. [Kinesis ] 5. (: StockTradeStream) 6. 1[] 7. [Kinesis ]

[Kinesis ] CREATING ACTIVE [] []

25

https://console.aws.amazon.com/console/homehttps://docs.aws.amazon.com/IAM/latest/UserGuide/console.htmlhttps://docs.aws.amazon.com/general/latest/gr/getting-aws-sec-creds.htmlhttps://docs.aws.amazon.com/general/latest/gr/getting-aws-sec-creds.htmlget-set-up-for-amazon-ec2.html#create-an-iam-userhttp://www.oracle.com/technetwork/java/javase/downloads/index.htmlhttp://www.oracle.com/technetwork/java/javase/downloads/index.htmlhttps://www.eclipse.org/downloads/sdk-for-java/eclipse/eclipse/https://github.com/awslabs/amazon-kinesis-clienthttps://console.aws.amazon.com/kinesis

Amazon Kinesis Data Streams 2: IAM

Kinesis Data Streams ()

[Kinesis ] [] 1

1 20 1 250 1 () 5,000

()

Kinesis Data Streams (p. 5) (p. 39)

2: IAM (p. 26)

2: IAM AWS AWS Identity and Access Management (IAM) AWS IAM


DescribeStream Kinesis data stream

PutRecordPutRecords Kinesis data stream Kinesis Data Streams

26

PoliciesOverview.html


DescribeStream Kinesis data stream

GetRecordsGetShardIteratorKinesis data stream Kinesis Data Streams

CreateTableDescribeTableGetItemPutItemScanUpdateItemAmazon DynamoDB

Kinesis (KCL) DynamoDB

DeleteItem Amazon DynamoDB


PutMetricData Amazon CloudWatch

KCL CloudWatch

IAM 1 2

IAM

1. Amazon (ARN) ARN [ ARN] []ARN

arn:aws:kinesis:region:account:stream/name

(us-west-2 )

AWS ID ()name

1: (p. 25) (StockTradeStream)

2. ()DynamoDB ARN

arn:aws:dynamodb:region:account:table/name

KCL StockTradesProcessor

3. IAM [] (https://console.aws.amazon.com/iam/home#policies) [] IAM [][]

4. [] [] 5. AWS [Amazon Kinesis]

27

using-regions-availability-zones.html#concepts-regions-availability-zonesusing-regions-availability-zones.html#concepts-regions-availability-zonesbilling/home?#/accountiam/home#policies


6. DescribeStreamGetShardIteratorGetRecordsPutRecord PutRecords

7. 1 ARN 8. []

AWS ARN

Amazon DynamoDB CreateTableDeleteItemDescribeTableGetItemPutItemScanUpdateItem 2 ARN

Amazon CloudWatch PutMetricData *

ARN (*) PutMetricData CloudWatch

9. [Next Step] 10. [] StockTradeStreamPolicy []

{ "Version": "2012-10-17", "Statement": [ { "Sid": "Stmt123", "Effect": "Allow", "Action": [ "kinesis:DescribeStream", "kinesis:PutRecord", "kinesis:PutRecords", "kinesis:GetShardIterator", "kinesis:GetRecords" ], "Resource": [ "arn:aws:kinesis:us-west-2:123:stream/StockTradeStream" ] }, { "Sid": "Stmt456", "Effect": "Allow", "Action": [ "dynamodb:*" ], "Resource": [ "arn:aws:dynamodb:us-west-2:123:table/StockTradesProcessor" ] }, { "Sid": "Stmt789", "Effect": "Allow", "Action": [ "cloudwatch:PutMetricData" ], "Resource": [ "*" ] } ]

28


}

IAM

1. https://console.aws.amazon.com/iam/ IAM 2. [] [] 3. [User name] StockTradeStreamUser 4. [] [] [: ]

5. [Attach existing policies directly] 6. [: ]

7. [] 8. [ ID] [] []

9.

~/.aws/credentials

[default]aws_access_key_id=access keyaws_secret_access_key=secret access key

IAM

1. IAM [] [] 2. [StockTradeStreamPolicy] [] 3. [StockTradeStreamUser] []

3: (p. 29)

3: the section called : (p. 24) () ()

1. 2. IDE

3.

Amazon Kinesis Client Library (KCL) AWS SDK Apache HttpCore Apache HttpClient

29

https://console.aws.amazon.com/iam/iam/home?#policies


Apache Commons Lang Apache Commons Logging Guava (Java Google ) Jackson Annotations Jackson Core Jackson Databind Jackson Dataformat: CBOR Joda Time

4. IDE IDE

(the section called 4: (p. 30))

(p. 30)

4: : Kinesis Data Streams (p. 24)

StockTrade

StockTrade () ID

JSON StockTrade 1 (:

{ "tickerSymbol": "AMZN", "tradeType": "BUY", "price": 395.87, "quantity": 16, "id": 3567129045}

StockTradeGenerator

StockTradeGenerator getRandomTrade()

StockTradesWriter

main StockTradesWriter Kinesis Data Streams

30


1. 2. AmazonKinesisClientBuilder 3. 4. AmazonKinesis 5. (

)6. StockTradeGenerator.getRandomTrade()

sendStockTrade 100

sendStockTrade StockTradesWriter

private static void sendStockTrade(StockTrade trade, AmazonKinesis kinesisClient, String streamName) { byte[] bytes = trade.toJsonAsBytes(); // The bytes could be null if there is an issue with the JSON serialization by the Jackson JSON library. if (bytes == null) { LOG.warn("Could not get JSON bytes for stock trade"); return; } LOG.info("Putting trade: " + trade.toString()); PutRecordRequest putRecord = new PutRecordRequest(); putRecord.setStreamName(streamName); // We use the ticker symbol as the partition key, explained in the Supplemental Information section below. putRecord.setPartitionKey(trade.getTickerSymbol()); putRecord.setData(ByteBuffer.wrap(bytes));

try { kinesisClient.putRecord(putRecord); } catch (AmazonClientException ex) { LOG.warn("Error sending record to Amazon Kinesis.", ex); }}

PutRecord API trade JSON

1

byte[] bytes = trade.toJsonAsBytes();

PutRecordRequest (putRecord )

PutRecordRequest putRecord = new PutRecordRequest();

PutRecord BLOB putRecord setXxxx()

putRecord.setStreamName(streamName);putRecord.setPartitionKey(trade.getTickerSymbol());putRecord.setData(ByteBuffer.wrap(bytes));

31


1 (p. 101)

putRecord (put )

kinesisClient.putRecord(putRecord);

if (bytes == null) { LOG.warn("Could not get JSON bytes for stock trade"); return;}

put try/catch

try { kinesisClient.putRecord(putRecord);} catch (AmazonClientException ex) { LOG.warn("Error sending record to Amazon Kinesis.", ex);}

KinesisData Streams put put

LOG.info("Putting trade: " + trade.toString());

Kinesis Data Streams API (PutRecord) PutRecords (p. 101)

1. (IAM ) ~/.aws/credentials

2. StockTradeWriter

StockTradeStream us-west-2

us-west-2

Feb 16, 2015 3:53:00 PM com.amazonaws.services.kinesis.samples.stocktrades.writer.StockTradesWriter sendStockTradeINFO: Putting trade: ID 8: SELL 996 shares of BUD for $124.18

32


Feb 16, 2015 3:53:00 PM com.amazonaws.services.kinesis.samples.stocktrades.writer.StockTradesWriter sendStockTradeINFO: Putting trade: ID 9: BUY 159 shares of GE for $20.85Feb 16, 2015 3:53:01 PM com.amazonaws.services.kinesis.samples.stocktrades.writer.StockTradesWriter sendStockTradeINFO: Putting trade: ID 10: BUY 322 shares of WMT for $90.08


5: (p. 33)

5: : Kinesis Data Streams (p. 24) (p. 30)1 KinesisClient Library (KCL) Kinesis Client Library 1.x (p. 118)

StockTradesProcessor

~/.aws/credentials RecordProcessorFactory RecordProcessor

StockTradeRecordProcessor RecordProcessorFactory (:

) KCL ()

Kinesis Data Streams RecordProcessor

StockTradeRecordProcessor

RecordProcessor initializeprocessRecordsshutdown 3

Kinesis Client Library initialize shutdown processRecords processRecord

processRecord reportStats resetStats

processsRecords processRecord

33


1 reportStats() resetStats()

1 checkpoint()

60 (p. 35)

StockStats

addStockTrade(StockTrade): StockTrade toString():

StockTradeRecordProcessor

1. processRecord StockTrade

StockTrade trade = StockTrade.fromJsonAsBytes(record.getData().array());if (trade == null) { LOG.warn("Skipping record. Unable to parse record into StockTrade. Partition Key: " + record.getPartitionKey()); return;}stockStats.addStockTrade(trade);

2. reportStats

System.out.println("****** Shard " + kinesisShardId + " stats for last 1 minute ******\n" + stockStats + "\n" + "****************************************************************\n");

3. resetStats stockStats

stockStats = new StockStats();

1. (p. 30)

2. (IAM ) ~/.aws/credentials

3. StockTradesProcessor

34


StockTradesProcessor StockTradeStream us-west-2

us-west-2

1 1

****** Shard shardId-000000000001 stats for last 1 minute ****** Most popular stock being bought: WMT, 27 buys. Most popular stock being sold: PTR, 14 sells. ****************************************************************

Kinesis Client Library 1.x (p. 118) KinesisClient Library 1 1 KCL KCL

KCL Kinesis Data Streams

(p. 30)Kinesis Data Streams API put Kinesis Data Streams ()

1 KCL KCL

6: () (p. 36)

35

Amazon Kinesis Data Streams 6: ()

6: () : Kinesis Data Streams (p. 24)

1 3 StockStats

1.

// Ticker symbol of the stock that had the largest quantity of shares sold private String largestSellOrderStock; // Quantity of shares for the largest sell order trade private long largestSellOrderQuantity;

2. addStockTrade

if (type == TradeType.SELL) { if (largestSellOrderStock == null || trade.getQuantity() > largestSellOrderQuantity) { largestSellOrderStock = trade.getTickerSymbol(); largestSellOrderQuantity = trade.getQuantity(); } }

3. toString

public String toString() { return String.format( "Most popular stock being bought: %s, %d buys.%n" + "Most popular stock being sold: %s, %d sells.%n" + "Largest sell order: %d shares of %s.", getMostPopularStock(TradeType.BUY), getMostPopularStockCount(TradeType.BUY), getMostPopularStock(TradeType.SELL), getMostPopularStockCount(TradeType.SELL), largestSellOrderQuantity, largestSellOrderStock); }

()

****** Shard shardId-000000000001 stats for last 1 minute ****** Most popular stock being bought: WMT, 27 buys. Most popular stock being sold: PTR, 14 sells. Largest sell order: 996 shares of BUD. ****************************************************************

7: (p. 37)

36


7: Kinesis data stream Amazon DynamoDB ""

1. 2. https://console.aws.amazon.com/kinesis Kinesis 3. (StockTradeStream)4. [] 5. https://console.aws.amazon.com/dynamodb/ DynamoDB 6. StockTradesProcessor

Kinesis Data Streams (processRecord(Record) ) Kinesis Data Streams Kinesis Data Streams

() 1 1 ( 1 1 )

() AmazonEC2 Auto Scaling

Amazon S3/DynamoDB/Amazon Redshift/Storm

AWS Kinesis Data Streams AWS

Kinesis Data Streams API Amazon Kinesis Data Streams

API AWS SDK for Java (p. 100)AWS SDK for Java Kinesis Data Streams API (p. 137) (p. 39)

37

https://console.aws.amazon.com/kinesishttps://console.aws.amazon.com/dynamodb/https://github.com/awslabs/amazon-kinesis-connectors


Kinesis Client Library Kinesis Client Library 1.x (p. 118)

(p. 160)

38


Amazon Kinesis Data Streams API AWS SDK for Java Kinesis data stream

Java Kinesis Data Streams API Kinesis Data Streams API AWS SDK

(p. 39) (p. 41) (p. 42) (p. 43) (p. 43) (p. 44) (p. 49) Amazon Kinesis Data Streams (p. 49) Amazon Kinesis Data Streams (p. 52) Amazon Kinesis Data Streams IAM (p. 78) (p. 82) Amazon Kinesis Data Streams VPC (p. 87) Kinesis (p. 88)

Kinesis data stream

Kinesis Data Streams Kinesis data stream Java

AmazonKinesisClientBuilder clientBuilder = AmazonKinesisClientBuilder.standard(); clientBuilder.setRegion(regionName);clientBuilder.setCredentials(credentialsProvider);clientBuilder.setClientConfiguration(config); AmazonKinesis client = clientBuilder.build();

39

sdk-for-java/developers/getting-started/


AWS General Reference Kinesis Data Streams

Kinesis Data Streams Kinesis Data Streams CreateStreamRequest

CreateStreamRequest createStreamRequest = new CreateStreamRequest();createStreamRequest.setStreamName( myStreamName );createStreamRequest.setShardCount( myStreamSize );

AWS 2 AWS 2 AWS 2 2 2

AWSKinesis DataStream (p. 5)

createStreamRequest createStream createStream ACTIVE describeStream describeStream describeStream try/catch

client.createStream( createStreamRequest );DescribeStreamRequest describeStreamRequest = new DescribeStreamRequest();describeStreamRequest.setStreamName( myStreamName );

long startTime = System.currentTimeMillis();long endTime = startTime + ( 10 * 60 * 1000 );while ( System.currentTimeMillis() < endTime ) { try { Thread.sleep(20 * 1000); } catch ( Exception e ) {} try { DescribeStreamResult describeStreamResponse = client.describeStream( describeStreamRequest ); String streamStatus = describeStreamResponse.getStreamDescription().getStreamStatus(); if ( streamStatus.equals( "ACTIVE" ) ) { break; } // // sleep for one second // try { Thread.sleep( 1000 ); } catch ( Exception e ) {} } catch ( ResourceNotFoundException e ) {}

40

#ak_region


}if ( System.currentTimeMillis() >= endTime ) { throw new RuntimeException( "Stream " + myStreamName + " never went active" );}

Kinesis Data Streams AWS AWS AWS 1 Kinesis Data Streams AWS

ListStreamsRequest listStreamsRequest = new ListStreamsRequest();listStreamsRequest.setLimit(20); ListStreamsResult listStreamsResult = client.listStreams(listStreamsRequest);List streamNames = listStreamsResult.getStreamNames();

ListStreamsRequest setLimit 20 listStreams setLimit KinesisData Streams listStreamsRequest listStreams listStreams ListStreamsResult getStreamNames streamNames Kinesis Data Streams getHasMoreStreams

while (listStreamsResult.getHasMoreStreams()) { if (streamNames.size() > 0) { listStreamsRequest.setExclusiveStartStreamName(streamNames.get(streamNames.size() - 1)); } listStreamsResult = client.listStreams(listStreamsRequest); streamNames.addAll(listStreamsResult.getStreamNames());}

getHasMoreStreams listStreamsRequest listStreams setExclusiveStartStreamName listStreams setExclusiveStartStreamName listStreams streamNames

listStreams

CREATING ACTIVE UPDATING DELETING

41


describeStream (p. 39)

1 ListShards

import software.amazon.awssdk.services.kinesis.KinesisAsyncClient;import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;

import java.util.concurrent.TimeUnit;

public class ShardSample {

public static void main(String[] args) {

KinesisAsyncClient client = KinesisAsyncClient.builder().build();

ListShardsRequest request = ListShardsRequest .builder().streamName("myFirstStream") .build();

try { ListShardsResponse response = client.listShards(request).get(5000, TimeUnit.MILLISECONDS); System.out.println(response.toString()); } catch (Exception e) { System.out.println(e.getMessage()); } }}

POM

4.0.0

kinesis.data.streams.samples shards 1.0-SNAPSHOT org.apache.maven.plugins maven-compiler-plugin 8 8

42

API_ListShards.html


software.amazon.awssdk kinesis 2.0.0

describeStream getShards 1 getHasMoreShards getStreamDescription true getShards getHasMoreShards false getShards EXPIRED (EXPIRED ) (p. 48)

DescribeStreamRequest describeStreamRequest = new DescribeStreamRequest();describeStreamRequest.setStreamName( myStreamName );List shards = new ArrayList();String exclusiveStartShardId = null;do { describeStreamRequest.setExclusiveStartShardId( exclusiveStartShardId ); DescribeStreamResult describeStreamResult = client.describeStream( describeStreamRequest ); shards.addAll( describeStreamResult.getStreamDescription().getShards() ); if (describeStreamResult.getStreamDescription().getHasMoreShards() && shards.size() > 0) { exclusiveStartShardId = shards.get(shards.size() - 1).getShardId(); } else { exclusiveStartShardId = null; }} while ( exclusiveStartShardId != null );

Kinesis Data Streams DeleteStreamRequest

DeleteStreamRequest deleteStreamRequest = new DeleteStreamRequest();deleteStreamRequest.setStreamName(myStreamName);client.deleteStream(deleteStreamRequest);

ResourceNotFound

43


Important

UpdateShardCount API

Amazon Kinesis Data Streams Kinesis Data Streams Kinesis Data Streams

2 1 2 2 1 1 1 2

()

() () Amazon CloudWatch IAM API Kinesis Data Streams IAM Amazon Kinesis Data Streams IAM (p. 78)

(p. 44) (p. 45) 2 (p. 46) (p. 47)

Amazon Kinesis Data Streams () ()

1

44

API_UpdateShardCount.html


Kinesis Data Streams Amazon CloudWatch 1

putRecordRequest.setPartitionKey( String.format( "myPartitionKey" ) );

Kinesis Data Streams MD5 MD5

ID ID getShardId putRecordResults putRecords putRecordResult putRecord

String shardId = putRecordResult.getShardId();

ID

Amazon Kinesis Data Streams MD5

shard.getHashKeyRange().getStartingHashKey();shard.getHashKeyRange().getEndingHashKey();

1 1/3 1 2/3

myStreamName shard splitShardRequest ID

SplitShardRequest splitShardRequest = new SplitShardRequest();splitShardRequest.setStreamName(myStreamName);splitShardRequest.setShardToSplit(shard.getShardId());

45

http://en.wikipedia.org/wiki/MD5http://en.wikipedia.org/wiki/MD5

Amazon Kinesis Data Streams 2

setNewStartingHashKey Kinesis Data Streams Kinesis Data Streams splitShard

BigInteger startingHashKey = new BigInteger(shard.getHashKeyRange().getStartingHashKey());BigInteger endingHashKey = new BigInteger(shard.getHashKeyRange().getEndingHashKey());String newStartingHashKey = startingHashKey.add(endingHashKey).divide(new BigInteger("2")).toString();

splitShardRequest.setNewStartingHashKey(newStartingHashKey);client.splitShard(splitShardRequest);

(p. 47)

2 2 1 1 2

2 2 2 2 1 276381 1 382454 2 1 276454

2 1 276381 1 455560 2 1 ( 382454)

OPEN () MD5 (CLOSED ) (p. 48)

CLOSED OPEN (CLOSED ) null

if( null == shard.getSequenceNumberRange().getEndingSequenceNumber() ) { // Shard is OPEN, so it is a possible candidate to be merged.}

CLOSED

shard.getHashKeyRange().getEndingHashKey();

2

46


2 myStreamName shard1 shard2 2

mergeShardsRequest setStreamName setShardToMerge setAdjacentShardToMerge 2 Kinesis Data Streams mergeShards

MergeShardsRequest mergeShardsRequest = new MergeShardsRequest();mergeShardsRequest.setStreamName(myStreamName);mergeShardsRequest.setShardToMerge(shard1.getShardId());mergeShardsRequest.setAdjacentShardToMerge(shard2.getShardId());client.mergeShards(mergeShardsRequest);

(p. 47)


(p. 47) (p. 48)

splitShard mergeShards (p. 39)

DescribeStreamRequest describeStreamRequest = new DescribeStreamRequest();describeStreamRequest.setStreamName( myStreamName );

long startTime = System.currentTimeMillis();long endTime = startTime + ( 10 * 60 * 1000 );while ( System.currentTimeMillis() < endTime ) { try { Thread.sleep(20 * 1000); } catch ( Exception e ) {} try { DescribeStreamResult describeStreamResponse = client.describeStream( describeStreamRequest ); String streamStatus = describeStreamResponse.getStreamDescription().getStreamStatus(); if ( streamStatus.equals( "ACTIVE" ) ) { break; } // // sleep for one second //

47


try { Thread.sleep( 1000 ); } catch ( Exception e ) {} } catch ( ResourceNotFoundException e ) {}}if ( System.currentTimeMillis() >= endTime ) { throw new RuntimeException( "Stream " + myStreamName + " never went active" );}

Kinesis Data Streams KinesisData Streams API getShardIterator getRecords (p. 137) Kinesis Client Library

Note

(p. 49)

OPEN CLOSED EXPIRED

OPEN: OPEN

CLOSED: CLOSED

EXPIRED: EXPIRED getStreamDescription().getShards EXPIRED (p. 49)

ACTIVE getRecordsResult.getNextShardIterator nullKinesis Client Library

48


Amazon Kinesis Data Streams Kinesis data stream Kinesis data stream 24 168

IncreaseStreamRetentionPeriod 168 DecreaseStreamRetentionPeriod 24 DescribeStream

AWS CLI

aws kinesis increase-stream-retention-period --stream-name retentionPeriodDemo --retention-period-hours 72

Kinesis Data Streams 24 48 23 55 24

Kinesis Data Streams DecreaseStreamRetentionPeriod

API

24 Amazon Kinesis Data Streams


Amazon Kinesis Data Streams AWS ()

(p. 50) (p. 50) (p. 50) Kinesis Data Streams (p. 51) AWS CLI (p. 51)

49

https://docs.aws.amazon.com/kinesis/latest/APIReference/API_IncreaseStreamRetentionPeriod.htmlhttps://docs.aws.amazon.com/kinesis/latest/APIReference/API_DecreaseStreamRetentionPeriod.htmlhttps://docs.aws.amazon.com/kinesis/latest/APIReference/API_DescribeStream.htmlhttps://docs.aws.amazon.com/kinesis/latest/APIReference/API_DecreaseStreamRetentionPeriod.htmlhttps://aws.amazon.com/kinesis/pricing/


Kinesis Data Streams API (p. 52)

Kinesis Data Streams AWS CLI Kinesis Data Streams API

: : : : :

AWS AWS () AWS () AWS Billing and Cost Management

() 50

aws: AWS AWS

1128 Unicode Unicode (_ . / = +- @)

50


0255 Unicode

Unicode (_ . / = + - @)

Kinesis Data Streams Kinesis Data Streams

1. Kinesis Data Streams

2. [] 3. [] []


2. [] 3. [] [] 4. [] []

[]

[] (p. 50)

5. []


2. [Stream List] 3. [] [] [] 4. [] []

AWS CLI AWS CLI

add-tags-to-stream

list-tags-for-stream

51

kinesis/add-tags-to-stream.htmlkinesis/list-tags-for-stream.html

Amazon Kinesis Data Streams Kinesis Data Streams API

remove-tags-from-stream

Kinesis Data Streams API Kinesis Data Streams API

AddTagsToStream

ListTagsForStream

RemoveTagsFromStream



CloudWatch (p. 52) Kinesis Data Streams Amazon CloudWatch

Kinesis (p. 62) Kinesis CloudWatch

API (p. 63) Kinesis Data Streams AWS CloudTrail API Amazon S3

Kinesis (p. 66) Kinesis Client Library (KCL) KCL

Kinesis (p. 74) Kinesis Producer Library (KPL) KPL

Amazon CloudWatch Amazon Kinesis DataStreams Amazon Kinesis Data Streams Amazon CloudWatch Kinesis CloudWatch PutRecords.Bytes GetRecords.Bytes

1 CloudWatch 2

Kinesis

52

kinesis/remove-tags-from-stream.htmlAPI_AddTagsToStream.htmlAPI_ListTagsForStream.htmlAPI_RemoveTagsFromStream.html

Amazon Kinesis Data Streams CloudWatch

() 1

() 1EnableEnhancedMonitoring

Amazon CloudWatch

Amazon Kinesis Data Streams Kinesis Data Streams 2 CloudWatch EnableEnhancedMonitoring

CloudWatch Amazon CloudWatch CloudWatch

(p. 53) (p. 58) Amazon Kinesis Data Streams (p. 60) Amazon Kinesis Data Streams (p. 61)

AWS/Kinesis

Kinesis Data Streams 1 CloudWatch

GetRecords.Bytes Kinesis MinimumMaximum Average GetRecords

: OutgoingBytes

: StreamName

: MinimumMaximumAverageSumSamples

:

53

https://docs.aws.amazon.com/kinesis/latest/APIReference/API_EnableEnhancedMonitoring.htmlcloudwatchcloudwatchhttps://docs.aws.amazon.com/kinesis/latest/APIReference/API_EnableEnhancedMonitoring.htmlhttps://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/cloudwatch_concepts.html#Statistic


GetRecords.IteratorAge GetRecords.IteratorAgeMilliseconds

GetRecords.IteratorAgeMillisecondsKinesis GetRecords ()GetRecords Minimum Maximum Kinesis

: IteratorAgeMilliseconds

: StreamName

: MinimumMaximumAverageSamples

:

GetRecords.Latency GetRecords

: StreamName

: MinimumMaximumAverage

:

GetRecords.Records MinimumMaximum Average GetRecords

: OutgoingRecords

: StreamName


: Count

GetRecords.Success GetRecords

: StreamName

: AverageSumSamples

: Count

54


IncomingBytes Kinesis PutRecord PutRecords MinimumMaximum Average put

: IncomingBytes

: StreamName


:

IncomingRecords Kinesis PutRecord PutRecords MinimumMaximum Average put

: IncomingRecords

: StreamName


: Count

PutRecord.Bytes PutRecord Kinesis

: StreamName


:

PutRecord.Latency PutRecord

: StreamName


:

PutRecord.Success Kinesis PutRecord Average

: StreamName

: AverageSumSamples

: Count

55


PutRecords.Bytes PutRecords Kinesis

: StreamName


:

PutRecords.Latency PutRecords

: StreamName


:

PutRecords.Records Kinesis PutRecords

: StreamName


: Count

PutRecords.Success Kinesis 1 PutRecords

: StreamName

: AverageSumSamples

: Count

ReadProvisionedThroughputExceeded GetRecords Average

Minimum 1

Maximum 0 ()

:ReadProvisionedThroughputExceeded

: StreamName


: Count

56


SubscribeToShard.RateExceeded 1 : StreamNameConsumerName

SubscribeToShard.Success SubscribeToShard 5 5 1

: StreamNameConsumerName

SubscribeToShardEvent.Bytes MinimumMaximum Average

: OutgoingBytes



:

SubscribeToShardEvent.MillisBehindLatestSubscribeToShard



:

SubscribeToShardEvent.Records MinimumMaximum Average

: OutgoingRecords



: Count

SubscribeToShardEvent.Success



: Count

57


WriteProvisionedThroughputExceededPutRecord PutRecords Average

Minimum

Maximum 0 ()

:WriteProvisionedThroughputExceeded

: StreamName


: Count

AWS/Kinesis

Kinesis 1 CloudWatch Kinesis Amazon CloudWatch Amazon CloudWatch 1

IncomingBytes PutRecord PutRecords MinimumMaximum Average put

: IncomingBytes

: StreamNameShardId


:

IncomingRecords PutRecord PutRecords MinimumMaximum Average put

: IncomingRecords

58

cloudwatch/pricing/


: StreamNameShardId


: Count

IteratorAgeMilliseconds GetRecords ()GetRecords Minimum Maximum Kinesis 0 ()

:GetRecords.IteratorAgeMilliseconds

: StreamNameShardId


:

OutgoingBytes MinimumMaximum Average GetRecords SubscribeToShard

: GetRecords.Bytes

: StreamNameShardId


:

OutgoingRecords MinimumMaximum Average GetRecords SubscribeToShard

: GetRecords.Records

: StreamNameShardId


: Count

59


ReadProvisionedThroughputExceeded GetRecords 1 1 5 1 1 2 MB Average

Minimum 1

Maximum 0 ()

:ReadProvisionedThroughputExceeded

: StreamNameShardId


: Count

WriteProvisionedThroughputExceededPutRecord PutRecords 1 1 1,000 1 1 1 MB Average

Minimum

Maximum 0 ()

:WriteProvisionedThroughputExceeded

: StreamNameShardId


: Count



StreamName Kinesis

ShardId Kinesis ID

60




GetRecords.IteratorAgeMilliseconds ( 24 7 ) 50% CloudWatch (p. 159)

ReadProvisionedThroughputExceeded Average

WriteProvisionedThroughputExceededReadProvisionedThroughputExceeded (PUT) Average

PutRecord.SuccessPutRecords.Success CloudWatch PUT KinesisProducer Library (KPL) PutRecords.Success

GetRecords.Success CloudWatch

CloudWatch CloudWatch API Kinesis Data Streams

CloudWatch

1. https://console.aws.amazon.com/cloudwatch/ CloudWatch 2. 3. 4. [ CloudWatch ] [Kinesis ] 5. [MetricName] [StreamName]

: [] [] CloudWatch 5 [] IncomingBytes CloudWatch [] GetRecords.Bytes

6. () CloudWatch

61

https://console.aws.amazon.com/cloudwatch/


AWS CLI

list-metrics get-metric-statistics

CloudWatch CLI

mon-list-metrics mon-get-stats

CloudWatch API

ListMetrics GetMetricStatistics

Amazon CloudWatch Kinesis Data Streams AWSKinesisAgent CloudWatch Kinesis Data Streams CPU CPU cloudwatch.endpoint (p. 107)

CloudWatch Kinesis Data Streams CloudWatch

BytesSent Kinesis Data Streams

:

RecordSendAttempts PutRecords ()

: Count

RecordSendErrors PutRecords ()

: Count

ServiceErrors () PutRecords

: Count

62

cloudwatch/list-metrics.htmlcloudwatch/get-metric-statistics.htmlcli-mon-list-metrics.htmlcli-mon-get-stats.htmlAPI_ListMetrics.htmlAPI_GetMetricStatistics.html

Amazon Kinesis Data Streams AWS CloudTrail Amazon


AWS CloudTrail Amazon Kinesis DataStreams API Amazon Kinesis Data Streams AWS CloudTrail Kinesis DataStreams API KinesisData Streams Kinesis Data Streams API Amazon S3 CloudTrail [Event history ()] CloudTrail IP KinesisData Streams

CloudTrail AWS CloudTrail User Guide

CloudTrail AWS Kinesis Data Streams CloudTrail AWS [Event history ()] AWS CloudTrail

CloudTrail Amazon S3 AWS AWS Amazon S3 AWS CloudTrail

CloudTrail CloudTrail Amazon SNS CloudTrail CloudTrail

Kinesis Data Streams CloudTrail

AddTagsToStream CreateStream DecreaseStreamRetentionPeriod DeleteStream DeregisterStreamConsumer DescribeStream DescribeStreamConsumer DisableEnhancedMonitoring EnableEnhancedMonitoring IncreaseStreamRetentionPeriod ListStreamConsumers ListStreams ListTagsForStream

63

view-cloudtrail-events.htmlview-cloudtrail-events.htmlcloudtrail-create-and-update-a-trail.htmlcloudtrail-aws-service-specific-topics.html#cloudtrail-aws-service-specific-topics-integrationsgetting_notifications_top_level.htmlreceive-cloudtrail-log-files-from-multiple-regions.htmlcloudtrail-receive-logs-from-multiple-accounts.htmlcloudtrail-receive-logs-from-multiple-accounts.htmlAPI_AddTagsToStream.htmlAPI_CreateStream.htmlAPI_DecreaseStreamRetentionPeriod.htmlAPI_DeleteStream.htmlAPI_DeregisterStreamConsumer.htmlAPI_DescribeStream.htmlAPI_DescribeStreamConsumer.htmlAPI_DisableEnhancedMonitoring.htmlAPI_EnableEnhancedMonitoring.htmlAPI_IncreaseStreamRetentionPeriod.htmlAPI_ListStreamConsumers.htmlAPI_ListStreams.htmlAPI_ListTagsForStream.html



MergeShards RegisterStreamConsumer RemoveTagsFromStream SplitShard StartStreamEncryption StopStreamEncryption UpdateShardCount

ID

AWS Identity and Access Management (IAM)

.

AWS

CloudTrail userIdentity

: Kinesis Data Streams Amazon S3 CloudTrail1 CloudTrail API

CreateStreamDescribeStreamListStreamsDeleteStreamSplitShardMergeShards CloudTrail

{ "Records": [ { "eventVersion": "1.01", "userIdentity": { "type": "IAMUser", "principalId": "EX_PRINCIPAL_ID", "arn": "arn:aws:iam::012345678910:user/Alice", "accountId": "012345678910", "accessKeyId": "EXAMPLE_KEY_ID", "userName": "Alice" }, "eventTime": "2014-04-19T00:16:31Z", "eventSource": "kinesis.amazonaws.com", "eventName": "CreateStream", "awsRegion": "us-east-1", "sourceIPAddress": "127.0.0.1", "userAgent": "aws-sdk-java/unknown-version Linux/x.xx", "requestParameters": { "shardCount": 1, "streamName": "GoodStream" }, "responseElements": null, "requestID": "db6c59f8-c757-11e3-bc3b-57923b443c1c", "eventID": "b7acfcd0-6ca9-4ee1-a3d7-c4e8d420d99b" },

64

API_MergeShards.htmlAPI_RegisterStreamConsumer.htmlAPI_RemoveTagsFromStream.htmlAPI_SplitShard.htmlAPI_StartStreamEncryption.htmlAPI_StopStreamEncryption.htmlAPI_UpdateShardCount.htmlcloudtrail-event-reference-user-identity.html



{ "eventVersion": "1.01", "userIdentity": { "type": "IAMUser", "principalId": "EX_PRINCIPAL_ID", "arn": "arn:aws:iam::012345678910:user/Alice", "accountId": "012345678910", "accessKeyId": "EXAMPLE_KEY_ID", "userName": "Alice" }, "eventTime": "2014-04-19T00:17:06Z", "eventSource": "kinesis.amazonaws.com", "eventName": "DescribeStream", "awsRegion": "us-east-1", "sourceIPAddress": "127.0.0.1", "userAgent": "aws-sdk-java/unknown-version Linux/x.xx", "requestParameters": { "streamName": "GoodStream" }, "responseElements": null, "requestID": "f0944d86-c757-11e3-b4ae-25654b1d3136", "eventID": "0b2f1396-88af-4561-b16f-398f8eaea596" }, { "eventVersion": "1.01", "userIdentity": { "type": "IAMUser", "principalId": "EX_PRINCIPAL_ID", "arn": "arn:aws:iam::012345678910:user/Alice", "accountId": "012345678910", "accessKeyId": "EXAMPLE_KEY_ID", "userName": "Alice" }, "eventTime": "2014-04-19T00:15:02Z", "eventSource": "kinesis.amazonaws.com", "eventName": "ListStreams", "awsRegion": "us-east-1", "sourceIPAddress": "127.0.0.1", "userAgent": "aws-sdk-java/unknown-version Linux/x.xx", "requestParameters": { "limit": 10 }, "responseElements": null, "requestID": "a68541ca-c757-11e3-901b-cbcfe5b3677a", "eventID": "22a5fb8f-4e61-4bee-a8ad-3b72046b4c4d" }, { "eventVersion": "1.01", "userIdentity": { "type": "IAMUser", "principalId": "EX_PRINCIPAL_ID", "arn": "arn:aws:iam::012345678910:user/Alice", "accountId": "012345678910", "accessKeyId": "EXAMPLE_KEY_ID", "userName": "Alice" }, "eventTime": "2014-04-19T00:17:07Z", "eventSource": "kinesis.amazonaws.com", "eventName": "DeleteStream", "awsRegion": "us-east-1", "sourceIPAddress": "127.0.0.1", "userAgent": "aws-sdk-java/unknown-version Linux/x.xx", "requestParameters": { "streamName": "GoodStream" }, "responseElements": null,

65

Amazon Kinesis Data Streams CloudWatch KCL

"requestID": "f10cd97c-c757-11e3-901b-cbcfe5b3677a", "eventID": "607e7217-311a-4a08-a904-ec02944596dd" }, { "eventVersion": "1.01", "userIdentity": { "type": "IAMUser", "principalId": "EX_PRINCIPAL_ID", "arn": "arn:aws:iam::012345678910:user/Alice", "accountId": "012345678910", "accessKeyId": "EXAMPLE_KEY_ID", "userName": "Alice" }, "eventTime": "2014-04-19T00:15:03Z", "eventSource": "kinesis.amazonaws.com", "eventName": "SplitShard", "awsRegion": "us-east-1", "sourceIPAddress": "127.0.0.1", "userAgent": "aws-sdk-java/unknown-version Linux/x.xx", "requestParameters": { "shardToSplit": "shardId-000000000000", "streamName": "GoodStream", "newStartingHashKey": "11111111" }, "responseElements": null, "requestID": "a6e6e9cd-c757-11e3-901b-cbcfe5b3677a", "eventID": "dcd2126f-c8d2-4186-b32a-192dd48d7e33" }, { "eventVersion": "1.01", "userIdentity": { "type": "IAMUser", "principalId": "EX_PRINCIPAL_ID", "arn": "arn:aws:iam::012345678910:user/Alice", "accountId": "012345678910", "accessKeyId": "EXAMPLE_KEY_ID", "userName": "Alice" }, "eventTime": "2014-04-19T00:16:56Z", "eventSource": "kinesis.amazonaws.com", "eventName": "MergeShards", "awsRegion": "us-east-1", "sourceIPAddress": "127.0.0.1", "userAgent": "aws-sdk-java/unknown-version Linux/x.xx", "requestParameters": { "streamName": "GoodStream", "adjacentShardToMerge": "shardId-000000000002", "shardToMerge": "shardId-000000000001" }, "responseElements": null, "requestID": "e9f9c8eb-c757-11e3-bf1d-6948db3cd570", "eventID": "77cf0d06-ce90-42da-9576-71986fec411f" } ]}

Amazon CloudWatch Kinesis Kinesis Client Library (KCL) for Amazon Kinesis Data Streams KCL Amazon CloudWatch CloudWatch []

66

https://docs.aws.amazon.com/kinesis/latest/dev/developing-consumers-with-kcl.htmlhttps://console.aws.amazon.com/cloudwatch/


Amazon CloudWatch

KCL CloudWatch Amazon CloudWatch Amazon CloudWatch API Amazon CloudWatch

(p. 67) (p. 67) (p. 67) (p. 68)

KCL

CloudWatch 2

CloudWatch NONESUMMARYDETAILED DETAILED CloudWatch NONE (p. 68)

KCL CloudWatch OperationWorkerIdentifierOperation ShardId

CloudWatch Amazon CloudWatch Amazon CloudWatch

WorkerIdentifier KCL ID WorkerIdentifier CloudWatch KCL WorkerIdentifier ID KCL ID KCL

KinesisClientLibConfiguration KCL MultiLangDaemon metricsLevel metricsEnabledDimensions MultiLangDaemon KCL .properties

67

publishingMetrics.htmlpublishingMetrics.htmlhttps://aws.amazon.com/cloudwatch/pricing/cloudwatch_concepts.html#Dimension


NONESUMMARY DETAILED 3 1 CloudWatch KCL OperationShardId WorkerIdentifier

KCL

KCL (p. 68) (p. 71) (p. 73)

KCL Amazon CloudWatch KCL

InitializeTask (p. 68) ShutdownTask (p. 69) ShardSyncTask (p. 70) BlockOnParentTask (p. 70)

InitializeTask

InitializeTask KCL Kinesis Data Streams

KinesisDataFetcher.getIterator.SuccessKCL GetShardIterator

: Detailed

: Count

KinesisDataFetcher.getIterator.Time KCL GetShardIterator

: Detailed

:

RecordProcessor.initialize.Time

: Summary

:

Success

: Summary

68


: Count

KCL

: Summary

:

ShutdownTask

ShutdownTask shutdown() 1 2

CreateLease.Success KCL DynamoDB

: Detailed

: Count

CreateLease.Time KCL DynamoDB

: Detailed

:

UpdateLease.Success

: Detailed

: Count

UpdateLease.Time

: Detailed

:

RecordProcessor.shutdown.Time

: Summary

:

Success

: Summary

: Count

69


KCL

: Summary

:

ShardSyncTask

ShardSyncTask Kinesis data stream KCL

CreateLease.Success KCL DynamoDB

: Detailed

: Count

CreateLease.Time KCL DynamoDB

: Detailed

:

Success

: Summary

: Count

: Summary

:

BlockOnParentTask

BlockOnParentTask KCL

Success

: Summary

: Count

: Summary

70


:

Amazon EC2 Kinesis data stream

RenewAllLeases (p. 71) TakeLeases (p. 72)

RenewAllLeases

RenewAllLeases

RenewLease.Success

: Detailed

: Count

RenewLease.Time

: Detailed

:

CurrentLeases

: Summary

: Count

LostLeases

: Summary

: Count

Success

: Summary

: Count

: Summary

:

71


TakeLeases

TakeLeases KCL KCL

ListLeases.Success KCL DynamoDB

: Detailed

: Count

ListLeases.Time KCL DynamoDB

: Detailed

:

TakeLease.Success KCL

: Detailed

: Count

TakeLease.Time

: Detailed

:

NumWorkers

: Summary

: Count

NeededLeases

: Detailed

: Count

LeasesToTake

: Detailed

: Count

TakenLeases

: Summary

: Count

TotalLeases KCL

72


: Detailed

: Count

ExpiredLeases

: Summary

: Count

Success TakeLeases

: Summary

: Count

TakeLeases

: Summary

:

ProcessTask

ProcessTask GetRecords processRecords

KinesisDataFetcher.getRecords.SuccessKinesis data stream GetRecords

: Detailed

: Count

KinesisDataFetcher.getRecords.TimeKinesis data stream GetRecords

: Detailed

:

UpdateLease.Success

: Detailed

: Count

UpdateLease.Time

: Detailed

73

https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html

Amazon Kinesis Data Streams CloudWatch KPL

:

DataBytesProcessed ProcessTask

: Summary

:

RecordsProcessed ProcessTask

: Summary

: Count

ExpiredIterator GetRecords ExpiredIteratorException

: Summary

: Count

MillisBehindLatest ()

: Summary

:

RecordProcessor.processRecords.Time processRecords

: Summary

:

Success

: Summary

: Count

: Summary

:

Amazon CloudWatch Kinesis Kinesis Producer Library (KPL) for Amazon Kinesis Data Streams Amazon CloudWatch CloudWatch [

74

https://docs.aws.amazon.com/kinesis/latest/dev/developing-producers-with-kpl.htmlhttps://console.aws.amazon.com/cloudwatch/


] AmazonCloudWatch

KPL CloudWatch Amazon CloudWatch Amazon CloudWatch API Amazon CloudWatch CloudWatch

(p. 75) (p. 75) Amazon CloudWatch (p. 76) (p. 76)

KPL KPL

KPL CloudWatch KPL KPL

CloudWatch 2

CloudWatch NONESUMMARYDETAILED DETAILED NONE

GLOBALSTREAMSHARD SHARD

SHARD ID 2 2 7 CloudWatch 1 1 1

MetricName (GLOBAL): Metric X Metric Y | | ----------------- ------------ | | | |StreamName (STREAM): Stream A Stream B Stream A Stream B

75

publishingMetrics.htmlhttps://aws.amazon.com/cloudwatch/pricing/


| | -------- --------- | | | |ShardID (SHARD): Shard 0 Shard 1 Shard 0 Shard 1

( Metric Y)

tuple: GLOBAL STREAM SHARD ID

Amazon CloudWatch KPL KPL KPL CloudWatch

N (N 1 60 )

CloudWatch

KPL 1 1 1 CloudWatch

User RecordsReceived

KPL

: Detailed

:

User RecordsPending

KPL PUT

: Detailed

:

76


User Records Put

KPL

: Summary

:

User Records DataPut

: Detailed

:

Kinesis RecordsPut

Kinesis Data Streams ( Kinesis Data Streams)

KPL

: Summary

:

Kinesis RecordsData Put


: Detailed

:

Errors by Code ErrorCode StreamName ShardId

Kinesis Data Streams API Kinesis Data Streams 1 Kinesis Data Streams

: Summary

:

All Errors

: Summary

:

77


Retries per Record 1

()

: Detailed

:

Buffering Time KPL

: Summary

:

Request Time PutRecordsRequests

: Detailed

:

User Records perKinesis Record


: Detailed

:

Amazon KinesisRecords perPutRecordsRequest

PutRecordsRequest Kinesis Data Streams

: Detailed

:

User Records perPutRecordsRequest

PutRecordsRequest 2

: Detailed

:

Amazon Kinesis Data Streams IAM

AWS Identity and Access Management (IAM)

AWS AWS AWS

78


AWS AWS AWS AWS

Kinesis Data Streams IAM Kinesis Data StreamsAPI AWS

Kinesis Client Library (KCL) AmazonDynamoDB Amazon CloudWatch KCL DynamoDB CloudWatch KCL CloudWatch KCL Kinesis Client Library 1.x (p. 118)

IAM

AWS Identity and Access Management (IAM) IAM

IAM Amazon DynamoDB Amazon DynamoDB IAM Amazon DynamoDB

IAM Amazon CloudWatch Amazon CloudWatch AWS

(p. 79) Kinesis Data Streams (p. 80) Kinesis Data Streams Amazon (ARN) (p. 80) Kinesis Data Streams (p. 80)

IAM 1 JSON

{ "Statement":[{ "Effect":"effect", "Action":"action", "Resource":"arn", "Condition":{ "condition":{ "key":"value" } } } ]}

79

https://aws.amazon.com/iam/https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started.htmlhttps://docs.aws.amazon.com/IAM/latest/UserGuide/UsingIAMWithDDB.htmlUsingIAMWithDDB.htmlUsingIAM.htmlUsingIAM.html


[Effect]: effect Allow Deny IAM API

[Action]: action API [Resource]:

Amazon ARN [Condition]: condition

IAM IAM Policy Generator IAM Policy Simulator

Kinesis Data Streams IAM IAM API Kinesis Data Streams API kinesis: : kinesis:CreateStreamkinesis:ListStreamskinesis:DescribeStream

"Action": ["kinesis:action1", "kinesis:action2"]

Get

"Action": "kinesis:Get*"

Kinesis Data Streams *

"Action": "kinesis:*"

Kinesis Data Streams API Amazon Kinesis API Reference

Kin

amazon kinesis data streams - 開発者ガイド · amazon kinesis data streams 開発者ガイド...

Documents