aws ops automator - amazon s3 · amazon web services – aws ops automator april 2018 page 4 of 28...

Copyright (c) 2018 by Amazon.com, Inc. or its affiliates.

AWS Ops Automator is licensed under the terms of the Amazon Software License available at

https://aws.amazon.com/asl/

AWS Ops Automator AWS Implementation Guide

Arie Leeuwesteijn

Mahmoud ElZayet

Ruald Andreae

July 2017

Last updated: April 2018 (see revisions)

https://aws.amazon.com/asl/

Amazon Web Services – AWS Ops Automator April 2018

of 28

Contents

Overview ................................................................................................................................... 3

Cost ........................................................................................................................................ 4

Architecture Overview........................................................................................................... 5

Solution Components ............................................................................................................... 6

Included Actions ................................................................................................................... 6

Role and Task Templates ...................................................................................................... 7

Task Execution Across Accounts .......................................................................................... 7

Performance .......................................................................................................................... 7

Daily Backups ........................................................................................................................ 8

Logging and Notifications ..................................................................................................... 8

Considerations .......................................................................................................................... 8

Regional Deployments .......................................................................................................... 8

AWS CloudFormation Templates ............................................................................................ 8

Automated Deployment ........................................................................................................... 9

What We’ll Cover ................................................................................................................... 9

Step 1. Launch the AWS Ops Automator Stack in the Primary Account ............................ 10

Step 2. Launch a Task Template in the Primary Account ................................................... 11

Step 3. Launch the Role Template in the Secondary Account(s) ........................................ 12

Step 4. Tag Your Resources .................................................................................................. 13

Setting the Tag Value ........................................................................................................ 13

Security .................................................................................................................................... 13

IAM Roles .............................................................................................................................14

Solution-Generated Roles ................................................................................................14

Additional Resources ............................................................................................................... 15

Appendix A: Task-Configuration Templates...........................................................................16

Common Template Parameters ...........................................................................................16

Amazon EC2 Create Snapshot Template ............................................................................. 17


of 28

Amazon EC2 Delete Snapshot Template ............................................................................ 18

Amazon EC2 Copy Snapshot Template .............................................................................. 18

Amazon Redshift Copy Snapshot Template ........................................................................19

Amazon Redshift Delete Snapshot Template ......................................................................19

Amazon DynamoDB Set Capacity Template ...................................................................... 20

Appendix B: Cross-Account Role ARN File ........................................................................... 20

Appendix C: Configuring Keys for Encrypting Snapshots ...................................................... 21

Encrypting Snapshots in the Primary Account.................................................................... 21

Encrypting Snapshots in a Secondary Account(s) .............................................................. 22

Appendix D: Resource Identification ..................................................................................... 22

Appendix E: Log Files ............................................................................................................. 23

Messages.............................................................................................................................. 24

Appendix F: Sample Deployment Configuration ................................................................... 24

Appendix G: Troubleshooting Capacity Issues ...................................................................... 26

Appendix H: Collection of Anonymous Data ......................................................................... 27

Send Us Feedback................................................................................................................... 27

Document Revisions ............................................................................................................... 28

About This Guide This implementation guide discusses architectural considerations and configuration steps for

deploying the Amazon Web Services (AWS) Ops Automator on the AWS Cloud. It includes

links to AWS CloudFormation templates that launch, configure, and run the AWS services

required to deploy this solution on AWS, using AWS best practices for security and

availability.

The guide is intended for IT infrastructure architects, administrators, and DevOps

professionals who have practical experience architecting on the AWS Cloud.

Overview Amazon Web Services (AWS) offers its customers several methods to help automate manual

tasks or processes such as archiving, backup and data recovery, resource selection and

http://aws.amazon.com/cloudformation/


of 28

scheduling, and configuration management. Automating these tasks can help increase

reliability, reduce costs, and reduce operational complexity so you can focus on delivering

applications and services at a high velocity. However, it can be a challenge to automate

operational tasks on various AWS resources across multiple regions and accounts.

To help customers more easily manage cross-account and cross-region automation, AWS

offers the AWS Ops Automator solution. This solution provides a core framework for

automated tasks, allowing you to focus on the development of new functionality rather than

underlying infrastructure operations. This framework includes task audit trails, logging,

resource selection, scaling, AWS API retries, completion handing for long tasks, and

concurrency handling.

The AWS Ops Automator is easy to deploy and enables customers to use time-based or event-

based triggers to automatically manage AWS resources. The solution includes actions to

automatically create, copy, and delete Amazon Elastic Block Store (Amazon EBS) snapshots;

copy and delete Amazon Redshift snapshots; and set the throughput capacity for Amazon

DynamoDB. You can use these initial actions as a reference to quickly and consistently

develop custom actions that extend the solution’s functionality to other AWS services.

The AWS Ops Automator can use resource tags to identify which resources will receive

automated actions, allowing you to customize automation at the individual-resource level.

Cost You are responsible for the cost of the AWS services used while running this reference

deployment. The total cost for running this solution depends on the number of actions you

run and what the actions do. As of the date of publication, the cost for running this solution

with default settings in the US East (N. Virginia) Region is approximately $10 per month.1

By default, this solution enables Auto Scaling for its Amazon DynamoDB tables to provide

sufficient read and write capacity for resource tracking and configuration data. If you use this

solution to manage a large number of resources, Auto Scaling can increase Amazon

DynamoDB charges.

This pricing does not reflect variable charges incurred from the executed actions, data

transfer fees, or snapshot storage costs. Prices are subject to change. For full details, see the

pricing webpage for each AWS service you will be using in this solution.

1 The cost estimate assumes 1 GB of Amazon DynamoDB data storage at the default throughput capacity, 1 GB of Amazon

CloudWatch Logs ingested, 1 million AWS Lambda function executions, and 1 million CloudWatch custom events generated.


of 28

Architecture Overview Deploying this solution with the default parameters builds the following environment in

the AWS Cloud.

Figure 1: AWS Ops Automator architecture

This solution includes an AWS CloudFormation template that you deploy in the primary

account. For guidance on choosing a primary account, see the Security section. This template

launches all solution components, including a set of microservices (AWS Lambda functions)

that manage triggering events, resource selection, task execution, concurrency control, and

completion; Amazon DynamoDB tables that store task-related data; Amazon CloudWatch for

logging; and Amazon Simple Notification Service (Amazon SNS) for push notifications.

The primary template also automatically generates additional AWS CloudFormation

templates in an Amazon Simple Storage Service (Amazon S3) bucket. These templates allow

you to create cross-account AWS Identity and Access Management (IAM) roles to perform

actions in secondary accounts, and to configure tasks. You can modify and build upon these

templates to create custom actions that extend the solution’s functionality.

During initial configuration of the primary AWS CloudFormation template, you define a tag

key you will use to identify resources that will receive automated actions. When you deploy a

task configuration template, the stack name you specify is used as the tag value that

identifies the task you want the solution to perform on the tagged resource. For example, a

user might assign the custom tag name (tag key) OpsAutomatorTaskList and create a task


of 28

stack called Delete7 that deletes a resource after seven days. To identify the resource for

deletion, the user adds the OpsAutomatorTaskList tag key with a value of Delete7.

The automated process begins when either a time-based or event-based trigger invokes an

instance of the main AWS Ops Automator Lambda function. This event handler function

identifies applicable resources, retrieves the specific task configuration from Amazon

DynamoDB, and passes this data to task execution handler Lambda functions. The task

execution handlers run the specified action code on the appropriate resources, and also

manage concurrency control and completion for long tasks. Amazon CloudWatch logs

information on the microservices. This solution also tracks all steps of the task execution

process, the selected resources, and the results of the actions, including possible errors, in a

DynamoDB table.

Figure 2: AWS Ops Automator task execution process

Solution Components

Included Actions The AWS Ops Automator includes a set of actions that you can use to configure automated

tasks. An action consists of metadata that specifies which resources and properties to select

from AWS services, the aggregation level of the selected resources, the permissions required

to access and modify that resource, and the code that implements the action logic. An action

can be used in multiple tasks.

To execute an action, you must launch the specific task configuration template. This creates

a new task that defines the time interval or resource event that triggers the action, the action

parameters that define how the task should be performed, the tags that determine which


of 28

resources will receive the actions, and the accounts and regions where the resources are

located. For more information about the task-configuration templates included with this

solution, see Appendix A.

Role and Task Templates When you deploy the AWS Ops Automator, the solution automatically creates an Amazon

Simple Storage Service (Amazon S3) bucket in the primary account that contains two types

of AWS CloudFormation templates for each action.

One template (role) is located in the Roles folder and creates the AWS Identity and Access

Management (IAM) roles necessary to perform the action in secondary accounts. You can

review and modify permissions in the role template before you launch the stack. The solution

also includes a single template that configures all IAM roles necessary to perform all solution

actions. Use this template if you want to configure tasks for all actions.

The other template (task-configuration) is located in the Configuration folder and creates

a task based on parameters you define. You can use these templates as a framework for your

own custom actions. For more information about the task-configuration templates, see

Appendix A.

If you delete the AWS Ops Automator stack in the primary account, all task stacks and

configurations will be deleted.

Task Execution Across Accounts To perform tasks on resources in secondary accounts, you must launch the applicable role

template in each secondary account after you launch the solution template in the primary

account. When the role stack is launched, it creates a cross-account role Amazon Resource

Name (ARN). When you launch the task-configuration template in the primary account, you

enter the appropriate cross-account role ARN(s) in the applicable template parameter to

allow the AWS Ops Automator to perform the specified task in those accounts.

Customers who want to perform tasks on resources in a large number of secondary accounts,

can upload a list of cross-account role ARNs to the solution’s Amazon S3 bucket. For more

information, see Appendix B.

Performance The AWS Ops Automator is designed to accommodate task processing for a large number of

resources. By default, this solution enables Auto Scaling for its Amazon DynamoDB tables to

provide sufficient read and write capacity to store tracking and configuration data for each

resource it manages. The primary solution template includes a Lambda size (MB)

parameter, which allows you to increase the memory of the main AWS Ops Automator


of 28

Lambda function. If you plan to process tasks for a large number of resources, or if you

encounter issues completing tasks for a large number of resources, we recommend you

increase this value. See Appendix G for detailed information.

Daily Backups The AWS Ops Automator creates a daily backup of the Amazon DynamoDB configuration

table. Configuration backup files are stored in the Backups folder in the solution’s Amazon

S3 bucket. The solution’s primary template includes a parameter where you define the

retention period for these backup files.

Logging and Notifications The AWS Ops Automator leverages Amazon CloudWatch Logs for logging. The solution logs

which AWS Lambda function handled each event; when each task was executed; which tasks

were executed; the state of executed tasks; whether each task completed; which resources

were selected for each task; the execution time of the next task; and debugging, warning, and

error messages. For more information, see Appendix E.

Warning and error messages are also published to a solution-created Amazon Simple

Notification Service (Amazon SNS) topic which sends messages to a subscribed email address

(see Subscribe to a Topic in the Amazon SNS Developer Guide). You can find the name of the

Amazon SNS topic in the Outputs tab of the primary solution stack.

Considerations

Regional Deployments The AWS Ops Automator uses Amazon Resource Names (ARNs) to allow the solution to

perform tasks in secondary accounts. In the AWS GovCloud (US) Region, ARNs have an

identifier that is different from the one in other AWS Regions. Because of this difference, the

AWS Ops Automator is not available in the AWS GovGloud (US) Region at this time.

AWS CloudFormation Templates This solution uses AWS CloudFormation to automate the deployment of the AWS Ops

Automator on the AWS Cloud. It includes the following AWS CloudFormation template,

which you can download before deployment:

ops-automator.template: Use this template to launch the Ops

Automator and all associated components. The default configuration

deploys AWS Lambda functions, Amazon DynamoDB tables, Amazon CloudWatch events,

View template

https://s3.amazonaws.com/solutions-reference/ops-automator/latest/ops-automator.template

http://docs.aws.amazon.com/sns/latest/dg/SubscribeTopic.html




of 28

AWS CloudFormation templates, AWS Identity and Access Management roles, an Amazon

Simple Storage Service (Amazon S3) bucket, and an Amazon Simple Notification Service

(Amazon SNS) topic.

Automated Deployment Before you launch the automated deployment, please review the architecture, configuration,

security, and other considerations discussed in this guide. Follow the step-by-step

instructions in this section to configure and deploy an AWS Ops Automator into your

account.

Time to deploy: Approximately 30 minutes

What We’ll Cover The procedure for deploying this architecture on AWS consists of the following steps. For

detailed instructions, follow the links for each step.

Step 1. Launch the AWS Ops Automator Stack in the Primary Account

• Launch the AWS CloudFormation template into your primary AWS account.

• Enter values for required parameters: Stack Name.

• Review the other template parameters, and adjust if necessary.

Step 2. Launch the Task Template in the Primary Account

• Launch the applicable task-configuration AWS CloudFormation template into the

primary account.

• Review the template parameters, and adjust if necessary.

Step 3. Launch the Role Template in the Secondary Account

• Launch the applicable role AWS CloudFormation template into the secondary account

with applicable resources.

• Enter values for required parameters: Stack Name.

Step 4. Tag Your Resources

• Apply the custom tag to applicable resources.


of 28

Step 1. Launch the AWS Ops Automator Stack in the Primary Account This automated AWS CloudFormation template deploys the AWS Ops Automator in your

primary account. Launch this template using an AWS Identity and Access Management

(IAM) role specifically created for this purpose. For more information, see the Security

section.

Note: You are responsible for the cost of the AWS services used while running this solution. See the Cost section for more details. For full details, see the pricing webpage for each AWS service you will be using in this solution.

1. Sign in to the AWS Management Console and click the button to

the right to launch the ops-automator AWS CloudFormation

template.

You can also download the template as a starting point for your own implementation.

2. The template is launched in the US East (N. Virginia) Region by default. To launch this

solution in a different AWS Region, use the region selector in the console navigation bar.

Note: This solution uses AWS Lambda, Amazon DynamoDB, and Amazon CloudWatch, which are currently available in specific AWS Regions only. Therefore, you must launch this solution an AWS Region where these services are available. 2

3. On the Select Template page, keep the default setting for Choose a Template and

select Next.

4. Under Parameters, review the parameters for the template and modify them as

necessary. This solution uses the following default values.

Parameter Default Description

Ops Automator Tag

Name

OpsAutomatorTaskList The tag key (name) that identifies applicable resources. The

tag value will contain the list of tasks to be performed on

tagged resources. See Step 4 for detailed information.

Clean up task

tracking table?

Yes Choose whether to clean the task tracking table.

Keep failed tasks? Yes Choose whether to store failed tasks in the Amazon

DynamoDB table.

Schedule active? Yes Choose whether to activate the scheduling task feature.

2 For the most current AWS service availability by region, see https://aws.amazon.com/about-aws/global-infrastructure/regional-

product-services/

Launch Solution

https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?&templateURL=https://s3.amazonaws.com/solutions-reference/ops-automator/latest/ops-automator.template


https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/

https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/




of 28


How long to keep

tasks?

168 The number of hours to keep a task before it is

automatically deleted

Days to keep

configuration

backups

7 The number of days to keep a configuration backup file

before it is automatically deleted

Lambda size (MB) 128 The memory size of the solution’s main AWS Lambda

function. If you plan to apply actions to a large number of

resources, increase the default size. For more information,

see Appendix G.

Log Retention Days 30 The number of days to keep logs before they are

automatically deleted

Send Anonymous

Usage Data

Yes Send anonymous data to AWS to help us understand AWS

Ops Automator usage across our customer base as a whole.

To opt out of this feature, select No.

For more information, see Appendix H.

5. Choose Next.

6. On the Options page, choose next.

7. On the Review page, review and confirm the settings. Be sure to check the box

acknowledging that the template will create AWS Identity and Access Management

(IAM) resources.

8. Choose Create to deploy the stack.

You can view the status of the stack in the AWS CloudFormation Console in the Status

column. You should see a status of CREATE_COMPLETE in roughly 10 minutes.

Step 2. Launch a Task Template in the Primary Account Before you configure a task, review the information in Appendix A for the applicable action.

1. In the primary account’s Amazon S3 console, navigate to the bucket for the AWS Ops

Automator solution stack.

Note: You can find the name of the S3 bucket in the AWS CloudFormation stack outputs tab. The bucket name is value of the TemplateBucketName key.

1. In the Configuration folder, select the applicable template.

2. Copy the Link value.

3. In the AWS CloudFormation console, select Create Stack.

4. Select Specify an Amazon S3 template URL.


of 28

5. Paste the template link into the text box and select Next.

6. Enter a Stack name.

7. Under Parameters, review the parameters for the template and modify them as

necessary. For more information, see Appendix A.

8. Select Next.

9. Select Next. Then, on the Review page, review and confirm the settings. Be sure to

check the box acknowledging that the template will create AWS Identity and Access

Management (IAM) resources.


Note: If you delete the AWS Ops Automator stack, all task stacks and configurations will be deleted.

Step 3. Launch the Role Template in the Secondary Account(s) Use this procedure to create roles to perform tasks on resources in secondary accounts.

Customers who want to perform tasks on resources in a large number of secondary accounts

can save a text file that contains a list of the cross-account role Amazon Resource Names. For

more information, see Appendix B.

11. In the primary account’s Amazon S3 console, navigate to the bucket for the AWS Ops

Automator solution stack.

Note: You can find the name of the S3 bucket in the AWS CloudFormation stack outputs tab. The bucket name is value of the TemplateBucketName key.

12. In the Roles folder, select the applicable template.

13. Select Download and note the location of the downloaded template.

14. In the secondary account’s AWS CloudFormation console, select Create Stack.

15. Select Upload a template to Amazon S3.

16. Select Choose File.

17. Navigate to the downloaded template and select Choose. Then, select Next.

18. Enter a Stack name and select Next.


of 28

19. Select Next. Then, on the Review page, review and confirm the settings. Be sure to

check the box acknowledging that the template will create AWS Identity and Access

Management (IAM) resources.


21. After the stack deploys, navigate to the stack Outputs tab and copy the Value of the

CrossAccountRoleArn key.

Step 4. Tag Your Resources When you deployed the AWS CloudFormation template, you defined the tag key for the

solution’s custom tag. For the AWS Ops Automator to recognize a resource, the tag key on

that resource must match the custom tag name stored in the solution’s Amazon DynamoDB

table. Therefore, it is important that you apply tags consistently and correctly to all applicable

resources. You can continue to use existing tagging strategies for your resources while using

this solution. For more information on resource tagging, see Tagging Your Amazon EC2

Resources, Tagging Resources in Amazon Redshift, and Tagging for DynamoDB.

On the AWS Management Console, use the Tag Editor to apply or modify tags for multiple

resources at a time. You can also apply and modify tags manually in the console.

Setting the Tag Value As you apply a tag to a resource, use the tag key you defined during initial configuration and

set the tag value to the name of an AWS Ops Automator task stack to perform that task on

the resource. For example, a user might define OpsAutomatorTaskList as the tag key.

Then, the user creates a stack called CopyResource. To identify the resources to be copied,

the user assigns the OpsAutomatorTaskList tag key with a value of CopyResource to

each resource.

To perform multiple tasks on a single resource, use a comma-separated list of those tasks as

the tag value. Continuing from the previous example, a user can assign the tag

OpsAutomatorTaskList tag key with the value CopyResource,DeleteResource to

identify resources to be copied, then deleted.

Security When you build systems on AWS infrastructure, security responsibilities are shared between

you and AWS. This shared model can reduce your operational burden as AWS operates,

manages, and controls the components from the host operating system and virtualization

layer down to the physical security of the facilities in which the services operate. For more

information about security on AWS, visit the AWS Security Center.

https://aws.amazon.com/answers/account-management/aws-tagging-strategies/

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html

http://docs.aws.amazon.com/redshift/latest/mgmt/amazon-redshift-tagging.html

http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Tagging.html

https://resources.console.aws.amazon.com/r/tags

http://aws.amazon.com/security/


of 28

IAM Roles AWS Identity and Access Management (IAM) roles enable customers to assign granular

access policies and permissions to services and users on the AWS Cloud. The AWS Ops

Automator creates an IAM role in the primary account that includes all the required

permissions to perform actions on specific resources.

When you launch a role AWS CloudFormation template in a secondary account, the solution

creates the applicable IAM role with least-privilege access in the secondary account. This IAM

role in the secondary account has a trust policy with the IAM role in the primary account,

allowing the primary account to access resources in the secondary account.

Solution-Generated Roles The AWS Ops Automator automatically creates stacks and IAM roles that can grant

additional permissions to the solution’s main AWS Lambda function. To mitigate the risk of

unauthorized access to the solution’s main Lambda function and roles, AWS recommends

that you deploy the solution in an isolated and tightly controlled management account, and

limit access to that account.

To further isolate the solution, AWS recommends that you create a separate IAM role with

the following permissions in the primary account:

{

"Version": "2012-10-17",

"Statement": [

{

"Sid": "Stmt1499433973000",

"Effect": "Allow",

"Action": [

"sns:CreateTopic",

"sns:DeleteTopic",

"sns:ListTopics",

"sns:GetTopicAttributes",

"sns:SetTopicAttributes",

"iam:CreateRole",

"iam:UpdateRole",

"iam:DeleteRole",

"iam:AttachRolePolicy",

"iam:DeleteRolePolicy",

"iam:GetRole",

"iam:PassRole",

"iam:ListRoles",

"iam:DetachRolePolicy",

"iam:PutRolePolicy",

"dynamodb:CreateTable",

"dynamodb:UpdateTable",

"dynamodb:DeleteTable",


of 28

"dynamodb:DescribeTable",

"events:*",

"logs:*",

"lambda:*",

"s3:CreateBucket",

"s3:PutObject",

"s3:DeleteBucket",

"s3:PutObjectAcl",

"s3:ListBucket",

"s3:GetObject",

"s3:GetLifecycleConfiguration",

"s3:PutLifecycleConfiguration",

"cloudformation:*",

"application-autoscaling:*"

],

"Resource": [

"*"

]

}

]

}

Use this role to launch the ops-automator AWS CloudFormation template. This restricts

unauthorized access to the solution’s AWS Lambda functions and roles.

For secondary accounts, use IAM roles to prevent non-administrators from creating,

deleting, or updating tags. AWS recommends you check the access levels created by the

solution templates and modify them, if necessary.

Additional Resources

AWS services

• AWS CloudFormation

• AWS Lambda

• Amazon DynamoDB

• Amazon EC2

• Amazon CloudWatch

http://aws.amazon.com/documentation/cloudformation/

https://aws.amazon.com/documentation/lambda/

https://aws.amazon.com/documentation/dynamodb/

https://aws.amazon.com/documentation/ec2/

https://aws.amazon.com/documentation/cloudwatch


of 28

Appendix A: Task-Configuration Templates The AWS Ops Automator automatically generates a separate AWS CloudFormation

template for each action. Review the template parameters and modify them as necessary.

Common Template Parameters Each action-specific template has a set of common parameters, as well as parameters that

are specific to the applicable action. Review the common parameters and modify them as

necessary.


Task description <Optional input> Description of the task. For example, Create a snapshot

every 30 minutes.

Task interval <Requires input> Enter the scheduled expression (Cron syntax) that specifies when

to run the task. For example, enter 0/30**** to run the task

every 30 minutes.

Tag filter <Optional input> Filter used to select resources. You can use this instead of adding

the task name to the list of values in the

OpsAutomatorTaskList tag. For example, Owner=DBAdmin,

Stack=Test. For more information on tag filters, see Appendix

D.

Note: For actions that can delete or terminate resources, you cannot use “*” as the name of the tag in the filter.

Timeout 60 The number of minutes the solution waits for a task to complete

before reporting timing out.

Note: This parameter will only show for actions that the solution checks for completion.

Regions <Default region> List of regions where the task will run. For example, us-east-

1, eu-west-1.

Note: Use this parameter for actions that use regional resources. This parameter will not show for actions that use global resources. For example, IAM and Amazon S3.

This account Yes Select Yes to allow the task to select resources in this account.

Note: If you set this parameter to No, you must

configure cross-account roles.

https://en.wikipedia.org/wiki/Cron


of 28


Cross account roles <Optional input> Comma-delimited list of cross-account roles used by the task. For

example,

arn:aws:iam::111122223333:role/CreateSnaphotRole.

Note: Enter the secondary account CrossAccountRoleArn value(s) in this parameter. For customers who use a cross-account role ARN file, leave this parameter blank. For more information on the cross-account ARN file, see Appendix B.

Timezone UTC The time zone used for scheduling the task

Task enabled Yes Select No to temporarily disable execution of the task

Enable debugging No Choose whether to log detailed debugging information

Amazon EC2 Create Snapshot Template The create snapshot template enables the solution to automatically create snapshots of

Amazon Elastic Block Store (Amazon EBS) volumes attached to Amazon Elastic Compute

Cloud (Amazon EC2) instances.

Review the action-specific parameters for the template and modify them as necessary. This

action template uses the following default values.


Copy root volume Yes Create a snapshot of a root Amazon EC2 instance volume

Copy data volumes Yes Create snapshots of Amazon EC2 instance data volumes

Copied instance tags <Optional input> Enter a tag filter to copy tags from the instance to the snapshot. For

example, enter * to copy all tags from the instance to the snapshot.

For more information on tag filters, see Appendix D.

Copied volume tags <Optional input> Enter a tag filter to copy tags from the volume to the snapshot. For

example, enter * to copy all tags from the volume to the snapshot.

For more information on tag filters, see Appendix D.

Snapshot tags <Optional input> Tags that will be added to snapshots. Use a list of

tagname=tagvalue pairs. For example, if you create a task called

DeleteEC2Snapshots, you can enter a value of

OpsAutomatorTaskList=DeleteEC2Snapshots in this

parameter to allow the AWS Ops Automator to delete the snapshot

based on the parameters specified in the DeleteEC2Snapshots

task.

Set snapshot name Yes Set the name of the snapshot

Snapshot name prefix <Optional input> Prefix of the snapshot name


of 28

Amazon EC2 Delete Snapshot Template The delete snapshot template enables the solution to automatically delete snapshots of

Amazon Elastic Block Store (Amazon EBS) volumes attached to Amazon Elastic Compute

Cloud (Amazon EC2) instances older than a customer-defined number of days. Or, customers

can configure this action to keep only the latest snapshots.




Retention days <Requires

input>

The retention period in days. Set this parameter to 0 to use

Retention count.

Retention count <Requires

input>

The number of snapshots to retain for a volume. The maximum

allowed value is 1000. Set this parameter to 0 to use Retention

days.

Note: If both Retention days and Retention count are set to 0, the solution will return an error. You must only specify one.

Amazon EC2 Copy Snapshot Template The copy snapshot template enables the solution to automatically copy snapshots of Amazon

Elastic Block Store (Amazon EBS) volumes attached to Amazon Elastic Compute Cloud

(Amazon EC2) instances from one AWS Region to another.




Copied tags <Optional input> Enter a tag filter to copy tags from the source

snapshot to the copied snapshot. For example,

enter * to copy all tags from the source snapshot

to the copied snapshot. For more information on

tag filters, see Appendix D.

Snapshot tags <Optional input> Tags to add to the copied snapshot. Use a list of

tagname=tagvalue pairs.

Destination

region

<Optional input> AWS Region to copy the snapshot to

Tag name for

copied snapshots

Ec2CopySnapshot:SnapshotCopied Tag to add to the copied snapshot to show that it

was copied. We recommend that you use the

default tag name unless you already use the same

name for other purposes.


of 28


Encrypted No To enable encryption of the copied snapshot, select

Yes.

KMS Key ID <Optional input> The full ARN of an AWS Key Management Service

(AWS KMS) customer master key (CMK) to use

when creating the snapshot copy. If you do not

specify a CMK, the solution will use the default

CMK for Amazon EBS.

For instructions on how to find a key ARN, see the

AWS KMS Developer Guide.

Note: If you specify an alternative CMK, it must exist in the same AWS Region that the snapshots are copied to. Also, the account or role that the Ops Automator uses, or an applicable cross-account role, must have permission to use the key. For more information, see Appendix C.

Amazon Redshift Copy Snapshot Template The Amazon Redshift copy snapshot template enables the solution to automatically copy

snapshots of Amazon Redshift clusters from one AWS Region to another.




Copied cluster tags <Optional input> Enter a tag filter to copy tags from the instance to the snapshot.

For example, enter * to copy all tags from the instance to the

snapshot. For more information on tag filters, see Appendix D.

Snapshot tags <Optional input> Tags to add to snapshot. Use a list of tagname=tagvalue

pairs. For example, if you create a task called DeleteRedShiftSnapshots, you can enter a value of

OpsAutomatorTaskList=DeleteRedShiftSnapshots in

this parameter to allow the AWS Ops Automator to delete the copied snapshot based on the parameters specified in the DeleteRedShiftSnapshots task.

Grant restore access

to accounts

<Optional input> Comma-delimited list of accounts that can restore the snapshot.

For example, 777788889999, 000000000000.

Amazon Redshift Delete Snapshot Template The Amazon Redshift delete snapshot template enables the solution to automatically delete

snapshots of Amazon Redshift clusters older than a customer-defined number of days. Or,

customers can configure this action to keep only the latest snapshots.



http://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html


of 28


Retention days <Requires input> The retention period in days. Set this parameter to 0 to use

Retention count.

Retention count <Requires input> The number of snapshots to retain for a volume. For example,

set this parameter to 2 to save the last two snapshots. Set this

parameter to 0 to use Retention days.

Note: If both Retention days and Retention count are set to 0, the solution will return an error. You must

only specify one.

Amazon DynamoDB Set Capacity Template The Amazon DynamoDB set capacity template enables the solution to automatically

provision throughput capacity for reads and writes to DynamoDB.




Table name <Requires input> The name of the Amazon DynamoDB table

Table read units <Requires input> The number of read units for the table

Table write units <Requires input> The number of write units for the table

To set the capacity for global secondary indexes on the DynamoDB table, modify the

applicable template parameters.

For more information on throughput capacity, see Provisioned Throughput in the Amazon

DynamoDB developer guide.

Appendix B: Cross-Account Role ARN File For customers who want to perform tasks on resources in a large number of secondary

accounts, the AWS Ops Automator enables customers to add cross-account roles in bulk

using a text file that contains a list of the Amazon Resource Names (ARNs) of the cross-

account roles.

To use this feature, you must create a text file for each specific task in the TaskRoles folder

in this solution’s Amazon Simple Storage Solution (Amazon S3) bucket. Each file must

contain a list of the cross-account ARNs for the specific task. Each cross-account role ARN

must be on its own line. Empty lines and lines starting with # (comments) will be ignored.

You must also save the text file as the exact name of the task.

http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ProvisionedThroughput.html


of 28

# CreateSnapshot cross account roles

# test account

arn:aws:iam::111122223333:role/CreateSnaphotRole

# acceptance accounts



# production accounts




Figure 3: Sample cross-account role ARN list file

Appendix C: Configuring Keys for Encrypting

Snapshots The AWS Ops Automator allows you to use AWS Key Management Service (AWS KMS) to

encrypt the snapshot copies this solution creates. The solution is designed to use the default

customer master key (CMK) for Amazon Elastic Block Store for copying snapshots, but you

can specify an alternate CMK. If you specify an alternate CMK, it must exist in the same AWS

Region that the snapshot is copied to. Also, the account or role that the AWS Ops Automator

uses, or applicable cross-account role, must have permission to use the role to encrypt copied

snapshots.

Encrypting Snapshots in the Primary Account Use this procedure to configure your CMK to encrypt snapshots in the primary account

where the AWS Ops Automator stack is deployed.

1. Sign into the AWS Management Console and open the AWS Identity and Access

Management (IAM) console.

2. In the left navigation pane, select Encryption keys.

3. Choose the destination AWS Region for your copied snapshots.

4. Verify that the desired key exists in the region. If the key does not exist in that region,

create a key.

5. Add a key policy to the key to allow the AWS Ops Automator’s IAM role to use the key to

encrypt snapshots. The name of the role is <stackname>-SchedulerRole-<id>.


http://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html


of 28

Encrypting Snapshots in a Secondary Account(s) Use this procedure to configure your CMK to encrypt snapshots in secondary account(s).

1. Sign into the AWS Management Console and open the AWS Identity and Access

Management (IAM) console.

2. In the left navigation pane, select Encryption keys.

3. Choose the destination AWS Region for your copied snapshots.

4. Verify that the desired key exists in the region. If the key does not exist in that region,

create a key.

5. Add a key policy to the key to allow the copy snapshot cross-account role in the

secondary account to use the key to encrypt snapshots.

6. In each secondary account, add the permission to use the key to the cross-account role

that allows the primary account to execute the copy snapshot task in the secondary

account.

a. Navigate to the IAM console.

b. Choose the cross-account role that allows the primary account to execute the copy

snapshot task in the secondary account. For more information, see Step 3.

c. Edit the cross-account role’s policy. Specify the key’s Amazon Resource Name and

the permitted actions. For more information, see Sharing Custom Encryption Keys

More Securely Between Accounts by Using AWS Key Management Service.

Appendix D: Resource Identification The AWS Ops Automator uses a custom tag key (name) to identify resources that will receive

automated actions. The default tag key is OpsAutomatorTaskList but you can modify it

during initial configuration of the solution’s AWS CloudFormation template. The value of the

OpsAutomatorTaskList tag contains a comma-separated list of tasks you want the

solution to perform on that resource. For more information, see Step 4.

The solution also enables you to identify resources based on a tag name, a tag value, or both,

using the Tag Filter parameter in the task-configuration templates. The parameter supports

exact string matching and substring matching (with wildcards), and is case-sensitive. The

following table gives acceptable input formats for the tag filter.

Tag filter format Description

* Any tagged resource


http://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html

https://aws.amazon.com/blogs/security/share-custom-encryption-keys-more-securely-between-accounts-by-using-aws-key-management-service/

https://aws.amazon.com/blogs/security/share-custom-encryption-keys-more-securely-between-accounts-by-using-aws-key-management-service/


of 28

Tag filter format Description

Note: For actions that can delete or terminate resources, you cannot use “*” as the name of the tag in the filter.

A* Tag keys that start with “A”

*A* Tag keys that contain “A”

*A Tag keys that end with “A”

\.*\d$ Tag keys that end with a digit

A=B Tag keys “A” with value “B”

A=B* Tag keys “A” with a value that starts with “B”

*=B Any tag with a value “B”

*=B* Any tag with a value that starts with “B”

*=\.*\d$ Any tag with a value that ends with a digit

A=B,C=D Tag keys “A” with value “B” or tag keys “C” with a value “D”

The following table gives examples of different tag filters and the resulting AWS Ops

Automator action.

Tag filter AWS Ops Automator Action

Owner=DBAdmin Perform the task on resources with the Owner tag key with a value of

DBAdmin.

Owner Perform the task on resources with the Owner tag key with any value.

*=DBAdmin Perform the task on resources with any tag key with a value of DBAdmin.

*test Perform the task on resources with a tag key that ends with test.

test* Perform the task on resources with a tag key that starts with test.

*=*test Perform the task on resources with any tag key with a value that ends

with test.

*=test* Perform the task on resources with any tag key with a value that starts

with test.

Owner=DBAdmin,Stack=Test Perform the task on resources with the Owner tag key with a value of

DBAdmin or resources with the Stack tag key with a value of Test.

Appendix E: Log Files The AWS Ops Automator creates a log group that contains the default AWS Lambda log

files and a log group that contains the following log files:


of 28

• AutomatorMain-yyyymmdd - Logs the output from the Lambda function that handled

the event.

• ScheduleHandler-yyyymmdd - Logs the output from Lambda functions that handle

time-based tasks.

• TaskTrackingHandler-yyyymmdd - Logs the output from Lambda functions that

handle event-based tasks.

• SelectResourcesHandler-<taskname>-yyyymmdd – Logs the output from the

Lambda function that selects resources for task execution.

• <Taskname>-yymmddhhmm-<unique task id> - Logs the output of the Lambda

function that executes the action. The unique task ID correspondents to the unique id for each executed task in the task tracking Amazon DynamoDB table.

• CompletionHandler-yyyymmdd – Logs the output from the Lambda function that

checks whether the action completed.

Messages This solution also logs error, warning, and debugging messages. Each message has the format: yyyy-mm-dd – hh:mm:ss.mmm - <type> - <text>. The type can have the

value INFO for informational messages, DEBUG for detailed debugging messages, WARNING for warning messages, or ERROR for error messages.

Warning and error messages are sent to an email address using Amazon Simple Notification

Service (Amazon SNS). The Amazon SNS topic is located in the AWS Ops Automator stack

output named IssueSNSTopic.

Appendix F: Sample Deployment Configuration The AWS Ops Automator enables customers to perform a sequence of tasks on resources in

their accounts. The following section shows how to configure a sequence of tasks that will

take snapshots of all Amazon Elastic Block Store (Amazon EBS) volumes at 1:00 AM daily

and retain the snapshots for seven days.

First, deploy the ops-automator template in your primary account. For more information,

see Step 1. Once the template is deployed, launch the Ec2DeleteSnapshot and

Ec2CreateSnapshot role templates in any secondary account(s) with applicable EBS

volumes. For more information, see Step 3. Copy the cross-account role Amazon Resource

Name(s). To perform tasks on resources in a large number of secondary accounts, save the

ARNs in a text file. For more information, see Appendix B.


of 28

Next, deploy the Ec2DeleteSnapshot configuration template in the primary account

using the following values:

Parameter Value

Stack name Delete7

Task description Delete a snapshot after 7 days

Task interval 0 2 * * ?

Tag filter (Leave blank)

Regions (Enter the applicable AWS Region(s). For example, us-

east-1, eu-west-1)

This account Yes

Cross-account roles (Enter the cross-account ARNs or leave blank to use a text

file.)

Timezone UTC

Task enabled Yes

Enable debugging No

Retention days 7

Retention count 0

Note: Set this parameter to 0 to retain snapshots

using Retention days.

Then, deploy the Ec2CreateSnapshot configuration template in the primary account

using the following values:

Parameter Value

Stack name BackupDaily

Task description Create a snapshot at 1 am daily

Task interval 0 1 * * ?

Tag filter (Leave blank)

Regions (Enter the applicable AWS Region(s). For example, us-

east-1, us-west-2)

This account Yes

Cross-account roles (Enter the cross-account ARNs or leave blank to use a text

file.)

Timezone UTC

Task enabled Yes


of 28

Parameter Value

Enable debugging No

Copy root volume Yes

Copy data volumes Yes

Copied instance tags *

Copied volume tags *

Snapshot tags OpsAutomatorTaskList=Delete7

Set snapshot name Yes

Snapshot name prefix ops-auto

When completely deployed using the configuration above, the AWS Ops Automator will do

the following:

1. Create a snapshot of any EBS volumes attached to Amazon Elastic Cloud Compute

(Amazon EC2) instances with the BackupDaily tag at 1 am. If a snapshot already exists

for the volume, the solution will take an incremental snapshot.

2. Copy the Amazon EC2 instance and volume tags to the snapshot.

3. Attach a new tag (OpsAutomatorTaskList=Delete7) to the snapshot. This tag is used

to identify applicable snapshots for deletion after the retention period (seven days).

4. After seven days, the solution will delete the snapshot.

Appendix G: Troubleshooting Capacity Issues When the automated process is triggered, an instance of the solution’s main AWS Lambda

function must select all applicable resources within a five-minute period. If the solution fails

to process a task for a large number of resources, such as copying or deleting a large amount

of Amazon Elastic Block Store (Amazon EBS) snapshots, it is often because the Lambda

function does not have enough memory to select those resources within the timeout period.

The primary solution template includes a Lambda size (MB) parameter, which allows you

to increase the memory of the main AWS Ops Automator Lambda function.

You can review the logs for your Lambda function in Amazon CloudWatch to see if

insufficient memory is causing execution issues. In the AWS Lambda console, choose the

function named <stackname>-SchedulerDefault. In the Monitoring tab under

Invocation duration, choose Jump to Logs to view the function’s log files directly in the

Amazon CloudWatch console. If there are entries that show execution times close to five

minutes, or executions that were timed out by the five-minute limit, then you need to increase


of 28

the memory size of the Lambda function. Use the Lambda size (MB) parameter in the

primary solution template to adjust this value.

Appendix H: Collection of Anonymous Data This solution includes an option to send anonymous usage data to AWS. We use this data to

better understand how customers use this solution and related services and products. When

enabled, the following information is collected and sent to AWS:

• Solution ID: The AWS solution identifier

• Unique ID (UUID): Randomly generated, unique identifier

• Timestamp: Data-collection timestamp

• Copy Snapshot Task Data: The number of snapshots copied

• Create Snapshot Task Data: The number of snapshots taken and the total size

• Delete Snapshot Task Data: The number of snapshots deleted

• Amazon DynamoDB Set Capacity Task Data: The number of old and new read

and write capacity units and global secondary indexes for DynamoDB tables

Note that AWS will own the data gathered via this survey. Data collection will be subject to

the AWS Privacy Policy. To opt out of this feature, set the Send Anonymous Usage Data

parameter to No.

Send Us Feedback We welcome your questions and comments. Please post your feedback on the AWS

Solutions Discussion Forum.

You can visit our GitHub repository to download the templates and scripts for this solution,

and to share your customizations with others.

http://aws.amazon.com/privacy/

https://forums.aws.amazon.com/forum.jspa?forumID=226

https://forums.aws.amazon.com/forum.jspa?forumID=226

https://github.com/awslabs/aws-ops-automator


of 28

Document Revisions

Date Change In sections

July 2017 Initial release --

December 2017 Added considerations for large numbers of resources; new

options for encryption and an AWS KMS customer master

key for EC2 snapshot copies

Performance; Amazon EC2 Copy

Snapshot Template; Appendix G

December 2017 Added instructions for configuring an alternative customer

master key for snapshot encryption

Appendix C

April 2018 Added clarification on AWS GovCloud (US) region

availability

Considerations

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Notices

This document is provided for informational purposes only. It represents AWS’s current product offerings

and practices as of the date of issue of this document, which are subject to change without notice. Customers

are responsible for making their own independent assessment of the information in this document and any

use of AWS’s products or services, each of which is provided “as is” without warranty of any kind, whether

express or implied. This document does not create any warranties, representations, contractual

commitments, conditions or assurances from AWS, its affiliates, suppliers or licensors. The responsibilities

and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of,

nor does it modify, any agreement between AWS and its customers.

The AWS Ops Automator is licensed under the terms of the Amazon Software License available at

https://aws.amazon.com/asl/.

aws ops automator - amazon s3 · amazon web services – aws ops automator april 2018 page 4 of 28...

Documents