yolo DocumentationRelease 0.3.0

Rackspace US, Inc.

Aug 17, 2018

Contents:

1 Getting Started 31.1 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 yolo.yaml configuration file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 Project structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81.4 Commands and workflows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2 Feature Roadmap 15

3 Indices and tables 17

i

ii

yolo Documentation, Release 0.3.0

yolo is a command line build/deployment tool for managing complete application stacks on AWS infrastructure.yolo can deploy entire services (infrastructure and code) with just a few commands. The following are supported:

• infrastructure deployments based on CloudFormation templates

• code deployments based on AWS Lambda Functions (plus API Gateway integration)

• code deployments based on static S3 website hosting (UI code, documentation, etc.)

yolo also takes a modern approach to credentials and secrets management to keep your deployment pipelines andapplication configurations secure.

yolo is written in Python and can be installed from the Python Package Index.

Contents

• Getting Started

– Installation

– yolo.yaml configuration file

* name section

* accounts section

* stages section

* templates section

* services section

* Context Variables

– Project structure

* Example: Python service

· lambda_main.py

· my-swagger-api.yaml

· requirements.txt

· setup.py

· yolo.yaml

* Example: S3 service

– Commands and workflows

* Deploy account-level infrastructure

* Deploy stage-level infrastructure

* Show infrastructure stack status

* Store parameters in SSM

* Build a Lambda application

* Create an application release

* List released application builds

* Deploy a Lambda service

Contents: 1

yolo Documentation, Release 0.3.0

* Deploy an S3 service

* Overview of commands

2 Contents:

CHAPTER 1

Getting Started

Start here to get up and running with yolo. The page contains information on how to install yolo, how to configureyolo for your projects, and how to run yolo commands.

1.1 Installation

The easiest way to get yolo is to install it from PyPI:

$ pip install yolo

1.2 yolo.yaml configuration file

yolo commands are driven primarily by a YAML configuration file. The following sections are required:

• name

• accounts

• stages

• templates

• services

yolo supports a limited set of context variables that can be used in configuration values in the stages, templates,and services sections.

1.2.1 name section

This section simply contains the name of your application. The name will be used to label various pieces of infras-tructure that yolo will create.

3

yolo Documentation, Release 0.3.0

1 name: 'MyApplication'

1.2.2 accounts section

This section contains details about the AWS accounts where your infrastructure is deployed. Define all of the relevantAWS account numbers and associated aliases for all of your infrastructure accounts here.

You must also define a default_region for each account. The default region is used when deploying accountlevel infrastructure templates.

Depending on how you choose to separate the various stages/environments of your application, it may be a good ideato make some deployments in separate accounts.

2 accounts:3 - name: 'DevAccount'4 account_number: '111222333444'5 default_region: 'us-east-2'6 - name: 'ProdAccount'7 account_number: '444555666777'8 default_region: 'us-east-1'

In the example above, we have separate accounts for development and production.

You must define at least one account.

1.2.3 stages section

This section contains details about the various environments (or “stages”) which yolo can deploy to. Define all ofthe relevant stages here, such as “dev”, “staging”, “production”, “QA”, etc.

You may also define a special stage called default. This stage definition can be used to create custom stages on thefly without having to explicitly define them in the configuration file. This is useful for creating stages for individualdevelopers.

9 stages:10 default:11 account: 'DevAccount'12 region: 'us-east-2'13 dev:14 account: 'DevAccount'15 region: 'us-east-2'16 prod:17 account: 'ProdAccount'18 region: 'us-east-1'19 protected: true

The name of a stage is used for many commands as the --stage parameter.

1.2.4 templates section

The templates section has two sub-sections: account and stage:

4 Chapter 1. Getting Started

yolo Documentation, Release 0.3.0

20 templates:21 account:22 path: 'cloudformation/account'23 params:24 AccountParam1: 'foo'25 AccountParam2: 'bar'26 stage:27 path: 'cloudformation/stage'28 params:29 StageParam1: 'baz'30 StageParam2: 'qux'

• path indicates a directory path relative to the yolo.yaml file where CloudFormation templates are to befound. This directory location should contain a master.json or master.yaml template file.

• params are input parameters to the respective template. If your templates do not require any parameters, enterparams: {}.

If you don’t use any CloudFormation templates to stand up your infrastucture, you still have to define the templatessection, but you can leave it empty: templates: {}.

1.2.5 services section

There is where the bulk is yolo configuration is located. This section contains details about how an application shouldbe built, configured, and deployed.

31 services:32 MyLambdaService:33 type: 'lambda-apigateway'34 build:35 working_dir: '.'36 dist_dir: 'dist'37 dependencies: 'requirements.txt'38 include:39 - 'myapplication/'40 - 'my-swagger-api.yaml'41 - 'lambda_main.py'42 deploy:43 apigateway:44 rest_api_name: 'myapplication-api'45 swagger_template: 'my-swagger-api.yaml'46 domains:47 - domain_name: '{{ stage.outputs.CustomDomain }}'48 base_path: '/'49 authorizers:50 - name: 'auth_token'51 type: 'TOKEN'52 providerARNs: []53 authType: 'token'54 authorizerUri: ''55 authorizerCredentials: '<IAM role ARN>'56 identitySource: 'method.request.header.Authorization'57 identityValidationExpression: '.*'58 authorizerResultTtlInSeconds: 059 integration:60 type: 'AWS'

(continues on next page)

1.2. yolo.yaml configuration file 5

yolo Documentation, Release 0.3.0

(continued from previous page)

61 uri: '<integration URI>'62 passthroughBehavior: 'WHEN_NO_MATCH | NEVER | WHEN_NO_TEMPLATES'63 credentials: '<IAM role ARN>'64

65 lambda_function_configuration:66 FunctionName: 'myapplication'67 Role: '{{ stage.outputs.LambdaExecutionRoleArn }}'68 Handler: 'lambda_main.lambda_handler'69 Description: 'My Application'70 Timeout: 3071 Memory: 51272 VpcConfig: {}73 Runtime: 'python3.6'74 TracingConfig:75 Mode: 'Active'76

77 parameters:78 stages:79 default:80 - name: 'log_level'81 value: 'debug'82 - name: 'db_username'83 value: 'admin'84 - name: 'db_password'85 value: 'password'86 - name: 'ssl_private_key'87 multiline: true88 dev:89 - name: 'log_level'90 value: 'info'91 prod:92 - name: 'db_username'93 value: 'prod-admin'94 - name: 'db_password'95 - name: 'log_level'96 value: 'warn'97 MyS3Service:98 type: 's3'99 bucket_name: 'my-hosting-s3-bucket'

Under services, we can have any number of services defined. Each service must be defined with a unique name(e.g., MyLambdaService) which will be used as the --service parameter in many yolo commands.

The details of service configuration can be quite complex. Let’s break it down:

• type: The type of service. Valid values are s3, lambda, and lambda-apigateway.

• bucket_name: The target S3 bucket where this application is hosted. Only relevant for s3-type services.

• build: Configuration used for build commands.

• build.working_dir: Directory to be used as the root directory for builds. All other paths in build areassumed to be relative to build.working_dir.

• build.dist_dir: Location for yolo to place built artifacts after a successful yolo build-lambdacommand. yolo push will collect build artifacts from this location.

• build.dependencies: File containing a list of build dependencies. For Python projects, this is the relativepath to a requirements.txt file.

6 Chapter 1. Getting Started

yolo Documentation, Release 0.3.0

• build.include: A list of relative files and directories to include in build artifacts.

• deploy: Configuration used for service deployment commands.

• deploy.apigateway: API Gateway-specific configuration. Only needed if the service type is lambda-apigateway.

• deploy.apigateway.rest_api_name: Unique name for the REST API. This will appear as the APIname under the API Gateway dashboard in the AWS console. is lambda-apigateway.

• deploy.apigateway.swagger_template: File path (relative to the yolo.yaml file) which containsthe API defintion in Swagger/OpenAPI format.

• deploy.apigateway.domains: List of custom domains and base path mappings to wire up Lambdafunction handlers to the API. See APIGateway.Client.create_base_path_mapping. You must define at least onemapping.

• deploy.apigateway.authorizers: Configuration for custom authorizers. This is useful if you need tointegrate your service with an existing identity provider to provide authentication/authorization. See APIGate-way.Client.create_authorizer.

• deploy.apigateway.integration: Configuration to wire up API Gateway requests to Lambda han-dlers. See APIGateway.Client.put_integration.

• deploy.lambda_function_configuration: Lambda Function runtime configuration. Only neededif the service type is lambda or lambda-apigateway. See Lambda.Client.create_function.

• deploy.parameters: Runtime environment/configuration parameters. Parameters defined in this sectionare stored encrypted in AWS’s SSM Parameter Store Even configuration parameters which are not strictly secretare stored encrypted anyway.

• deploy.parameters.stages: Sub-section for parameters for a specific stage.

• deploy.parameters.stages.default: Special stage where common parameters can be defined inorder to prevent duplication. Define parameters here which are common to all other stages.

• deploy.parameters.stages.<stage>: List of parameters for a specific stage. Each parameter definition consists of an object with the following keys:

– name: [REQUIRED] Name of the parameter.

– value: [OPTIONAL] Default value for the parameter.

– multiline: [OPTIONAL] Boolean value indicating if this parameter consists of multiple lines.This is useful for inputting multiline secrets such as private keys and SSL certificates.

Note: deploy.apigateway.authorizers[].name needs to match what’s specified in thesecurityDefinitions section of the Swagger API specification.

1.2.6 Context Variables

There are handful of “context” variables which can be used to dynamically populate values in parts of the yolo.yamlconfiguration. Context variables may only be used in the templates and services sections.

Context variables are rendered using Jinja templating and can used to replace a complete or partial string value. Forexample, 'prefix-{{ stage.outputs.MyOutput }}-suffix'.

The following contex variables are supported:

• account.name: The alias of AWS account which is currently being used to run a command. Account namesare defined in the accounts section of the yolo.yaml file.

1.2. yolo.yaml configuration file 7

yolo Documentation, Release 0.3.0

• account.account_number: The account number of the current AWS account. Typically a 12-digit integer.

• account.outputs.<output-name>: A specific output from the account CloudFormation stack. Seetemplates. The <output-name> is the name of an item defined in the Outputs section of the CloudForma-tion template. This is useful for dynamically referencing critical pieces of infrastructure– such as IAM roles orKMS keys–without having to hard-code the values.

• stage.name: The name of the current stage, such as dev, prod, or bob.

• stage.region: The AWS region where the current stage is hosted.

• stage.outputs.<output-name>: Similar to account.outputs.<output-name>, except that itis a reference to an output from the stage CloudFormation stack. See templates.

Often multiple context variables can be used together to assemble complex values. For example, an AWS ARN for agiven Lambda Function can be dynamically generated as follows:

arn:aws:lambda:{{ stage.region }}:{{ account.account_number }}:function:{{ stage.→˓outputs.FunctionName }}

1.3 Project structure

yolo strives to not be too prescriptive about project structure and aims to be flexible enough to support commonpatterns (and some uncommon ones) for structuring projects.

The following are some examples you can follow, whether you’re starting a new project or retrofitting an existingproject with yolo.

1.3.1 Example: Python service

dist/lambda_function.zip

lambda_main.pymyapplication/api.pyutil.pydb.py

my-swagger-api.yamlrequirements.txtsetup.pyyolo.yaml

Note the highlighted files above are created automatically by yolo build-lambda.

lambda_main.py

AWS Lambda needs a “handler” or “entry point” for execution. It’s a good practice to keep this file separate from therest of your codebase and as small as possible for a couple of reasons:

1. It prevents Lambda-specific patterns and constraints from propagating through your code base. This is importantin the event that you’re either

(a) porting an existing code base to run on Lambda; or

(b) porting a Lambda-ready code base to run on another platform (such as Docker/ECS)

8 Chapter 1. Getting Started

yolo Documentation, Release 0.3.0

2. It keeps things clean so that your application components are nicely decoupled and can be updated indepen-dently.

Listing 1: Example:

from myapplication import api

def lambda_handler(event, context):"""Main entry point of Lambda function.

:param dict event:Dictionary containing the entire request template. This can vary wildlydepending on the template structure and contents.

:param context:Instance of an AWS Lambda Python Context object, as described onhttp://docs.aws.amazon.com/lambda/latest/dg/python-context-object.html.

"""return api.process_event(event, context)

my-swagger-api.yaml

Applications based on Lambda and API Gateway require a detail API specification to be provided. The simplest wayto provide that definition is to use Swagger.

Below is an example Swagger API specification, written in YAML.

swagger: '2.0'basePath: '/myapi/v1'info:

version: 'v1'title: My APIdescription: This is the long description of my API

schemes:- https

consumes:- application/json

produces:- application/json

securityDefinitions:auth_token:type: "apiKey"name: "Authorization"in: "header"

paths:/stuff:get:

description: Get a list of stuffoperationId: 'myapplication.api.get_stuff'security:

- auth_token: []parameters:

- name: limitdescription: Used for pagination to limit pages sizesin: querytype: integerrequired: false

(continues on next page)

1.3. Project structure 9

yolo Documentation, Release 0.3.0

(continued from previous page)

default: 100- name: offsetdescription: Used for pagination, indicates the index of the first element

→˓to be returnedtype: integerin: queryrequired: false

responses:200:description: Success

404:description: Not Found

504:description: Gateway Timeout

500:description: Internal Server Error

To build out your own API specification, here are some helpful links:

• Swagger (OpenAPI) Specification

• Swagger Editor

requirements.txt

This is the kind of basic requirements file that you will find in most Python projects. If your Lambda applicationrequires any third-party dependencies, specify those a requirements.txt file and yolo build-lambda will bundlethem up for you.

This file needs to be referenced in the build section of your service config.

setup.py

setup.py is not strictly required for yolo to bundle and deploy your application, however most Python projectswill include one.

This file is only mentioned in this documentation to illustrate common patterns for laying out Python projects.

yolo.yaml

This file describes the general architecture of your service, as well as how to configure and deploy it. Virtuallyeverything you need (with this exception of infrastructure templates) to deploy your service is contained in this file.

See yolo.yaml.

1.3.2 Example: S3 service

An S3 service is essentially just a collection of static assets which are stored in S3. Examples of S3 services include:

• Statically hosted UI applications (see http://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html)

• Documentation

• Images, videos, or other large static assets

10 Chapter 1. Getting Started

yolo Documentation, Release 0.3.0

Todo: We need some simple, concrete examples of S3 services.

TODO: This section needs more detail.

1.4 Commands and workflows

The following command sequences are list more or less order of relevance for standing up new projects.

1.4.1 Deploy account-level infrastructure

Generally the first thing you’ll want to do is deploy any account-level infrastructure. This serves as a baseline for allother infrastructure. Account-level infrastructure typically includes resources such as:

• KMS encryption keys

• DNS hosted zones

• S3 buckets

• IAM roles

Account-level infrastructure should rarely change and should never be deleted (except in the case of a complete tear-down).

Listing 2: Example:

$ yolo deploy-infra --account DevAccount

What’s going on here?

• The command above will create or update a CloudFormation stack using the templates and parameters definedin the templates.account.

• DevAccount must be a valid account alias defined in the accounts section.

You should repeat this process for each relevant account. This command can be used for new deployments or to updateexisting deployments.

1.4.2 Deploy stage-level infrastructure

Similar to the command above, the following command will deploy stage-level infrastructure.

$ yolo deploy-infra --stage dev

Stage-level infrastructure can change slightly more frequently that account-level infrastructure but less frequently thanapplication code. Stage-level infrastructure typically includes resources such as:

• SNS topics

• SQS queues

• Various types of storage resources (such as S3 buckets, EBS volumes, or DynamoDB tables)

• Autoscaling groups

• EC2 instances (virtual machines)

1.4. Commands and workflows 11

yolo Documentation, Release 0.3.0

• CloudFront CDN configurations

• Elastic Load Balancers

• DNS record sets

• IAM roles

• Security groups

These infrastructure stacks should be designed to completely stand-alone, with the exception of some inputs from theaccount-level infrastructure.

Stage-level stacks should not be dependent on one another, nor should they create any naming conflicts with a targetAWS account. Dynamically naming resources by using the stage.name context variable variable as a templateparameter is a good practice.

1.4.3 Show infrastructure stack status

Show an overview of the deploy stacks (for all accounts and stages):

$ yolo status

1.4.4 Store parameters in SSM

yolo manages the storage and usage of configuration (especially secrets) through AWS SSM.

First, define the parameters your application needs in the services.<service>.parameters section. For parameterswhich are not secret, you may define a default value. For parameters which are secret (such as passwords and APIkeys), do not specify a default value.

Next, run the following command to upload parameters into a centralized location in SSM:

$ yolo put-parameters --service MyLambdaService --stage dev --use-defaultsSetting "db_password" to "password"Setting "log_level" to "info"Setting "db_username" to "admin"Enter "ssl_private_key" multiline value (ctrl+d when finished):-----BEGIN RSA PRIVATE→˓KEY-----ryxjSsshEa9Ml08TA1YPjrEfQXRmdeVf9PJdSgV3zKI5+UV/g+J3MMxLZ/CRjhnn...rjfwMC1qbyhXz/5so17CfdMCgYAv0ypMF4SU0ao73zObHFV08e7ced==-----END RSA PRIVATE KEY-----Environment configuration complete!

When you deploy an application, yolo will copy the centrally stored parameters to another location in SSM in orderto lock down the configuration for a given deployment. This makes deployments less prone to mutation.

1.4.5 Build a Lambda application

To build a Lambda Function code package, run:

$ yolo build-lambda --service MyLambdaService --stage dev

This will generate a code package (zipfile) which can be upload to AWS Lambda.

12 Chapter 1. Getting Started

yolo Documentation, Release 0.3.0

1.4.6 Create an application release

Once you’ve locally built a code package, create a release with it and ready it for deployment:

$ yolo push --service MyLambdaService --stage dev

This will place all the necessary build artifacts in an S3 and ready them for deployment.

This commands works for s3 services as well as lambda and lambda-apigateway services.

Note: This may seem like an extra and unncessary step, but if you need to rollback a deployment to fix a regression, itcan be handy to reference a previous build in a central location (S3) without having to recreate the build from source.

1.4.7 List released application builds

To list the builds which have been pushed for release, run:

$ yolo list-builds --service MyLambdaService --stage devBuild Timestamp---------------------------------------- --------------------------b407b7ff4cdb61a4ffc7146b367a76a4a347e320 2017-10-09_12-51-54-143005876af6fe66c2e58f70b1c3b56dc88553c7bd4063 2017-10-09_11-57-37-302699370dfc794cf89ab07f93edcf2128b0327344b884 2017-10-09_11-33-19-07122918241c3aa4d7c223c2333a6cc8050c4c9fede333 2017-10-09_10-31-04-20288118241c3aa4d7c223c2333a6cc8050c4c9fede333 2017-10-09_10-25-00-666010

Take note of the Build version you want to deploy.

1.4.8 Deploy a Lambda service

To deploy a build of a Lambda-based service, run:

$ yolo deploy-lambda --service MyLambdaService --stage dev --version→˓18241c3aa4d7c223c2333a6cc8050c4c9fede333

For lambda services, this command will create or update the target Lambda Function with the release code and adjustspecific configuration options (such as memory and timeout settings). For lambda-apigateway services, this com-mand will create/update API Gateway integrations and API definitions in addition to the Lambda Function updates.

Note: Note that in the yolo list-builds example above there can be multiple builds for the same version. Ifyou deploy from a version with multiple builds, yolo will choose the most recent build according to its timestamp.

You also have the ability to deploy a new version of your Lambda Function from a local build, without having to yolopush before deployment. It’s useful for quick iterations.

$ yolo deploy-lambda --service MyLambdaService --stage dev --from-local

1.4.9 Deploy an S3 service

To deploy a build of an S3-based service, run:

1.4. Commands and workflows 13

yolo Documentation, Release 0.3.0

$ yolo deploy-s3 --service MyS3Service --stage dev --version→˓025c43fdfe41e54f5a8ab80723a7b1a9983df6c2

S3 service deployments are significantly simpler than those for Lambda-based services. Deployment simply syncsartifacts from the build location (in S3) to the target S3 bucket indicated by the bucket_name defined in the serviceconfiguration.

1.4.10 Overview of commands

$ yolo -hUsage: yolo [OPTIONS] COMMAND [ARGS]...

Manage infrastructure and services on AWS for multiple accounts/stages.

(Or, "yolo everything into prod".)

Options:-h, --help Show this message and exit.

Commands:build-lambda Build Lambda function packages.clear-config Clear cached configuration for `yolo`.deploy-baseline-infra DEPRECATED: Use `yolo deploy-infra` instead.deploy-infra Deploy infrastructure from templates.deploy-lambda Deploy Lambda functions for services.deploy-s3 Deploy a built S3 application.list-accounts List AWS accounts.list-builds List the pushed builds for a service/stage.list-lambda-builds DEPRECATED: Use `yolo list-builds` instead.list-s3-builds DEPRECATED: Use `yolo list-builds` instead.login Login with and cache Rackspace credentials.push Push a local build, ready it for deployment.push-lambda DEPRECATED: Use `yolo push` instead.put-parameters Securely store service/stage parameters.run Run a script with AWS account credentials.shell Launch a Python shell with AWS credentials.show-config Show currently cached configuration.show-outputs Show infrastructure stack outputs.show-parameters Show centralized config for a service/stage.show-service Show service configuration for a given stage.status Show infrastructure deployments status.upload-s3 DEPRECATED: Use `yolo push` instead.

14 Chapter 1. Getting Started

CHAPTER 2

Feature Roadmap

The following features are planned for yolo, more or less in order of importance:

• Terraform support: Currently only CloudFormation templates are supported. It would be useful for projectswhich already have infrastructure defined in Terraform templates to be able to take advantage of yolo withneeding to port templates to CloudFormation.

• ECS support: Currently yolo only supports building/deploying backend applications based based on LambdaFunctions running Python 2.7 or Python 3.6. Add ECS support for build/deploy commands would allow yoloto deploy virtually any application that can’t or doesn’t run in Lambda. This could be useful for retrofittingyolo deployment methods onto existing “legacy” projects.

• Basic AWS credentials support: Currently, yolo only supports fetching AWS credentials through Rackspace’sFanatical AWS API (which requires a Rackspace API key. This is a good feature, but it only helps for a smallof use cases and many people could benefit from more general AWS support.

• Additional runtime support for Lambda: AWS Lambda support Python 2.7/3.6, Node.js 4.3.2/6.10.3, Java 8,and .NET Core 1.0.1 (C#). yolo currently only supports Python 2.7/3.6. It would be useful to expand supportand enable more use cases.

15

yolo Documentation, Release 0.3.0

16 Chapter 2. Feature Roadmap

CHAPTER 3

Indices and tables

• genindex

• modindex

• search

17