microservices for java architects · what are aws lambdas? •you provide custom code -> aws...

AD23 DevOps Engineering 3:00 PM

AD23 -‐ AWS Lambda: Best Practices and Common Mistakes

Presented by:

Derek Ashmore Asperitas Consulting

Brought to you by:

888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ [email protected] -‐ https://agiledevopswest.techwell.com/

Derek Ashmore Derek Ashmore is a senior technology expert with more than thirty years of experience in a wide variety of technologies and industries. Derek is currently focusing on microservices architectures, cloud computing, and migrating applications to the cloud. Derek's approach allows companies to increase speed to market while also increasing application availability. Derek routinely speaks at JavaOne, DevNexus, the Chicago Coders Conference, and many others. His books include The Java EE Architect's Handbook and Microservices for Java EE Architects.

AWS Lambda: Best Practices and Common Mistakes

Given by Derek C. AshmoreDevOps WestJune 5, 2019

©2018 Derek C. Ashmore, All Rights Reserved 1

http://www.asperitasconsulting.com/

Who am I?• Professional Geek

since 1987

• Java/J2EE/Java EE since 1999

• AWS since 2010

• Specialties• Cloud

Workshops

• Cloud-native Applications

• Yes – I still code!


http://www.amazon.com/gp/product/0972954880/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0972954880&linkCode=as2&tag=dvtpresscom-20


Discussion Resources

• This slide deck– https://www.slideshare.net/derekashmore/presentations

• Sample code on my Github– https://github.com/Derek-Ashmore/

• Slide deck has hyper-links!

– Don’t bother writing down URLs


http://www.slideshare.net/derekashmore

https://github.com/Derek-Ashmore/

Agenda

The “What”

and “Why” of AWS Lambda

Code-Level Tips

Operation and Design

Habits

When to use

Lambdas

Summary / Q&A


What are AWS Lambdas?

• You provide custom code -> AWS runs it– Java, Node.js, Python

• Computing power with less management– AWS manages the hardware– AWS autoscales that hardware– AWS maintains that hardware

• Lambdas are event driven– API Gateway (e.g. RESTful Web Service call)– Many more

• Lambdas are stateless• Not to be confused with “Lambda Expressions” in Java 8


Lambda Event Sources

• API Gateway

• SNS Messaging Subscriptions

• Schedule

• Storage writes – S3, DynamoDB, Kenesis


• SES Email receipt

• Cloudwatch– Schedule, Events, log entries

• Cognito (Security)

• CloudFormation– Creation script

What’s the Business Benefit

• Less Maintenance Hassle• Unlimited Parallelism• Current cost advantage

– Don’t pay for idle– CPU cost currently lower

• Free tier– 1 M executions and 400K compute seconds per month– Memory allocated determines allowed free-tier runtime

• 20 cents per 1 M executions + memory/runtime cost

– Administration cost• No O/S upgrades, server backups, etc.


https://aws.amazon.com/lambda/pricing/

There’s no free lunch

• Less control over environment– Harder to tune– Memory and time limits on execution

• Few Environment amenities– No connection pooling, session support, caching

• Proprietary Interface– Potential Technical Lock-in

• No Guarantee that AWS cost will be constant– Potential Business Risk

• Modern version of CGI


Agenda

The “What”


Code-Level Tips


Habits

When to use

Lambdas

Summary / Q&A


What Makes a “Best Practice”?

• Makes Support Easier• Increases Reuse

• Increases Performance

• Minimizes Resource Consumption

– Labor

– Runtime


Let’s start with Low-Hanging Fruit


Report Inputs/Env on Exception

• Place a Try / Catch in your handler

– Python Example

– Java Example

• Also check your arguments with a clear error message


def crossAccountHandler(event, context):

try:

………………

except Exception as e:

e.args += (event,vars(context))

raise

https://github.com/Derek-Ashmore/AWSDevOpsUtilities/blob/master/samples/cross-account/crossAccount.py

https://github.com/Derek-Ashmore/AWSLambdaExamples/blob/master/aws-email-collector/src/main/java/org/force66/aws/email/collector/EmailCollectorLambda.java

Check Arguments Up Front

• Check your arguments with a clear error message

– Python Example

– Java Example


def crossAccountHandler(event, context):

try:

if 'Assumed_Role' in event:

…………………

else:

raise Exception('Assumed_Role not provided as argument')

except Exception as e:


https://github.com/Derek-Ashmore/AWSLambdaExamples/blob/master/aws-email-collector/src/main/java/org/force66/aws/email/collector/EmailCollectorLambda.java

Specify Lambda Source Repo

• Explicitly put the source repository name in the Lambda comments– In most organizations, the repository name isn’t obvious

– Others changing your code need it

– You don’t want source control to be out of date


"""

secretLambda.py

……………

Source Control: https://github.com/Derek-Ashmore/AWSDevOpsUtilities

"""

Separate Lambda from Business Logic

• Make business logic reusable– Callable by other applications

– Usable on premises

• Easier to locally develop and debug– Lambda-specific logic is thin!


def startStopHandler(event, context):

try:

executeStopStart(datetime.datetime.now()

, os.getenv('Scheduled_StartTime', ‘’)

, os.getenv('Scheduled_StopTime', ‘’)

, os.getenv('Scheduled_StartStop_Days', 'M,T,W,R,F’))

……………

return 0;

This is low-hanging fruit that will be appreciated by your fellow developers!


• Log All Inputs and Environment on Exception

• Check all arguments up front• Document the source repo at the top.

• Repo readme can have other developer specifics

• Separate Lambda code from business logic

• Now let’s talk design and operations….

Agenda

The “What”


Code-Level Tips


Habits

When to use

Lambdas

Summary / Q&A


Automate builds and deployments!


Lambda Copies Everywhere!

• Changes / Bug Fixes need to be deployed everywhere

• Solving with automation solves the wrong problem!


One Copy for All!

• Scalable – only need to add accounts over time• Bugfixes in one place• Configuration usually in common DynamoDB table(s)• Sample in Python here



Cross-Account Execution

• Algorithm is

– Assume a remote-account role using STS

• The response has temporary credentials

– Create a session using the remote account creds

– Do work in the remote account

• Example here: Derek-Ashmore/AWSDevOpsUtilities (Github)



For workloads over 5 min

• Executor that invokes lambda asynchronously for each account

• Sample in Python here


https://github.com/Derek-Ashmore/AWSDevOpsUtilities/blob/master/productivity/lambda-controller/lambdaController.py

Limit Custom Nesting to One Level

• Debugging with nested executions is

– Time consuming and difficult

– Can’t do locally

– Absolutely requires unique correlation id for the entire transaction

• Allows you to tell invocation history for one logical transaction

– Instead of deep custom nesting, use AWS Step Functions

• Use Step Functions if you need more


Nested Calls using AWS Step Functions

• AWS Step Functions

– Uses a State Machine model

• Think turn-style to get access to train– States are “Locked” and “Unlocked”

– Locked → Payment input allowed, then “Unlocked”

– Unlocked → One person allowed through, then “Locked”

– Automatically provides correlation between invocations

• Unified logs for the entire transaction


https://en.wikipedia.org/wiki/Finite-state_machine

Operations and Design Habits


• Automate Builds and Deployments• Only install Lambda’s Once• Limit Lambda nesting to One Level

• Step functions if you need more

• Now let’s talk dependencies and secrets

Use Configuration Injection

• No environment specifics hardcoded in the Lambda deployment

• Use Environment Variables on the Lambda Definition– No un-encrypted secrets (e.g. database password)

• Use Arguments in the triggering event– No un-encrypted secrets

• Anti-Example– Splunk forwarding Lambda with hard-coded Splunk channels


Providing Secrets to Lambdas

• Secrets are needed items like credentials of any type.

• Use IAM Roles to grant permission to read secrets

• Options are:

– Use KMS

• Encrypt credential and base64 encode it– Place encrypted version in environment variable

• Sample Lambda and Encryption Script (here)

– Use a Digital Vault (e.g. AWS Secrets Manager)

• Sample Lambda here


https://github.com/Derek-Ashmore/AWSDevOpsUtilities/tree/master/samples/kms-secrets

https://github.com/Derek-Ashmore/AWSDevOpsUtilities/blob/master/samples/secrets-manager/secretLambda.py

AWS Secrets Manager

• Use IAM Roles to grant permission to read secrets

• You don’t need a “secret” to get a “secret”!


Avoid Heavy-Footprint Dependencies

• Minimizes load time

– Mitigates cold-start problem

• Java

– Use Guice over Spring

• Python

– Use AWS provided deps first (list is here)


https://github.com/google/guice

https://gist.github.com/gene1wood/4a052f39490fae00e0c3

Don’t Hog Resources

• 512 Mb temp space per invocation

• 1,024 file descriptors

• 1,024 Threads and processes (combined)

• 6 MB payload size (synchronous)

• 128 KB payload size (asynchronous)

• 1000 concurrent lambda executions per account per region


Idempotence

• Possible for your Lambda to be invoked multiple times for the same event– Prevent repeat actions from having a different effect.

• Options– Record the event id –> Skip repeated events

• Most event sources provide a unique request id– Lambda invoking lambda does not!

• Negatively affects performance– 1 extra read

– 1 extra write

• Need to roll-off old events

– Insure that the effect is the same each time• Not perfect → You don’t control invocation order


Agenda

The “What”


Code-Level Tips


Habits

When to use

Lambdas

Summary / Q&A


Suitable workloads for Lambda’s

• Workloads that – Take less than 15 min– Are stateless– Idempotent

• Evaluate cost with calculator: https://dashbird.io/lambda-cost-calculator/• Typical examples

– Streaming data processors– Dynamo DB Change Processors– AWS-specific DevOps Tasks

• Security Enforcement• Uptime Scheduling• AWS Change Event processing

– CloudFormation Macros


https://dashbird.io/lambda-cost-calculator/

Lambda can herd the cats!

• Using Lambda to enforce security– Automatic Remediation

• Unlike AWS Config, Lambdas can take action!

• Unwanted port exposures– Unauthorized exposure of 0.0.0.0/0 to the world

• Decentralized Management– Empowers the organization

– Improves speed to market• Less bottleneck by admin groups

– Still keeps the enterprise secure


Lambdas and Microservices


Using Lambdas as Microservices

• Lambda / API Gateway is a deployment option for microservices– No differences in design principles

• Single purpose• Self-contained

– Still design for failure• Don’t “assume” that Lambda outages can’t happen

– A Lambda might need external resources that aren’t available

• Off Limits: Coding patterns that use state– Lambdas must be stateless– Fail fast patterns

• Service Call Mediator• Circuit Breaker

– Performance Patterns• Expiring Cache (API Gateway allows request caching)


Lambda Alternatives

• Use Kubernetes

– Operates as a Lambda type service in Kubernetes

– Functions get shut down when not used – like Lambda behind the scenes

– Examples

• Fission

• Knative

• Kubeless

• Serverless workloads are Containerized

– Run on premises or on any Cloud


Further Reading

• This slide deck– https://www.slideshare.net/derekashmore/presentations

• AWS Lambda Reading List– http://www.derekashmore.com/2016/04/aws-lambda-reading-list.html

• Amazon’s Published Best Practice List– https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html


http://www.slideshare.net/derekashmore

http://www.derekashmore.com/2016/04/aws-lambda-reading-list.html

https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html

Questions?

• Derek Ashmore:– Blog: www.derekashmore.com

– LinkedIn: www.linkedin.com/in/derekashmore• Connect Invites from attendees welcome

– Twitter: https://twitter.com/Derek_Ashmore

– GitHub: https://github.com/Derek-Ashmore

– Book: http://dvtpress.com/


http://www.derekashmore.com/

http://www.linkedin.com/in/derekashmore

https://twitter.com/Derek_Ashmore

https://github.com/Derek-Ashmore

http://dvtpress.com/

http://www.amazon.com/gp/product/0972954880/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0972954880&linkCode=as2&tag=dvtpresscom-20


microservices for java architects · what are aws lambdas? •you provide custom code -> aws...

Documents