aws lambda and serverless framework: lessons learned while building a serverless company
TRANSCRIPT
AWS Lambda &Serverless framework Lessons learned while building a serverless company
26/01/2017@Podgeypoos79 - @loige
Luciano Mammino
● @loige on Twitter
● Lmammino on GitHub
● Blog at loige.co
● Co-author of:
Node.js Design Patterns (Second Edition)
Padraig O’Brien (a.k.a “Podge”)
● @podgeypoos79 on Twitter
● Does not have a book (yet…)
● Organiser of Nodeschool Dublin and Dun Laoghaire
● Organiser of Dublin Serverless meetup
● He is writing his own database, unicorndb �
Agenda
● Planet 9 Energy● Serverless● Security● Quality● Developer Experience● Costs● Lessons learned
17.520 * * 24
Meter readings Customer size Data versions
≈4 billion data pts / year (tot.)
SMALL customer ≈3 mln data pts / year MEDIUM customer ≈20 mln data pts / year BIG customer ≈110 mln data pts / year
( Per customer / year )
● Limited number of “Full stack” engineers
● Write & deploy quality code as fast as possible
● Adopt hot and relevant technologies
● No servers... or more to the point, NO CALLS AT 2 AM!
Our requirements
Early experiments
Current Technology
“Serverless”Meaning
Serverless frameworkAWS Lambda
What does “serverless” mean?
Why Serverless framework?
Anatomy of Serverless.yml
Serverless.yml header
Defining functions and events
Why?
What is a lambda?
● Function as a service (FAAS)
● Pay for invocation / processing time
● Virtually “infinite” auto-scaling
● Focus on business logic, not on serversdaaaa!
Lambda as micro-services
● Events are first-class citizens● Every lambda scales independently● Agility (develop features quick and in an isolated fashion)
Classic micro-services concerns:
● Granularity (how to separate features? BDD? Bounded Contexts?)● Orchestration (dependencies between lambdas, service discovery…)
Anatomy of a Lambda in Node.js
Some use cases
● REST over HTTP (API Gateway)
● SNS messages
● Schedule/Cron
● DynamoDB data changes
● S3 files changes
● IoT
HTTP REQUEST - API Call
POST /hello/Elvis?foo=bar
{ “test”: “body”}
SecurityAuthentication AuthorizationSensitive data
Authentication
(JWT tokens) Custom“authorizer lambda”
“Who is the current user?”
Users
API 1
Authentication
AuthorizerAPI 2
API 3
CredentialsAPI
Request
JWT token
JWT tokenVerify credentials
Validate token& extract userId
User: Podge Pass: Unicorns
Authorization
“Can Podge trade for Account23 ?”
User Action Resource
Authorization
● Custom ACL library imported by every lambda
● Built on top of:
○ node-acl
○ Knex.js
● Persistence in Postgres
User Action Resource
Podge trade Account23
Podge changeSettings Account23
Luciano delete *
... ... ...
ACL Lib Example
Environment variables
QualityTesting
Continuous IntegrationDeployment
Testing
JEST Node-TAP
Unit Testing
● Split business logic into small testable modules
● Use dependency injection for external resources
(DB, Filesystem, etc.)
● Mock dependencies with Sinon
● Aim for 100% coverage
● Use child_process.exec to launch “serverless invoke local”
● Use node-tap to make assertions on the output
● Test environment recreated locally with docker (e.g. Postgres test DB).
● Cannot always test locally (Redshift, SNS, SQS…)
Functional Testing
Functional test with sls invoke local
Building the project
Babel custom preset (transpile to Node 4.3.2)
Build process
JSdoc
Up to 90% code size reductionUsing Webpack 2 “tree-shaking”
Continuous Integration
Git-Flow● Check code style (ESLint)
● Run unit tests
● Build the project
● Run functional tests
● If commit is on “master”:
Create deployable artifact
● Develop features/fixes locally
using git branches
● Push to GitHub
Deployments
local test dev qa production
Deploy lambdas and React apps
Downloading the artifact from CircleCI
Deploying the code using Serverless
Developer ExperienceLocal development
DebuggingMonitoring
● Develop locally - Invoke local.
● Deploy to AWS.
● Invoke on AWS and stream the logs.
● This is a SLOW feedback loop.
● Lots of requests to improve local dev.
● Plugins are helping to improve this.
Local development
Serverless Plugins
Serverless Offline
Serverless Api Gateway Logs
Serverless Mocha
Monitoring
CostThe formula
Cost forecasts
$ = Time * Memory * Invocation
Our current Lambda costs
Lessons learned( we found out…)
DEBUGGING
How do we do debug then...
● console.log … �
● Using debug module
● Enabling detailed logs only when needed (per feature)
Learn Cloudformation
Api GatewayCustom domain mapping
It is a manual step…
With Api Gateway...You have 10 30 seconds!
AWS Lambda limitations
● 512 MB of TMP space
● Lambdas can only execute for 300 seconds
● Request Response size of 6 MB
● Event size of 128 Kb
● Max 50 MB for deployment package
AWS Soft limits
http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html
Cold starts...
☃�
https://www.iopipe.com/2016/09/understanding-aws-lambda-coldstarts/
serverlessbeer.com
is a real thing!
Well… it’s not, but it’s a good idea!
(read the tutorial)
Recap
● Serverless architectures are COOL!
○ Infinite scalability at low cost
○ Managed service
○ Still has some limitations
● Managing a project might be hard but:
○ Technology progress and open source projects (Serverless.com) will make things easier
Planet 9 Energy tech-team
We are hiring :)
Peter Saŝa Podge Luciano YOU?
Alberto Joe DomagojHugh Gus Ruth
Check outLoki.js