@awscloud es designing next-gen apps: serverless€¦ · building blocks for serverless...
TRANSCRIPT
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Ignacio García @igngar_cloud
Designing next-gen apps: ServerlessOctober 2018
AWS Solutions Architect
@awscloud_es
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
• Introduction to Serverless
• Use case: Microservices
• Viesgo: Datalake
• Recap - resources
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Serverless means…
F l ex i b l e S ca l i n g
N o I d l e C a p a c i t y
$
H i g h Ava i l a b i l i t y
N o S e r v e r M a n a g e m e n t
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Serverless Growth
Hundreds of thousands of active customers
From students building Alexa Skills to Enterprise.
3X year-over-year usage growth
Vibrant startup ecosystem
stdlib, IOpipe, Serverless.com, FaunaDB
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Building blocks for serverless applications
AWS Lambda Amazon DynamoDB
Amazon SNS
Amazon API Gateway Amazon SQS Amazon Kinesis
Amazon S3
Orchestration and State Management
API Proxy and GraphQL Messaging and Queues Analytics
Monitoring and Debugging
Compute Storage Database
AWS X-RayAWS Step Functions Amazon Cognito
User Management and IdP
AWS AppSync Amazon Athena
AWS Lambda@Edge Amazon Aurora Serverless
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A typical day for a developer…
Manager: “We need an app to let our customers send feedback– our competitors just launched the same thing, so I need it fast. I don’t want to pay a lot for it, especially when no one is using it. But remember that we’re growing, so make sure it scales great and is easy to manage and operate. And you’re on your own – sorry!”
Developer “Not a problem. I’ll make it serverless…”
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A typical day for a Serverless developer…
Serverless web app with
…an API ???
…access to existing data stored in ???
…static content served by ???
…dynamic content/business logic encoded as ???
… authenticate/authorize application ???
… orchestrate application logic ???
Joe promised h is boss an app.Now what?
Step 1: Find out what to use!
Joe we needs
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon API Gateway
Create a unified
API frontend for
multiple micro-
services
Authenticate and
authorize requests
to a backend
DDoS protection
and throttling for
your backend
Throttle, meter,
and monetize API
usage by third-
party developers
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Building an API with Amazon API Gateway
Internet
Mobile Apps
Websites
Services
AWS Lambda functions
API Gateway Cache
Endpoints on Amazon EC2
All publicly accessible endpoints
Amazon CloudWatch Monitoring
Amazon CloudFront
Any other AWS service
Endpoints on Amazon VPC
CognitoAuthorizer
CustomAuthorizer
API Authorization
PrivateAPI endpoints
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
API Configuration
You can create APIs
Define resources within an API
Define methods for a resource
• Methods are Resource + HTTP verb
Pet Store
/pets
/pets/{petId}
• GET
• POST
• PUT
API Configuration can be deployed to a
stage
Stages are different environments
For example:
• Dev (e.g. awsapigateway.com/dev)
• Beta (e.g. awsapigateway.com/beta)
• Prod (e.g. awsapigateway.com/prod)
• As many stages as you need
Pet Store
dev
beta
gamma
prodCanary releases within a stage!
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS API Gateway customers
“With Amazon API Gateway and AWS Lambda, the user
experience is up to 90% faster. That's for both
photographers uploading images and the editorial team processing them.”
Marco ViganòHead of Digital Development
Using a common API layer powered by Amazon API
Gateway and Network Load Balancer, Fox can completely decouple the frontend of FOX
NOW from the backend content system.
NH uses API Gateway for their NH PCI Bubble workload.
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Business logic encoded in AWS Lambda
Changes in data state
Requests to endpoints
Changes in resource state
EVENT SOURCE SERVICES (ANYTHING)FUNCTION
Node.jsPythonJavaC#GolangPowershell
SLA: 99.95% Monthly Uptime Percentage
https://aws.amazon.com/lambda/sla/
Up to 15 min
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lambda execution models
Asynchronous (event) Poll-based
Amazon SNS
AWS Lambda function
Amazon S3
reqs
AmazonAPI Gateway
AWS Lambda function
/order
Synchronous (push)
Amazon DynamoDB
Amazon Kinesis
changes
AWS Lambda service
function
Amazon SQS
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Tweak your function’s computer power
Lambda exposes only a memory control, with the % of CPU core and network capacity allocated to a function proportionally
Is your code CPU, Network or memory-bound? If so, it could be cheaper to choose more memory.
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Concise function logic
• Separate Lambda handler (entry point) from core logic
• Use functions to TRANSFORM, not TRANSPORT
• Dynamic logic via configuration• Per function – Environment variables
• Cross function – Amazon Parameter Store/Secrets Manager
• Read only what you need. For example:• Properly indexed databases
• Query filters in Aurora
• Use S3 select
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Lambda customers
Seamlessly ramps up to peak traffic of
22K concurrent requests
Quick time to market: 8 weeks from conception to production
Ableto scale up to 20,000 concurrent Lambda executions in
testing
One simulation of 20 million mortgages ran in 1.5 hours, more
than 4X faster than existing process
Enel uses AWS Lambda for their Factelec workload to generate
electronic bills
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ExtractImageMetadata
ImageTypeCheck
TransformMetadata
Rekognition Thumbnail
StoreMetadata
Start
End
AWS Step Functions
• Execute One or One Million• Scales out• Doesn’t lose state• Deals with errors/timeouts• Auditable
• Easy to build! • Defined in JSON – see your progress in the console.
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Step Functions
State Types
Task: a single unit of work “I want to sequence functions”
Retry: If fails, retry “I want to retry functions”
Choice: a single unit of work “I want to select functions based on data”
Parallel: fork and join data across tasks “I want to run functions in parallel”
Pass: passes inputs to its outputs ”I want to pass this till the next step”
Wait: wait until n seconds. “I have code that runs for hours”
Fail: Stops and mark as failure “I want try/catch/finally”
Success: Stops an execution successfully ”I finished!”
ExtractImageMetadata
ImageTypeCheck
TransformMetadata
Rekognition Thumbnail
StoreMetadata
Start
End
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
No orchestration in code
STA
RT
JOB
JOB
#X
STA
RTE
D
HT
TP P
OST
HT
TP P
OST
AR
E W
E TH
ERE
YET?
NO
PE!
WE’
RE
DO
NE!
ZzZz
OR
time.sleep(10)
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
No orchestration in code – use AWS Step Functions!
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Step Functions customers
Custom monitoring and alarming with step functions.
“Developers do not have to be experts in workflow, and the
developers who wrote the Lambda function can easily create the Step
Functions workflow. We can put more developers on projects, and that will
ultimately enable the creation of more internal solutions that will
drive efficiency and productivity. ”
Paul BrownSenior Developer Manager
Coca-Cola uses Step Functions for:
Vending pass worklow
Nutrition information validation workflow
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Accessing stored data in Amazon DynamoDB
Dead Simple• GetItem(primaryKey)
• PutItem(item)
Amazon DynamoDB Accelerator
Amazon DynamoDB
Applications
Amazon DynamoDB - StreamsMore than a hundred thousand AWS customers use Amazon DynamoDB!
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Dynamo DB customers
“We recently wrote a zero-infrastructure service using AWS Lambda and Amazon
DynamoDB,”
Capital One uses DynamoDB to reduce latency for their
mobile applications by moving their mainframe transactions
to a serverless architecture for unbound scale.
Samsung Electronics uses DynamoDB for their petabyte
size mobile app backups, resulting in consistent high
performance and cost savings.
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Serving static content with S3 and Cloudfront
Amazon CloudFront Amazon S3
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Managing Access with Cognito
Web and Mobile Apps
Amazon Cognito
Developers focus on what is special about
their app
Cognito handles auth and identity
Federation
Managed User Directory
Hosted UI
AWS Credentials
Standard Tokens
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Bringing all that together…
CloudFront
Content Delivery
ApplicationLayer
PersistencyLayer
S3
Static Content
APILayer
API GatewayDynamoDB
AWS Lambda
Cognito
AWS Step Functions
AWS Step Functions
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CloudFormation templateAWSTemplateFormatVersion: '2010-09-09'Resources:
GetHtmlFunctionGetHtmlPermissionProd:Type: AWS::Lambda::PermissionProperties:Action: lambda:invokeFunctionPrincipal: apigateway.amazonaws.comFunctionName:
Ref: GetHtmlFunctionSourceArn:
Fn::Sub: arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${ServerlessRestApi}/Prod/ANY/*ServerlessRestApiProdStage:
Type: AWS::ApiGateway::StageProperties:DeploymentId:
Ref: ServerlessRestApiDeploymentRestApiId:
Ref: ServerlessRestApiStageName: Prod
ListTable:Type: AWS::DynamoDB::TableProperties:ProvisionedThroughput:
WriteCapacityUnits: 5ReadCapacityUnits: 5
AttributeDefinitions:- AttributeName: id
AttributeType: SKeySchema:- KeyType: HASH
AttributeName: idGetHtmlFunction:
Type: AWS::Lambda::FunctionProperties:Handler: index.gethtmlCode:
S3Bucket: flourish-demo-bucketS3Key: todo_list.zip
Role:Fn::GetAtt:- GetHtmlFunctionRole- Arn
Runtime: nodejs4.3GetHtmlFunctionRole:
Type: AWS::IAM::RoleProperties:ManagedPolicyArns:- arn:aws:iam::aws:policy/AmazonDynamoDBReadOnlyAccess
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRoleAssumeRolePolicyDocument:
Version: '2012-10-17'Statement:- Action:
- sts:AssumeRoleEffect: AllowPrincipal:
Service:- lambda.amazonaws.com
ServerlessRestApiDeployment:Type: AWS::ApiGateway::DeploymentProperties:RestApiId:
Ref: ServerlessRestApiDescription: 'RestApi deployment id: 127e3fb91142ab1ddc5f5446adb094442581a90d'StageName: Stage
GetHtmlFunctionGetHtmlPermissionTest:Type: AWS::Lambda::PermissionProperties:Action: lambda:invokeFunctionPrincipal: apigateway.amazonaws.comFunctionName:
Ref: GetHtmlFunctionSourceArn:
Fn::Sub: arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${ServerlessRestApi}/*/ANY/*ServerlessRestApi:
Type: AWS::ApiGateway::RestApiProperties:Body:
info:version: '1.0'title:
Ref: AWS::StackNamepaths:
"/{proxy+}":x-amazon-apigateway-any-method:x-amazon-apigateway-integration:
httpMethod: ANYtype: aws_proxyuri:
Fn::Sub: arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${GetHtmlFunction.Arn}/invocationsresponses: {}
swagger: '2.0'
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CloudFormation templateAWSTemplateFormatVersion: '2010-09-09'Resources:
GetHtmlFunctionGetHtmlPermissionProd:Type: AWS::Lambda::PermissionProperties:Action: lambda:invokeFunctionPrincipal: apigateway.amazonaws.comFunctionName:
Ref: GetHtmlFunctionSourceArn:
Fn::Sub: arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${ServerlessRestApi}/Prod/ANY/*ServerlessRestApiProdStage:
Type: AWS::ApiGateway::StageProperties:DeploymentId:
Ref: ServerlessRestApiDeploymentRestApiId:
Ref: ServerlessRestApiStageName: Prod
ListTable:Type: AWS::DynamoDB::TableProperties:ProvisionedThroughput:
WriteCapacityUnits: 5ReadCapacityUnits: 5
AttributeDefinitions:- AttributeName: id
AttributeType: SKeySchema:- KeyType: HASH
AttributeName: idGetHtmlFunction:
Type: AWS::Lambda::FunctionProperties:Handler: index.gethtmlCode:
S3Bucket: flourish-demo-bucketS3Key: todo_list.zip
Role:Fn::GetAtt:- GetHtmlFunctionRole- Arn
Runtime: nodejs4.3GetHtmlFunctionRole:
Type: AWS::IAM::RoleProperties:ManagedPolicyArns:- arn:aws:iam::aws:policy/AmazonDynamoDBReadOnlyAccess
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRoleAssumeRolePolicyDocument:
Version: '2012-10-17'Statement:- Action:
- sts:AssumeRoleEffect: AllowPrincipal:
Service:- lambda.amazonaws.com
ServerlessRestApiDeployment:Type: AWS::ApiGateway::DeploymentProperties:RestApiId:
Ref: ServerlessRestApiDescription: 'RestApi deployment id: 127e3fb91142ab1ddc5f5446adb094442581a90d'StageName: Stage
GetHtmlFunctionGetHtmlPermissionTest:Type: AWS::Lambda::PermissionProperties:Action: lambda:invokeFunctionPrincipal: apigateway.amazonaws.comFunctionName:
Ref: GetHtmlFunctionSourceArn:
Fn::Sub: arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${ServerlessRestApi}/*/ANY/*ServerlessRestApi:
Type: AWS::ApiGateway::RestApiProperties:Body:
info:version: '1.0'title:
Ref: AWS::StackNamepaths:
"/{proxy+}":x-amazon-apigateway-any-method:x-amazon-apigateway-integration:
httpMethod: ANYtype: aws_proxyuri:
Fn::Sub: arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${GetHtmlFunction.Arn}/invocationsresponses: {}
swagger: '2.0'
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Serverless Application Model (SAM)
CloudFormation extension optimized for serverless
New serverless resource types: functions, APIs, and tables
Supports anything CloudFormation supports
Open specification (Apache 2.0)github.com/awslabsserverless-application-model
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SAM template
AWSTemplateFormatVersion: "2010-09-09"Transform: AWS::Serverless-2016-10-31
Resources: GetHtmlFunction:Type: AWS::Serverless::FunctionProperties:
CodeUri: s3://sam-demo-bucket/todo_list.zipHandler: index.gethtmlRuntime: nodejs4.3Policies: AmazonDynamoDBReadOnlyAccessEvents:GetHtml:
Type: ApiProperties:
Path: /{proxy+}Method: ANY
ListTable:Type: AWS::Serverless::SimpleTable
Tells CloudFormation this is a SAM template it needs to “transform”
Creates a Lambda function with the referenced managed IAM policy, runtime, code at the referenced zip location, and handler as defined. Also creates an API Gateway and takes care of all mapping/permissions necessary
Creates a DynamoDB table with 5 Read & Write units
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Meet SAM CLI – Local API Gateway/Lambda
CLI tool for local testing of serverless apps
Works with Lambda functions and “proxy-style” APIs
Response object and function logs available on your local machine
Supports all native runtimes
github.com/awslabs/aws-sam-cli
R u n L o c a l l y
D e b u g L o c a l l y
Nuevos Modelos de Negocio apalancados en la tecnología. Gestor de servicios energéticos integral
Generación distribuida en BTClientes finales o
pequeñas plantas de energía limpia
P2PEconomía colaborativa también en el
sector eléctrico
AlmacenamientoLa energía almacenada se convierte en energía
eléctrica cuando sea necesario
Coche eléctricoNo solo un sistema de movilidad alternativo, sino sistemas de
almacenamiento móvil que pueden interactuar con la red
Edificios automatizadosSistemas energéticamente eficientes que actúan
directamente en el edificio y brindan flexibilidad a la red
Virtual energy plantsTecnología de generación
distribuida con un potencial de crecimiento significativo
Smart homesHogares equipados con
dispositivos electrónicos (por ejemplo, iluminación, calefacción) que se pueden controlar de forma
remota
Relaciones con clienteLos minoristas totalmente digitales ofrecen una experiencia
completamente nueva a sus clientes
BlockchainUso potencial de la tecnología en toda la cadena
de valor
Porque un DataLake……..y porque Serverless
Reducción del time to
market no servidores que
gestionar
Escala bajo demanda 24 horas pero no se paga
cuando no esta funcionandoDisponibilidad y
tolerancia al fallo
Menor Equipo de operación
Flujo de datos analitico……..entran datos y sales respuestas que se convierten en datos nuevamente
Ingesta
Orquestar y Tranformar
DATOS
RESPUESTAS
Almacena Analiza y Procesa
VisualizarConsumir
SERVERLESS
Flujo de datos analítico Ingesta y Almacena…….. como entran los datos y se persisten en el lake
SchemaDefinido
por el usuario
Organización del Lake
Nuestro S3 como “source of truth”
• Landing: Donde llegan la información en crudo• Trasnformed : Donde se almacena transformada• Enriched Donde se enriquece la información tras procesos
de transformación
Landing y transforma están gobernado por el catalogo de Glue
Notificación
• Eventos de cargas Ok o no OK
Tópico denotificaciones
Flujo de datos analítico Analiza y Procesa……..como se procesan , se analizan y enriquecen
Amazon EC2
Amazon
EMR
Op
cio
ne
s d
ep
en
die
nd
o
de
la n
ec
es
ida
d
Analizar
• Athena con objetivo exploratorios y autoservicio• Redshift para nuestro procesos y reporting formal y
corporativo
Orquestación y transformación
• Step funciones como orquestador de la transformación.
• Varias posibilidades para realizar la transformación
• Lambdas, procesos simples
• EC2 , con procesos en Java y Python, lo mas
habitual.
• EMR para grandes volúmenes de información
• Glue (a futuro)SERVERLESS
Viesgo identifica tres niveles de usuarios informacionales con diferentes capacidades y diferentes necesidades que requerirán modelos de autoservicio y gobierno diferente.
USUARIOS DE NEGOCIO
POWER USERS
DATA SCIENTIST
Su necesidad está basada en el consumo de información directo a través de herramientas de visualización (Qisght & QlikView & Qlik Sense & Power Bi) .
Evolución deseada - Se pretende incrementar el autoconsumo de estos usuarios y dotarlos de la capacidad de generar informes AD-HOC sin necesidad de colaboración
de las áreas de IT.
Son usuarios con conocimientos técnicos de gestión de información (SQL), tienen la necesidad generar nuevos KPIs y de hacer segmentaciones avanzadas. Sus análisis buscan dar explicación a la realidad de la empresa.
Evolución deseada - Se pretende dotar a estos usuarios de cierta libertad para la realización de análisis ad-hoc con acceso directo a un SET de información mas amplio que los usuarios de negocio y con un espacio de desarrollo propio para realizar análisis . Además deben disponer de un procedimiento estándar para industrializar e incorporar nuevos datos al catalogo de datos oficial.
Son los mas avanzado tecnológicamente y los que realizan os análisis mas sofisticados, suelen realizar análisis de tipo predictivo o segmentación en base nuevas
dimensiones.
Evolución deseada - Se pretende dotar a estos usuarios de capacidades tecnológicas de Advanced Analytics y espacios de desarrollo para realizar sus análisis. Tienen
acceso a un conjunto de información mas amplio que los Power Users y pueden incluir datos externos en sus análisis. El resultado de sus análisis puede ser
industrializado.
Flujo de datos analítico Visualiza y consume……..que tipos usuarios no encontramos y que nos demandan
Flujo de datos analítico: Visualiza y consume……..múltiples tipos de usuarios , múltiples tipos de herramientas, múltiples necesidades
USUARIOS DE NEGOCIO
POWER USERS
DATA SCIENTIST
+
-
Gold y Silver – Gobierno del dato……..como permitimos no comprometer la agilidad con un proceso gobierno del datos riguroso y formal
Nuestro S3 como “source of truth”
Silver
Lugar especifico para proyectos o usuarios donde se almacena información especifica del proyecto o del usuario. Son sanbboxesdonde se puede tener información no gobernada por la organización.
Golden
Lugar donde se establecen los maestros corporativos y donde se ha definido una política de gobierno del dato. Hay un responsable , se tiene definido su linaje..etc etc.
Monitorizar este proceso……..como monitorizar
SERVERLESS
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Additional Resources - Whitepaper
Serverless Architectures with AWS Lambda
November 2017
bit.ly/ServerlessLambda
Optimizing Enterprise Economics with Serverless
Architectures
October 2017
bit.ly/ServerlessEconomics
Serverless Applications Lens
November 2017
bit.ly/ServerlessLens
Serverless Streaming Architectures and Best
Practices
June 2018
bit.ly/StreamingServerless
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
@awscloud_es
Página de eventos y webinars AWS Iberia: https://aws.amazon.com/es/about-aws/events/eventos-es