handle your lambdas - from event-based processing to continuous integration / munich aws user group...
TRANSCRIPT
Handle your Lambdas:From event-based processing to Continuous
Integration
Sergii Khomenko, Data [email protected], @lc0d3r
Munich AWS User Group - Mar 17 2016
Sergii Khomenko
2
Data scientist at one of the biggest fashion communities, Stylight.
Data analysis and visualisation hobbyist, working on problems not only in working time but in free time for fun and personal data visualisations.
Speaker at Berlin Buzzwords 2014, ApacheCon Europe 2014, Puppet Camp London 2015, Berlin Buzzwords 2015 , Tableau Conference on Tour 2015, Budapest BI Forum 2015, Crunchsconf 2015, FOSDEM 2016, PyData Amsterdam 2016
Profitable LeadsStylight provides its partners with high-quality leads enabling partner shops to leverage Stylight as a ROI positive traffic channel.
InspirationStylight offers
shoppable inspiration that
makes it easy to know what to
buy and how to style it.
Branding & ReachStylight offers a unique opportunity for brands to reach an audience that is actively looking for style online.
ShoppingStylight helps users search
and shop fashion and lifestyle products smarter across
hundreds of shops.
3
Stylight – Make Style HappenCore Target Group
Stylight help aspiring women between 18 and 35 to evolve their style through shoppable inspiration.
Stylight – acting on a global scale
Experienced & Ambitious Team
Innovative cross-functional organisation with flat hierarchy builds a unique team spirit.
• +200 employees• 40 Engineers/PhDs• 28 years average age
• 63% female• 23 nationalities• 0 suits
5
6
A Brief History of Computing
7
8
9Charles Babbage's machine - 1832
101940 – 1956: First Generation – Vacuum Tubes
111956 – 1963: Second Generation – Transistors
121964 – 1971: Third Generation – Integrated Circuits
131972- … : Fourth Generation – Microprocessors
14
15
Immutable infrastructure
16
Infrastructure as Code
17
N e e d t o u p g r a d e ? N o p r o b l e m . B u i l d a n e w , u p g r a d e d s y s t e m a n d t h r o w t h e o l d o n e a w a y . N e w a p p
r e v i s i o n ? S a m e t h i n g . B u i l d a s e r v e r ( o r i m a g e ) w i t h a n e w
r e v i s i o n a n d t h r o w a w a y t h e o l d o n e s .
18
19
20
21
CloudFormation
22
Terraform
23
24
Terraform
Kubernetes and Docker {Swarm, Compose}
Serverless architecture
25
26
27
28
29
30
31
Installation
32
export GOPATH=$HOME/go
export PATH=$PATH:$GOPATH/bin
go get github.com/apex/apex/cmd/apex
export AWS_REGION=eu-west-1
33
34
35
36
main.go
37
main.py
Python dependencies
38
pip install requests -t functions/dependency
39
apex invoke
Outcome
40
• all Lambdas in one place with version control • integration tests with real events • proper CI/CD setup
41
CircleCI deployments
42
CircleCI deployments
43
CircleCI deployments
44
Cloud functions
Use-caseCloudWatch alerting
45
46
47
48
Use-case of outlier detection
49
50
51
custom unificationpipeline
DepartmentsBusiness
Intelligence
internal processes variety of event types and structures
52
The Future is Now
53
54
AWS Lambda in FRA
55
AWS Lambda in FRA
Related links
57
1. The Babbage Engine
2. The Five Generations of Computers
3. IBM System 360
4. https://github.com/cloudtools/troposphere
5. https://www.terraform.io/
6. (Docker Compose + Docker Swarm) or Kubernetes
7. Google Cloud Functions
8. https://github.com/apex/apex
9. https://github.com/apex/apex/tree/master/_examples/python/functions/dependency
10.Streaming Data Processing with Amazon Kinesis and AWS Lambda
58
USE CODE - AWS20