the future is an event max
TRANSCRIPT
The future is an eventADDRESSING MODERN DATA CHALLENGES
MAX COTTICA
Computer Shop Clerk
(5 years)
IT Development Manager
(10 years)
SQL DBA/Developer
(5 years)
Data Warehouse Developer/Junior Manager
(6 years)
Data Warehouse Manager
(5 years)
Global Data Integration Senior Manager
(4 years)
Head of Data Science
and Solutions
(1 year)
About me
1980s 1990s 2000s 2010s 2017
1980
‘80s
‘90s
‘00s
‘10s“My goal is to use predictive analytics in conjunction with scenario based algorithms to produce prescriptive analytics and actionable events.”
“My goal was to use aggregated reports to produce KPI reports andto use predictive models to estimate future growth.”
“My goal was to aggregate and integrate data and offer analytics
based on different segmentations like time or geography .”
“My goal was to produce ad-hoc reports for a large variety ofdepartments using a centralized media for distribution.”
“My goal was to produce reports.”
TIER 1
TIER 0
TIER 2
TIER 3
Descriptive Analytics
Diagnostic Analytics
Predictive Analytics
Prescriptive Analytics
What happened
Roll-ups and how, when and where
Identify problems and fire alerts
Why and forecasting
What will happen if…Next best action
Max vs. Analytics in the years:
Modern data
Getting the flow right – 1997 to 2017
Capture Store Transform Publish
Modern Data Does not like
batch
Batch it’s all it knows
Getting the flow right - Today
Maybe not a lake but just a river Real customer value is in speed streams
Decisions in real time
Aligning to IoT speed
Challenges and trends:
SVOT vs MVOT
Switching from Lambda to Kappa architecture
Kafka for the win
Speed modelling
OTS model for analytics
Replay and Search Analytics
Real time application
SVOT vs MVOT
Truth
Finance
HR
IT
Sales
IT
Finance
Sales
HR
Truth Truth
Truth Truth
Truth
The future is an event - Lambda
Kafka for the winApache Kafka is an open-source stream processingplatform developed by the Apache Software Foundationwritten in Scala and Java. Originally developed by LinkedIn as a messaging queue system.
The future is an event - Kappa
Speed modelling
Traditional dimensional model
Surrogate Key DoorID DoorLocation DoorStatus Timestamp IsLatestRecord DWHField1 DWHFiled2 DWHField3
1 1 North Closed 12:00:01 N 22/03/1965 123456 userA
2 2 South Open 12:00:02 N 22/03/1965 123456 userA
3 1 North Open 12:00:03 N 22/03/1965 123456 userA
4 1 North Closed 12:00:04 N 22/03/1965 123456 userA
5 2 South Closed 12:00:05 N 22/03/1965 123456 userA
6 2 South Open 12:00:06 N 22/03/1965 123456 userA
7 2 South Closed 12:00:07 N 22/03/1965 123456 userA
8 1 North Open 12:00:08 N 22/03/1965 123456 userA
9 1 North Closed 12:00:09 Y 22/03/1965 123456 userA
10 2 South Open 12:00:11 Y 22/03/1965 123456 userA
Data streams
Micro-dimensions
Data streams
Surrogate Key DoorID DoorLocation DWHField1 DWHFiled2 DWHField3
1 1 North 22/03/1965 123456 userA
2 2 South 22/03/1965 123456 userA
3 1 North 22/03/1965 123456 userA
4 1 North 22/03/1965 123456 userA
5 2 South 22/03/1965 123456 userA
6 2 South 22/03/1965 123456 userA
7 2 South 22/03/1965 123456 userA
8 1 North 22/03/1965 123456 userA
9 1 North 22/03/1965 123456 userA
10 2 South 22/03/1965 123456 userA
DoorID DoorStatus Timestamp
1 Closed 12:00:01
1 Open 12:00:03
1 Closed 12:00:04
1 Open 12:00:08
1 Closed 12:00:09
DoorID DoorStatus Timestamp
2 Open 12:00:02
2 Closed 12:00:05
2 Open 12:00:06
2 Closed 12:00:07
2 Open 12:00:11
Entities or partitions
OTS model for green field analytics
Enterprise
Business driven
Design
12 to 24 months
E2E solution
Small scale
Architecture
Subject driven
6 to 12 months
Qlik / Tableau
Exploration
Discovery
Insight
3 to 6 months
SQL, R, Python
Operational
Tactical
Strategic
CLOUD
Replay, Untapped and Search Analytics Capability to replay past transactions in a segmentations and predictive context
Fine tune algorithms for prescriptive analytics
Anomaly detection techniques to align past to present
Video, Audio and other media
We now have the technology to leverage this data
Needs specific use cases
Using NLP for Google like searches
Results are of various nature from a list to actual insights to graphs
Elastic/Solr will play a big role
CV
OnlineProfiles
Cover Letter
ML
NLP
DL
Job Specs
AIMatches
WEB UI
BOT
Mobile
EmployersEmployees
MicroServices
APIs
BUILDING A REAL TIME SOLUTION (TO DISRUPT THE RECRUITMENT INDUSTRY)
“Oh yes, my company is doing Agile…”
“Does your CEO have a backlog ?”
“Of course not !”
“I have got news for you, you are not doing Agile…”