bigdata analytics and our iot gateway
TRANSCRIPT
2 May 2023 1
Aemro Amare, Freek van Gool
Bigdata Analytics and our IoT Gateway :Toon
Agenda• A little bit about us• Who is Quby & what is Toon?• Our big data journey• Data collection & services• Architecture & technologies• Demo• Questions
@javafreekNL @twoxey
A little bit about us
Freek Aemro
@javafreekNL @twoxey
First attemptHome control box
@javafreekNL @twoxey
PivotEnergy stick
What is Toon?
Meter Adapter
Gas Sensor
Electricity Sensor
Z-Wave
Phillips Hue
WiFi Router
Z-wave
Smart plugand smoke detectors
Central heating boiler
Boiler adaptor
Toon®
Toon connections - Data sources
The service center
Service Center
Toon Displays
Mobile devices
ProductApplication
s
Back OfficeApplication
s
Mobile Backend
Internet
Internet
VPNAccess
Data collector
Firstpartnership
Exponential growth
2013 30.0002014 90.0002015 190.0002016 300.0002017 (1.000.000)
2013 2014 2015 2016 2017
2 May 2023 12
Our journey towards bigdata 2014
• 30K customers • 40 types of
sensor data from 3K customers
• up to 1 minutes resolution
2012
• 4K customers • Aggregated
Energy data collected(DAY,WEEK,MONTH)
• Benchmarking functionality for customers
2013
• 90K customers • Data collection
from 30K customers
• Event and Measurement data
2 May 2023 13
Steps moving towards bigdata(cont’d)2017
• 300K customers • Near Real Time
aggregation with segmentation
• 400 signal/5 minutes
• Sharing Measurement data with tenants
2015
• 190K customers • Apache storm for
Near Real Time Energy consumption (3K customers)
• Data analysis using Cloudera stack
• Up to 10 seconds resolution
2016
• 330K customers• Collect data
from 230K customers anonymously
• Data as a service
• Analytics platform as a service
Toon®
Types of data
More than 300 types of sensor and user interaction data. If you want to know more please contact us : [email protected]
Energy CH-boiler (OT) Thermostat
Gas consumption
Electricity consumption
Water temperature
Service messages
Burner information
Room temperatureThermostat
programManual settings
Plug-dataElectricity
consumptionOn/off
Water pressure
Energy production
Energy feed-in
Boiler status
Touch eventsThermostat
Mobile
1 2 3 4 5
Thermostat setting
Hue data
2 May 2023 15
Anonymous Data Collection Service Center Customer
Always anonymous data collected
Customer may opt in for services and
send keys+ =
>Customer may opt out and
change keysData can’t be de-anonymized
again
2 May 2023 16
Sample Services with Toon’s Data?1. Benchmarking with daily energy meter data
2 May 2023 17
Sample Services with Toon’s Data? ( cont’d)2. Nearly Real Time Forecasting : Energy data Aggregation
2 May 2023 18
Sample Services with Toon’s Data? ( cont’d)3. Home Appliances Energy Consumption Detection: Energy Data disaggregation
2 May 2023 19
Sample Services with Toon’s Data ?(cont’d) 4. Boiler Maintenance Prediction
(KetelIQ)
2 May 2023 20
Sample Services with Toon’s Data? ( cont’d)5. Customer’s Behavioral studies
Click data Thermostat programs and co2 emission
21
What are we collecting? 24K measurements / day / customer 600KB (zipped) 230K customers
22
Arch. Principle : lambda Arch.
2 May 2023 23
Messaging
Data warehouse
Tools and Technologies
Analytic Clusters
Tools and Frameworks
Containers & micro services
Mon
itorin
g an
d al
ert
24
Data flow (partial view)
2 May 2023 27
Things to mention • ssl mutual authentication• raw-data stored partitioned in better way• Customers can control access to their own
data
2 May 2023 29
What are we doing with all that data?• Firehose – data as a service for B2B (and
researchers)• Analytic platform as a service• New data driven services for Toon users• Improve existing services• Marketing strategy (understand your customers)
2 May 2023 30
2 May 2023 31