the internet of everywhere—how ibm the weather company scales
TRANSCRIPT
![Page 1: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/1.jpg)
![Page 2: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/2.jpg)
![Page 3: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/3.jpg)
![Page 4: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/4.jpg)
EverywhereDefined• 26Bforecasts/dayor250,000/second– vs 3.5BGooglequeriesdaily
• 2.2billionuniquelocations• 200kpersonalweatherstations• 200Mactivemobileusers• Petabytesofdatagenerateddaily
![Page 5: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/5.jpg)
OurBrands
![Page 6: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/6.jpg)
Over
30BillionServed
![Page 7: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/7.jpg)
FlightRouting
![Page 8: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/8.jpg)
EnergyTrading
![Page 9: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/9.jpg)
Insurance
![Page 10: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/10.jpg)
WeatherAlerting
![Page 11: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/11.jpg)
DecisionsatScale
101001110100101
101001110100101010100101011001101010101011100000011010110010
![Page 12: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/12.jpg)
WhoAreYou?
RDBMS?
![Page 13: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/13.jpg)
WhoAreYou?
?
![Page 14: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/14.jpg)
SocialWeather
![Page 15: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/15.jpg)
SocialWeather
RDBMS
![Page 16: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/16.jpg)
SocialWeather
RDBMS SELECTcount(*)FROMwx_reportsGROUPBYtime/300000*300000
![Page 17: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/17.jpg)
SocialWeather
Live Reporting
ETL
![Page 18: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/18.jpg)
SocialWeather
Live
Reporting
SqoopM/R
![Page 19: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/19.jpg)
ScalingwithSpark
Live
Reporting
![Page 20: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/20.jpg)
EasingtheTransition
101001110100101
101001110100101010100101011001101010101011100000011010110010
![Page 21: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/21.jpg)
EasingtheTransition
101001110100101
101001110100101010100101011001101010101011100000011010110010
![Page 22: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/22.jpg)
EasingtheTransition
101001110100101
101001110100101010100101011001101010101011100000011010110010
101001110100101010100101011001101010101011100000011010110010
10100,11101,0010101010,01010,1100110101,01010,1110000001,10101,...
![Page 23: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/23.jpg)
EasingtheTransition
101001110100101
101001110100101010100101011001101010101011100000011010110010
101001110100101010100101011001101010101011100000011010110010
10100,11101,0010101010,01010,1100110101,01010,1110000001,10101,...
![Page 24: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/24.jpg)
ScalingwithSpark
Live
![Page 25: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/25.jpg)
ScalingwithSpark
Live
Reporting
![Page 26: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/26.jpg)
BatchAggregationval wx_reports = // load data from database
val sql = new org.apache.spark.sql.SQLContext(sc)import sql.implicits._
wx_reports.toDF.registerTempTable("wx_reports")
val counts = sql("select count(*) from wx_reports group by timestamp / 300000 * 300000")
![Page 27: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/27.jpg)
StreamingAggregationval wx_reports = // load from streaming source
wx_reports.foreachRDD { rdd =>val sql = SQLContext.getOrCreate(rdd.sparkContext)import sql.implicits._rdd.toDF.registerTempTable("wx_reports")val count = sql("select count(*) from wx_reports")
}
![Page 28: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/28.jpg)
DataScienceRoles
Data Scientist Data Engineer
![Page 29: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/29.jpg)
DataScienceRoles
Data Scientist Data Engineer
Machine learningexpert
![Page 30: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/30.jpg)
DataScienceRoles
Data Scientist Data Engineer
Machine learningexpert Scalablealgorithms expert
![Page 31: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/31.jpg)
DataScienceRoles
Data Scientist Data Engineer
Buildspipelines thatworkonherlaptop
![Page 32: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/32.jpg)
DataScienceRoles
Data Scientist Data Engineer
Rewritesherpipelinestoscalebetter
![Page 33: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/33.jpg)
CollaborativeDataScience
![Page 34: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/34.jpg)
TheAnalyticsOS
Notebooks StreamAnalytics
BatchAnalytics
![Page 35: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/35.jpg)
But…
![Page 36: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/36.jpg)
TheRealWorld(EnterpriseVersion)
![Page 37: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/37.jpg)
TheRealWorld(StartupVersion)
Application MySQL
![Page 38: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/38.jpg)
Step1:PickaProblemtoSolve
![Page 39: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/39.jpg)
Step2:BuildaDataLake
![Page 40: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/40.jpg)
Step3:SetupSpark
• Directdownload• Hadoop distribution(Hortonworks,Cloudera,etc)
• Managedservice(ElasticMapReduce,Databricks,BlueMix,etc)
![Page 41: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/41.jpg)
Step4:StartCollectingData• Options:– Sqoop tomoveRDBMStables– Flume/FluentD tomovelogs– ImportfromSpark-supporteddatasources– UsingSparkStreamingattachedtoaqueue– …
![Page 42: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/42.jpg)
Step5:UseaNotebook
![Page 43: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/43.jpg)
FinalThoughts
![Page 44: The Internet of Everywhere—How IBM The Weather Company Scales](https://reader031.vdocuments.site/reader031/viewer/2022030215/588a415a1a28abc6168b74f5/html5/thumbnails/44.jpg)
ThankYou!
Robbie Strickland@rs_atl
(we’rehiring!)