big data : the missing puzzle of mobile computing
TRANSCRIPT
PowerPoint Presentation
From WSN to Mobile Computing, Big Data : The Missing PuzzleJazz WangYao-Tsung [email protected]
WHO AM I ? JAZZ
/
http://trac.nchc.org.tw/cloud
FOSSDebian/UbutnuAccess GridMotion/VLCRed5Debian RouterDRBL/ClonezillaHadoop
DRBL/ClonezillaPartclone/TuxbootHadoop Ecosystem
TRTC WSU/Haduzilla / Hadop4Win / Ezilla
DECLARATION
I'm NOT Export of Mobile Computing.My current research topics are:Cloud Computing and Big Data.I worked on few WSN projects years ago.In this talk, I'm trying to sharethe magic journey of sensor data to mobile computing.
Agenda From WSN to IoT? Cloud and Big Data Smart City AppHow to do in future What is Cloud ? Conclusion
This is a story about Data Flow
SensorNetwork
SmartGrid
Magic Journey of Data
Internet of Things
Open Data
Big Data
Mobile Computing
Cloud Computing
HISTORY
(Wireless Sensor Network)
SensorNetwork
Camera
InfraredLight
Photo-meter
Thermo-meter
Hygro-meter
WaterGauge
Speaker
Micro-phone
Video
Audio
Data
(Ecology Grid)
http://ecogrid.nchc.org.tw
1st camera
2nd camera
3rd camera
Optical connector
Field server
Taipower Admin office
KenTing Ecosite Geography Topology
EcoGrid Website PC
HiNet ADSL
512K/512K x 2
KenTing Ecosite PC
MPEG-1
Disk
Disk
Tape
LAN
Current Usage:90G @ 2004/09/01
NCHC KM Storage
KenTing Dataflow and Storage
Underwater Camera
Site A
Site B & C
Site B & C
(Agriculture Grid)
http://sensor.nchc.org.tw/tounan
AP Client
DataLogger
Sensor
Field Server
Pigeon House
AccessPoint
VideoServer
Camera
Router
ADSLmoderm
AP Client
DataLogger
Sensor
AccessPoint
VideoServer
Camera
Router
Bridge
Bridge
TANET
Internet
Special Education School
NCHC
Electric Wire Post
Electric Wire Post
Passive Field Server
(TRTC Neihu WSU)
http://www.flickr.com/photos/rail02000/5292653579/lightbox/
Wireless Fast Roaming
AP #1
AP #2
WSU
TRENDS
Internet of Things
Attacks on Mobile and Embedded Systems: Current TrendsbyMocana
http://www.techbang.com/posts/1258-wifi-scales-through-the-internet-to-manage-your-weight
Smart Grid
NIST
The Trends of Cloud Computing
Google Trends
200720112012
http://www.google.com/trends
Open Data
http://data.gov.uk/
Open Data
http://www.mygonews.com/news/detail/news_id/101779
Ocean Database
http://oceandb.info
Agenda From WSN to IoT? What is Cloud ?
WHAT
What is Cloud Computing ?
http://www.youtube.com/watch?v=bJLSAcU6O3U http://www.youtube.com/watch?v=VIMtd3nfPqc 8
Paradigm Shift of Cloud Business Model !!
Office 2007 Google Docs / Office 365
Outlook Webmail Mail Web Apps Mail Mobile Apps
PC / Server Hosting / Colocation Amazon EC2 / S3
PC / Server NB / Tablet Pad / Mobile
National Definition of Cloud ComputingNIST
5 Characteristics
4 Deployment Models
3 Service Models
1. On-demand self-service.
2. Broad network access
3. Resource pooling
4. Rapid elasticity
5. Measured Service
Enterprise iskey market
Private Cloud
4 Deployment Models of Cloud Computing
Public Cloud
Target Market is S.M.B.
HybridCloud
Dynamic Resource Provisioningbetween public and private cloud
Community Cloud
Academia
3 Service Models of Cloud Computing
SaaSSoftware as a Service
PaaSPlatform as a Service
IaaSInfrastructure as a Service
2 perspectives : Services vs Technologies?Cloud computing hype spurs confusion, Gartner sayshttp://www.computerworld.com/s/article/print/9115904 (Cloud Computing)http://www.cc.ntu.edu.tw/chinese/epaper/0008/20090320_8008.htm
One key spirit of Cloud Computing!!
Anytime
Anywhere
With Any Devices
Accessing Services
Cloud Computing =~ Network Computing =~
Key spirit of Cloud ~!!Everything as a Service !!
WHO
Supply Chain of Cloud Industry !!
\\
Who are the Cloud Service Providers ?
Enterprise iskey market
Private Cloud
Public Cloud
Target Market is S.M.B.
HybridCloud
Community Cloud
Academia
WHEN
Source: http://www.cnet.co.uk/i/c/blg/cat/software/cloudcomputing/clouds1.jpg
200689
GoogleEric SchmidtSES'06
Cloud Computing
2006824
AmazonElastic Compute Cloud
The Wisdom of Clouds (Crowds)
Mobile Cloud Service
Share Service Software
Personal Software
Physical
Mobile Mail
Web Mail
Mailbox
Mobile TV
Web TVEx. Youtube
Setop Box
TV
M-Office
Google Docs
Office
Typer Writer
Flash Wengo
Skype
PBX
Telephone
Blog
BBS
Bullet Borad
Evolution of Cloud Services
WHY
Why are they named by SMART ?!
Smart Phone
Smart Grid
Smart Home
Smart Car
Smart City
Smart Meter
SMART ?
Wisdom
Knowledge
Data
Can Machine understand You?
http://www.ettoday.net/news/20120215/25085.htm
Data, Information, Knowledge, Wisdom
http://www.pursuantgroup.com/blog/tag/dikw-model/
Agenda From WSN to IoT? Cloud and Big Data What is Cloud ?
Key Driving Forces of Cloud Computing
Mobile Service
Cost Down
Data Explore
"It's the economy, stupid"James Carville42- 1992
"It's STILL the economy, stupid"- 2002
"It's the data, stupid"- 2007
Data Explosion!!2007
The Expanding Digital Universe, A Forecast of Worldwide Information Growth Through 2010,
March 2007, An IDC White Paper - sponsored by EMC
http://www.emc.com/collateral/analyst-reports/expanding-digital-idc-white-paper.pdf
2007IDC20102006
2006 161 EB
2010 988 EB ()
Extracting Value from Chaos,
June 2011, An IDC White Paper - sponsored by EMC
http://www.emc.com/collateral/about/news/idc-emc-digital-universe-2011-infographic.pdf
IDC
2006 161 EB
2007 281 EB2008 487 EB2009 800 EB (0.8 ZB)
2010 988 EB ()
2010 1200 EB (1.2 ZB)
2011 1773 EB ()
2011 1800 EB (1.8 ZB)
Data expanded 1.6x each year !!1.6
TBPB'Big Data' = few dozen TeraBytes to PetaBytes in single data set.
What is Big Data?!
100TB
100TB
100TB
http://en.wikipedia.org/wiki/Big_data
Gartner Big Data Model ?
Volume (amount of data)
Velocity (speed of data in/out)
Variety (data types, sources)
Batch ()
Realtime ()
TB
EB
Unstructured
Semi-structured
Structured
PB
[1] Laney, Douglas. "3D Data Management: Controlling Data Volume, Velocity and Variety" (6 February 2001)[2] Gartner Says Solving 'Big Data' Challenge Involves More Than Just Managing Volumes of Data, June 2011
12D of Information Management? 12
Source: Gartner (March 2011), 'Big Data' Is Only the Beginning of Extreme Information Management, 7 April 2011, http://www.gartner.com/id=1622715
Big Data
HOW
Devices share the wisdom of Cloud
New IT Architecture toward Cloud Computing !!
1. 6. (
2.
3.
4.
5.
Three Key Technologies !! vs.
IaaS
Infrastructure as a Service
PaaS
Platform as a Service
SaaS
Software as a Service
Virtualization
Big Data
Web 2.0
Mobile Service
Cost Down
Data Explore
Agenda From WSN to IoT? Cloud and Big Data Smart City AppWhat is Cloud ?
http://www.postscapes.com/anatomy-of-a-smart-city-full
http://www.postscapes.com/anatomy-of-a-smart-city-full
200850%
http://www.postscapes.com/anatomy-of-a-smart-city-full
http://www.postscapes.com/anatomy-of-a-smart-city-full
2011
http://www.postscapes.com/anatomy-of-a-smart-city-full
http://www.postscapes.com/anatomy-of-a-smart-city-full
http://www.postscapes.com/anatomy-of-a-smart-city-full
http://www.postscapes.com/anatomy-of-a-smart-city-full
6D of Smart Cities
SmartLivingSmartGovernanceSmartPeopleSmartEconomySmartEnvironmentSmartMobilitySustainablehttp://en.wikipedia.org/wiki/Smart_city
EducationTrafficHousingFoodEntertainmentClothing
Ubiquitous Computing
http://www.ipeen.com.tw/map/#loc=
http://www.cwb.gov.tw/township/
http://rent.591.com.tw/
http://1968.freeway.gov.tw/
http://www.datanami.com/datanami/2012-08-01/what_it_takes_to_deliver_real-time_traffic_info.html
http://hypercities.ats.ucla.edu/
http://earth.wra.gov.tw/water/index.html
Agenda From WSN to IoT? Cloud and Big Data Smart City AppHow to do in future What is Cloud ?
The SMAQ stack for big data
The SMAQ stack for big dataEdd Dumbill22 September 2010http://radar.oreilly.com/2010/09/the-smaq-stack-for-big-data.htmlhttp://smashingweb.ge6.org/wp-content/uploads/2011/10/apache-php-mysql-ubuntu.png
LAMP
SMAQStorage, MapReduce and Query
The SMAQ stack for big data
The SMAQ stack for big dataEdd Dumbill22 September 2010http://radar.oreilly.com/2010/09/the-smaq-stack-for-big-data.html
The SMAQ stack for big data
The SMAQ stack for big dataEdd Dumbill22 September 2010http://radar.oreilly.com/2010/09/the-smaq-stack-for-big-data.html
The SMAQ stack for big data
The SMAQ stack for big dataEdd Dumbill22 September 2010http://radar.oreilly.com/2010/09/the-smaq-stack-for-big-data.html
Three Core Technologies of Google ....
Google ....
Google shared their design of web-search engine
SOSP 2003 :
The Google File System
http://labs.google.com/papers/gfs.html
OSDI 2004 :
MapReduce : Simplifed Data Processing on Large Cluster
http://labs.google.com/papers/mapreduce.html
OSDI 2006 :
Bigtable: A Distributed Storage System for Structured Data
http://labs.google.com/papers/bigtable-osdi06.pdf
Open Source Mapping of Google Core Technologies
Google
Hadoop Distributed File System (HDFS)
Sector Distributed File System
Hadoop MapReduce API
Sphere MapReduce API, ...
HBase, Hypertable
Cassandra, ....
S = Storage
Google File System
To store petabytes of data
MapReduce
To parallel process data
Q = Query
BigTable
A huge key-value datastore
Hadoophttp://hadoop.apache.org
HadoopApache Top Level
Hadoop is Apache Top Level Project
Yahoo!
Major sponsor is Yahoo!
Doug CuttingGoogle Filesystem
Developed by Doug Cutting, Reference from Google Filesystem
JavaHDFSMapReduce API
Written by Java, it provides HDFS and MapReduce API
2006Yahoo
Used in Yahoo since year 2006
It had been deploy to 4000+ nodes in Yahoo
Petabyte
Design to process dataset in Petabyte
FacebookLast.fmJoost are also powered by Hadoop
Hadoop
map reduce
HDFS
Map
Reduce
Hadoop
Sector / Spherehttp://sector.sourceforge.net/
Developed by National Center for Data Mining, USA
C/C++ Hadoop
Written by C/C++, so performance is better than Hadoop
Google File SystemMapReduce
Provide file system similar to Google File System and MapReduce API
UDT
Based on UDT which enhance the network performance
Open Cloud TestbedMalStone
Open Cloud Consortium provide Open Cloud Testbed and develop MalStone toolkit for benchmark
Why we choice Hadoop? Good Ecosystem!
http://rationalintelligence.com/wp_log/?p=104
Agenda From WSN to IoT? Cloud and Big Data Smart City AppHow to do in future What is Cloud ? Conclusion
IoT is the source of Big DataCloud Computing is the StoreBig Data is the key to Mobile ComputingSMAQ is the way to achieve Wisdom
SMAQ
Questions?Slides - http://trac.nchc.org.tw/cloudJazz WangYao-Tsung [email protected]
Column 1
User50
Promoter35
Developer15