state of the data universe
DESCRIPTION
Big Data: State of the Data Universe - What the best are doing and why your data is your future Business is being driven by data at an ever increasing rate. If you aren't gathering and making sense of more data than your competition then you are behind. Smart businesses are developing mechanisms to generate more detailed data. Customers are starting to expect more tailored products, increased support, and an overall better experience. How does one keep up? Kenny Gorman will speak to the state of the data universe. What are businesses doing? What challenges are they facing and how are they tackling these challenges? What technologies do they use? What technologies are still hype? Attendees will leave armed with the ability to tell fact from fiction, and understand how companies that are successfully using data to gain advantage are doing it.TRANSCRIPT
![Page 1: State of the Data Universe](https://reader033.vdocuments.site/reader033/viewer/2022060201/5599e5071a28ab447a8b461f/html5/thumbnails/1.jpg)
State of the Data UniverseWhat the best are doing and why your data is your
future
Kenny GormanChief Technologist; Data at Rackspace
Co-Founder, ObjectRocket
@rackspace @kennygorman
![Page 2: State of the Data Universe](https://reader033.vdocuments.site/reader033/viewer/2022060201/5599e5071a28ab447a8b461f/html5/thumbnails/2.jpg)
Big DataWTF?
![Page 3: State of the Data Universe](https://reader033.vdocuments.site/reader033/viewer/2022060201/5599e5071a28ab447a8b461f/html5/thumbnails/3.jpg)
Big DataIt’s just data. It’s really important, and there
is probably lots of it, so maybe we call it:
‘really important data’
![Page 4: State of the Data Universe](https://reader033.vdocuments.site/reader033/viewer/2022060201/5599e5071a28ab447a8b461f/html5/thumbnails/4.jpg)
Focus on using data to be competitive
Forget how big it is or isn’t
![Page 5: State of the Data Universe](https://reader033.vdocuments.site/reader033/viewer/2022060201/5599e5071a28ab447a8b461f/html5/thumbnails/5.jpg)
Data is your competitive advantage
We're entering a new world in which data may be more important than software.
- Tim O'Reilly
![Page 6: State of the Data Universe](https://reader033.vdocuments.site/reader033/viewer/2022060201/5599e5071a28ab447a8b461f/html5/thumbnails/6.jpg)
The changing landscape
Today
DSS
BI
Big Data ????Analytics
????NoSQL
RDBMS NewSQL
![Page 7: State of the Data Universe](https://reader033.vdocuments.site/reader033/viewer/2022060201/5599e5071a28ab447a8b461f/html5/thumbnails/7.jpg)
![Page 8: State of the Data Universe](https://reader033.vdocuments.site/reader033/viewer/2022060201/5599e5071a28ab447a8b461f/html5/thumbnails/8.jpg)
● Operational Stores(PostgreSQL,MongoDB)
● Big Data (Hadoop)
● Streaming (Kafka)
Level Set
![Page 9: State of the Data Universe](https://reader033.vdocuments.site/reader033/viewer/2022060201/5599e5071a28ab447a8b461f/html5/thumbnails/9.jpg)
Value of the data?
1. Business Intelligence
2. Product Improvement
3. Operationalization
System Types
![Page 10: State of the Data Universe](https://reader033.vdocuments.site/reader033/viewer/2022060201/5599e5071a28ab447a8b461f/html5/thumbnails/10.jpg)
● Beyond traditional BI
● Capture important data, instrument everything important
● Pick your systems wisely, the right tool for the job
● Build automation & product based on data.
● Augment and extend existing systems.
Succeeding
![Page 11: State of the Data Universe](https://reader033.vdocuments.site/reader033/viewer/2022060201/5599e5071a28ab447a8b461f/html5/thumbnails/11.jpg)
Examples
![Page 12: State of the Data Universe](https://reader033.vdocuments.site/reader033/viewer/2022060201/5599e5071a28ab447a8b461f/html5/thumbnails/12.jpg)
![Page 13: State of the Data Universe](https://reader033.vdocuments.site/reader033/viewer/2022060201/5599e5071a28ab447a8b461f/html5/thumbnails/13.jpg)
Rush Hour Rewards is a service that helps you earn money back from your energy company by using less energy when everyone else is using more.
Using data to be more efficient with finite product availability, optimizing overall cost of goods, and thus increasing profitability
Nest
![Page 15: State of the Data Universe](https://reader033.vdocuments.site/reader033/viewer/2022060201/5599e5071a28ab447a8b461f/html5/thumbnails/15.jpg)
How it works(I think)
Thermostat -> API -> DB <- API <- Austin Energy
Austin Energy -> API -> Thermostat
![Page 16: State of the Data Universe](https://reader033.vdocuments.site/reader033/viewer/2022060201/5599e5071a28ab447a8b461f/html5/thumbnails/16.jpg)
{ "hvac_ac_state": "False",
"heat_pump_comp_threshold": "-31.5", "fan_cooling_enabled": "True", "leaf_away_high": "26.111",
"compressor_lockout_timeout": "0", "gear_threshold_low": "0.0",
"lower_safety_temp_enabled": "True", "postal_code": "78730", "learning_mode": "True", "country_code": "US",
"heat_x3_source": "lp", "fan_timer_duration": "900",
"backplate_serial_number": "02BA03AC031406Y4", "hvac_wires": "Heat,Cool,Fan,Rc,Star",
"humidifier_type": "unknown", "target_change_pending": "False",
"sunlight_correction_active": "False", "logging_priority": "informational",
"temperature_lock": "False", "dual_fuel_breakpoint_override": "none",
"has_x3_heat": "False", "alt_heat_x2_delivery": "forced-air",
"maint_band_lower": "0.56", "auto_away_learning": "ready",
"device_locale": "en_US", "learning_time": "1794",
"timestamp": "1405831101750", "time_to_target_training": "ready",
"has_fan": "True", "auto_dehum_state": "False",
"star_type": "unknown", "backplate_model": "Backplate-2.8",
"heat_x2_source": "lp", "aux_heat_source": "electric", "filter_changed_date": "0",
"equipment_type": "electric", "dehumidifier_orientation_selected": "unknown",
"forced_air": "True", "name": "Master",
"can_cool": "True", "aux_lockout_leaf": "10.0",
"filter_reminder_level": "0", "humidifier_state": "False",
"error_code": "", "leaf_threshold_cool": "23.888",
"has_x2_cool": "False", "hvac_pins": "W1,Y1,Rc,G,Star",
"creation_time": "1400446808068", "heat_pump_comp_threshold_enabled": "False",
"pin_star_description": "none", "compressor_lockout_enabled": "False", "learning_days_completed_cool": "60",
"away_temperature_low_enabled": "True", "note_codes": "[]",
"leaf_threshold_heat": "1000.0", "y2_type": "unknown",
"cooling_source": "electric", "leaf": "False",
"auto_dehum_enabled": "False", "alt_heat_x2_source": "gas",
"hvac_aux_heater_state": "False", "learning_days_completed_heat": "2",
"has_humidifier": "False", "gear_threshold_high": "0.0",
"current_schedule_mode": "COOL", "target_temperature_type": "cool",
"backplate_bsl_info": "BSL", "version": "-978861903",
"fan_cooling_readiness": "ready", "battery_level": "3.864",
"temperature_lock_high_temp": "22.222", "humidity_control_lockout_end_time": "0",
"target_temperature_high": "24.0", "ob_persistence": "True",
"hvac_safety_shutoff_active": "False", "schedule_learning_reset": "False",
"pin_y2_description": "none", "is_on_stand": "False",
"emer_heat_source": "electric", "filter_reminder_enabled": "False", "compressor_lockout_leaf": "-17.8", "aux_heat_delivery": "forced-air", "away_temperature_high": "24.444",
"fan_duty_start_time": "0", "learning_days_completed_range": "0", "target_humidity_enabled": "False",
"switch_system_off": "False", "sunlight_correction_ready": "True",
"sunlight_correction_enabled": "True", "has_emer_heat": "False",
"safety_temp_activating_hvac": "False", "has_dual_fuel": "False",
"heatpump_setback_active": "False", "has_heat_pump": "False",
"fan_control_state": "False", "model_version": "Display-2.8",
"has_aux_heat": "False", "current_version": "4.2.4",
"away_temperature_high_enabled": "False", "can_heat": "True",
"alt_heat_delivery": "forced-air", "current_humidity": "41", "target_humidity": "35.0",
"upper_safety_temp": "35.0", "heater_delivery": "forced-air",
"where_id": "00000000-0000-0000-0000-000100000006", "backplate_mono_version": "4.0.21",
"has_fossil_fuel": "True", "mac_address": "18b4302ae97f",
"serial_number": "02AA01AC041402HT", "type": "TBD",
"lower_safety_temp": "4.444", "hvac_heater_state": "False",
"humidity_control_lockout_start_time": "0", "fan_mode": "auto",
"filter_changed_set_date": "0", "range_enable": "True",
"heatpump_savings": "off", "radiant_control_enabled": "False", "temperature_lock_low_temp": "20.0",
"pin_ob_description": "none", "auto_away_reset": "False",
"humidity_control_lockout_enabled": "False", "fan_duty_cycle": "3600", "heatpump_ready": "False",
"preconditioning_enabled": "False", "hvac_fan_state": "True",
"preconditioning_ready": "True", "target_time_confidence": "0.0",
"local_ip": "192.168.1.144", "pin_w1_description": "heat",
"current_temperature": "22.43", "has_air_filter": "True",
"cooling_x2_source": "electric", "hvac_alt_heat_state": "False",
"heat_pump_aux_threshold": "10.0", "rssi": "58.0",
"fan_timer_timeout": "0", "has_alt_heat": "False",
"leaf_schedule_delta": "1.11", "backplate_bsl_version": "2.1", "user_brightness": "medium",
"preconditioning_active": "False", "pin_w2aux_description": "none", "pin_rc_description": "power", "has_dehumidifier": "False", "maint_band_upper": "0.56",
"target_temperature": "22.778", "leaf_learning": "ready",
"emer_heat_delivery": "forced-air", "pin_y1_description": "cool", "capability_level": "4.0",
"pin_rh_description": "none", "available_locales": "en_US,fr_CA,es_US,en_GB",
"dehumidifier_state": "False", "hvac_emer_heat_state": "False", "dehumidifier_type": "unknown",
"nlclient_state": "", "hvac_heat_x2_state": "False",
"upper_safety_temp_enabled": "False", "learning_state": "slow",
"hvac_heat_x3_state": "False", "hvac_cool_x2_state": "False", "fan_cooling_state": "True", "fan_duty_end_time": "0",
"auto_away": "0", "alt_heat_source": "gas",
"heat_link_connection": "0", "temperature_lock_pin_hash": "", "cooling_delivery": "unknown",
"heat_pump_aux_threshold_enabled": "True", "leaf_away_low": "16.67",
"heat_x3_delivery": "forced-air", "ob_orientation": "O", "touched_by": "{}",
"temperature_scale": "F", "emer_heat_enable": "False",
"backplate_mono_info": "TFE (BP_D2) 4.0.21 (root@bamboo) 2014-05-02 16:54:17", "auto_away_enable": "True", "pin_g_description": "fan",
"click_sound": "on", "hvac_alt_heat_x2_state": "False", "target_temperature_low": "20.0",
"has_x2_heat": "False", "away_temperature_low": "10.0",
"time_to_target": "0", "heat_x2_delivery": "forced-air", "cooling_x2_delivery": "unknown", "dual_fuel_breakpoint": "-1.0",
"_id": "ObjectId(53cb494e2239c261ac83dfe0)", "heater_source": "lp",
"pin_c_description": "none", "has_x2_alt_heat": "False"
}
![Page 17: State of the Data Universe](https://reader033.vdocuments.site/reader033/viewer/2022060201/5599e5071a28ab447a8b461f/html5/thumbnails/17.jpg)
Results
![Page 18: State of the Data Universe](https://reader033.vdocuments.site/reader033/viewer/2022060201/5599e5071a28ab447a8b461f/html5/thumbnails/18.jpg)
Why this is the future● It’s may be big data, but maybe not.
● Multiple entities, communicating via API
● Multiple layers of data analytics
● Very competitive
![Page 19: State of the Data Universe](https://reader033.vdocuments.site/reader033/viewer/2022060201/5599e5071a28ab447a8b461f/html5/thumbnails/19.jpg)
Rackspace Cloud Insights
How do I cut down the number of false positives in my monitoring solution?
![Page 20: State of the Data Universe](https://reader033.vdocuments.site/reader033/viewer/2022060201/5599e5071a28ab447a8b461f/html5/thumbnails/20.jpg)
Rackspace Cloud Insights● Pattern & anomaly detection algorithms
● Variation of Bollinger Bands algorithm
● First order differencing
● Belief network & Vector Similarity
![Page 21: State of the Data Universe](https://reader033.vdocuments.site/reader033/viewer/2022060201/5599e5071a28ab447a8b461f/html5/thumbnails/21.jpg)
Rackspace Cloud Insights
![Page 22: State of the Data Universe](https://reader033.vdocuments.site/reader033/viewer/2022060201/5599e5071a28ab447a8b461f/html5/thumbnails/22.jpg)
Rackspace Cloud Insights
metrics store -> analysis componentry -> alerting engine
https://developer.rackspace.com/blog/rackspace-cloud-intelligence-insights-in-monitoring/
![Page 23: State of the Data Universe](https://reader033.vdocuments.site/reader033/viewer/2022060201/5599e5071a28ab447a8b461f/html5/thumbnails/23.jpg)
Why this is the future● Answers a very simple, but powerful use
case
● Heterogeneous solution
● Very competitive
![Page 24: State of the Data Universe](https://reader033.vdocuments.site/reader033/viewer/2022060201/5599e5071a28ab447a8b461f/html5/thumbnails/24.jpg)
● Your brilliant data scientist spends all his/her time wrangling data not producing insights
● Technology choices gone wrong
● Capture lots of data, but never analyze it
● Waiting too long to start capturing data
Data Anti-Patterns
![Page 25: State of the Data Universe](https://reader033.vdocuments.site/reader033/viewer/2022060201/5599e5071a28ab447a8b461f/html5/thumbnails/25.jpg)
Data is a precious thing and will last longer than the systems themselves.
- Tim Berners-Lee