Big Data Overview

○ What is Big Data?○ Big Data in a real life.○ Big Data problems & challenges?○ Ways to solve these problems.○ Big Data tools and technologies.

Crucial Questions

What is Big Data?○ Too big to fit in

memory○ Too fast data

acquisition requirements

○ Too fast data processing

○ Too complex for traditional processing

Big Data: numbers & facts701,389 Facebook logins

38,194 posts to instagram

2.4 Million search queries

2.78 Million video views

Example: car fleet management○ 1M car profiles

○ Daily reports○ Track position

by request○ Keep history in

database

Real-time car fleet management○ 1K cars connected in real time

○ Gather data via OBD2 scanners in real-time○ Gather data from cars’ GPS sensors in real-

time○ Store the data for future processing○ Real-time calculation to predict traffic,

engine problems, accidents

What can we do with (Big)Data?○ Data ingestion & acquisition

○ Data storage (search, transfer, sharing)○ Data processing & analysis○ Data visualization

Data Ingestion & Acquisition○ Extract:

RDBMS, file systems, messaging systems, sensors, log files○ Transform:

Filter, encode/decode, aggregate, validate○ Load:

Data warehouse, messaging system

Data StorageBig Data storage challenges:○ Size (keep and search

huge amount of data)○ Speed (data acquisition,

data search)○ Availability (fault

tolerance, partition tolerance)

○ Consistency: all nodes see the same data at the same time

○ Availability: every request gets a response (success or failure)

○ Partition tolerance: system works despite of network failures

CAP Theorem

Streaming vs Batch processing

Batch Batch

Stream

Data

Data processing: Lambda Architecture

Data processing: Kappa Architecture

Data processing: MapReduce

Data Visualization

Everything as a Service

Example: Amazon Web Services

Q & A ?○ What is Big Data?○ Big Data in a real life.○ Big Data problems & challenges?○ Ways to solve these problems.○ Big Data tools and technologies.