big data seminar br-new
TRANSCRIPT
326-Sept-2013
Agenda
What is Big Data
Big Data Characteristics
Brief history
Use Cases
who are the players
Challenges to deal Big data with traditional approach
Research & analysis
Action plans and future approach
Q & A
importance of Big Data in Financial service
426-Sept-2013
Big Data
Big data technologies describe a new generation of technologies and architectures, designed to economically extract value from very large volumes of a wide variety of data, by enabling high-velocity capture, discovery, and/or analysis--Idc
Big data[ is the term for a collection of datasets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis, and visualization--Wiki.
Every day, we create 2.5 quintillion bytes of data — so much that 90% of the data in the world today has been created in the last two years alone. This data comes from everywhere: sensors used to gather climate information, posts to social media sites, digital pictures and videos, purchase transaction records, and cell phone GPS signals to name a few. This data is big data--IBM
526-Sept-2013
History Behind
Data process done by processors Users Becomes processors and generate their own data into the systems
Usage of Social networking sites Smart phones
Machines accumulated the data Humidity, temperature Electricity usage Satellites
Google published a paper in 2003 about their Distributed File Systems, computation towards unstructured data
increasing internet, bandwidth speedStorage mechanisms implemented a lot
626-Sept-2013
Velocity
volumev
variety
Characteristics 12 terabytes of tweets created
each day Airline jets collects 10 terabytes
of sensor data for every 30 mins of flying time
Data will grow 800% over next 5 years-Gartner
text, Sensors data, audio, video, click streams, RFID, GPS devices, log files and more.
80% of data is unstructured or semi structured
How much data
How fast data is processed
Various types of data
Facebook has an average of 3.2 billion likes and comments are posted every day 575 photos uploaded,8500 likes and 7800 comments by Instagram users every
second
726-Sept-2013
Use cases
Banking/insurance/finance
Telecommunications
Life science/Healthcare
Government
Retail
Energy
Media
Manufacturing
826-Sept-2013
Players
Global threat analytics Virus analysis
intrusion detection and prevention
forensic analysis
Customer sentiment Network analysis
Major credit card issuer Recommendation engine Fraud detection & prevention
Electronic manufacturer Click stream analysis Quality profiling
DNA based relationship discovery
Recommendation engine
Leading retailer
Customer behavior analysis Brand monitoring
Information retrieval and extraction of research project
Large scale audio feature analysis
926-Sept-2013
Traditional systems
failed to analyze the un structured and semi structured data
CPU cannot handle the Big data
RDBMS handles schema based table like structure
Reading or writing more amount of data to the system is very time consuming
1126-Sept-2013
Research & Analysis
There will be a shortage of talent necessary for organizations to take advantage of big data. By 2018, the United States alone could face a shortage of 140,000 to 190,000 people with deep analytical skills as well as 1.5 million managers and analysts with the know-how to use the analysis of big data to make effective decisions- McKinsey
the digital universe will about double every two years-- Idc
Big data investments in 2013 continue to rise, with 64 percent of organizations investing or planning to invest in big data technology Gartner
Global spending on big data by organizations will exceed $31 billion in 2013, finds a new market forecast by ABI Research. The spending will grow at a CAGR of 29.6% over the next five years, reaching $114 billion in 2018- ABI Research
1326-Sept-2013
Action plans
Distributed File Systems
No-SQL database
parallel processing
Schema on Read rather than schema on write
Machine learning techniques
implementation of Data analytic tools for unstructured data