1 cc-by-nc-sa 3.0 license, 2015 hope bay technologies, inc. 和沛科技 big data philosophy ben jai...

1CC-BY-NC-SA 3.0 License, 2015 Hope Bay Technologies, Inc. 和沛科技

Big Data PhilosophyBen Jai

CEO, Hope Bay Technologies, Inc.


What is Big Data?

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications.

Translation: DB2/Oracle/MSSQL/MySQL can’t handle it


Difference from just Data?

Volume (amount of data) Velocity (speed of data in and out) Variety (range of data types and sources) Veracity (truthfulness of the data)

Translation: DB2/Oracle/MSSQL/MySQL can’t handle it


Is that It?

Big, fast, unstructured, uncertain, so?

Conventional wisdom: I can solve the problem if my computer is 1000 times bigger and faster!• I have a question looking for an answer.

Unconventional wisdom: I can find the pattern if my data set is 1000 times bigger!• I have a data set looking for useful information.


Example 1: Machine Translation

Conventional method: dictionary + grammar rules + exception database…• Good enough initially, but will not improve• Exceptions grow naturally and DB can’t follow

Statistical machine translation: compare same article in different languages and build statistical models• Need a LOT of data to see initial results but

will grow automatically without human knowledge of the languages


How big is BIG?

47.5

48.5

49.5

50.5

51.5

52.5

53.5

AE BLEU[%]

+weblm =219B words of web data!

BLEU% score – Arabic to English


Example 2: Spell Check

Conventional method: compare words against dictionary and find minimal editing distance. But:• Proper names are not in the dictionary• Word may be in the dictionary but not what the

user wants• Can’t break ties

Google method: look at what others do• Britney Spears brittany spears, brittney spears,

britany spears, britny spears, briteny spears, britteny spears, brine spears, …


300 Ways to Spell Her Name


Also Look at the Context

Kofi Annan is correct

Kofee Annan Kofi Annan

Kofee Shop Coffee Shop


Example 3: Detecting Defaults

Conventional method: look at FICO score MIT method: use machine learning to look at

100’s of variables in customer activities

NTU professor used a similar method to reduce return rate in TV shopping channel


Example 4: Who’s in Love?


Amazing Results

Big Data System Architecture

Data Source

Data Source

Data Source

Data Source

Data Storage Platform

Data Analysis Platform

Data Analysis Application


Levels of Big Data Processing

STORAGE is what you can ACCESS DATA is what you can PROCESS INFORMATION is what you can

UNDERSTAND INTELLIGENCE is what you can USE


Amazing Results

Intelligence

Data Storage Platform

Data Analysis Platform

Data Storage

Data Analysis Tool

Data Analysis ApplicationData Information

Big Data System Components

Data Source

Data Source

Data Source

Data Source


Who Can Use Big Data?

Those who owns a lot of data Those who can see the information Those who can convert the information

into intelligence Those who can monetize the intelligence


Who Can Make Money?

Those who owns a lot of data Those who can see the information Those who can convert the information

into intelligence Those who can monetize the intelligence


It’s Just the Beginning…

1952: UNIVAC I predicted the landslide victory of Dwight D. Eisenhawer


[email protected]

Q&A

1 cc-by-nc-sa 3.0 license, 2015 hope bay technologies, inc. 和沛科技 big data philosophy ben jai...

Documents