Bridging Innovation and Products in the Internet AgeP.P.S. NarayanMarch 2015

Bio

Work■ VP of Engineering @ Yahoo■ Technical Manager @ Bell Labs

Computer Science/Engg■ Masters: IIT Bombay■ Bachelors: Bombay University

Publications/Patents■ SIGMOD, VLDB, CloudDB, WebDB,

Internet Computing■ YDN, Tumblr Blog Posts■ 4 Patents and 7 Pending

Engineer at heart!!

Views are mine and mine alone. Not of my employer, co-workers, buddies or family.

Definitions

Researchn.noun

■ Careful study of a given subject, field, or problem, undertaken to discover facts or principles.

Sciencen.noun

■ The observation, identification, description, experimental investigation, and theoretical explanation of phenomena.

Innovationn.noun

■ a new method, idea, product, etc.

Engineeringn.noun

■ The application of scientific and mathematical principles to practical ends such as the design, manufacture, and operation of efficient and economical structures, machines, processes, and systems.

Problem Horizon

1970s

Enterprise,Finance

Data Recording

1960s 1980s 1990s 2000s

Market

Application

Form Factor

Computing Paradigm

Defense, Academia

Communication Consumer Daily Habit

Niche Computing

Reporting Predictive Analytics

Computing Evolution

Real-time Analytics

MobileInternet

2010s

Ubiquitous Computing

Cloud, Rich Media

Exhibit: Evolution of Database Systems

Flat file (VSAM, ISAM)

1970s 1980s 1990s 2000s

Non-relational Databases

Relational Database and DBMS

NoSQL, Distributed Databases

Scal

e

Open Source Databases

Commercial Databases

VSAM, ISAM, IDMS, Oracle

Version 1

IBM DB2, DBASE, PARADOX, Microsoft-SQL-server, Teradata,

Sybase

MySQL, PostgreSQL

HBase, Cassandra, MongoDB,

CouchDB, Redis, Riak, HyperTable, Cobbler, Neo4J,

GraphDB, MariaDB

BerkeleyDB

Aster-Data, MarkLogic, CouchBase,

Netezza, Oracle-NoSQL,

DynamoDB, HP-Vertica, EMC-

Greenplum

2010s

NuoDB, VoltDB, MySQL-Cluster,

Tokutek, Aerospike

SAP-HANA, Amazon RDS, SQL-

Azure, Google-Spanner

FoundationDB

NewSQL Databases

Emergent Database Class

Informix, Ingres, IBM-Infosphere,

Versant, ObjectivityDB,

Time Horizon

Exhibit: Packaged Software Release Cycles

1980 1990 2000 2010

Oracle 1Oracle 2

Oracle 3Oracle 4Oracle 5Oracle 6

Oracle 7Oracle 8Oracle 8i

Oracle 9iOracle 10gOracle 11g

Oracle 12c

Exhibit: Mobile Technology Evolution

http://www.slideshare.net/cpqd/1-a-vision-on-the-evolution-to-5-g-networks

Exhibit: AWS Pace of Innovation

http://www.slideshare.net/AmazonWebServices/spot301-aws-innovation-at-scale-aws-reinvent-2014

Exhibit: AWS Services

Yahoo Stream Ads

45 days to execute- create the format- bootstrap demand- create the buy side- book-to-bill flow- serving side arch- instrumentation

Launched in 43 days

Product with revenues projected greater than Twitter!!

Re-Invention

ReInvention Cycle - Long (10 Yrs)UDB•K-V Store•Commodity Storage•Geo-Replicated

•Lumpy Scalability•Consistency•Durability

•JSON•Ordered Tables•REST APIs

•Pluggable Storage Engine•Eventual, Timeline Consistency•Quorum vs. Message Bus

•PNUTs

Sherpa•Elastic Service•Structured Storage•Geo-Replicated•Multi-Tenant

Virtuous Cycle - Quick (1 Yr) Sherpa•Elastic Service•Structured storage•Commodity Storage•Geo-Replicated

•Cost per writes•Disk Latency•Automation

•Low Latency•Write heavy workloads•Inter-operability with Hadoop

•LSM Engine•Zero MQ Messaging•Protocol Buffer

Sherpa LL•Low latency•High throughput

•In-Memory•Memcache•Redis•PCIe

“Open Source” Research

History of Innovation

Internet Age of Innovation

Voldemort

Research Model

Which Model Works?

Traditional Avant Garde Best of Both Worlds

● long term view 10-15 years out

● academia-like, pioneering work

● “Research”● Publications

● short-term view mostly*● “blurry line”, “not treated

as a separate activity”● “Engineering”● Product

● mid-term view 2-3 years● “importing new ideas and

implementing them”● “Science”● Impact on Product

*

Comparisons

Observations

■ Close-ended Research: Proliferation of niche problems. Niche problems need niche solutions

■ Shrinking time-to-market: Product and Innovation time has shrunk from decades to years to quarters to months to weeks/days

■ Continuous Feedback: Services oriented world allows for Research to get inputs quickly and innovate

■ Collaboration and shared problems: leading to “Open Source” research models

■ Traditional Industrial Research model is dead