structure data 2014: inverting 80/20: beyond bespoke big data, ari gesher

Wednesday, March 26, 14

INVERTING 80/20: BEYOND BESPOKE BIG DATA

Ari GesherPalantir Technologies

Wednesday, March 26, 14SLIDES BY 3/14 at NOON - TBD.....

INVERTING 80/20Beyond Bespoke Big Data

(or how operating systems change everything)

mailto: [email protected]

twitter: @alephbass

Ari Gesher, Engineering Ambassador

Wednesday, March 26, 14! •! Hiring, hiring, hiring - the only way to have this problem (and it’s a good one to have) is to have a hiring machine that’s kicking ass.! •! Design for growth - make sure your systems (IT, food, etc) can handle exponential growth. Build small, but design scalable! •! Be most worried about long-lead time resources: hiring, leadership, and real estate! •! Culture: how it changes, how to preserve it! •! Discard your old biases - project managers enable scale, but process hinders at small scale

Wednesday, March 26, 14Artillery tables, calculated by people called ‘computers’

Wednesday, March 26, 14Eniac, 1945

OS/360 becomes THE computer


Yak Shavers, Inc

BEFORE OPERATING SYSTEMS


Yak Shavers, Inc


• Programmers had to write the minutiae of hardware, error codes, etc


Yak Shavers, Inc



• Writing directly for the hardware of a single computer


Yak Shavers, Inc




• Programmers started with nothing but primitive functionality


Yak Shavers, Inc




• Programmers started with nothing but primitive functionality

• Very little of the effort went into building the actual intent of the software


yak shaving, n


yak shaving, n1. Any apparently useless activity which, by allowing you to overcome intermediate difficulties, allows you to solve a larger problem.


yak shavingAny apparently useless activity which, by allowing you to overcome intermediate difficulties, allows

you to solve a larger problem.


Yaks keep hair

AFTER OPERATING SYSTEMS


Yaks keep hair


• OS API is a clean abstraction, hardware agnostic


Yaks keep hair



• Hardware can change and code still works


Yaks keep hair




• Common functionality (GUI, networking, files) shared by all programs


Yaks keep hair




• Common functionality (GUI, networking, files) shared by all programs

• Majority of effort goes into primary purpose


THE RIGHT ABSTRACTION


abstraction creates efficiency


abstraction creates SAFETY


abstraction creates innovation


big data, n1. Data sets that are too large and complex to manipulate or interrogate with standard methods or tools (OED, 2013).

2. A set of technologies and tools designed to address large-scale data analysis problems.

3. A IT buzzword from the early teens of the 21st century.


OPERATING SYSTEM ABSTRACTIONS



• Device Drivers



• Device Drivers

• Virtual Machine



• Device Drivers

• Virtual Machine

• File system



• Device Drivers

• Virtual Machine

• File system

• Permissions / Access Controls



• Device Drivers

• Virtual Machine

• File system


• Networking



• Device Drivers

• Virtual Machine

• File system


• Networking

• GUI Toolkits / UI Frameworks


Yaks keep hair

BIG DATA ABSTRACTIONS


Yaks keep hair


• Data Integration Standards / Pipelines


Yaks keep hair



• Virtual object model / Composition Rules


Yaks keep hair




• Object revisioning


Yaks keep hair





• Permissions / Access Control


Yaks keep hair






• Data Exchange / Translation


Yaks keep hair






• Data Exchange / Translation

• Interactive, integrated, extensible visualization & analysis environment


Wednesday, March 26, 14MOVIE ON NEXT SLIDE - NO AUDIO

Wednesday, March 26, 14MOVIE FILE _ NO AUDIO

FAMILIAR


INTERACTIVE


EXPRESSIVE


Questions?

[email protected]


structure data 2014: inverting 80/20: beyond bespoke big data, ari gesher

Documents

hardware agnostic hardware

operating systems programmers

minutiae of hardware

operating systems os

data sets

yak shavers

clean abstraction

yak shaving