dremel: interactive analysis of web-scale...
TRANSCRIPT
Dremel: Interactive Analysis of Web-Scale Datasets CS 744 BIG DATA
PHIL MARTINKUS
Motivation LargeScaleDatamustbeaccessibletoanalystsandengineers
Interactivequeriesareimportantfordataexploration,monitoring,onlinecustomersupport,rapidprototyping,debuggingofdata,andothertasks
Manydatabasesrequireacostlyloadingphase
Webdataisoftennon-relational
The Solution: Dremel Dremelisasystemthatsupportsinteractiveanalysisofverylargedatasetsoversharedclustersofcommoditymachines.
HasbeeninproductionatGooglesince2006 Canoperateondatainplaceusingadistributedstoragesystem
Usesanovelcolumnarstorageformatfornesteddata
Providesahigh-levelSQL-likelanguageforinteractivequeries
Data Model Strongly-typednestedrecords Recordsconsistofoneormorefields
Fieldscanberequired,optionalorrepeated
Data Model Example EachRecordrepresentsadocument
RequiredDocIdfield Linksisanoptionalgroupwithtwonestedrepeatedfields.
NameisarepeatedgroupwithanestedLanguagegroup.
Nested Columnar Storage Allvaluesofafieldarestoredconsecutivelyinblocks
Goalsforthestoragesystem:◦ Losslessrepresentationofrecordstructureincolumnarformat
◦ Fastencodings◦ Efficientrecordassembly
Repeatedrecordsarehandledwithrepetitionanddefinitionlevels
Repetition Levels Usedtodisambiguateoccurrencesofthesamefieldwithinthesamerecord
Tellusatwhatrepeatedfieldinthefield'spaththevaluehasrepeated
Repetition Level Example
Definition Levels Wheneveranoptionalorrepeatedfieldisnotpresentinarecord,thesystemstoresaNULL.
Tellushowmanyfieldsinthefield'spaththatcouldbeundefined(becausetheyareoptionalorrepeated)areactuallypresentintherecord.
MostlyusefulfordistinguishingNULLvalues.
Definition Level Example
Splitting Records into Columns Recursivealgorithmcomputeslevelsforeachfield
Atreeoffieldwritersmatchthestructureofthefieldschema
ManydatasetsatGooglearesparse
Record Assembly Goalistoreconstructrecordsgivenasubsetoffields
Finitestatemachine(FSM)readsvaluesandappendstooutputrecords
AnFSMstatecorrespondstoafieldreader
TheFSMistraversedfromthestartstatetotheendstateforeachrecord
Query Language BasedonSQL Designedforcolumnarnestedstorage
Query Execution UsesaTreearchitecture Rootreceivesincomingqueries
Intermediateserversrewritethequery
Leafserversaccessdata Eachserverhasaninternaltreecorrespondingtoaphysicalqueryexecutionplan.
Query Execution Example
Queryisrewritten
Querysenttoroot
Querysenttoleafnodes
Asetofiteratorsscantheinputcolumninlockstepandemitresultswithannotatedrepetitionanddefinitionlevelswithoutactuallyassemblingtherecords
Query Dispatcher Dremelisamulti-usersystemthatexecutesqueriessimultaneously
Thequerydispatcherschedulesqueries Dealingwithstragglers◦ Disproportionallyslowprocessesarerescheduledonanotherserver◦ Aparameterspecifiestheminimumpercentageoftabletsthatmustbescannedbeforereturningaresult
Experiments
Questions?