cs 4604: introducon to database management systemscs4604/spring16/lectures/...prakash 2016 vt cs...

CS4604:Introduc0ontoDatabaseManagementSystems

B.AdityaPrakashLecture#10:QueryProcessing

Prakash2016 VTCS4604 2

Outline

§  introduc?on§  selec?on§  projec?on§  join§  set&aggregateopera?ons


Introduc0on

§  Today’stopic:QUERYPROCESSING§  Somedatabaseopera?onsareEXPENSIVE§  Cangreatlyimproveperformancebybeing“smart”–  e.g.,canspeedup1,000,000xovernaïveapproach


Introduc0on(cnt’d)

§ Mainweaponsare:– cleverimplementa?ontechniquesforoperators– exploi?ng“equivalencies”ofrela?onaloperators– usingsta?s?csandcostmodelstochooseamongthese.


AReallyBadQueryOp0mizer

§  ForeachSelect-From-Wherequeryblock– docartesianproductsfirst–  thendoselec?ons–  etc,ie.:

•  GROUPBY;HAVING•  projec?ons•  ORDERBY

§  Incrediblyinefficient– Hugeintermediateresults!

× σpredicates

tables …


Cost-basedQuerySub-System

Query Parser

Query Optimizer

Plan Generator Plan Cost Estimator

Query Plan Evaluator

Catalog Manager

Usuallythereisaheuris?cs-basedrewri?ngstepbeforethecost-basedsteps.

Schema Sta?s?cs

Select * From Blah B Where B.blah = blah

Queries


TheQueryOp0miza0onGame§  “Op?mizer”isabitofamisnomer…§  Goalistopicka“good”(i.e.,lowexpectedcost)plan.–  Involveschoosingaccessmethods,physicaloperators,operatororders,…

– No?onofcostisbasedonanabstract“costmodel”


Rela0onalOpera0ons§  Wewillconsiderhowtoimplement:–  Selec3on(σ)Selectsasubsetofrowsfromrela?on.–  Projec3on(π)Deletesunwantedcolumnsfromrela?on.–  Join()Allowsustocombinetworela?ons.–  Set-difference(-)Tuplesinreln.1,butnotinreln.2.–  Union(∪)Tuplesinreln.1andinreln.2.–  Aggrega3on(SUM,MIN,etc.)andGROUPBY

§  Recall:opscanbecomposed!§  Later(akerspringbreak),we’llseehowtoop3mizequerieswithmanyops

▹◃


SchemaforExamples

§  Similartooldschema;rnameaddedforvaria?ons.§  Sailors:–  Eachtupleis50byteslong,80tuplesperpage,500pages.–  N=500,pS=80.

§  Reserves:–  Eachtupleis40byteslong,100tuplesperpage,1000pages.– M=1000,pR=100.

Sailors(sid:integer,sname:string,ra3ng:integer,age:real)Reserves(sid:integer,bid:integer,day:dates,rname:string)


SimpleSelec0ons

§  Oftheform§  Ques?on:howbesttoperform?

SELECT*FROMReservesRWHERER.rname<‘C%’

σ R attr valueop R. ( )


SimpleSelec0ons

§  A:Dependson:– whatindexes/accesspathsareavailable– whatistheexpectedsizeoftheresult(intermsofnumberoftuplesand/ornumberofpages)


SimpleSelec0ons

§  Sizeofresultapproximatedas sizeofR*reduc3onfactor– “reduc?onfactor”isalsocalledselec3vity.– es?mateofreduc?onfactorsisbasedonsta?s?cs–wewilldiscussshortly.


Alterna0vesforSimpleSelec0ons

§ Withnoindex,unsorted:– Mustessen?allyscanthewholerela?on– costisM(#pagesinR).For“reserves”=1000I/Os.


SimpleSelec0ons(cnt’d)

§ Withnoindex,sorted:– costofbinarysearch+numberofpagescontainingresults.

– Forreserves=10I/Os+⎡selec?vity*#pages⎤


SimpleSelec0ons(cnt’d)

§ Withanindexonselec?onauribute:– Useindextofindqualifyingdataentries,–  thenretrievecorrespondingdatarecords.–  (Hashindexusefulonlyforequalityselec?ons.)


UsinganIndexforSelec0ons

§  Costdependson#qualifyingtuples,andclustering.– Cost:• findingqualifyingdataentries(typicallysmall)• pluscostofretrievingrecords(couldbelargew/oclustering).


Selec0onsusingIndex(cnt’d)

Index entries

Data entries

direct search for

(Index File) (Data file)

Data Records

data entries

Data entries

Data Records

CLUSTEREDUNCLUSTERED


Selec0onsusingIndex(cnt’d)–  Inexample“reserves”rela?on,if10%oftuplesqualify(100pages,10,000tuples).• Withaclusteredindex,costisliulemorethan100I/Os;•  ifunclustered,couldbeupto10,000I/Os!unless…


Selec0onsusingIndex(cnt’d)§  Importantrefinementforunclusteredindexes:

1.Findqualifyingdataentries.2.Sorttherid’softhedatarecordstoberetrieved.3.Fetchridsinorder.Thisensuresthateachdatapageislookedatjustonce(though#ofsuchpageslikelytobehigherthanwithclustering).


GeneralSelec0onCondi0ons

§  Q:Whatwouldyoudo?(day



§  Q:Whatwouldyoudo?§  A:trytofindaselec?ve(clustering)index.Specifically:

(day



§  Converttoconjunc3venormalform(CNF):–  (day



§  AB-treeindexmatches(aconjunc?onof)termsthatinvolveonlyauributesinaprefixofthesearchkey.–  Indexonmatchesa=5ANDb=3,butnotb=3.

§  ForHashindex,musthaveallauributesinsearchkey

(day


TwoApproachestoGeneralSelec0ons

§  Firstapproach:Findthecheapestaccesspath,retrievetuplesusingit,andapplyanyremainingtermsthatdon’tmatchtheindex

§  Secondapproach:getridsfromfirstindex;ridsfromsecondindex;intersectandfetch.

SKIP


TwoApproachestoGeneralSelec0ons

§  Firstapproach:Findthecheapestaccesspath,retrievetuplesusingit,andapplyanyremainingtermsthatdon’tmatchtheindex:– Cheapestaccesspath:AnindexorfilescanwithfewestI/Os.

– Termsthatmatchthisindexreducethenumberoftuplesretrieved;othertermshelpdiscardsomeretrievedtuples,butdonotaffectnumberoftuples/pagesfetched.

SKIP


CheapestAccessPath-Example§  Considerday<8/9/94ANDbid=5ANDsid=3.

§  AB+treeindexondaycanbeused;– then,bid=5andsid=3mustbecheckedforeachretrievedtuple.

§  Similarly,ahashindexoncouldbeused;– Then,day


CheapestAccessPath-cnt’d

§  Considerday<8/9/94ANDbid=5ANDsid=3.

§  HowaboutaB+treeon?§  HowaboutaB+treeon?§  HowaboutaHashindexon?

SKIP


Intersec0onofRIDs

§  Secondapproach:ifwehave2ormorematchingindexes(w/Alterna?ves(2)or(3)fordataentries):– Getsetsofridsofdatarecordsusingeachmatchingindex.

– Thenintersectthesesetsofrids.– Retrievetherecordsandapplyanyremainingterms.

SKIP


Intersec0onofRIDs(cnt’d)

§  EXAMPLE:Considerday


TheProjec0onOpera0on

§  Issueisremovingduplicates.§  Basicapproach:sor?ng– 1.ScanR,extractonlytheneededaurs(why?)– 2.Sorttheresul?ngset– 3.RemoveadjacentduplicatesCost:Reserveswithsizera?o0.25=250pages.With20bufferpagescansortin2passes,so1000+250+2*2*250+250=2500I/Os

SELECTDISTINCTR.sid,R.bidFROMReservesR


Projec0on

§  Canimprovebymodifyingexternalsortalgorithm(seechapter13):– ModifyPass0ofexternalsorttoeliminateunwantedfields.

– Modifymergingpassestoeliminateduplicates.Cost:forabovecase:read1000pages,writeout250inrunsof40pages,mergeruns=1000+250+250=1500.

SKIP


DiscussionofProjec0on

§  Ifanindexontherela?oncontainsallwantedauributesinitssearchkey,candoindex-onlyscan.– Applyprojec?ontechniquestodataentries(muchsmaller!)


DiscussionofProjec0on

§  Ifanordered(i.e.,tree)indexcontainsallwantedauributesasprefixofsearchkey,candoevenbeuer:– Retrievedataentriesinorder(index-onlyscan),discardunwantedfields,compareadjacenttuplestocheckforduplicates.

AB-treeindexmatches(aconjunc?onof)termsthatinvolveonlyauributesinaprefixofthesearchkey.–  Indexonmatchesa=5ANDb=3,butnotb=3.

ForHashindex,musthaveallauributesinsearchkey


Joins

§  Joinsareverycommon.§  Joinscanbeveryexpensive(crossproductin

worstcase).§  Manyapproachestoreducejoincost.


Joins

§  Jointechniqueswewillcover:– Nested-loopsjoin–  Index-nestedloopsjoin– Sort-mergejoin– Hashjoin


EqualityJoinsWithOneJoinColumn

§  Inalgebra:RS.Common!Mustbecarefullyop?mized.R×Sislarge;so,R×Sfollowedbyaselec?onisinefficient.

§  Remember,joinisassocia?veandcommuta?ve.

SELECT*FROMReservesR1,SailorsS1WHERER1.sid=S1.sid

▹◃


EqualityJoins

§  Assume:– MpagesinR,pRtuplesperpage,mtuplestotal– NpagesinS,pStuplesperpage,ntuplestotal–  Inourexamples,RisReservesandSisSailors.

§ Wewillconsidermorecomplexjoincondi?onslater.

§  Costmetric:#ofI/Os.Wewillignoreoutputcosts.


Nestedloops

§  Algorithm#0:(naive)nestedloop(SLOW!)

R(A,..)

S(A, ......) m

n


Nestedloops

§  Algorithm#0:(naive)nestedloop(SLOW!)foreachtuplerofR

foreachtuplesofSprint,iftheymatch

R(A,..)

S(A, ......) m

n


Nestedloops

§  Algorithm#0:(naive)nestedloop(SLOW!)foreachtuplerofR

foreachtuplesofSprint,iftheymatch

R(A,..)

S(A, ......) m

n

outer relationinner relation


Nestedloops

§  Algorithm#0:whyisitbad?§  howmanydiskaccesses(‘M’and‘N’arethenumberofblocksfor‘R’and‘S’)?

R(A,..)

S(A, ......) M pages,

m tuples N pages,

n tuples


Nestedloops

§  Algorithm#0:whyisitbad?§  howmanydiskaccesses(‘M’and‘N’arethenumberofblocksfor‘R’and‘S’)?M+m*N

R(A,..)

S(A, ......) M pages,

m tuples N pages,

n tuples


SimpleNestedLoopsJoin

§  Actualnumber(pR*M)*N+M=100*1000*500+1000I/Os.– At10ms/IO,Total:???

§ Whatifsmallerrela?on(S)wasouter?

§ Whatassump?onsarebeingmadehere?


SimpleNestedLoopsJoin

§  Actualnumber§  (pR*M)*N+M=100*1000*500+1000I/Os.– At10ms/IO,Total:~6days(!)

§ Whatifsmallerrela?on(S)wasouter?– slightlybeuer

§ Whatassump?onsarebeingmadehere?– 1bufferforeachtable(and1foroutput)


Nestedloops

R(A,..)

S(A, ......) M pages,

m tuples N pages,

n tuples

•  Algorithm #1: Blocked nested-loop join –  read in a block of R

•  read in a block of S –  print matching tuples COST?


Nestedloops

R(A,..)

S(A, ......) M pages,

m tuples N pages,

n tuples

•  Algorithm #1: Blocked nested-loop join –  read in a block of R

•  read in a block of S –  print matching tuples COST= M+M*N


Nestedloops

R(A,..)

S(A, ......) M pages,

m tuples N pages,

n tuples

•  Which one should be the outer relation?

COST= M+M*N


Nestedloops

R(A,..)

S(A, ......) M pages,

m tuples N pages,

n tuples

•  Which one should be the outer relation? •  A: the smallest (page-wise)

COST= M+M*N


Nestedloops

R(A,..)

S(A, ......) M pages,

m tuples N pages,

n tuples

•  M=1000, N=500 •  Cost = 1000 + 1000*500 = 501,000 •  = 5010 sec ~ 1.4h COST= M+M*N


Nestedloops

R(A,..)

S(A, ......) M pages,

m tuples N pages,

n tuples

•  M=1000, N=500 - if smaller is outer: •  Cost = 500 + 1000*500 = 500,500 •  = 5005 sec ~ 1.4h COST= N+M*N


Nestedloops

R(A,..)

S(A, ......) M pages,

m tuples N pages,

n tuples

•  What if we have B buffers available?


Nestedloops

R(A,..)

S(A, ......) M pages,

m tuples N pages,

n tuples

•  What if we have B buffers available? •  A: give B-2 buffers to outer, 1 to inner, 1 for

output


Nestedloops

R(A,..)

S(A, ......) M pages,

m tuples N pages,

n tuples

•  Algorithm #1: Blocked nested-loop join –  read in B-2 blocks of R

•  read in a block of S –  print matching tuples COST= ?


Nestedloops

R(A,..)

S(A, ......) M pages,

m tuples N pages,

n tuples

•  Algorithm #1: Blocked nested-loop join –  read in B-2 blocks of R

•  read in a block of S –  print matching tuples COST= M+M/(B-2)*N


Nestedloops

R(A,..)

S(A, ......) M pages,

m tuples N pages,

n tuples

•  and, actually: •  Cost = M + ceiling(M/(B-2)) * N

COST= M+M/(B-2)*N


Nestedloops

R(A,..)

S(A, ......) M pages,

m tuples N pages,

n tuples

•  If smallest (outer) fits in memory •  (ie., B= N+2), •  Cost =? COST= N+N/(B-2)*M


Nestedloops

R(A,..)

S(A, ......) M pages,

m tuples N pages,

n tuples

•  If smallest (outer) fits in memory •  (ie., B= N+2), •  Cost =N+M (minimum!) COST= N+N/(B-2)*M


Nestedloops-guidelines

§  pickasouterthesmallesttable(=fewestpages)

§  fitasmuchofitinmemoryaspossible§  loopovertheinner


§  useanexis?ngindex,orevenbuildoneonthefly

§  cost:M+m*c(c:look-upcost)

IndexNLjoin

R(A,..)

S(A, ......) M pages,

m tuples N pages,

n tuples


§  cost:M+m*c(c:look-upcost)§  ‘c’dependswhethertheindexisclusteredornot.

IndexNLjoin

R(A,..)

S(A, ......) M pages,

m tuples N pages,

n tuples


Joins



Sort-mergejoin

R(A,..)

S(A, ......) M pages,

m tuples N pages,

n tuples

•  sort both on joining attributed •  scan each and merge •  Cost, given B buffers?


Sort-mergejoin

R(A,..)

S(A, ......) M pages,

m tuples N pages,

n tuples

•  Cost, given B buffers? •  ~ 2*M*logM/logB + 2*N* logN/logB + M + N


Sort-MergeJoin§  Usefulif


Sort-MergeJoin§  Usefulif– oneorbothinputsarealreadysortedonjoinauribute(s)

– outputisrequiredtobesortedonjoinauributes(s)

§  “Merge”phasecanrequiresomebacktrackingifduplicatevaluesappearinjoincolumn


ExampleofSort-MergeJoin

sid sname rating age22 dustin 7 45.028 yuppy 9 35.031 lubber 8 55.544 guppy 5 35.058 rusty 10 35.0

sid bid day rname28 103 12/4/96 guppy28 103 11/3/96 yuppy31 101 10/10/96 dustin31 102 10/12/96 lubber31 101 10/11/96 lubber58 103 11/12/96 dustin


ExampleofSort-MergeJoin

§  With35,100or300bufferpages,bothReservesandSailorscanbesortedin2passes;totaljoincost:7500.

§  (while Block Nested Loop (BNL) cost: 2,500 to 15,000 I/Os)


Sort-mergejoin

§ Worstcaseformergingphase?

§  Cost?


Refinements

§  Alltherefinementsofexternalsor?ng§  plusoverlappingofthemergingofsor?ngwiththemergingofjoining.


Joins



§  hashjoin:usehashingfunc?onh()– hash‘R’into(0,1,...,‘max’)buckets– hash‘S’intobuckets(samehashfunc?on)–  joineachpairofmatchingbuckets

Hashjoins

R(A, ...) S(A, ......) 0

1

max


– howtojoineachpairofpar??onsHr-i,Hs-i?– A:buildanotherhashtableforHs-i,andprobeitwitheachtupleofHr-i

Hashjoin-details

R(A, ...) S(A, ......)

Hr-0

0

1

max

Hs-0


Hashjoin-details

§  Inmoredetail:§  Choosethe(page-wise)smallest-ifitfitsinmemory,do~NL– and,actually,buildahashtable(withh2()!=h())– andprobeit,witheachtupleoftheother


§  whatifHs-iistoolargetofitinmain-memory?

§  A:recursivepar??oning§  moredetails(overflows,hybridhashjoins):inbook

§  costofhashjoin?(ifwehaveenoughbuffers:)3(M+N)(why?Seenextslide)

Hashjoindetails


CostofHash-Join

§  Inpar??oningphase,read+writebothrelns;2(M+N).Inmatchingphase,readbothrelns;M+NI/Os.

§  Inourrunningexample,thisisatotalof4500I/Os.


§  [costofhashjoin?(ifwehaveenoughbuffers:)3(M+N)]

§ Whatis‘enough’?sqrt(N),orsqrt(M)?

Hashjoindetails


§  [costofhashjoin?(ifwehaveenoughbuffers:)3(M+N)]

§  Whatis‘enough’?sqrt(N),orsqrt(M)?§  A:sqrt(smallest)(why?)–  Becauseyouonlyneedenoughmemorytoholdthehashtablepar??onsofthesmallertableinmemorysoB>sizeofsmaller/B-1èB~sqrt(size-of-smaller)

Hashjoindetails Details


Sort-MergeJoinvs.HashJoin

§  Givenaminimumamountofmemorybothhaveacostof3(M+N)I/Os.

(min.memoryforsort-merge=sqrt(largertable)usingaggressiverefinements---intextbook)(min.memoryforhash=sqrt(smallertable)---seepreviousslides)


Sort-MergevsHashjoin

§  HashJoinPros:– ??– ??– ??

§  Sort-MergeJoinPros:– ??



§  HashJoinPros:– Superiorifrela?onsizesdiffergreatly– Showntobehighlyparallelizable(beyondscopeofclass)

§  Sort-MergeJoinPros:– ??



§  HashJoinPros:– Superiorifrela?onsizesdiffergreatly– Showntobehighlyparallelizable(beyondscopeofclass)

§  Sort-MergeJoinPros:– Lesssensi?vetodataskew– Resultissorted(mayhelp“upstream”operators)– goesfasterifoneorbothinputsalreadysorted


GeneralJoinCondi0ons

§  Equali?esoverseveralauributes(e.g.,R.sid=S.sidANDR.rname=S.sname):– allpreviousmethodsapply,usingthecompositekey



§  Inequalitycondi?ons(e.g.,R.rname<S.sname):§  whichmethodss?llapply?– NL–  indexNL– Sortmerge– Hashjoin



§  Inequalitycondi?ons(e.g.,R.rname<S.sname):§  whichmethodss?llapply?– NL (probably,thebest!)–  indexNL (onlyifclusteredindex)– Sortmerge (doesnotapply!)(why?)– Hashjoin (doesnotapply!)(why?)


SetOpera0ons

§  Intersec?onandcross-product:specialcasesofjoin

§  Union(Dis?nct)andExcept:similar;we’lldounion:

§  Effec?vely:concatenate;usesor?ngorhashing§  Sor?ngbasedapproachtounion:– Sortbothrela?ons(oncombina?onofallauributes).– Scansortedrela?onsandmergethem.– Alterna3ve:MergerunsfromPass0forbothrela?ons.

SKIP


SetOpera0ons,cont’d

§  Hashbasedapproachtounion:– Par??onRandSusinghashfunc?onh.– ForeachS-par??on,buildin-memoryhashtable(usingh2),scancorrespondingR-par??onandaddtuplestotablewhilediscardingduplicates.

SKIP


AggregateOpera0ons(AVG,MIN,etc.)

§ Withoutgrouping:–  Ingeneral,requiresscanningtherela?on.– GivenindexwhosesearchkeyincludesallauributesintheSELECTorWHEREclauses,candoindex-onlyscan.

SKIP


Summary§  Avirtueofrela?onalDBMSs:–  queriesarecomposedofafewbasicoperators–  Theimplementa?onoftheseoperatorscanbecarefullytuned

–  Importanttodothis!§  Manyalterna?veimplementa?ontechniquesforeachoperator–  Nouniversallysuperiortechniqueformostoperators.

“it depends” [Guy Lohman (IBM)]


Summarycont’d

§  Mustconsideravailablealterna?vesforeachopera?oninaqueryandchoosebestonebasedonsystemsta?s?cs,etc.–  Partofthebroadertaskofop?mizingaquerycomposedofseveralops.

cs 4604: introducon to database management systemscs4604/spring16/lectures/...prakash 2016 vt cs...

Documents