in search of petabyte databases
TRANSCRIPT
![Page 1: In Search of PetaByte Databases](https://reader036.vdocuments.site/reader036/viewer/2022062600/586a2e8b1a28ab5e1d8be4f1/html5/thumbnails/1.jpg)
GigaByte
TeraByte
PetaByte
ExaByte
In Search of PetaByte Databases
Jim GrayTony Hey
![Page 2: In Search of PetaByte Databases](https://reader036.vdocuments.site/reader036/viewer/2022062600/586a2e8b1a28ab5e1d8be4f1/html5/thumbnails/2.jpg)
GigaByte
TeraByte
PetaByte
ExaByte
The Cost of Storage(heading for 1K$/TB soon)
y = 6.7x
y = 17.9x
0100200300400500600700800900
1000
0 20 40 60GB
$ IDE
SCSI
Price vs disk capacity
6
0
5
10
15
20
25
30
35
40
0 10 20 30 40 50 60GB
$
IDE
SCSI
k$/TB
12/1/1999
y = 3.8x
y = 13x
0100200300400500600700800900
1000
0 20 40 60 80Raw Disk unit Size GB
$
SCSI
IDE
Price vs disk capacity
0
5
10
15
20
25
30
35
40
0 20 40 60 80Disk unit size GB
$
SCSI
IDE
raw k$/TB
9/1/2000
y = 2.0x
y = 7.2x
0
200
400
600
800
1000
1200
1400
0 50 100 150 200Raw Disk unit Size GB
$ SCSI
IDE
Price vs disk capacity
0.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
8.0
9.0
10.0
0 50 100 150 200Disk unit size GB
$ SCSI
IDE
raw k$/TB
9/1/2001
![Page 3: In Search of PetaByte Databases](https://reader036.vdocuments.site/reader036/viewer/2022062600/586a2e8b1a28ab5e1d8be4f1/html5/thumbnails/3.jpg)
GigaByte
TeraByte
PetaByte
ExaByteSummary• DBs own the sweet-spot:
– 1GB to 100TB• Big data is not in databases
HPTS does not own high performance storage (BIG DATA)
• We should• Cost of storage is people:
–Performance goal:1 Admin per PB
![Page 4: In Search of PetaByte Databases](https://reader036.vdocuments.site/reader036/viewer/2022062600/586a2e8b1a28ab5e1d8be4f1/html5/thumbnails/4.jpg)
GigaByte
TeraByte
PetaByte
ExaByteState is Expensive
• Stateless clones are easy to manage– App servers are middle tier
• Cost goes to zero with Moore’s law.– One admin per 1,000 clones.– Good story about scaleout.
• Stateful servers are expensive to manage– 1TB to 100TB per admin – Storage cost is going to zero(2k$ to 200k$).
• Cost of storage is management cost
![Page 5: In Search of PetaByte Databases](https://reader036.vdocuments.site/reader036/viewer/2022062600/586a2e8b1a28ab5e1d8be4f1/html5/thumbnails/5.jpg)
GigaByte
TeraByte
PetaByte
ExaBytePersonal 100 GB today
The Personal Petabyte (someday)• It’s coming (2M$ today…2K$ in 10 years)
• Today the pack rats have ~ 10-100GB– 1-10 GB in text (eMail, PDF, PPT, OCR…)– 10GB – 50GB tiff, mpeg, jpeg,…– Some have 1TB (voice + video).
• Video can drive it to 1PB.• Online PB affordable in 10 years.• Get ready: tools to capture, manage,
organize, search, display will be big app.
![Page 6: In Search of PetaByte Databases](https://reader036.vdocuments.site/reader036/viewer/2022062600/586a2e8b1a28ab5e1d8be4f1/html5/thumbnails/6.jpg)
GigaByte
TeraByte
PetaByte
ExaByte
10 TBAn Image Database: TerraServer
• Snapshot of the USA (1 meter granularity)– 10,000,000,000,000 (=10^13) sq meters– == 15TB raw (some duplicates)– == 5 TB cooked
• 5x compression• + Image pyramid• + gazetteer
• Interesting things: – Its all in the Database– Clustered (allows flaky hardware, online upgrade)
– Triplexed – snapshot each night
![Page 7: In Search of PetaByte Databases](https://reader036.vdocuments.site/reader036/viewer/2022062600/586a2e8b1a28ab5e1d8be4f1/html5/thumbnails/7.jpg)
GigaByte
TeraByte
PetaByte
ExaByteDatabases (== SQL)
• VLDB survey (Winter Corp).• 10 TB to 100TB DBs.
– Size doubling yearly– Riding disk Moore’s law– 10,000 disks at 18GB is 100TB cooked.
• Mostly DSS and data warehouses.• Some media managers
![Page 8: In Search of PetaByte Databases](https://reader036.vdocuments.site/reader036/viewer/2022062600/586a2e8b1a28ab5e1d8be4f1/html5/thumbnails/8.jpg)
GigaByte
TeraByte
PetaByte
ExaByteDB iFS
• DB2: leave the files where they live– Referential integrity between DBMS and FS.
• Oracle: put the files in the DBMS– One security model– One storage management model– One space manager– One recovery manger– One replication system– One thing to tune.– Features: transactions,….
![Page 9: In Search of PetaByte Databases](https://reader036.vdocuments.site/reader036/viewer/2022062600/586a2e8b1a28ab5e1d8be4f1/html5/thumbnails/9.jpg)
GigaByte
TeraByte
PetaByte
ExaByteInteresting facts
• No DBMSs beyond 100TB.• Most bytes are in files.• The web is file centric• eMail is file centric.• Science (and batch) is file centric.• But….• SQL performance is better than CIFS/NFS..
– CISC vs RISC
![Page 10: In Search of PetaByte Databases](https://reader036.vdocuments.site/reader036/viewer/2022062600/586a2e8b1a28ab5e1d8be4f1/html5/thumbnails/10.jpg)
GigaByte
TeraByte
PetaByte
ExaByte BarBar: the biggest DB
• 350 TB• Uses Objectivity™• SLAC events• Linux cluster scans DB looking for patterns
![Page 11: In Search of PetaByte Databases](https://reader036.vdocuments.site/reader036/viewer/2022062600/586a2e8b1a28ab5e1d8be4f1/html5/thumbnails/11.jpg)
GigaByte
TeraByte
PetaByte
ExaByte
300 TB (cooked)Hotmail / Yahoo
• Clone front ends ~10,000@hotmail.
• Application servers– ~100 @ hotmail – Get mail box– Get/put mail– Disk bound
• ~30,000 disks
• ~ 20 admins
![Page 12: In Search of PetaByte Databases](https://reader036.vdocuments.site/reader036/viewer/2022062600/586a2e8b1a28ab5e1d8be4f1/html5/thumbnails/12.jpg)
GigaByte
TeraByte
PetaByte
ExaByteAOL (msn)
(1PB?)• 10 B transactions per day (10% of that)• Huge storage• Huge traffic• Lots of eye candy• DB used for security/accounting.• GUESS AOL is a petabyte
– (40M x 10MB = 400 x 1012)
![Page 13: In Search of PetaByte Databases](https://reader036.vdocuments.site/reader036/viewer/2022062600/586a2e8b1a28ab5e1d8be4f1/html5/thumbnails/13.jpg)
GigaByte
TeraByte
PetaByte
ExaByteGoogle
1.5PB as of last spring• 8,000 no-name PCs
– Each 1/3U, 2 x 80 GB disk, 2 cpu 256MB ram
• 1.4 PB online.• 2 TB ram online• 8 TeraOps • Slice-price is 1K$ so 8M$.• 15 admins (!) (== 1/100TB).
![Page 14: In Search of PetaByte Databases](https://reader036.vdocuments.site/reader036/viewer/2022062600/586a2e8b1a28ab5e1d8be4f1/html5/thumbnails/14.jpg)
GigaByte
TeraByte
PetaByte
ExaByte
ComputationalScience
• Traditional Empirical Science – Scientist gathers data by direct observation– Scientist analyzes data
• Computational Science– Data captured by instruments
Or data generated by simulator– Processed by software– Placed in a database– Scientist analyzes database– tcl scripts
• on C programs – on ASCII files
![Page 15: In Search of PetaByte Databases](https://reader036.vdocuments.site/reader036/viewer/2022062600/586a2e8b1a28ab5e1d8be4f1/html5/thumbnails/15.jpg)
GigaByte
TeraByte
PetaByte
ExaByteAstronomy
• I’ve been trying to apply DB to astronomy• Today they are at 10TB per data set• Heading for Petabytes• Using Objectivity• Trying SQL (talk to me offline)
![Page 16: In Search of PetaByte Databases](https://reader036.vdocuments.site/reader036/viewer/2022062600/586a2e8b1a28ab5e1d8be4f1/html5/thumbnails/16.jpg)
GigaByte
TeraByte
PetaByte
ExaByteFast Moving Objects
• Find near earth asteroids:SELECT r.objID as rId, g.objId as gId, r.run, r.camcol, r.field as field, g.field as gField,
r.ra as ra_r, r.dec as dec_r, g.ra as ra_g, g.dec as dec_g,sqrt( power(r.cx -g.cx,2)+ power(r.cy-g.cy,2)+power(r.cz-g.cz,2) )*(10800/PI()) as distance
FROM PhotoObj r, PhotoObj g WHERE
r.run = g.run and r.camcol=g.camcol and abs(g.field-r.field)<2 -- the match criteria-- the red selection criteriaand ((power(r.q_r,2) + power(r.u_r,2)) > 0.111111 )and r.fiberMag_r between 6 and 22 and r.fiberMag_r < r.fiberMag_g and r.fiberMag_r < r.fiberMag_iand r.parentID=0 and r.fiberMag_r < r.fiberMag_u and r.fiberMag_r < r.fiberMag_zand r.isoA_r/r.isoB_r > 1.5 and r.isoA_r>2.0-- the green selection criteriaand ((power(g.q_g,2) + power(g.u_g,2)) > 0.111111 )and g.fiberMag_g between 6 and 22 and g.fiberMag_g < g.fiberMag_r and g.fiberMag_g < g.fiberMag_iand g.fiberMag_g < g.fiberMag_u and g.fiberMag_g < g.fiberMag_zand g.parentID=0 and g.isoA_g/g.isoB_g > 1.5 and g.isoA_g > 2.0-- the matchup of the pairand sqrt(power(r.cx -g.cx,2)+ power(r.cy-g.cy,2)+power(r.cz-g.cz,2))*(10800/PI())< 4.0and abs(r.fiberMag_r-g.fiberMag_g)< 2.0
• Finds 3 objects in 11 minutes• Ugly,
but consider the alternatives (c programs an files and…)
–
![Page 17: In Search of PetaByte Databases](https://reader036.vdocuments.site/reader036/viewer/2022062600/586a2e8b1a28ab5e1d8be4f1/html5/thumbnails/17.jpg)
GigaByte
TeraByte
PetaByte
ExaByte
Particle Physics – Hunting the Higgs and Dark Matter
• April 2006: First pp collisions at TeV energies at the Large Hadron Collider in Geneva
• ATLAS/CMS Experiments involve 2000 physicists from 200 organizations in US, EU, Asia
• Need to store,access, process, analyse 10 PB/yr with 200 TFlop/s distributed computation
• Building hierarchical Grid infrastructure to distribute data and computation
• Many 10’s of million $ funding – GryPhyN, PPDataGrid, iVDGL, DataGrid, DataTag, GridPP
ExaBytes and PetaFlop/s by 2015
![Page 18: In Search of PetaByte Databases](https://reader036.vdocuments.site/reader036/viewer/2022062600/586a2e8b1a28ab5e1d8be4f1/html5/thumbnails/18.jpg)
GigaByte
TeraByte
PetaByte
ExaByteAstronomy: Past and
Future of the Universe• Virtual Observatories – NVO, AVO, AstroGrid
– Store all wavelengths, need distributed joins– NVO 500 TB/yr from 2004
• Laser Interferometer Gravitational Observatory– Search for direct evidence for gravitational waves– LIGO 250 TB/yr, random streaming from 2002
• VISTA Visible and IR Survey Telescope in 2004– 250 GB/night, 100 TB/yr, Petabytes in 10 yrs
New phase of astronomy, storing, searching and analysing Petabytes of data
![Page 19: In Search of PetaByte Databases](https://reader036.vdocuments.site/reader036/viewer/2022062600/586a2e8b1a28ab5e1d8be4f1/html5/thumbnails/19.jpg)
GigaByte
TeraByte
PetaByte
ExaByte
Engineering, Environment and
Medical Applications• Real-Time Health Monitoring– UK DAME project for Rolls Royce Aero Engines– 1 GB sensor data/flight, 100,000 engine hours/day
• Earth Observation – ESA satellites generate 100 GB/day– NASA 15 PB by 2007
• Medical Images to Information– UK IRC Project on mammograms and MRIs– 100 MB/mammogram, UK 3M/yr, US 26M/yr– 200 MB/patient, Oxford 500 women/yr
Many Petabytes of data of real commercial interest
![Page 20: In Search of PetaByte Databases](https://reader036.vdocuments.site/reader036/viewer/2022062600/586a2e8b1a28ab5e1d8be4f1/html5/thumbnails/20.jpg)
GigaByte
TeraByte
PetaByte
ExaByte
Grids, Databases and Cool Tools
• Scientists:– will build Grids based on Globus Open Source m/w– will have instruments generating Petabytes of data– will annotate their data with XML-based metadata
Realize a version of Licklider and Taylor’s original vision of resource sharing and the ARPANET
• TP and DB community:- Should assist in developing Grid Interfaces to DBMS- Should develop ‘Cool Tools’ for Grid Services
There will be commercial Grid applications and viable business opportunities
![Page 21: In Search of PetaByte Databases](https://reader036.vdocuments.site/reader036/viewer/2022062600/586a2e8b1a28ab5e1d8be4f1/html5/thumbnails/21.jpg)
GigaByte
TeraByte
PetaByte
ExaByteSummary• DBs own the sweet-spot:
– 1GB to 100TB• Big data is not in databases• HPTS crowd is not really high
performance storage (BIG DATA)• Cost of storage is people:
–Performance goal:1 Admin per PB