millennium data dissemination databases and other services 2012-12-17millennium workshop1
TRANSCRIPT
![Page 1: Millennium Data Dissemination databases and other services 2012-12-17Millennium Workshop1](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f265503460f94c3e238/html5/thumbnails/1.jpg)
Millennium Data Dissemination
databases and other services
2012-12-17 Millennium Workshop 1
![Page 2: Millennium Data Dissemination databases and other services 2012-12-17Millennium Workshop1](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f265503460f94c3e238/html5/thumbnails/2.jpg)
Raw data:Particles
FOF groups and Subhalos
Density fields
Subhalo merger trees
Synthetic galaxies (SAM)
Mock catalogues
Mock images
2012-12-17 Millennium Workshop 2
![Page 3: Millennium Data Dissemination databases and other services 2012-12-17Millennium Workshop1](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f265503460f94c3e238/html5/thumbnails/3.jpg)
SUBFIND Tree Builder
Halo Merger Trees
SAMSSP
FOF GroupFinder
(Sub-)HalosFOF GroupsSimulation Particles
Gadget
MoMaF’
SAM Galaxies
Light-cone Galaxies
SkyMaker’SExtractor
Synthetic Images
SExtractor Catalogues
PSF, Noise,Background
Scaling
IGM
MRObs
2012-12-17 Millennium Workshop 3
![Page 4: Millennium Data Dissemination databases and other services 2012-12-17Millennium Workshop1](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f265503460f94c3e238/html5/thumbnails/4.jpg)
Virgo-Millennium Database
• Online access to Millennium simulations etal– portal: http://www/mpa-garching.mpg.de/millennium
• Databases – public: http://gavo.mpa-garching.mpg.de/Millennium – private + MyDB:
http://gavo.mpa-garching.mpg.de/MyMillennium
• Improved database access– MillenniumTAP batch queries – http://galformod.mpa-garching.mpg.de/millenniumtap
2012-12-17 Millennium Workshop 4
![Page 5: Millennium Data Dissemination databases and other services 2012-12-17Millennium Workshop1](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f265503460f94c3e238/html5/thumbnails/5.jpg)
Millennium data model (old)
5 5
![Page 6: Millennium Data Dissemination databases and other services 2012-12-17Millennium Workshop1](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f265503460f94c3e238/html5/thumbnails/6.jpg)
last progenitor main leaf and root
Trees in Database
2012-12-17 Millennium Workshop 6
![Page 7: Millennium Data Dissemination databases and other services 2012-12-17Millennium Workshop1](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f265503460f94c3e238/html5/thumbnails/7.jpg)
Merger trees
particles halos
2012-12-17 Millennium Workshop 7
![Page 8: Millennium Data Dissemination databases and other services 2012-12-17Millennium Workshop1](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f265503460f94c3e238/html5/thumbnails/8.jpg)
Joining Garching and Durham
8 8
![Page 9: Millennium Data Dissemination databases and other services 2012-12-17Millennium Workshop1](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f265503460f94c3e238/html5/thumbnails/9.jpg)
2012-12-17 Millennium Workshop 9
![Page 10: Millennium Data Dissemination databases and other services 2012-12-17Millennium Workshop1](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f265503460f94c3e238/html5/thumbnails/10.jpg)
Most recent data sets(@MPA)
• New simulations: – MR-XXL: Angulo etal 2012– Millennium WMAP7 (Thomas etal, MilliGas?)
• Updated semi-analytics, with (scaling to) WMAP7, MCMC– Guo etal 2011, 2012– Henriques etal 2012/3
• New synthetic light cone catalogues– Henriques etal 2012– also some all-sky (SDSS depth)
• Millennium Run Observatory– Overzier etal 2012, images+Sextractor catalogues
• Most loaded in new databases– soon: MR-XXL trees, Millennium WMAP7 (sim +scaled, halos + galaxies)
• New DB functions:– spatial queries– possibly: scaling, …
2012-12-17 Millennium Workshop 10
![Page 11: Millennium Data Dissemination databases and other services 2012-12-17Millennium Workshop1](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f265503460f94c3e238/html5/thumbnails/11.jpg)
Millennium Workshop 11
AllSky(Paritosh Verma)
2012-12-17
“Red”“Blue”
“All” “All”smoothed
![Page 12: Millennium Data Dissemination databases and other services 2012-12-17Millennium Workshop1](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f265503460f94c3e238/html5/thumbnails/12.jpg)
Most recent data sets(@MPA)
• New simulations: – MR-XXL: Angulo etal 2012– Millennium WMAP7 (Thomas etal, MilliGas?)
• Updated semi-analytics, with (scaling to) WMAP7, MCMC– Guo etal 2011, 2012– Henriques etal 2012/3
• New synthetic light cone catalogues– Henriques etal 2012– also some all-sky (SDSS depth)
• Millennium Run Observatory– Overzier etal 2012, images+Sextractor catalogues
• Most loaded in new databases– soon: MR-XXL trees, Millennium WMAP7 (sim +scaled, halos + galaxies)
• New DB functions:– spatial queries– possibly: scaling, …
2012-12-17 Millennium Workshop 12
![Page 13: Millennium Data Dissemination databases and other services 2012-12-17Millennium Workshop1](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f265503460f94c3e238/html5/thumbnails/13.jpg)
Spatial queries in the databaseSpatial3D library (with Tamas Budavari)
• Searches for objects in spatial subsets of simulation box is hard
• C# library for optimising such spatial queries using PH space filling curve
• Soon in Millennium DBs (needs documentation)
2012-12-17 Millennium Workshop 13
![Page 14: Millennium Data Dissemination databases and other services 2012-12-17Millennium Workshop1](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f265503460f94c3e238/html5/thumbnails/14.jpg)
with cover as ( -- some details left outselect * from dbo.fSimulationCoverSphere(
Simulation::Millennium,Sphere3D::newInstance(0,0,0,20),6) )
select f.* from cover c , millennium..fof f where f.snapnum=67 and f.phkey between c.keymin and keymax and c.shape.ContainsPoint(f.x, f.y, f.z)=1
14
![Page 15: Millennium Data Dissemination databases and other services 2012-12-17Millennium Workshop1](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f265503460f94c3e238/html5/thumbnails/15.jpg)
Query shape syntax
• Primitives :– box[ 0,0,0,10,10,10 ], sphere[10,10,10,20]– cone[0,0,0,1,1,1,.1,600] vertex,direction,angle,min-
depth,max-depth, – conesegment[0,0,0,1,1,1,.1,500,600]
vertex,direction,angle,min-depth,max-depth• Combinations :
– union/intersection/difference [ shape (, shape)* ]
15
![Page 16: Millennium Data Dissemination databases and other services 2012-12-17Millennium Workshop1](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f265503460f94c3e238/html5/thumbnails/16.jpg)
courtesy RaulAngulo
used in MXXL browser
![Page 17: Millennium Data Dissemination databases and other services 2012-12-17Millennium Workshop1](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f265503460f94c3e238/html5/thumbnails/17.jpg)
In the future: light cones in SQL(requires galaxy catalogues with dM/dz)
2012-12-17 Millennium Workshop 17
![Page 18: Millennium Data Dissemination databases and other services 2012-12-17Millennium Workshop1](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f265503460f94c3e238/html5/thumbnails/18.jpg)
Usage statistics• 589 registered users• > 17.3 million queries (7532/day)• > 287 billion rows (137 million/day)
queries,rows,users
2012-12-17 Millennium Workshop 18
![Page 19: Millennium Data Dissemination databases and other services 2012-12-17Millennium Workshop1](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f265503460f94c3e238/html5/thumbnails/19.jpg)
Distributions over users
2012-12-17 Millennium Workshop 19
![Page 20: Millennium Data Dissemination databases and other services 2012-12-17Millennium Workshop1](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f265503460f94c3e238/html5/thumbnails/20.jpg)
Millennium TAP
• New DB access service– All MyMillennium users have access
• Batch queue– longer timeout– capability to kill queries– history
• Implements IVOA’s Table Access Protocol (TAP)– AJAX frontend: TAPFace– TAP aware tools can query it, TOPCAT
• Some differences– schema vs database (Non SQLServer DBs can be served)– no mydb (yet)
• Demo this afternoon
2012-12-17 Millennium Workshop 20
![Page 21: Millennium Data Dissemination databases and other services 2012-12-17Millennium Workshop1](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f265503460f94c3e238/html5/thumbnails/21.jpg)
It’s not always as simple as it seems.One should try to understand the data when using the database
thanks to
Kevin Bundy, Tommaso Treu, Richard Ellis
http://astro.berkeley.edu/~kbundy/millennium/
2012-12-17 Millennium Workshop 21
![Page 22: Millennium Data Dissemination databases and other services 2012-12-17Millennium Workshop1](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f265503460f94c3e238/html5/thumbnails/22.jpg)
Other Millennium data dissemination services
• portal: http://galformod.mpa-garching.mpg.de
• Simulation service(s)– SeSam online (Neistein, Weinmann , Bourges)– L-Galaxies online (see talk by Matthias Egger)
• Image Browsers: – may help define where to point your virtual
telescope.– MR-XXL browser (public | interactive) (Angulo), – MRObs image browser (public | private)
(Overzier etal)
2012-12-17 Millennium Workshop 22
![Page 23: Millennium Data Dissemination databases and other services 2012-12-17Millennium Workshop1](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f265503460f94c3e238/html5/thumbnails/23.jpg)
Service vs other data sets
• Databases– HydroClusters database (Dolag)
• Virtual telescopes– Smac (Dolag, Reynecke, Bourges)– PHOX (Biffi etal)
• Image Browsers: – Phoxy browser + PHOX
2012-12-17 Millennium Workshop 23
![Page 24: Millennium Data Dissemination databases and other services 2012-12-17Millennium Workshop1](https://reader035.vdocuments.site/reader035/viewer/2022062721/56649f265503460f94c3e238/html5/thumbnails/24.jpg)
Thank you.
2012-12-17 Millennium Workshop 242012-12-17 Millennium Workshop 24