cooking with sloan: special dr6 recipes jordan raddick, ani thakar alex szalay, maria...

Cooking with Sloan:Special DR6 Recipes

Jordan Raddick, Ani Thakar

Alex Szalay, Maria Nieto-Santisteban, Nolan Li,

Wil O’Mullane, Adrian Pope, Tamas Budavari, George Fekete,

Sam Carliles

JHU

Brian Yanny, Svetlana Lebedeva

FermiLab

Jim Gray

Microsoft Research

SDSS catalog data demo, March 30, 2007 2Jordan Raddick and Ani Thakar

Outline

• SDSS-II and DR6• CAS Data Access

– SkyServer, ImgCutout, CasJobs– Help resources and sample queries– Restricted collab access– SDSS and other datasets

• VO services• EPO content


SDSS-II

• Legacy– Continuation of SDSS-I (fill out 10k sq.deg.)– Completeness is same as for SDSS-I– Flux limits are the same – Target all galaxies with r_petro < 17.77, plus LRGs

• SEGUE– Detailed 3-d map of the Galaxy

• Spectra of 240,000 stars in the disk and spheroid• Age, composition and phase space distribution

– CAS component cataloged in SegueDRx DB

• Supernova Survey– Repeated scans of SDSS Southern Stripe over 3 mths/yr– Data not available in CAS yet, will be on DAS soon


SDSS Data Overview

Data Archive Server (DAS)Data Archive Server (DAS)FITS files (raw data)

Images, spectra, corrected frames, atlas images, binned images, masks

Online form-based accessRsync and wget file retrieval

Data Archive Server (DAS)Data Archive Server (DAS)FITS files (raw data)

Images, spectra, corrected frames, atlas images, binned images, masks

Online form-based accessRsync and wget file retrieval

Catalog Archive Server (CAS)Catalog Archive Server (CAS)Science parameters extracted to

catalogsStuffed into relational DBMS (SQL

Server)Heavily indexed, optimizedOnline access via SkyServer

Several levels of access, query tools

Catalog Archive Server (CAS)Catalog Archive Server (CAS)Science parameters extracted to

catalogsStuffed into relational DBMS (SQL

Server)Heavily indexed, optimizedOnline access via SkyServer

Several levels of access, query tools

SDSSData

Releasewww.sdss.org

das.sdss.org/DRx-cgi-bin/DAS

www.sdss.orgdas.sdss.org/DRx-cgi-bin/DAS

cas.sdss.org/collab/

skyserver.sdss.org

cas.sdss.org/collab/

skyserver.sdss.org


CAS Datasets (databases)

• BestDRx– Latest, greatest calibration of the data– Photometric and spectroscopic objects– The default and most accessed (by far) dataset

• TargDRx– The calibration from which spectroscopic targets

were chosen

• RUNS– All the runs (processings) other than Best, Target

• SegueDRx– New with DR6 (SDSS-II)


CAS Data AccessCAS Data Access

• SkyServer: http://cas.sdss.org/collabdr6– Also pwd access at http://cas.sdss.org/collabdr6pw

for non-collab IPs– See message sdss-archive/2935 for uname/pwd

• CasJobs– DR6 and DR6QA targets point to BestDR6 DB– TARGDR6, SEGUEDR6 targets– Need to have “collab” privilege set in user profile


CAS Data Access

• SkyServer– Synchronous access – need only web browser– Meant to support several levels of users– Visual tools to browse image and catalog data– API access, e.g. emacs interface, sqlcl (command-

line)– Strict limits on execution time and output size

(but higher for collaboration)


Navigating SkyServer

• Major sections linked from main page


Finding your favorite object

• Get position from SIMBAD



• Advanced Tools -> Navigate• Enter RA/Dec in DMS or HH:MM:SS



• Click to get position, magnitude• Notebook “shopping cart”



• Explore – full SDSS data for object• Chart – create finding chart


Searching the data

• Search for data that meets certain criteria• Return matching data, and only matching data• Example: search for galaxies

with spectra in Abell 168– RA = 18.84, Dec = 0.26,

radius = 3 arcmin– Limit to galaxies g < 18– Return RA, Dec,

magnitudes, redshift


Searching the Data

• Tools ->ImagingQuery

• Get data asHTML or CSV

• Constraintsby positionor imaging


SQL

• SQL – computer language for searching databases

• Advantages– You get longer queries and more results returned– You can write more complex queries– You can use command-line tools


SQL Concepts

• Data are stored in a database• Similar data types are stored in tables

– photoObj (photometry), specObj (spectroscopy), etc.

• A small part of the photoObj table:


SQL Concepts

• Within a table:– Horizontal rows are individual data points, or

records– Vertical columns are types of data, or columns

• A request to a database to return data is called a query

• Queries usually request data that meets certain constraints


SQL as a foreign language

• Languages have grammar and vocabulary• Dutch grammar

– In sub-clause, verb goes at the end

• English:– I think that astronomy is fun.

• Dutch:– Ik denk dat het sterrenkunde leuk is.


SQL Grammar

• Select – choose which columns of data you want to see

• From – choose the table(s) from which you want to retrieve data

• Where – set constraints on the search


Dutch vocabulary


SQL vocabulary

• Go to Help -> Schema Browser


Learning SQL

• Go to Help -> Sample SQL Queries• Go to Tools -> Search -> SQL• Copy, paste, modify


Visual search results

• Go to Advanced Tools ->Image List

• Use query to fill form

• Query must have:select x, ra, dec


CasJobs

• Go to Search Tools -> CasJobs• Best method for long or complex queries


CasJobs

• Register and log in

• Quick mode: 1 minute

• Submit: 24 hours

• Results stored in MyDB


CasJobs: other features

• MyDB database to save results of your queries– Define your own functions, procedures too– Share your tables with collaborators (groups)

• History of • Output data as FITS/CSV/VOTable• Plot data in browser• Table Import (upload) for your own data• Groups to share your results with collaborators• Command-line access Java tool also

downloadable


Collab-only access

• collabdrx SkyServer sites– http://cas.sdss.org/collabdrx– http://cas.sdss.org/collabdrxpw

• collab privilege in CasJobs– Restricted data, longer queues (DR6QA 24-hr

queue)– If you have collab set, you will see these queues– If you don’t have it, email [email protected]


SkyServer help

• Schema Browser

• FAQ• Glossary – of

SDSS terms• Algorithms• Helpdesk


CasJobs help

• Shares SQL IntroSQL Intro, Schema BrowserSchema Browser with SkyServer

• Has its own FAQFAQ page– Lists differences between CasJobs and SkyServer

due to distributed query execution

• Advanced CasJobs QueriesAdvanced CasJobs Queries page– Neighbor searches with fixed and variable search

radii– Cursors– Compound queries


Find the right tool for the job

• When you want to browse images, or when you know a position…

• ToolsToolsVisual ToolsVisual Tools– Find images and data: NavigateNavigate– Browse objects one at a time: ExploreExplore– Find objects on a frame, compare your own

observations: Finding ChartFinding Chart– View multiple objects or thumbnail search results:

Image ListImage List


Finding the right tool (contd.)

• When you don’t know specific objects…• ToolsToolsSearchSearch

– Find objects in a patch of sky: Radial or Rectangular– Simple search for imaging data: Imaging Query Imaging Query

FormForm– Simple search for spectral data: Spectro QuerySpectro Query form– More sophisticated searches: SQLSQL – Long, complex queries: Long, complex queries: CasJobsCasJobs


Finding the right tool (contd.)

• When you have a list of objects, and you want to see what SDSS knows about them…

• ToolsToolsObject CrossidObject Crossid– Imaging crossid: UploadUpload– Spectro crossid: SpecListSpecList


Data access: general tips

• Use collab or astro sites– Less “frills”, more direct access to tools– Some extra features– e.g. Imaging/Spectro Form Query not on public

page

• Each release has separate sites– http://cas.sdss.org/collab/ (the public release)– http://cas.sdss.org/collabdr6/ (not yet public)

• Use ContactContact link to contact help-desk (not direct E-mail)


CAS Dos and Don’ts

• Do notDo not submit a query unless you have some idea how long it will take!– It could tie up the server for hours (sometimes

days)!– Use “select count(*)” or “select top 10” instead

• Do use predefined views for commonly-used objects and parameters– PhotoObjPhotoObj, PhotoPrimaryPhotoPrimary for photometry– PhotoTagPhotoTag if you only need popular fields– SpecObjSpecObj for spectra


Educational Resources

• Extensive education content in SkyServer– Use ProjectsProjects link on the menu at the top– K-12 and college level student exercises and

teacher resources


Educational projects

• Go to public site – http://cas.sdss.org • Go to

Projects• Great as

labs for intro classes


Focused practice

• Get into groups of two• Share a computer• Consider some task you do often in research

or teaching– Some ideas in your handout

• Use SkyServer or CasJobs to accomplish the task

• We will answer questions, offer advice• You have 45 minutes… go!

cooking with sloan: special dr6 recipes jordan raddick, ani thakar alex szalay, maria...

Documents