scaling nvo services to the teragrid roy williams conrad steenberg matthew graham joe jacob ray...
TRANSCRIPT
![Page 1: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/1.jpg)
Scaling NVO Services to the Teragrid
Roy Williams
Conrad Steenberg
Matthew Graham
Joe Jacob
Ray Plante
![Page 2: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/2.jpg)
NESSSINVO Extensible Secure Scalable Service Infrastructure
• Services are science-oriented• Services are made by trusted developers from the
science community• Web forms OR command line (Python API)• Built-in security (X.509 certificates)• Very large jobs can be run• Easy to get a certificate• No complex install needed by client• Different levels of certificate get different service• Is installed on Teragrid• Services can be part of a workflow
![Page 3: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/3.jpg)
Desired Characteristics of NVO Services• Service oriented architecture
• Services should be easily and quickly deployable and usable on workstations or supercomputers
• Services deployed, managed, and upgraded by their developers• Service developers/deployers are trusted users
• Service developer acts as a broker between computing customer and computer center
• Service users authenticated with “graduated security”• Easy to start, but great power is possible
• Asynchrony for compute intensive jobs• Jobs submitted to batch queue• Unique sessionID may be used to monitor job & return results
• From “clicking” to “scripting”• Services may be accessed by clicking on a web page or with scripted client codes• Authentication for web clicking comes from a certificate store• Scripted access requires a certificate (strong or weak) straight from the client
• Services as workflow components• A service user may be another service (a computer, not a human!)
![Page 4: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/4.jpg)
A “Graduated Security” Model
Web form - anonymous access, small jobsSome science....
Get NVO weak certificate - access logged, but identity not verified
More science....
Full TeraGrid account - browser accessBig-iron computing....
Scripted accessPower user
Portal-Based
![Page 5: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/5.jpg)
Traditional Grid Security
client
Show us your Certificate!I will do exactly what you want.
![Page 6: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/6.jpg)
Graduated Security
clientMay I have your Request and your Certificate?
![Page 7: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/7.jpg)
This is a US driver’s licence. In the US it proves identity strongly. It is like a strong certificate.
This is a loyalty card where I buy food.(You can put a false address on the application.)It is like a weak certificate.
This is a $50 gift card at a bookstore.It does not prove my identity in any way.It is like an anonymous certificate.
CertificatesThe Virtual Observatory as a Virtual Organization
![Page 8: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/8.jpg)
service implementationweb formspython APIgraduated security
certificatescertificate chainsroot certificatesproxy certificatesproxy certificate chains2nd level proxy chainsxformssecure https redirectionteragrid security policecaltech security policeNCSA security policechown directory ownershipNFS root-squashingPBS stdout permissionspubcookie
![Page 9: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/9.jpg)
A proxy is a copy of a certificate with a 24-hour expiry date
It is safer than sending the full certificate.
A proxy can come from a certificate storereleased by username/password
A proxy can be built with a local tool eg nesssi_proxy_init or globus_proxy_init
Proxy Certificates
![Page 10: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/10.jpg)
Web Portal
client
certificaterepository
nesssiweb portal
nesssi
node
node
node
node
web form SOAP http queue
fetchproxy
select useraccount
sandboxstorage
open http
certificatepolicies
![Page 11: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/11.jpg)
Commandline Portal
client nesssi
node
node
node
node
Teragridcluster
certificatepolicies
queue
select useraccount
sandboxstorage
Secure SOAP
certificate
open http
buildproxy
![Page 12: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/12.jpg)
Exercise: Running a Nesssi Service
see http://us-vo.org/nesssi
![Page 13: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/13.jpg)
The NVO Certificate Authority
The NVO now has a certificate authority
![Page 14: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/14.jpg)
Getting an NVO login
![Page 15: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/15.jpg)
The Web Portal
![Page 16: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/16.jpg)
Getting a proxy certificate
% cd $NVOSS_HOME% source bin/setup.csh [snip]All set up for the 2006 NVO Summer School.% cd nesssi% java NesssiInit YourUsername YourPassword /tmp/x509up_u501% ls -l /tmp/x*-rw------- 1 roy wheel 2231 Sep 1 12:40 /tmp/x509up_u501
web portal
command line
is this your UID?
![Page 17: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/17.jpg)
SessionID and Sandbox
• Identify which job we are talking about• 32 character hex string eg cb28d0753a7fec9a485981f741d425ec
• Used to monitor a running jobsessionID = nesssiServer.cutout.init()msg = server.cutout.monitor(sessionID)
• Used to form URL where results appear, eg• http://dtf-test1.sdsc.teragrid.org:8080
/clarens/shell/cb/cb28d0753a7fec9a485981f741d425ec/cutouts/index.html
• If you lose the sessionID, you lose your job
![Page 18: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/18.jpg)
<NesssiMonitor>
<Service>Cutout</Service>
<Uname>ux400560</Uname>
<SessionID>774daf5ef52facc68cb03db4b1fdc815</SessionID>
<Sandbox>http://dtf-test1.sdsc.teragrid.org:8080/clarens/shell/77/774daf5ef52facc68cb03db4b1fdc815</Sandbox>
<Result>http://dtf-test1.sdsc.teragrid.org:8080/clarens/shell/77/774daf5ef52facc68cb03db4b1fdc815/cutouts/index.html</Result>
<QueueStatus>149.envoy.cacr.calte roy batch C8845cb 11516 1 -- -- 60:00 R --
</QueueStatus></NesssiMonitor>
Monitoring a Nesssi job
service name
running as this user
session ID
sandbox URL
results URL
queue status(R = running)
![Page 19: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/19.jpg)
Example: SleepyAdd
nesssiServer=nesssi.client('https://dtf-test1.sdsc.teragrid.org:8443/clarens/',debug=0)# nesssiServer=nesssi.client('https://dtf-test1.sdsc.teragrid.org:8443/clarens/',debug=0)
sessionID = nesssiServer.sleepyadd.init()print "Your session ID is", sessionID
# Run: sleep 30 seconds then add 52 and 344nesssiServer.sleepyadd.run(sessionID, "-time 30 -n 52 -m 344")
web portal
command line
![Page 20: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/20.jpg)
Monitoring the Run
Key n is 52Key m is 344Key time is 30Sleeping for 30 secondsWaking up...Sum of 52 and 344 is 396
<NesssiMonitor><Service>Sleepyadd</Service><Uname>ux400560</Uname><SessionID>a3a167a383111c0cbd6941325b8659aa</SessionID><Result>http://dtf-test1.sdsc.teragrid.org:8080/clarens/shell/a3/a3a167a383111c0cbd6941325b8659aa/batch.out</Result><Sandbox>http://dtf-test1.sdsc.teragrid.org:8080/clarens/shell/a3/a3a167a383111c0cbd6941325b8659aa</Sandbox><QueueStatus>305875.dtf-mgmt1.sds ux400560 dque Ca3a167 -- 1 -- -- 18:00 Q --</QueueStatus></NesssiMonitor>
![Page 21: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/21.jpg)
Mosaic Service
nesssiServer=nesssi.client('https://envoy.cacr.caltech.edu:8443/clarens/',debug=0)
mosaic_loc = "-ra 49.1 -dec 60.1 -rawidth 0.5 -decwidth 0.5 -filt f -bgcorr 0"
session = nesssiServer.dpossMosaic.mosaic(mosaic_loc)print "Your session ID is %s." % session
msg = dbsvr.dpossMosaic.monitor(session)print msg
![Page 22: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/22.jpg)
nesssiServer.dpossMosaic.mosaic (“-ra 49.1 -dec 60.1 -rawidth 0.5 -decwidth 0.5 -filt f -bgcorr 0”)
![Page 23: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/23.jpg)
Coadd Service
nesssiServer=nesssi.client('https://envoy.cacr.caltech.edu:8443/clarens/',debug=0)
# Initialize the servicesessionID = nesssiServer.hyperatlas.init()print "Session id is ", sessionID
# Arguments for service, the coaddition to doargs = "-bandpass z1 -ra 170.08 -dec 13.275 -rawidth 1.0 -decwidth 1.0"
![Page 24: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/24.jpg)
-bandpass z1 -ra 170.08 -dec 13.275 -rawidth 1.0 -decwidth 1.0
![Page 25: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/25.jpg)
Cutout Service
nesssiServer=nesssi.client('https://envoy.cacr.caltech.edu:8443/clarens/',debug=0)sessionID = nesssiServer.cutout.init()print "Session id is ", sessionID
# Upload locations fileremoteinputfile = "/shell/%2s/%s/inputfile.xml" % (sessionID[0:2], sessionID)nesssiServer.upload_file(inputfile, remoteinputfile)
# Arguments for service, surveys to use and cutout sizeargs = "-surveys PQ:gr,PQ:gi,PQ:z1,PQ:z2,SDSS:r,SDSS:i,SDSS:z,2MASS:k,2MASS:h "args += "-size 64"
# Run servicenesssiServer.cutout.run(sessionID, args)
![Page 26: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/26.jpg)
Cutout Monitoring
![Page 27: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/27.jpg)
cutouts from Palomar-Quest, SDSS, 2MASSof sources from Veron quasar catalog
![Page 28: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/28.jpg)
![Page 29: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Matthew Graham Joe Jacob Ray Plante](https://reader036.vdocuments.site/reader036/viewer/2022062511/5514d7c555034693478b52f9/html5/thumbnails/29.jpg)