asynchronous services from nvo roy williams conrad steenberg craig miller matthew graham joe jacob...
TRANSCRIPT
Asynchronous services from NVO
Roy Williams
Conrad Steenberg
Craig Miller
Matthew Graham
Joe Jacob
Julian Bunn
Why Nesssi?
• Graduated security• Anonymous, Registered, Known
• Multiple interfaces• Fat browser, Web proxy, Scripted
• Astrogrid pattern• Nesssi services can easily be Astrogrid services• Can be discovered by VO registry• Can be used as part of Astrogrid workflow
• Multiple implementations• cacr.caltech.edu and sdsc.teragrid.org
• Some useful services• Mosaic, Cutouts, Synoptic coaddition• Mashup of VO SIAP services
• Teragrid acceptance of security model• Server runs a job as somebody else• Anonymous access to TeraGrid!!
A “Graduated Security” Model
Web form - anonymous access, small jobsSome science....
Get NVO weak certificate - access logged, but identity not verified
More science....
Full TeraGrid account - browser accessBig-iron computing....
Scripted accessPower user
Portal-Based
Request and Certificate are judged *together*
"nesssi_strong_cert_max_time" : 216000,"nesssi_weak_cert_max_time" : 3600,"nesssi_anon_max_time" : 900,"nesssi_anon_user" : ”nvo","nesssi_weak_user" : "nvo",
Graduated security• No certificate gets 15 CPU-minutes from community account
• Just switch on Javascript or connect to portal
• Weak certificate gets 1 CPU-hour from community account
• In exchange for registering name/email• Can do scripted access
• Strong (gridmapped) certificate gets infinity from own account
• Get this one from TeraGrid HQ
Multiple Interfaces
• Fat Browser• https: and browser managed PKCS12 certs
• Thin Browser• Web Proxy works dynamically with cert authority
• Commandline with Python• No Globus install needed!
• Astrogrid• Application Description allows publish/discover/workflow
Fat Browser Portal
Browser nesssi
node
node
node
node
Teragridcluster
certificatepolicies
queue
select useraccount
sandboxstorage
JSON-RPC
certificate
open http
Certificate Authority
load certificate
Web Portal
client
certificateservices
nesssiweb portal
nesssi
node
node
node
node
web form SOAP http queue
fetchproxy
select useraccount
sandboxstorage
open http
certificatepolicies
Commandline Portal
client nesssi
node
node
node
node
Teragridcluster
certificatepolicies
queue
select useraccount
sandboxstorage
XML-RPC
proxy
open http
buildproxy
Certificate Authority
get certificate
Astrogrid
workflow
AG server
node
node
node
node
Grid cluster
queue
sandboxstorage
SOAPRequest/response
workflow
workflow
SessionID and Sandbox
• Identify which job we are talking about• 32 character hex string eg cb28d0753a7fec9a485981f741d425ec
• Used to monitor a running jobsessionID = remoteserver.cutout.init()msg = remoteserver.cutout.monitor(sessionID)
• Used to form URL where results appear, eg• http://dtf-test1.sdsc.teragrid.org:8080
/clarens/shell/cb/cb28d0753a7fec9a485981f741d425ec/cutouts/index.htm
• If you lose the sessionID, you lose your job
Mosaic Service
nesssiServer=nesssi.client('https://envoy.cacr.caltech.edu:8443/clarens/',debug=0)
mosaic_loc = "-ra 49.1 -dec 60.1 -rawidth 0.5 -decwidth 0.5 -filt f -bgcorr 0"
session = nesssiServer.dpossMosaic.mosaic(mosaic_loc)print "Your session ID is %s." % session
msg = dbsvr.dpossMosaic.monitor(session)print msg
Repeat the monitoring
Cutout Service
nesssiServer=nesssi.client('https://envoy.cacr.caltech.edu:8443/clarens/',debug=0)sessionID = nesssiServer.cutout.init()print "Session id is ", sessionID
# Upload locations filenesssiServer.upload_file(“inputfile.xml”, ”inputfile.xml”)
# Arguments for service, surveys to use and cutout sizeargs = "-surveys PQ:gr,PQ:gi,PQ:z1,PQ:z2,SDSS:r,SDSS:i,SDSS:z,2MASS:k,2MASS:h "args += "-size 64"
# Run servicenesssiServer.cutout.run(sessionID, args)
Making a Service
• Developer builds script• Keyword-value pairs on command line [+uploaded files]• Sandbox location given on cmdline -- all files staged there• Make index.htm in sandbox for progress• Make Nesssi connector for init(), upload(), run(), monitor()• Make Astrogrid service description
• Nesssi admin installs your service• Interview first• Symlink to code• Code is cached, restart server after edit• Developer gets right to restart server (running jobs not affected)
• Service instantiations farmed out to cluster with PBS
Server side code
Application example:mosaic.py -sandbox my-output-dir \
-ra 123 -dec 22.7 \ -rawidth 0.4 -decwidth 0.4 \ -survey SDSS:i -bgcorr 1It should:
(1) Use keyword-value arguments and uploaded files(2) Read/Write results in the given sandbox directory(3) Write a progress file in sandbox/index.htm(4) Estimate limits for anon/weak/strong certs
Astrogrid services are also driven by keyword-value pairs-- this is why every Nesssi service can be an Astrogrid service