the software infrastructure of seti @ home ii
Post on 22-Jan-2016
37 Views
Preview:
DESCRIPTION
TRANSCRIPT
The software infrastructure
of SETI@home II
David P. AndersonSpace Sciences Laboratory
U.C. Berkeley
Public-resource computing
Home PCsbusiness
academic
Advantages:• scale• free• growth• public education• no institutional policy issues
Challenges:• low bandwidth at client• costly bandwidth at server• firewall/NAT issues• sporadic connection• untrustworthy, insecure clients• server security• heterogeneity• must recruit participants
your computers
Achievements of SETI@home
• 1,000,000 years of CPU time in 3 years
• Sustained 30 TeraFLOPs• 1.5E21 floating-point operations• 3,600,000 users in 226 countries• 40 Terabytes of data processed• 3 billion “events” detected• Solved scaling, security problems
SETI@home II• Broadband pulse search on
existing data• Parkes observatory: Southern sky• Multi-beam receivers• Wider frequency band• Use KL transform• Data archival on clients
SETI@home software shortcomings
• Monolithic client and server• Limited communication model• Limited computation/data model• Ad hoc accounting model
PRC platform goals
Research lab X
University Y Public project Z
projects
applications
resource pool
• Participants install one program, select projects, specify constraints• Projects are autonomous• Advantages of a shared platform:
• Better instantaneous resource utilization• Better long-term resource utilization• Faster/cheaper for projects, software is better• Easier for projects to get participants• Participants learn more
Distributed computing platforms
• Academic and open-source– Globus– Cosm– XtremWeb– Jxta
• Commercial– Entropia– United Devices– Avaki
BOINC(Berkeley Open Infrastructure for Network
Computing)
• Overall structure• Storage model• Computation model• Programming interface• Operational interface• Participant’s view
Overall structure
• Project:
• Participant:
Scheduling server (C++)
BOINC DB(MySQL)
Project work manager
data server (HTTP)
App agentApp agent
App agent
data server (HTTP)data server
(HTTP)
Web interfaces
(PHP)
Core agent (C++)
lib
Storage model• Files: input, output, executables• Created by client or project• Files are immutable• File transfer by HTTP• File attributes:
– Name– URL list– Persistent– Upload-when-present– executable– MD5 checksum– Digital signature
<file_info> <name>protein_db.12</name> <persistent/> <url>http://a.b/c</url> <url>ftp://x.y/z</url> <md5_cksum>fw7398h</md_cksum> <nbytes>4782747</nbytes></file_info>
File management
• Implicit– Executables, input and output files are
transferred pursuant to computation• Explicit
– Clients report persistent files– Scheduling server maintains DB of
files on active hosts– Project can request upload, download,
delete
Workunits
<file_info> <name>out123</name> <url>http://…</url></file_info><workunit> <file_assoc> <file_name>out123</file_name> <app_name>input</app_name> </file_assoc></workunit>
• Represents inputs to a computation
• Components:– Cmdline args, environment vars– Expected resource usage– Description of input files
Results
<file_info> <name>out123</name> <generated_locally/> <upload_when_present/> <url>http://…</url></file_info><result> <file_assoc> <file_name>out123</file_name> <fd>1</fd> </ file_assoc ></result>
<file_info> <name>out123</name> <url>http://…</url> <md5_cksum>182aed847</md5_cksum></file_info>
• Represents results of a computation• Components:
– Which host did the computation– Exit status– Stderr output– CPU time– Output file description
• Template• Actual
Work sequences(long computations with big footprints)
• Results can be linked into sequences• Result is sent to host that handled
predecessor• If result times out, sequence is
shifted to another host
Upload state
Check for abort
Hosts and scheduling
• Host measurements– CPU performance (integer/FP/memory)– RAM, cache, disk free/total– On/idle/connected statistics– Network bandwidth statistics
• Workunit properties– RAM/disk/computation requirements
• Scheduling policy– Client: project quotas; high/low water marks– Server: workunit feasibility test; prioritization
Accounting and result validation
• Standardized unit of credit (CPeUro?)– CPU time * (int+FP+mem)
• Result validation (optional):– Compare redundant results, flag
incorrect results• Granted credit:
– Minimum of claimed credit among correct results
Programming interfaces
• Application– May be multi-file; any executable– API for interaction with core client (optional)– Checkpoint/restart: MFILE class– Graphics: render to shared memory
• Software development tools– Version management– Web-based bug tracking
Operational interfaces
• Operations– Add/manage app versions– Create workunits/results– Query results– Query client problems
• Interfaces– C++ libraries– Scriptable apps– Web-based
Participant preferences
• Examples:– Work only while computer idle– Confirm before connecting– Don’t work if running on batteries– High, low water marks– Limits on disk space, bandwidth– Application-specific preferences– List of projects + authenticators + % allocation
• Edited via Web interface• Can define multiple “preference sets”
Participation
• Initial project registration:– Create account on project web site– Authenticator is emailed– Install core client, enter authenticator
• Subsequent projects:– Create account on project web site– Authenticator is emailed– Add project to preferences on home
site
Core client• Goals
– Concurrent communicate/compute– Obey user preferences– Application, screensaver or service– Multi-platform; multiprocessor-
capable
• FSM structurefile transfers
running applications
wait()
active sockets
select()
HTTP transactions
main looppoll
Scheduler requests
Conclusion• BOINC features
– Multiproject, multi-app open PRC platform– Simple/small but general
• BOINC status– Mostly feature-complete– Client runs on Linux, Solaris, Windows,
MacOS X– http://boinc.sourceforge.net
• Projects:– SETI@home Arecibo (later this year)– Other SETI@home (Parkes etc.)– Climate modeling, other science projects– Genetic art
top related