“Bookshelf.exe”: Executable Extensions of the Bookshelf

Marius Eriksen and Igor MarkovUniversity of Michigan, EECS

Outline• Making the bookshelf more dynamic

– Implementations + data = algorithm evaluations– Potentially more than that…

• Related efforts– SatEx, PUNCH, NEOS, OmniFlow

• Proposed solution: ``Bookshelf.exe’’– Services offered:

• online execution, ASP, sharing of results, flow scripting,…– Interfaces

• Support for optimization-specific concepts• Power versus ease-of-use and modularity

– Architecture and implementation details

Making the Bookshelf More Dynamic

• Existing bookshelf:– A static collection of data and implementations– Manual addition of material– No automatic evaluations, reporting of results

• Those limitations are not fundamental– Artifacts of the current implementation

• Proposal for improvement: “Bookshelf.exe”

Related Efforts • SatEx http://www.lri.fr/~simon/satex/satex.php3

– Specialized in satisfiability problems• PUNCH http://punch.ece.purdue.edu

– Very broad selection of software• From StarOffice and gnulot to Capo and Feng Shui

– Local to Purdue Univ.• NEOS http://www-neos.mcs.anl.gov/neos

– Open-source, distributed architecture– Used primarily for linear and non-linear optimization

• OmniFlow: DAC 2001, Brglez and Lavana– http://www.cbl.ncsu.edu/OpenProjects/OmniFlow– Distributed Collaborative Design Framework for VLSI– GUI-based flow control, chaining of design tools

SatEx• Continual evaluation and ranking of submitted

implementations– Results produced and posted on the Web automatically– Intuitive interface

• Popular– 93,707 hits since March, 2000– 23 SAT provers, 32,610 executions

• Limited scalability– Runs on one workstation – 529 days, 17 hours, 53 minutes and 20 seconds of CPU

time– Specialized to one application

PUNCH• Very general execution framework

– Everything from domain-specific algorithm implementations to GUI-based office applications

– Custom-designed filesystem (restricts to Purdue hosts)– 241,458 executions in 5 years

• 8,152 on VLSI CAD related applications– 20+ publications

• Only maintainer can add executables• No support for evaluation and chaining/flows

– No stats for results of runs (cf. “top 20” on SatEx )– No MIME-like data types (even IE/netscape need them!)

• Difficult to use when multiple tools are involved

NEOS• Open-source, distributed framework• Wide use, a solid library of implementations• Adding new implementations requires maintainer

intervention (but less than on PUNCH)– Each new implementation must come with a host– Distributed maintenance

• Loose data typing– No type system for data and implementations

• Compare to MIME

NEOS (what can be improved)

• Independent eval. and verification of results– E.g., PUNCH offers a WL-eval. From the bookshelf

• Real-time on-line reporting of results + stats• Provisions for chaining multiple

implementations– Scripts to control the execution and evaluation flows

• Provisions for pairing exec’s with benchmarks– Goal: SatEx-like evaluation,

but for multiple data types

OmniFlow• Context: collaborative VLSI Design

– collaboration in terms of computational resources

– no provisions for sharing results on the Web• Distributed over multiple hosts• Provides GUI-based flow control

– supports chaining of design tools– several hard-coded conditions for flow control– no support for execution conditional on results– no scripting language; limited by GUI

• Cannot dynamically add hosts

Bookshelf.exe• Best of SatEx, PUNCH, NEOS and OmniFlow

– Reporting style similar to SatEx (+ alternatives)– The versatility of PUNCH– Scalability and distributed nature of NEOS (or better)– Flow control as in OmniFlow or better

• New features– MIME-like data types and optimization-specific concepts – Automatic submission of binaries and source code– Chaining of implementations; scripting for flow control– Support for use models with proprietary data or code

Bookshelf.exe (2)• Scalability

– Computation is distributed (unlike in SatEx)– Maintenance is automated (unlike in NEOS)

• Support for multiple use models– “adapts to users”– Multiple levels of expertise– Multiple levels of commitment – Sharing of public data– Hiding/protection of proprietary data– “Screen-saver” mode, similar to SETI@Home,

Entropia, etc

Bookshelf.exe (3)

• Provides basic algorithm evaluation infrastructure to research groups – Avoid large maintenance costs– Just as usable for larger groups

End-user Interface

• Web site– Using CGI/PHP for interactive content and

database interaction– Submission through the web site (email is ok)– Dynamic addition of computation hosts– Stores data locally or uploads on demand– Automatic Web-based reports and statistics

End-user Interface (2)• User accounts (including anonymous)

– Every user has private resource-limited “workspace” – Job submissions must be from accounts– Results of jobs stored on accounts– Anonymous jobs possible, but can use fewer resources

• users are encouraged to register

• Many policy questions (no roadblocks seen)– e.g., can different owners chain their jobs?

• Transparent error diagnostics– Greatly improves learning curve

End-user Interface (3)

• Result notification (similar to NEOS)– E-Mail– Web reports

• Script composer for making run flows without needing to know a scripting language– less flexibility, but better learning curve– GUI implemented as form that POSTs scripts

Script Composer• HTML Form interface – basic flow with

fundamental predicates and functions– All API functions

• E.g. “run algorithm ‘optimizer 2’ and store the results”– Several common Perl functions– Conditional statements

• E.g. “verify if quality is more than 60”– Iteration support

• E.g. “iterate until quality is 100 or the number of iterations is 100”, “run randomized solver 3 timers, report the best result”

– Start with one ‘step’ – add an arbitrary number of steps

Optimization-specific Concepts in Bookhelf.exe

• Type system (similar to MIME) for all types of submissions

• Rules for matching submissions of different types– E.g., match a placer with a LEF/DEF benchmark

• Types include– Algorithm implementations

• Deterministic and randomized optimizers, etc.

– Input data, results– Common benchmarks

Architecture Overview

Component Types

• Where relevant, components also have types

• Data– Data format

• E.g. Cadence LEF/DEF, DIMACS CNF

• Implementations– Architecture (solaris, linux, ‘source code’, etc.)– Class (optimizer, randomized solver, etc.)

Component Descriptors• Unique id, name and description for each component• Owner field• Component history

– E.g. usage of a certain implementation, piece of data or benchmark

• Component specific fields– Associated expert for implementation, incremental diff for

data• Component version• Component changelog

Data Controller• Backing store

– Permanent storage of all involved components• Backed by database and file system

– Data, results suited for database– Implementations suited for file system storage

• Components may be deferred to local storage– Proprietary, non-disclosed components on a host owned

by the proprietor of the component• All data stored with complimentary descriptor

Job Controller• Responsible for communicating with and

controlling all agents (and thus jobs and implementations)

• Initiates jobs• Delivers user instructions• Handles data retrieval and storage

– Interfaces with data controller• Keeps track of node load, etc. for proper

distribution

Host Job Controller• Resides on each node in the distributed network

– Basic token of presence of DS-OSS• Reports statistics to job controller

– Load, memory usage• Signals when ready to take jobs• Launches agents as necessary• Performs self upgrades

– New software revisions, bugfixes

The Agent

• Responsible for running job on node• Provides environment for script to run in

– API– Script sandbox/jail

• For security reasons– Provides common callbacks if not provided by

scripts• Data storage/retrieval

The Agent (2)

• Input/Output of data from script• Job notification and control

– Start, stop, termination• Maintains local demands

– Resource usage– Stop on local demand

Scripts

• In Perl• Interfaces directly with an algorithm, or

expert handling it• API support

– Full Perl library– Job control functionality– Data storage and retrieval

Scripts (2)

• Checks for conflicting input data types and algorithm input type– Generates appropriate, understandable errors

• A number of automatic tasks– Pass thru data storage (e.g. results of a single

run)– Can provide all steps automatically, leaving the

user to do exactly what he/she wants

The Expert

• Wrappers for proprietary implementation interfaces– Knows how to deal with a particular interface

• E.g. command line options, configuration files– Translates from well known parameters

(specified from the API) to proprietary interface

In Conclusion

• DSS-OS provides features that are missing in current systems– Existing systems are limited by their

architecture and thus cannot provide these needed features.

• Provides an infrastructure that is scalable– Computationally scalable– Scalable to proprietary vendors

In Conclusion (2)

• Scripting support allows for flexible execution flows– Conditional flows, iterative flows

• Provides an established infrastructure for smaller research groups– Enjoying advantages that only large

proprietary, high-maintenance systems previously could provide.