Software Project Brent Fultz

California Institute of Technology

• ARCS Project• Intellectual Coherence• Distributed Computing• Tasks Underway• Coordination between Caltech and SNS

ARCS Spectrometer

Moderator

Shutter

Guides

Choppers

Sample

Detector Array Beamstop

ARCS Spectrometer

Headstock

Neck

Frets

Pickups

Volume/ToneSolidBody

Tailpiece(Whammy Optional)

ARCS Baseline Review • Hardware

- Detectors inside vacuum - Shielding

• Project Management - Project Execution Plan- Project Baseline Plan- Memorandum of Agreement (Caltech-SNS)- Budget Authority

• Software - Plan

Dates: Review: March Committee Report: May DOE Instructions: September ARCS Response: November

Preamp Prototype

Hardware Issue #2: Shielding Design

Effort on:

• Beamstop design

• Crosstalk between instruments

• Background from proton transport line and choppers

Project Schedule

Software ProjectPlan

Software Roadmap v. 1.0

Package Rating(-2 -1 0 +1 +2)

0 neutral-2 very negative+2 very positive

Strength of Opinion(0 1 2 3)0 none1 weak

2 average3 strong

ISAWGenieHomerLampTreat

Software Survey

.

.

. 10 responses, from serious users.very detailed comments.

User Ratings of Software Packages

ISAW Genie Homer Lamp Treat Mscat Mflcour Maxent

+0.5 +2.33 +2.5 -0.3 -3.5 0.0 -1.0 +1.3

Mslice Tobyfit Tobyplot Chop Gulp Unisoft+3.6 +3.0 +2.4 +3.2 +4.3 +2.0

McStas Vitess Ideas Matlab IDL

+4.0 +1.7 +1.3 +2.8 +1.4

User Opinion (–2 to +2)Strength of Opinion (0 to +3)Number of RespondentsRating = UO*SO/N

Software Survey Results

Interesting:CS: ISAW, Unisoft, Computational Crystallographic Toolkit User: ISAW, DAVE, Mslice, McStas, Gulp, Chop

Problems:1. Code Quality and User Opinion are

not necessarily correlated (McStas is an exception)2. We are still looking at details, but we can already see that there

will be more rewriting than expected

Issues in Code Development

• X hours to develop a code for yourself3 X hours to make it convenient for others9 X hours to make it run for others on several platforms

• Adding personnel to a late software project makes the project later. (Brooks Law)

Architectural Coherence

Left: Saint-Sernin, Toulouse, Romanesque Pilgrimage Church, c. 1080-1120

Right: Notre-Dame, Amiens, French Gothic Cathedral, begun 1220

Inconsistencies in Architecture

Cathedrals --- Charming :-)

Software --- Annoying :-(

Enforce Coherence -- Write the Manual Now

(Brooks, Mythical Man Month)

• Define needs and scope• Reference for developers• Identify and show interrelationships• Standardize notation• Documentation

Experimental Inelastic Neutron Scattering

Contents: Theory of neutron scattering Theory of excitations in condensed matter Data analysis procedures Software architecture Reference manual

Book Project Plan: We write it as we learn it LATEX Presently ~130 pages, on www Intellectual follow-on to diffraction textbook Publish electronically?

library.caltech.edu

Caltech Library System

Caltech Collection of Open Digital Archives (CODA)

Coda: Mus. A passage of more or less independent character introduced after the completion of the essential parts of a movement, so as to form a more definite and satisfactory conclusion. -- Oxford English Dictionary

Caltech CODA uses the Open Archives Initiative - Protocol for Metadata Harvesting (OAI-PMH). Archives in production are registered as OAI Data Providers. As registered Data Providers, they may participate in federated searching interfaces and other end user services offered by a growing number of OAI Service Providers. Several OAI Service Providers are listed below. 

Caltech CODA runs on freely available electronic archiving software. The Caltech ETD collection runs on ETD-db software developed at Virginia Tech; all other collections run on EPrints software developed at the University of Southampton.

In Production

Python Modern language with clear syntax

Open sourceObject-orientedInterpretedVery high level data types“Easy” interface to C++

C++ Industry standard language with cryptic syntax

Low-level control allows optimum machine performance Compiled and dangerous, but the only choice

XML Syntax for organizing data (does no work itself)

Java Some similarities to Python. The choice for browser applets. Java 1.4 integrates XML-RPC scripting.

Key Concept: Data Analysis as a Web Service

• Data analysis is a service controlled by a web browser

• Computation is arranged by the serveruser’s web browser issues commands and receives results

XML and the Network

User’s Web BrowserWeb Server

Database Server

Beowulf Cluster

• XML-RPC is an open standard for remote procedure calls• The user’s web browser issues commands to a server• The server distributes the work to appropriate computers

The User Interface

• Users navigate the web site with the left frame (content is generated dynamically by the web server).

• Users select data and calculational tools, and add them to the Java applet on the right

The User Interface (“Labview”-like)

• The user “wires” the boxes together to represent data flows

• We intend to allow users to insert new code in empty boxes, and archive sessions and procedures

Data Analysis Execution

• User hits “Run”

• Applet translates wiring diagram into XML-RPC commands

• Server receives commands, arranges Python script, starts data processing.

Data Analysis as a Web Service

• The server can provide access to the best combination of hardware and software

• Experimental data and analysis codes reside on the servers, so little data bandwidth is needed

• Platform independent (a huge savings in effort!)• Computing resources can be changed without affecting

the user• Clean separation of GUI from analysis code• One web portal for all SNS instruments?

Software Project

• Decisions MadePython (for users and instrument scientists)C++ (core modules) Web service using remote procedure calls

• Immediate Tasks to Test ArchitectureDemonstration web portal (Java/XML - XML/Python)Rebinning software (Python/C++)Lattice dynamics (Python/C++)

• Contingencies minimized after we write some test modules

• A baseline plan is possible by January, 2003.

Software Roadmap v. 1.0

Born - von Kármán Lattice Dynamics

Simplicity: Complexity:

1. Undergrowth of indices for tensor quantities:

2. Crystal structure:

€

mκ ˙ ̇ u κl = − Vκ ,l

∑ κl,κ ' l 'uκ ' l '

€

uκl

€

V κl ,κ ' l '

lattice, l, basis, allowed elements depend on symmetry

J. M. Ziman, Electrons and Phonons

Data Rebinning

• Account for incident flux• Remove background• Convert from time to energy• Correct for detector efficiency• Bin into rings of constant scattering angle• Convert from angle to momentum • Subtract multiphonon and multiple scattering• Correct for absorption

Data Rebinning – Tim’s Test

Tested rebin procedure in several forms.

• 300 seconds with IDL • 60 seconds with IDL using a DLL compiled from C++• 2 seconds with C++ “Rebinner” class

C++PythonC++PythonC++PythonC++Python

First-Order Approx. for SNS – Caltech Interaction

• Universities codify science

• National Labs generate science

Detectors Dispersions

SNS Caltech

?

Model Fails in Second Order

Project Phase of SNS• User interface requires tremendous care from SNS and Caltech • Common standards and documentation• Analysis software and hardware now at Caltech, but must migrate to SNS, ORNL resources

Operations Phase of SNS• Role of Caltech and broad community in future software maintenance or development?

Electronic Publishing

server

user

documentation

library

instrumentand data

?

?

codes

?

Discussion on SNS - Caltech Interactions

• Coordination of software projects at Caltech and SNS? Scope and scope management.

• Expectations of users for software support by SNS and instrument scientists.

• Consistency of GUI for SNS instruments? Single web portal?• Who maintains the code? • Standards for maintainable code with "open source coalition"?• Issues with distributed computing. Lab policies, security, graphics, user

permissions. • Issues with releasing software to run on users' machines • Status of storage and archiving of raw data by SNS?• Institutional arrangements with the ORNL supercomputing center?• On-line control of a neutron spectrometer. Technical and policy issues.

SNS - Caltech Interactions

• Communication is the key. Thanks for coming!

• Please visit the ARCS wiki• http://viz.cacr.caltech.edu:8000/arcs/1

Agenda8:00 AM Breakfast or coffee 8:30 AM Introductions 8:40 AM Overview (Brent Fultz)9:20 AM Directions (Michael Aivazis)10:10 AM Break10:30 AM Demo of web portal for data analysis

(Jonny Lin, Tabitha Swan-Wood)10:50 AM Physical concepts as software classes (Tim Kelley)

12:00 Lunch at the Athenaeum (hosted)

Discussionwho …what …why …when … howNeXus classes for analysis classes (Ray Osborn)

6 PM Heritage Wines?7 PM Dinner at Cafe Santorini (no host)