DANSEDistributed Data Analysis for

Neutron Scattering Experiments

Michael M. McKerns, Michael A.G. Aivazis, Tim M. Kelley, June Kim, and Brent Fultz

Materials Science and Applied PhysicsCalifornia Institute of Technology

AbstractThe DANSE system will merge the various computational tasks of neutron scattering

into a unified, component based run-time environment. Standard components will implement data analysis, visualization, modeling, and instrument simulation for all areas of neutron scattering. A core technology of DANSE is an open source framework that supports the components and mediates their interactions. Within the DANSE environment, users will be able to mix and match different software components without compilation, and execute calculations seamlessly across distributed resources. DANSE will provide tools to help instrument scientists and expert users migrate their existing routines (written in any number of languages) to components, and an interface that will allow new and casual users to access a stock set of standard analysis applications or configure their own new computing procedures for novel experiments. The modular structure of DANSE parallels the steps of data analysis performed by scientists, thus making it a natural environment for creating flexible computing procedures. DANSE will lower barriers to sharing software, and extend the experimentalist’s toolkit with capabilities of analysis and interpretation such as high-performance simulations (band structure, molecular dynamics, etc.), co-analysis of data from multiple experiments, and real-time feedback for experimental control.

An introduction to DANSE• DANSE is a community organizing project with the potential to

provide a unique facility/user interaction:– a single environment for data analysis, visualization, modeling, and instrument

simulation for all areas of neutron scattering– a collaborative effort between software professionals, neutron scattering

scientists, and facilities– provides tools for remote collaboration and co-analysis– support from members of the international community and from the directors of

SNS, IPNS, HFIR/CNS, Lujan Center, NCNR– potentially the software environment for all instruments at the SNS

• DANSE provides a unified component-based runtime environment for computational neutron scattering:– open-source framework provides seamless use of distributed and high-

performance resources– a flexible, extensible, dynamic, interactive, cross-platform, cross-compiler,

object-oriented software architecture– integration of legacy codes and community-standard software– well suited for the development of new science, standard stock computation,

quality and plausibility assessment, and as a educational tool

Tools for each level of user

• Beginning student– user of prepackaged tools and documentation as a learning environment

• Visiting scientist– user of prepackaged and specialized analysis tools

• Instrument scientist– author of prepackaged specialized tools

• Analysis expert– author of analysis, modeling or simulation software

• Established researcher– collaboration coordinator, designer of new analysis procedures

• Software integrator– responsible for extending software with new technology

• Framework maintainer– responsible for maintaining and extending the DANSE infrastructure

Encourages Better Science

• More science from experiment execution– Single crystals on chopper spectrometers

– Feedback control for engineering diffraction

– Alter experiment depending on results:• visualization of science trends, not data trends e.g., see structure, not I(Q)

• on-demand modeling, ab-initio calculations

• reality checks against scattering theory

• Better science by planning experiments– Plausibility tests before submitting a proposal

– Assessment of sample plus instrument

– Contingency planning using prior simulations

– Assessments of trends in previous data

Facilitates New Science

• New science with better data analysis– FEM calculations of strains in microstructures

– Monte-Carlo inversions of S(Q,E) to obtain parameters of structure and dynamics models

– Model refinements with multiple data sets.

• New science by leveraging theory– VASP, CASTEP, ABINIT are commodities today; use them for assessing

structures and dynamics.

– Micromechanics – correlations of local strains

– Phase diagrams – thermodynamic functions

– Ab-initio calculations of spin interactions

– Soft matter structure – atomic force fields guided by diffraction

• Simulation and plausibility testing on virtual instruments

Ni

Pd

Pt

100

102

104

106

108

1010

1012

1014

1016

1018

1020

1022

Z (

for

10

0 m

od

es)

300250200150100500

Temperature [K]

1.6

1.5

1.4

1.3

1.2

1.1

1.0

Z (fo

r 1 m

ode)

Phonon Partition Function fcc Ni

for E,g_E in spectrum: Z *= one_osc(E,T) ** g_E

Built on the Pyre integration architecture• Pyre is a robust, stable foundation

– 75,000 lines of Python; 30,000 lines of C++– multiply leveraged DoE ASCI project

• Pyre is a software architecture:– a specification of the organization of the software system– a description of the crucial structural elements and their

interfaces– a specification for the possible collaborations of these

elements – a strategy for the composition of structural and

behavioral elements• Pyre is multi-layered

– flexibility– complexity management– robustness under evolutionary pressures

• Pyre is a component framework

application-general

application-specific

framework

computational engines

Component architecture

componentcomponent

bindingsbindings

librarylibrary

extension

componentcomponent

bindingsbindings

custom codecustom code

core

facilityfacility

framework

facilityfacility

facilityfacilityfacilityfacility

componentcomponent

bindingsbindings

custom codecustom code

service

requirementrequirement

implementationimplementation

packagepackage

• The integration framework is a set of co-operating abstract services

FORTRAN/C/C++FORTRAN/C/C++

pythonpython

ANL

LANL

NIST

ISIS

java

F77

IDL

Matlab

ISAW

GSAS

DAVE

Mslice

…

A Path for Software of Today

Finer-GrainedInteroperableComponents

NeXusReaderNeXusReader SelectorSelector

BckgrndBckgrnd

SelectorSelector

SelectorSelector

EnergyEnergy NeXusWriterNeXusWritertimestimes

instrument infoinstrument info

raw countsraw counts

filenamefilename

time intervaltime interval

energy binsenergy bins filenamefilename

Component dataflow

• Granularity allows reusability of object-oriented components

Component Templates Standard Data Streams

Python objects

Standard communication protocol between components that can reside anywhere

Data Flow Paradigm

histogramstables

meta-data

CodePlace

Name Place

Initiate, terminate, error

properties

'''Multiphonon.pyCalculates the multiphonon scattering, using a phonon DOS...'''from mpFunctions import *

def run(All_Inputs_List): """Multiphonon.py main loop...""" # check user inputs for validity, get data from disk checkUserInput(input_arglist) setup_arglist = setupRun(run_arglist)

# 1-phonon quantities, multiphonon terms single_arglist = onePhonon(arglist) multi_arglist = multiPhonon(N_arglist)

# prepare results for output, send to disk, etc. output_arglist = prepareResults(result_arglist) outputResults(output_arglist) return

if __name__ == '__main__': """Run main loop if launched standalone.""" from mpUserInput import * run(All_Inputs_List)

Encapsulation

Abstraction

Launched standalone orInside Analysis Procedure

Component implementation strategy• Write engine

– custom code, third party libraries– modularize by providing explicit support for life cycle management– implement handling of exceptional events

• Construct python bindings– select entry points to expose

• Integrate into framework– construct object oriented veneer– extend and leverage framework services

• Cast as a component– provide object that implements component interface– describe user configurable parameters– provide meta data that specify the IO port characteristics– code custom conversions from standard data streams into lower level data

structures

Flexibility through the use of scripting• Scripting enables us to

– organize large numbers of user tunable parameters– allow the runtime environment to discover new capabilities without the need

for recompilation or relinking– compose computations at runtime

• The interpretive environment:– Python is

• a modern object oriented language• robust, portable, mature, well supported, well documented• easily extensible• rapid application development

– has been extended to support for parallel programming– has no measurable impact on either performance or scalability

Encapsulating critical technologies• Extensibility

– new algorithms and analysis engines– technologies and infrastructure

• High end– visualization– easy access to large data sets

• single runs, backgrounds, archived data• metadata

– distributed computing– parallel computing

• Flexibility:– interactivity: web, GUI, scripts– must be able to do almost everything on a laptop

Data Analysis as a Distributed Service

• Data analysis is a service controlled by the user • User’s laptop issues commands and receives results • Computation is arranged by your client software

Support for distributed computing• We are in the process of migrating the existing support for distributed

processing into gsl, a new package that completely encapsulates the middleware

• Provide both user space and grid-enabled solution• User space:

– ssh, scp– pyre service factories and component management

• Web services– pyglobus

• Advanced features– dynamic discovery for optimized deployment– reservation system for computational resources

Fultz/Aivazis

Billinge

Strengthening the neutron community

Ustundag

Kienzle

Butler

Fultz/Trouw

3 SNS instruments on-line in 20062 IDT instruments

PROTONSPROTONS

Engineering Diffractometer – BL 9Engineering Diffractometer – BL 9

Areas for User and Instrument Support Areas for User and Instrument Support

SANS – BL 6SANS – BL 6

Cold Neutron Chopper Spectrometer – BL 5Cold Neutron Chopper Spectrometer – BL 5

Magnetism – BL 4a Liquids – BL 4b Reflectometers

Magnetism – BL 4a Liquids – BL 4b Reflectometers

High Pressure Diffractometer – BL 3High Pressure Diffractometer – BL 3

Backscattering Spectrometer – BL 2Backscattering Spectrometer – BL 2

Disordered Materials Diffractometer – BL 1bDisordered Materials Diffractometer – BL 1b

ARCS Spectrometer – BL 18ARCS Spectrometer – BL 18

High Resolution Chopper Spectrometer – BL 17High Resolution Chopper Spectrometer – BL 17

Single Crystal Diffractometer – BL 12Single Crystal Diffractometer – BL 12

Fundamental Physics Beamline – BL 13Fundamental Physics Beamline – BL 13

Powder Diffractometer – BL 11aPowder Diffractometer – BL 11a

Software needs to be on-line to support BL 2, 4a, 4b, 5, 18

Crystal model C1XX, C1XY…

Calculate force constant matrix Phi_{alpha beta}(0 l_ kappa kappa_)

Sweep reciprocal space

Calculate dynamical Matrix D(q)

Diagonalize D(q)

Update DOS histogram

Output DOS

Initial guess

Compare with experimental DOS

Powellminimize

n

y Ouput force

constants

End?

RMS Converged

?