E. Matias and S. LomasFor the Project Team

BigBANGWIDTH CLS IBM University of Alberta University of Western Ontario

eScience – The CLS Case Study

Agenda

• What is eScience

• The Canadian Light Source Facility

• Traditional User Access Model

• CLS Remote Access Project

• User Configurable Light Paths

• Lessons Learned

• Conclusion

What is eScience?

• Providing a means for scientists to do better science, faster and at lower cost using advanced networking.

In addition to CLS remote access are there other examples?– SETI– Radio Telescopes– Neptune– Grid Computing….

Examples…. Neptune

Neptune @ University of Victoria• Seafloor observatory• Remote Telemetry System (800km ring)• Joint Canada/US Project

Examples …. Grid Computing

• What is a grid– Clusters of clusters linked together– Generally geographically distributed

• Some examples:– WestGrid (covering major universities in western

Canada is an example)– CERN - LHC (27km accelerator) is an example

of an experimental setup where no one county has sufficient processing capabilities

Examples …. LHC– LHC at CERN is an example of an grid application where

no one county has sufficient processing capabilities– 15 million gigabytes of data per year– In 2006 LHC Tier 1 Grid was tested– TRIUMF is a Canadian Tier 1 Centre for LHC Experiments

Courtesy TRIUMF

Examples….. WestGrid

WestGrid and WestGrid II

• $50M Project• Diverse cluster of machines and

resources for high-performance computing

Example Resources:• 1680 processor IBM eServer Blade

Centre• 256 processor HP XC Cluster• 144 processor HP AlphaServer• 20 – 4 processor HP ES40s• 256 processor SGI Origin • Two 64 processor IBM p595• Cray XD1 system• 56 processor IBM JS20

• 30 Terabyte disk storage• 360 Terabyte tape storage

• Partners Institutions:– University of Victoria– University of British Columbia– Simon Fraser University – University of Northern British Columbia– TRIUMF – University of Lethbridge – The Banff Centre – University of Calgary– University of Alberta– Athabasca University– University of Saskatchewan– University of Regina– University of Manitoba– University of Winnipeg– Brandon University

• Industry Partners:– HP, IBM and SGI

Agenda

• What is eScience

• The Canadian Light Source Facility

• Traditional User Access Model

• CLS Remote Access Project

• User Configuration Light Paths

• Lessons Learned

• Conclusion

Canada’s Synchrotron Facility

• Located on the University of Saskatchewan Campus

• One of the largest national science projects in a generation

• Accelerator Complex Consists of:– Linac Accelerator (200 MeV)– Booster (200 MeV to 2.9 GeV) Ring – Storage Ring (170.88 m - 2.9 GeV) – Two diagnostic beamlines

• Seven beamlines in operation or commissioning• Six beamlines under construction• Six beamlines being proposed (not-funded)

Machine Layout

Beamlines

• Operate in parallel harnessing synchrotron light generated by the circulating electron beam

• Each beamline is tailored to a class of experiments• Beam-teams of users provide scientific direction• Average beamline contains 50 to 150 motors• Data rates are different from beamline to beamline

• Operate in parallel harnessing synchrotron light generated by the circulating electron beam

• Each beamline is tailored to a class of experiments• Beam-teams of users provide scientific direction• Average beamline contains 50 to 150 motors• Data rates are different from beamline to beamline

CMCF 2 Beamline

Applications: Proteins & Disease

Applications: Medical Studies

Applications: Mining & Petroleum

Applications: Better Satellites, Planes and Cars

Applications: Nano-Technology

Agenda

• What is eScience

• The Canadian Light Source Facility

• Traditional User Access Model

• CLS Remote Access Project

• User Configurable Light Paths

• Lessons Learned

• Conclusion

Traditional User Access Model

• Academic Users– The user applies for beam-time once every six months– Peer review (Independent Committee)– Safety review (CLS)– Technical feasibility review (CLS) – Time scheduled in 8 hour slices usually multiple shifts

• Commercial Users– Straight fee-for-service arrangement without peer review

A good target for web based automation…..

Automation?

• This sounds like an electronic document management problem?

• Yes, it is.• Good target for outsourcing.• The selected platform:

– Cronus Technologies “C-Factor” and– Oracle.

• Hosted off-site by Cronus

Managed by our User Services Office

Next Project?

• This time gain the flexibility of SOA

• Look at developing the infrastructure for remote access and to interweave different users.

• We have three beamlines that by their nature are good targets for remote access.

Agenda

• What is eScience

• The Canadian Light Source Facility

• Traditional User Access Model

• CLS Remote Access Project

• User Configurable Light Paths

• Lessons Learned

• Conclusion

Project Terms of Reference

• Major project funding from CANARIE with additional funding from:– IBM– Canadian Light Source– Bigbangwidth– University of Western Ontario

• UP & UML software engineering approach• All custom developed software will be open source• Project duration 14 months• Project is completed before the target beamlines

– Therefore concentration is on reusable components

• Service Oriented Architectures and Web Services

Project Delivery Team• CANARIE Funded Project

• Canadian Light Source– Dionisio Medrano (System Analyst)*– Daron Chabot (System Analyst)*– Jason Chan (Intern)*– Elder Matias (Project Leader/Manager)– Michel Fodje (CMCF Beamline Scientist)– Renfei Feng (VESPERS Beamline Scientist)– Jason Cyrenne (Networking)– Bob Harvey (Networking)– Russ Berg (EPICS/CMCF)

• IBM Canada– Chris Armstrong (System Architect)*– John Haley (System Analyst/Architect)*

• University of Western Ontario– Marina Fuller (Requirements and Testing)*– Stewart McIntyre (User Champion VESPERS)– Gary Good (System Support)

• Alberta Synchrotron Institute– Ernst Bergman (User Champion CMCF)

• Big Bangwidth– Stuart Lomas (Networking)– Steve Hyatt (UCLP WebServices Software)

* Full Time

High Level Functional Requirements (Long Term)

Experiment Configuration- Control beam line- Manage experiments (add, edit, delete)- View experiment results/data- Save experiment results- Run experiments on simulated. Beam Line

Sample Management

- Enter sample- Track sample- Sample Management

User Office- View user details- Create new project- View current projects- View completed projects- Schedule projects- Login- Logout

Analyze and Process Results- Send result to be analyzed- Analyze results (ie. Sample scoring)- Process results/data

Sample Management

Kronos System- User accounts- Experiments accounts- Beam time schedule

Beam Lines

Experiment setup/results

External Systems- interface into other systems

Remote Access & Control

Protein Solvers

- Analyzed results (PX)

ELF

- Analyzed results (PX)

Create 3D Models

- Analyzed results (PX and Vespers)

Quartz

- Analyzed results (Vespers)

Australian Map Code

- Analyzed results (Vespers)

BigBangWidth

- Access to networking web service

Beamline Scheduling

- Schedule Beamline

Component Overview

• Challenge:– Support dynamic reconfiguration– Support on-line changes in a 24/7 environment– Provide flexibility and ease in reconfiguring the environment– Separation of meta data from presentation

• Solution– XML based configuration information instead of hard-coding

Ha

rdw

are C

on

trolle

rs

EPICSUser

Interfaces

Users Beam Line

NetworkMessageServices

UIServices

ExperimentResults

Performance Metrics and Diagnostic Logging

Metadata

Component Overview

• Challenge:– Web-standards are still immature,– Must have a real-time feel to the user– Diverse client hardware/software independently selected at each university

• Solution– Thin-client browser (Java Script)– AJAX used to provide real-time like interface with Spring Framework– Identified a single supported browser (Fire-fox) all others at users own risk

Ha

rdw

are C

on

trolle

rs

EPICSUser

Interfaces

Users Beam Line

NetworkMessageServices

UIServices

ExperimentResults

Performance Metrics and Diagnostic Logging

Metadata

Component Overview

• Challenge:– Requires secure data transfer– Real-time performance – Guaranteed Quality of Service– Users located at major research Universities, Institutes in Canada and Australia

• Solution– LightPath and LighPath Accelerator Technology– CANet4 with International connections

Ha

rdw

are C

on

trolle

rs

EPICSUser

Interfaces

Users Beam Line

NetworkMessageServices

UIServices

ExperimentResults

Performance Metrics and Diagnostic Logging

Metadata

Component Overview

• Challenge:– Robust Reliable– User performance requirements are unknown– Support on-line changes, since the Facility Operates 24/7 with limited outage periods– Common interface presented to the user

• Solution– Websphere Hosted– Provides Services for Managing Users and Presentation of Data to the User– Spring Framework and Custom Java Classes

Ha

rdw

are C

on

trolle

rs

EPICSUser

Interfaces

Users Beam Line

NetworkMessageServices

UIServices

ExperimentResults

Performance Metrics and Diagnostic Logging

Metadata

Component Overview

• Challenge:– The “glue” that ties things together– Ability to Interface to both Internal and External Services– Reliable, flexible, ability to deal with services connecting and disconnecting gracefully

• Solution– Provides internal and external services to communicate with other systems, analysis

codes etc.– Web-services for diverse and distributed services

Ha

rdw

are C

on

trolle

rs

EPICSUser

Interfaces

Users Beam Line

NetworkMessageServices

UIServices

ExperimentResults

Performance Metrics and Diagnostic Logging

Metadata

Component Overview

• Challenge:– Understand where the system fails– Achieve performance objectives– Allocate resources to performance only where there is a clear measurable benefit

• Solution– Build in some basic auditing to determine bottlenecks and trace faults

Ha

rdw

are C

on

trolle

rs

EPICSUser

Interfaces

Users Beam Line

NetworkMessageServices

UIServices

ExperimentResults

Performance Metrics and Diagnostic Logging

Metadata

Component Overview

• Challenge:– Several Terra bytes of data– Several megabytes per experiment– Provide good performance

• Solution– Storage Area Network (SAN)– Light-paths to permit the rapid transfer of data to the user home institution

Ha

rdw

are C

on

trolle

rs

EPICSUser

Interfaces

Users Beam Line

NetworkMessageServices

UIServices

ExperimentResults

Performance Metrics and Diagnostic Logging

Metadata

Component Overview

• Challenge:– Control diverse hardware– Implement motion control and data acquisition algorithms– Support both local and remote access

• Solution– EPICS (framework extensively used at synchrotrons around the world)– Integrate vendor and other libraries as needed

• Legacy System?

Ha

rdw

are C

on

trolle

rs

EPICSUser

Interfaces

Users Beam Line

NetworkMessageServices

UIServices

ExperimentResults

Performance Metrics and Diagnostic Logging

Metadata

EPICS Overview

Application Software

EDMMatlabSpecIDL

Labview

NetworkDeviceDrivers

ChannelAccessServer

Network

Channel

Access

Network

Hardwired DriverVME, Firewire

Serial DriveVME, Firewire

Scanner

StateMachineEngine

Hardw

are Co

ntrollers

Metadata

Fault Logging

EPICS Provides: Supervisory Control and Data Acquisition (SCADA) or (Distributed Control System) DCS functions

• Originally written by the US Department of Energy in 1980s• Widely used at large scale science facilities around the world

InterfaceRemote Access

EPICS

ProfibusTCP/IP

Siemens S7/300 PLC

ModbusTCP/IP

GPIB

RS-232

Channel A

ccess Protocol

IOC

IOC

IOC

IOC

State MachineEngine

CA

CA

CA

CA

CA

Single BoardComputer

CA

Operator WorkstationUser Applications

TouchPanels

CA

CA

Telemecanique Momentum PLC

VME

IOC

CA

Logical Component Diagram

ExperimentServices

BeamlineServices

BeamlineConfig. Data

XML/file

ExperimentMetadata

jdbc

Channel (Adaptor)

Other ControlSoftware

Channel (CAJ) Epicstcp

(various)

ExperimentData

file

Services

(various)

UI PresentationServices

PresentationConfig. Data

XML/file

BrowserUI

XML/HTTP(S)

(Adaptor)

CollaborationService

TBD

- analysis- 3rd party analysis- system services (e.g. LDAP)

DataComponent

FunctionalComponent

protocol

Dependency:

Layout

SamplePosition

Detector

Camera

Move Stop

Start Scan

Source

Off

CurrentZoom

Current X

Current H

Estimated Scan Time

Step Across (microns)

Y

Z

X

45º

H

Stop Scan

Exposure (seconds)

Number of Steps Across

Step Down (microns)

Number of Steps Down

Future X

Future H

Scan

On

Y is a function of H Z is a function of H

FutureZoom

CurrentFocus

FutureFocus

Height (microns)

Source Status

Remote Beamline Access Prototype Architecture

BrowserJavascript/Ajax

Netw

ork

Web Server

Other Services- SOAP, etc.

RDBMS

ApplicationLogic SOAP

Beamline - EPICS

J2EE - SPRING

restUI - jaxscript

Persistence - Spring DB

(DERBY)

Spring MVCEPICS Spring

Bridge

POJOs

Simplified Business Model for Prototype

-name-organizationName-startDate-attributes*-notes

Project

-person-role

ProjectRole

1*

-startDateTime-endDateTime-attributes*-notes

Experiment

1

*-name-id-shippedDate-receivedDate-attributes*-notes

Sample

-startCollectionTimestamp-endCollectionTimestamp-fileName-notes-stateValue

ExperimentData

*1

+setSamplePos()+setSlitSize()+setScanDuration()+startScan()+stopScan()

-stateAttributes-currentState

BeamlineControl

*1

*1

*

1

*

1

1

*

Application Screen Shots: Meta Data

Application Screen Shot: Experiment

Agenda

• The Canadian Light Source Facility

• Traditional User Access Model

• What is eScience

• CLS Remote Access Project

• User Configurable Light Paths

• Lessons Learned

• Conclusion

Network Architecture

LPA816e

LPA816e

LPA816e

LPA816e

LPA816e

LPA816e

LPA816e

LPA816e

LPA816eUBC

SFU

U of Lethbridge

U of Calgary

U of Alberta

UWO

U of Toronto

McGill

NRC-BRI

request

control

control

Control(UCLP)

CANARIEand ORAN

Lightpath Network

DomainManager

LPA816e

VESPERS Beamline Computer

CMCF Beamline Computer

Canadian Light SourceSaskatoon

Lightpath Accelerator network.

This solution allows any one research workstation, at right, to connect to one of the two beamline computers at the CLS.

This solution can be extended to allow multiple connections through each

LPA816e Lightpath Director.

October, 2006

BigBangwidth Lightpath Accelerator

BigBangwidth Lightpath Accelerator extends and automates Advanced Networking Lightpaths

LPA816e lightpath cross-connect

Software and hardware system

Extends Advanced Network Lightpaths across the LAN• “Last Mile Solution” for lightpaths

Interfaces to CANARIE UCLP• Provides an automated front end, no IT

Also works without UCLP, in LAN or WAN

This graph shows the traffic each second as a stream is recognized and moved from the LAN (blue) to a lightpath (red).

Lightpath Accelerator Operation

Lightpath Accelerator controls a software virtual cross-connect that commands UCLP.

Integrating LPA with UCLP

In effect, CA*Net4 is treated as a single lightpath cross-connect

real device real devicevirtual device

domain manager process

UCLPcommands

Agenda

• The Canadian Light Source Facility

• Traditional User Access Model

• What is eScience

• CLS Remote Access Project

• User Configurable Light Paths

• Lessons Learned

• Conclusion

Lessons Learned

• The technology is powerful but very complex• Web-services and the associated technology are

not mature this was a problem in this project– Standards are a moving target

• When evaluating frameworks, the products were evolving faster than the selection process

• Engaged consultants and collaborators with previous experience (IBM, and BigBangwidth in our case)

• Concepts are not completely new

The End

Thank you.