The Boulder AreaTeragrid (BAT)“Step up to the Plate”

Marla Meehl

Peter O’Neil

Jim Van Dyke

Outline

Motivations for connecting to the Teragrid

The pyramid and hierarchies of networkingCENIC (California) and I-Wire (Illinois)

NSF ANIR’s hierarchy for funding

Distributed Terascale Facility vision

Pacific Light Rail/Teragrid access to DTF

Applications and benefits

Costs and potential funding partners

Motivations

Make UCAR/NCAR, NOAA/NIST, and CU-Boulder (BAT) facilities (supercomputers, mass storage, visualization, models, datasets, and networking technologies,) available to the Teragrid community

Enable advanced scientific application services

Investment in infrastructure to position the BAT for emerging federal program funding

Motivations

Initiate the BAT collaboration in national and international “experimental” grid networking

Accelerate exchange of technical expertise

Natural extension of BRAN

Prudent and scalable path for long term

Political clout of being “on” the Teragrid

PITAC Report ConsiderationsPresident's Information Technology Advisory Committee (PITAC)

NSF Centers 1 (no more) generation behind ASCI

Support for massive petabyte databases

Near futureBandwidth across country as fast if not faster than bandwidth within centers

10GHz processor’s by end of decade for petaflop computing

LSN projects and funding driven by PITAC report(s)

NSF Review of NCAR/SCD

“…advanced networking capabilities will be required to make this data available to the community that NCAR serves.”“NCAR must lead the way to this future.”

be a “player” on the national stage - not follow or lag behind“SCD will need to increase its investment in research – research carefully directed at the most critical problems faced by the computational atmospheric sciences community.”“The panel recognizes that the provision of computing, datasets, data analysis, and networking services to the university community is central to the mission of SCD.”“NCAR must continue to develop its networking capabilities, otherwise, it will not be able to serve its proper role as a data repository for the atmospheric sciences community.”

Our Challenge“The world has seemingly grown smaller through astonishing advances in telecommunications, but we have, at the same time, a vastly greater appreciation of the complexity and interrelatedness of the physical and human spheres that form the coupled Earth system. NSF now uses new terms such as ‘planetary metabolism’ and planetary ecology’ to capture the need to think in a more integrated sense about humanity’s relationship with the natural world. These scientific challenges are indeed grand.”

NCAR Strategic Plan - 2001

SCD’s Challenge

“New to SCD is a focus on using this [intellectual] leadership as a transformational engine for NCAR and its community through convergence of elements of the information technology revolution, such as collaborative environments and connect to NSF’s Teragrid of distributed computing and data services.”

NCAR Strategic Plan - 2001

CENIC Pyramid

Future Networks Operational High Performance (Production) Networks

Experimental Infrastructure Networks

Research Networks

Networking – ANIR/NSF

High Performance (Production) Networks

Abilene, vBNS+, FedNets (ESnet, DREN, NREN) Essential Tool for Research and Education

Always available and dependable 24/7

High Performance

International Connectivity

Exciting Future

NSF support for focused Activities e.g. middleware, measurement initiatives, network simulation

Networking – ANIR/NSF

Experimental Networks – PLR/NLR, I-Wire, TeragridHigh performance trials of cutting-edge networks

Based on advanced application needs unsupported by existing production network services

Ultra High Speed – one or more 10-40 Gig waves

Robust enough to support application-dictated development of application software toolkits, middleware,

computing, and networking

Provide delivered experimental services on a persistent basis, yet encourage experimentation with innovative and

novel concepts

International Connectivity

Networking – ANIR/NSF

Research Net – Point to Point Waves, DTFExperiment with Disruptive Technologies

Design of Experiments

Implementation of Experiments

Evaluation of Results

Smaller-scale network prototypes which enable basic scientific and engineering network research and testing of component technologies, protocols, and network architectures

Networking – ANIR/NSF

“The Network is the Supercomputer”

Distributed/Extended Machine Rooms (MR2MR)

DTFDTF

12/5/01

Critical Mass SitesTop 10 Res. Univ.:Next 15 Res. Univ:Key Centers, Labs:

Intl. 10gig &

DTF

Increasingly with the broadband & even private waves – fiber needed for e2e experimental/developmental networks

draft 12/4/01

Critical Mass SitesTop 10 Res. Univ.:Next 15 Res. Univ:Centers, Labs:Intl. 10gig & Key Hubs

• Leverage Regional Connections• Incent fatter/dedicated pipes• Enable significant e2e• Connect Scientists/Labs/Devices• Establish Tera/MetaPop Centers

Enabling New Class of Applications

Data intensive computingCollaboration technologyDistance visualizationWorkflow management and collaborative problem solving environmentsManagement of large-scale, distributed, multi-institutional systems, e.g., GridSensornetHierarchical data delivery

Applications

TurbulenceBig Data

1000 year climate data

Earth Systems GridData Portals

Atmospheric reanalysis

Windows to the UniverseNSF Cyber-infrastructure – eScience

NCAR is being asked to be a leader here

Doppler radar networks

What If DTF Fails?The data, network, and application focused research and development could still make the Teragrid a success

The BAT could be a big part of this success

Custom networks (tailor-made)

Efficient networks that are geographically placed

Data repositories

Agile optical transport networks

Enhancing and scaling networks

Benefit to Non-Teragrid BAT Users

Benefits all users if we can access big datasets on the NCAR Mass Storage System and other storage systemsPre-positioning to be highly competitive in e-Science funding environmentAccess to broader data repositoriesTestbeds for refining standards and technology in a limited environment for application dictated services on a broader basis (in cooperation with private sector)Positions Colorado for state-level advanced networking efforts

Additional Benefits to UCAR Universities of Experimental Network “The pursuit of knowledge drives a researcher’s experimental design, which in turn, determines the scientific resources required, which then drives the information resources and services required. Or, that is how it should be, from an application’s point of view. A major complaint from application scientists is that historic funding mechanisms for FedNets create the opposite order, whereby networks define the limits of the applications. Furthermore, end-to-end requirements have not been addressed; the problem of routinely getting from science machines in the sites/campuses to the high-performance wide-area network is unsolved.”

NSF CISE Grand Challenges in e-Science Workshop Report 2002

Experimental Networks to Incubate Paradigm Shift“Networks should be described as collections of application services rather than by their circuits, their theoretical bandwidth or their architectures, and experimental networks are the only likely means for incubating this paradigm shift…e-Science developers care only about services delivered at the application level – such as observed data transfer rates, video frame rates, reliable multicasting, and inter-organizational security and authentication capabilities. Delivery of application services requires a vertical integration effort – from the network infrastructure level all the way to the application layer, requiring a ‘paradigm shift’ in the way the Nation thinks about high-performance networks.”

NSF CISE Grand Challenges in e-Science Workshop Report 2002

Cost Sharing and Funding Opportunities

UCAR/NCAR, NOAA/NIST, and CU Boulder cost share for experimental/ research effortsNSF ANIR and ATM support for NCAR inter-connectivity to Teragrid

Year 3 Teragrid funds ($37.5M)January solicitation for TeraPoPs as core nodes of “experimental” network infrastructure for optical application transferMarch solicitation for experimental network research

Potential DOE funding

Funding Criteria

Setting priorities between areas of research ultimately requires tradeoffs between different goals, quality of life, and expanding the frontiers of human knowledge and understanding

The allocation of funds to research is primarily a political processPromote tools, technologies, or facilities that can accelerate the pace of discovery in geosciencesEnable cyber-infrastructure for integrated and interdisciplinary studies

• CRA Testimony to Senate Advisory Committee

Budget Assumptions

Two cost options

Includes cost of spare equipment

Boulder fiber interconnect costs undetermined

No direct attachment to the DTF

Teragrid/Light Rail Cost Summary – Worst Case

Item NCAR Only One-Time

NCAR Recurring Shared One-Time

Shared Recurring

Denver Node $1M (10 yr IRU) $200K $333K $67K

Juniper Router $500K (3 yr life) $50K $167K $17K

Light Rail Sub-Total

$1.5M $250K $500K $84K

Boulder-Denver Dark Fiber

$500K (10 yr IRU) $75K $167K $25K

DWDM Equipment

$1M (3 yr life) $100K $333K $33K

NCAR Juniper Router

$1M (3 yr life) $100K $333K $33K

Boulder-Denver Sub-Total

$2.5M $275K $833K $91K

TOTAL $4M $525K $1.33M $175K

Teragrid/Light Rail Cost Summary –Current Realistic Estimate

Item NCAR Only One-Time

NCAR Recurring

Shared One-Time

Shared Recurring

Denver Node $1M (10 yr IRU) $200K $333K $67K

10 GigE Switch $200K (3 yr life) $20K $67K $7K

Light Rail Sub-Total

$1.2M $220K $400K $74K

Boulder-Denver Dark Fiber

$150K (10 yr IRU)

$30K $50K $10K

10 GigE Switch $800K (3 yr life) $80K $267K $27K

Boulder-Denver Sub-Total

$950K $110K $317K $37K

TOTAL $2.15M $330K $717K $110K

Other Cost ConsiderationsSavings

UCAR/NCAR/NOAA FRGP OC12 = $100,000/year

CU-Boulder FRGP/4-campus OC12 = $100,000/year

Summary

Facilitates/defines the BAT national presenceParticipation strongly encouraged by SDSC, NCSA, PSC, NSF, and Light Rail Critical to ensure long term funding opportunitiesValuable long term investment

incremental cost of network upgrades minimized

“We can’t afford not to participate”