05NERSC2005 Annual Report

National Energy ResearchScientific Computing Center

05NERSC2005 Annual Report

National Energy ResearchScientific Computing Center

Lawrence Berkeley National Laboratory / 1 Cyclotron Road / Berkeley CA / 94720

LBNL-60296

The Year in Perspective 2

Research News 4

The Heat Is On 5

Burning Questions 9

Combustion Up Close 11

Hailstones in Hell 13

A Perfect Liquid 15

Whispers from Underground 18

Breaking Up Is Hard to Calculate 20

Talent Scouting 23

Surface Charge 25

Magnetic Disks in Space 26

Proteins in Motion 30

NERSC Users Honored 34

The NERSC Center 35

Science-Driven Computing 36

DOE Greenbook Published 36

NERSC’s Five-Year Plan 36

DOE Review of NERSC 37

Organizational Changes 38

Science-Driven Systems 40

Two New Clusters: Jacquard and Bassi 40

New Visual Analytics Server:DaVinci 42

NERSC Global Filesystem 42

Integrating HPSS into Grids 44

Bay Area MAN Inaugurated 45

Another Checkpoint/RestartMilestone 46

Science-Driven SystemArchitecture 46

Proposals for New SystemEvaluated 47

SC05 Conference Honor 47

Science-Driven Services 48

Support for Large-Scale Projects 48

Archiving Strategies for Genome Researchers 49

User Survey Provides ValuableFeedback 50

Science-Driven Analytics 51

Appendix A: NERSC Policy Board 54

Appendix B: NERSC Client Statistics 55

Appendix C: NERSC Users Group Executive Committee 56

Appendix D: Supercomputing Allocations Committee 57

Appendix E: Office of Advanced ScientificComputing Research 58

Appendix F: Advanced Scientific ComputingAdvisory Committee 59

Appendix G: Acronyms and Abbreviations 60

TABLE OF CONTENTSTable of Contents iii

During the spring of 2006, we findourselves reflecting on the “early days”when NERSC moved to Berkeley adecade ago. We and many of theNERSC staff can now look back overa full ten years of our professionallives connected to NERSC. Memoriesof 1996, from the first system instal-lation in a not yet completed computerroom to the dedication of the CrayT3E, come back and remind us howfar NERSC has progressed sincethen. So many exciting events havehappened during that time, it wouldbe impossible to list them all here.

Instead we will proudly repeat whatSecretary of Energy Samuel Bodmanrecently said about NERSC: “NERSChas a well-earned reputation for pro-

viding highly reliable systems, fastturnaround on critical projects, anddedicated support for users.” He madethese comments when announcing arenewed allocation of NERSC computerresources to the U.S. Army Corps ofEngineers to simulate Gulf Coast hur-ricanes. In the aftermath of hurricaneKatrina, the Corps plans to use theNERSC cycles for simulations of theTexas, Louisiana, and Mississippicoastlines. These simulations willallow researchers to produce moreaccurate models for calculating theeffects of future hurricanes. “Becausethese simulations could literally affectthe lives of millions of Americans, wewant to ensure that our colleagues inthe Corps of Engineers have accessto supercomputers which are up to

the task,” the Secretary stated, givingNERSC credit for its proven record ofdelivering highly reliable productionsupercomputing services.

We intentionally chose not to com-memorate ten years in Berkeley witha big celebration this year. With rapidnew developments expected in the nextcouple of years, our focus must be onthe future, not on reminiscing. In 2005NERSC set the stage for the next fiveyears of its development through aseries of important planning activities.Through the well-established Greenbookprocess, our active user communityprovided their input to the planningprocess. NERSC management thendeveloped a new five-year plan for2006 to 2010, which is discussed in

THE YEAR IN PERSPECTIVE

The Year in Perspective2

detail in this annual report. This planwas then thoroughly reviewed in aprogrammatic review by DOE. We canbe proud about the outcome of thisreview of our plans, which were fullyendorsed: “NERSC is a strong, produc-tive, and responsive science-drivencenter that possesses the potential tosignificantly and positively impact sci-entific progress.… NERSC is extremelywell run with a lean and knowledgeablestaff.” This strong endorsement, togetherwith the continued support of our pro-gram office at DOE and the good budgetnews for 2007 and beyond, gives usthe most positive outlook we havehad in the last several years.

One of the reasons for excitement inthe near future is the upcoming intro-duction of the NERSC-5 system in2006 and 2007. As we eagerly awaitthe completion of our contract negoti-ations, we can confidently state thatNERSC-5 will be a major step forwardin providing a new capability for ourusers, unlike any of the upgrades inthe past half decade. The new systemwill provide 16 teraflop/s sustainedperformance on our benchmarks (morethan 100 teraflop/s peak), which isabout an order of magnitude moresustained performance than NERSCoffers today in early 2006.

In addition to our increase of capabilitycomputing in the near future, NERSCadded significant capacity in 2005 byintroducing two new clusters, named“Jacquard” and “Bassi.” This increasein capacity was highly welcomed by theNERSC community. Our users makethe most progress in computationalscience with exactly this type of sys-tem that provides high performance

in a reliable, predictable environment.One of our users, Robert Duke of theUniversity of North Carolina, commentedon the two systems as follows: “I haveto say that both of these machines arereally nothing short of fabulous. WhileJacquard is perhaps the best-perform-ing commodity cluster I have seen,Bassi is the best machine I haveseen, period.” By delivering this quali-ty, NERSC makes it possible for itsusers to concentrate on their science.Not surprisingly, our user communityenjoys continued and unparalleledscientific productivity. In 2005 wewere able to list more than 1100 sci-entific publications that were writtenon the basis of simulations carriedout at NERSC.

We also saw a long-term goal accom-plished that will significantly set apartthe NERSC production environmentfrom other centers. In early 2006,NERSC deployed the NERSC GlobalFilesystem (NGF) into production, pro-viding seamless data access from allof the Center’s computational andanalysis resources. NGF is intendedto facilitate sharing of data betweenusers and between machines. NGF’ssingle unified namespace makes iteasier for users to manage their dataacross multiple systems. Users nolonger need to keep track of multiplecopies of programs and data, and theyno longer need to copy data betweenNERSC systems for pre- and post-pro-cessing. NGF provides several otherbenefits as well: storage utilization ismore efficient because of decreasedfragmentation, and computationalresource utilization is more efficientbecause users can more easily runjobs on an appropriate resource.

Thus today NERSC users have the ben-efit of scalable high-end capabilitycomputing in Seaborg (and the soon-to-be-added NERSC-5), along with reliablecapacity machines, in an integratedenvironment that offers a global file-system, analytics support, and a seem-ingly infinite mass storage system (nowclose to 40 petabytes). Our five-yearplan will move this integrated systemenvironment to the near petaflop/sperformance level, which we will reachin 2010 with the planned introductionof NERSC-6. Scalability to tens of thou-sands of processors, both for applica-tions and systems software, managingpetabytes of data, and at the sametime continuing the expected level ofsupport, reliability, and quality of serv-ice, will be the big challenges ahead.Thanks to the ongoing support from ourprogram management at the Office ofAdvanced Scientific ComputingResearch at DOE, the NERSC budgethas been set at a level that makesthese ambitious plans feasible. Thuswe are confidently looking forward toanother year of both scientific andcomputing accomplishments at NERSC.As always, this progress would not bepossible without the NERSC staff,who continue to tirelessly dedicatetheir time, skill, and effort to makeNERSC the best scientific computingresource in the world. Our specialthanks to all of you.

Horst D. Simon, NERSC CenterDivision Director

William T. C. Kramer, Division Deputyand NERSC Center General Manager

The Year in Perspective 3

RESEARCH NEWS

Research at NERSC spans a wide range of topics, from subatomic particles to the formation of the

cosmos, and from alternative energy sources to the behavior of proteins. This year’s Research

News presents a sampling of important investigations and discoveries made using NERSC’s com-

putational resources.

Global warming and energy issues were frequently in the news this past year. The article “Delayed

reactions” describes three investigations into the dynamics and results of global warming; they con-

clude that climate change is no longer avoidable. “Burning questions” and “Combustion up close”

report on breakthrough studies of the basic processes of combustion; studies such as these con-

tribute to the scientific understanding needed if we are to improve the efficiency of our energy use.

“Hailstones in hell” discusses an important issue in the design of fusion reactors, a long-anticipated

alternative source of energy that is moving closer to realization.

Computational studies of matter and its behavior cover the widest possible range of scales, from

subatomic to molecular to nanoscale to cosmic. “A perfect liquid” reports on the remarkable newly

created state of quark-gluon matter. “Whispers from underground” relates how ghostly geoneutri-

nos hint at Earth’s inner secrets. “Breaking up is hard to calculate” tells of the first complete

numerical solution of the fragmentation of a system with four charged particles. At the nanoscale,

“Talent scouting” and “Surface charge” discuss important discoveries about catalysts. And at the

cosmic scale, “Magnetic disks in space” explores the origins of instability in accretion disks.

Finally, computing is becoming an indispensable tool for biology and medicine. The article “Proteins

in motion” tells how biochemists are systematically simulating the unfolding pathways of all known

protein folds, hoping to discover the general principles of protein folding, one of the basic mecha-

nisms of life.

Research News 5

A significant rise in sea level isinevitable by the end of this century;ocean temperatures over the last 40years have risen far above naturallyoccurring climate cycles; and contin-ued greenhouse gas emissions mayoverwhelm the capacity of land andocean to absorb excess carbon diox-ide, thus speeding up global warming.Those are the conclusions of threerecent climate modeling studies, allpointing to the inevitability of globalwarming and the need to develop mit-igation strategies.

Even if the concentrations of all green-house gases had leveled off in theyear 2000, we would still be commit-ted to a warmer Earth and rising sealevels in this century, according to astudy by a team of climate modelers atthe National Center for AtmosphericResearch (NCAR).1 The models wererun on supercomputers at NCAR andseveral DOE labs, including NERSC,and on the Earth Simulator in Japan.

The modeling study quantifies the rel-ative rates of sea level rise and globaltemperature increase that we arealready committed to in the 21st cen-tury. Even if no more greenhousegases were added to the atmosphere,

globally averaged surface air tempera-tures would rise about 0.5° C (1° F)and global sea levels would rise another11 centimeters (4.3 inches) fromthermal expansion alone by 2100.

“Many people don’t realize we arecommitted right now to a significantamount of global warming and sealevel rise because of the greenhousegases we have already put into theatmosphere,” said lead author GeraldMeehl. “Even if we stabilize green-house gas concentrations, the climatewill continue to warm, and there will beproportionately even more sea levelrise. The longer we wait, the more cli-

mate change we are committed to inthe future.”

The half-degree temperature rise issimilar to that observed during thesecond half of the 20th century, butthe projected sea level rise is morethan twice the 5-centimeter (2-inch)rise that occurred during that period.These numbers do not take intoaccount fresh water from melting icesheets and glaciers, which could atleast double the sea level risecaused by thermal expansion alone.

Though temperature rise shows signsof leveling off 100 years after stabi-

The heat is onNew studies reveal the inevitable and possibly self-perpetuating consequences of global warming

400

300

200

%

100

2100208020602040Year

202020001980

PCM sea levelCCSM3 sea levelPCM temperatureCCSM3 temperature

0

–100

FIGURE 1. Percent increase ofglobally averaged surfaceair temperature and sealevel rise from the two mod-els computed relative to val-ues for the base period1980–1999, for the experi-ment in which greenhousegas concentrations and allother atmospheric con-stituents were stabilized atthe end of the 20th century.

1 G. A. Meehl, W. M. Washington, W. D. Collins, J. M. Arblaster, A. Hu, L. E. Buja, W. G. Strand, and H. Teng, “How much more global warming and sea level rise?”Science 307, 1769 (2005). Research funding: NSF, BER, NCAR, MEXT. Computational resources: NCAR, NERSC, LANL, NCCS/ORNL, ES.

Research News6

lization in the study, ocean waterscontinue to warm and expand, caus-ing global sea level to rise unabated(Figure 1).

“With the ongoing increase in concen-trations of greenhouse gases, everyday we commit to more climate changein the future,” Meehl explained. “Whenand how we stabilize concentrationswill dictate, on the time scale of acentury or so, how much more warmingwe will experience. But we are alreadycommitted to ongoing large sea levelrise, even if concentrations of green-house gases could be stabilized.”

The inevitability of the climatechanges described in the study is theresult of thermal inertia, mainly fromthe oceans, and the long lifetime ofcarbon dioxide and other greenhousegases in the atmosphere. Thermalinertia refers to the process by whichwater heats and cools more slowlythan air because it is denser than air.

This study quantifies future committedclimate change using coupled globalthree-dimensional climate models.Coupled models link major componentsof Earth’s climate in ways that allowthem to interact with each other. Meehland his NCAR colleagues ran the samescenario a number of times and aver-aged the results to create ensemblesimulations from each of two globalclimate models. Then they comparedthe results from each model.

The scientists also used the twomodels to compare possible 21stcentury climate scenarios in whichgreenhouse gases continue to buildin the atmosphere at low, moderate,or high rates. The worst-case sce-nario projects an average temperaturerise of 3.5° C (6.3° F) and sea levelrise from thermal expansion of 30centimeters (12 inches) by 2100(Figure 2). All scenarios analyzed inthe study will be assessed by interna-

tional teams of scientists for the nextreport by the Intergovernmental Panelon Climate Change, due out in 2007.

The NCAR team used the ParallelClimate Model (PCM), developed byNCAR and the Department of Energy,and the new Community ClimateSystem Model (Version 3). TheCCSM3 was developed at NCAR withinput from university and federal cli-mate scientists around the countryand principal funding from the NationalScience Foundation (NCAR’s primarysponsor) and the Department of Energy.The CCSM3 shows slightly highertemperature rise and sea level risefrom thermal expansion and greaterweakening of the thermohaline circu-lation in the North Atlantic. Otherwise,the results from the two models aresimilar. Three of the researchers laterperformed the same analysis using12 different general climate models,and all showed similar results.

The warming of the oceans over thepast 40 years has not been uniform,and another recent study2 investigatedwarming at different depths on anocean-by-ocean basis, using observa-tional data as well as simulationsfrom PCM and the HadCM3 modelfrom Britain’s Hadley Centre. Theresearchers noted that the verticalstructure of the temperature data

varies widely by ocean, and theyexplored three possible causes for thewarming trend: (1) natural variabilityinternal to the coupled ocean– atmos-phere system; (2) external naturalvariability, such as solar or volcanicforcing; and (3) forcing arising fromhuman activity (emission of green-house gases and sulfate aerosols).

The study showed that the warmingtrend is much stronger than would beexpected from natural variability. InFigure 3, the hatched blue regionshows simulated natural internal vari-ability; the green triangles representcombined solar and volcanic forcing;and the red dots represent observedtemperature trends. On the time andspace scales used in this study, thesolar plus volcanic signals are gener-ally indistinguishable from the naturalinternal variability, but the observedtemperatures bear little resemblanceto the natural variations. Figure 4shows that the actual temperaturetrends do match the simulated resultsof greenhouse gases and sulfateaerosols, indicating that ocean warm-ing is caused by human activities.

“This is perhaps the most compellingevidence yet that global warming ishappening right now and it shows thatwe can successfully simulate its pastand likely future evolution,” said Tim

50

40

30

cm

20

2200215021002050Year

200019501900

PCM 20th stabilizationPCM B1PCM A1BPCM A2CCSM3 20th stabilizationCCSM3 B1CCSM3 A1BCCSM3 A2

10

0

FIGURE 2. Globally averagedsea level rise from thermalexpansion for various sce-narios of greenhouse gasbuildup.

2 T. P. Barnett, D. W. Pierce, K. M. AchutaRao, P. J. Gleckler, B. D. Santer, J. M. Gregory, and W. M. Washington, “Penetration of human-induced warming into theworld’s oceans,” Science 309, 284 (2005). Research funding: NOAA, BER, DEFRA, GMRDP. Computational resources: NCAR, NERSC, ORNL, LANL, Hadley Center.

Research News 7

Barnett, lead author of the study anda research marine physicist at theScripps Institution of Oceanography.Barnett said he was “stunned” by theresults because the computer modelsreproduced the penetration of thewarming signal in all the oceans.“The statistical significance of theseresults is far too strong to be merelydismissed and should wipe out muchof the uncertainty about the reality ofglobal warming.”

A third study, based on a climatemodel that includes the effects ofEarth’s carbon cycle, indicates thereare limits to our planet’s ability toabsorb increased emissions of car-bon dioxide.3 If the current release ofcarbon from fossil fuels continuesunabated, by the end of the centurythe land and oceans will be less ableto take up carbon than they are today,the model indicates.

“If we maintain our current course offossil fuel emissions or accelerateour emissions, the land and oceanswill not be able to slow the rise ofcarbon dioxide in the atmosphere theway they’re doing now,” said Inez Y.Fung at the University of California,Berkeley, who is director of theBerkeley Atmospheric SciencesCenter, co-director of the newBerkeley Institute of the Environment,and professor of earth and planetaryscience and of environmental sci-ence, policy and management. “It’sall about rates. If the rate of fossilfuel emissions is too high, the carbonstorage capacity of the land andoceans decreases and climate warm-ing accelerates.”

Currently, the land and oceans absorbabout half of the carbon dioxide pro-duced by human activity, most of itresulting from the burning of fossilfuels, Fung said. Some scientistshave suggested that the land andoceans will continue to absorb more

700500300200125

753010

700500300200125

753010

Dep

th (

m)

0.40.20.0 0.40.20.0Signal strength (C)

–0.2 0.40.20.0–0.2–0.2

NorthAtlantic

NorthIndian

NorthPacific

SouthAtlantic

SouthIndian

SouthPacific

700500300200125

753010

700500300200125

753010

Dep

th (m

)

0.40.20.0 0.40.20.0Signal strength (C)

–0.2 0.40.20.0–0.2–0.2

NorthAtlantic

NorthIndian

NorthPacific

SouthAtlantic

SouthIndian

SouthPacific

FIGURE 3. Observed ocean temperature trends from the last 40 years (red dots), projected ontoa PCM simulation of natural variability internal to the coupled ocean–atmosphere system(blue hatched area) and combined solar and volcanic forcing (green triangles). Actual temper-atures are much higher than would be expected from natural variability.

FIGURE 4. Observed ocean temperature trends (red dots) projected onto a PCM simulation ofthe results of greenhouse gases and sulfate aerosols (green hatched area). There is excellentagreement at most depths in all oceans, indicating that ocean warming is the result ofhuman activities. Similar results were obtained with the HadCM3 model.

3 I. Y. Fung, S. C. Doney, K. Lindsay, and J. John, “Evolution of carbon sinks in a changing climate,” Proceedings of the National Academy of Sciences 102, 11201(2005). Research funding: NSF, NASA, BER, LBNL, WHOI. Computational resources: NCAR, NERSC.

Research News8

and more CO2 as fossil fuel emis-sions increase, making plants flourishand the oceans bloom. Fung’s com-puter model, however, indicates thatthe “breathing biosphere” can absorbcarbon only so fast. Beyond a certainpoint, the planet will not be able tokeep up with carbon dioxide emis-sions (Figure 5).

“The reason is very simple,” Fungsaid. “Plants are happy growing at acertain rate, and though they canaccelerate to a certain extent withmore CO2, the rate is limited by meta-bolic reactions in the plant, by waterand nutrient availability, et cetera.”

Fung and her colleagues found thatincreasing temperatures and droughtfrequencies lower plant uptake of CO2

as plants breathe in less to conservewater. At some point, the rate of fos-sil fuel CO2 emissions will outstripthe ability of the vegetation to keepup, leading to a rise in atmosphericCO2, increased greenhouse tempera-tures and increased frequency ofdroughts. An amplifying loop leads toever higher temperatures, moredroughts, and higher CO2 levels.

The oceans exhibit a similar trend,Fung said, though less pronounced.There, mixing by turbulence in theocean is essential for moving CO2

down into the deep ocean, away fromthe top 100 meters of the ocean, wherecarbon absorption from the atmos-phere takes place. With increasedtemperatures, the ocean stratifiesmore, mixing becomes harder, andCO2 accumulates in the surfaceocean instead of in the deep ocean.This accumulation creates a backpressure, lowering CO2 absorption.

In all, business as usual would leadto a 1.4° C (2.5° F) rise in global tem-peratures by the year 2050. This esti-mate is at the low range of projectedincreases for the 21st century, Fungsaid, though overall, the model is inline with others predicting largeecosystem changes, especially in thetropics.

With voluntary controls that flattenfossil fuel CO2 emission rates by theend of the century, the land andoceans could keep up with CO2 levelsand continue to absorb at their cur-rent rate, the model indicates.

“This is not a prediction, but a guide-line or indication of what could hap-pen,” Fung said. “Climate predictionis a work in progress, but this modeltells us that, given the increases ingreenhouse gases, the Earth willwarm up; and given warming, hotplaces are likely to be drier, and theland and oceans are going to take incarbon at a slower rate; and there-fore, we will see an amplification oracceleration of global warming.”

“The Earth is entering a climate spacewe’ve never seen before, so we can’tpredict exactly what will happen,” sheadded. “We don’t know where thethreshold is. A two-degree increase inglobal temperatures may not soundlike much, but if we’re on the thresh-old, it could make a big difference.”

Fung and colleagues have worked forseveral decades to produce a model ofthe Earth’s carbon cycle that includesnot only details of how vegetation takesup and releases carbon, but alsodetails of decomposition by microbesin the soil, the carbon chemistry ofoceans and lakes, the influence of rainand clouds, and many other sources

90S

60S

30S

30N

60N

90NA B

0

30E 30E60E 60E

–100

0–5

00–3

50–2

50–2

00–1

50–1

00 –50 50 10

015

020

00

–100

00

–500

0

–250

0

–100

0–5

00–1

50 150

1000

2500

500050

0

90E 90E120E 120E150E 150E150W150W 120W 120W90W 90W60W 60W30W 30W0 0180 18018090S

60S

30S

30N

60N

90N

0

FIGURE 5. The impact of carbon–climate feedback on carbon storage for the years 2001–2100 (a) in the oceans and (b) on land. The yellow,green, and blue areas indicates less carbon storage, while the orange and red areas indicate more carbon storage.

Research News 9

and sinks for carbon. The model takesinto account thousands of details,ranging from carbon uptake by leaves,stems, and roots to the different waysthat forest litter decomposes, day-nightshifts in plant respiration, the salinityof oceans and seas, and effects oftemperature, rainfall, cloud cover andwind speed on all these interactions.

All of today’s climate models are ableto incorporate the climate effects ofcarbon dioxide in the atmosphere, butonly with concentrations of CO2 speci-fied by the modelers. Fung’s modeldoes not specify atmospheric CO2 lev-els, but rather predicts the levels,given fossil fuel emissions. Theresearchers used observations of the

past two centuries to make sure thattheir model is reasonable, and thenused the model to project what willhappen in the next 100 years.

The climate model coupled with thecarbon cycle has been her goal fordecades, as she tried to convince cli-mate modelers that “whether plantsare happy or not happy has an influ-ence on climate projections. To includeinteractive biogeochemistry in climatemodels, which up to now embrace pri-marily physics and dynamics, is new.”

She admits, however, that much workremains to be done to improve model-ing. Methane and sulfate cycles mustbe included, plus effects like changes

in plant distribution with rising tem-peratures, the possible increase infires, disease, or insect pests, andeven the effects of dust in the oceans.

“We have created a blueprint, interms of a climate modeling frame-work, that will allow us to go beyondthe physical climate models to moresophisticated models,” she said.“Then, hopefully, we can understandwhat is going on now and what couldhappen. This understanding couldguide our choices for the future.”

This article written by: NCAR Media Relations

Office, Robert Sanders (UC Berkeley), Mario

Aguilera and Cindy Clark (Scripps Institution of

Oceanography), John Hules (Berkeley Lab).

Controlling fire to provide heat andlight was one of humankind’s firstgreat achievements, and the basicchemistry of combustion—what goesin and what comes out—was estab-lished long ago. But a complete quan-titative understanding of what hap-pens during the combustion processhas remained as elusive as the ever-changing shape of a flame.

Even a simple fuel like methane(CH4), the principal component of nat-ural gas, burns in a complexsequence of steps. Oxygen atomsgradually replace other atoms in thehydrocarbon molecules, ultimatelyleaving carbon dioxide and water.

Methane oxidation involves about 20chemical species for releasing energy,as well as many minor species thatcan become pollutants. Turbulence candistort the distribution of species andthe redistribution of thermal energywhich are required to keep the flameburning. These turbulence–chemistryinteractions can cause the flame toburn faster or slower and to createmore or less pollution.

The holy grail of combustion sciencehas been to observe these turbulence–chemistry effects. Over the past fewdecades amazing progress has beenmade in observational techniques,including the use of lasers to excite

molecules of a given species and pro-duce a picture of the chemical distri-bution. But laser imaging is limited inthe species and concentrations thatcan be reliably observed, and it is dif-ficult to obtain simultaneous imagesto correlate different chemical species.

Because observing the details of com-bustion is so difficult, progress in com-bustion science has largely coincidedwith advances in scientific computing.For example, while basic concepts forsolving one-dimensional flat flamesoriginated in the 1950s, it only becamepossible to solve the 1D flame equa-tions some 30 years later using Cray-1supercomputers. Those calculations,

Burning questionsNew 3D simulations are closing in on the holy grail of combustion science:turbulence–chemistry interactions

1 cm

Research News10

which are routine on personal com-puters today, enabled a renaissancein combustion science by allowingchemists to observe the interrelation-ships among the many hypothesizedreaction processes in the flame.

Simulating three-dimensional turbulentflames took 20 more years of advancesin applied mathematics, computer sci-ence, and computer hardware, partic-ularly massively parallel systems. Butthe effort has been worth it. New 3Dsimulations are beginning to providethe kind of detailed information aboutthe structure and dynamics of turbulentflames that will be needed to designnew low-emission, fuel-efficient com-bustion systems.

The first 3D simulation of a laboratory-scale turbulent flame from first princi-ples—the result of a SciDAC-fundedcollaboration between computationaland experimental scientists at BerkeleyLab—was featured on the cover of theJuly 19, 2005 Proceedings of theNational Academy of Sciences (Figure6).4 The article, written by John Bell,

Marc Day, Ian Shepherd, MatthewJohnson, Robert Cheng, Joseph Grcar,Vincent Beckner, and Michael Lijewski,describes the simulation of “a laboratory-scale turbulent rod-stabilized premixedmethane V-flame.” This simulation wasunprecedented in several aspects—thenumber of chemical species included,the number of chemical reactionsmodeled, and the overall size of theflame.

This simulation employed a differentmathematical approach than has typi-cally been used for combustion. Mostcombustion simulations designed forbasic research use compressible flowequations that include sound waves,and are calculated with small timesteps on very fine, uniform spatialgrids—all of which makes them verycomputationally expensive. Becauseof limited computer time, such simu-lations often have been restricted toonly two dimensions, to scales lessthan a centimeter, or to just a fewcarbon species and reactions.

In contrast, the Center for Computa-tional Sciences and Engineering (CCSE),under Bell’s leadership, has developed

an algorithmic approach that combineslow Mach-number equations, whichremove sound waves from the compu-tation, with adaptive mesh refinement,which bridges the wide range of spa-tial scales relevant to a laboratoryexperiment. This combined methodol-ogy strips away relatively unimportantaspects of the simulation and focus-es computing resources on the mostimportant processes, thus slashingthe computational cost of combustionsimulations by a factor of 10,000.

Using this approach, the CCSE teamhas modeled turbulence and turbu-lence–chemistry interactions for athree-dimensional flame about 12 cm(4.7 in.) high, including 20 chemicalspecies and 84 fundamental chemi-cal reactions. The simulation wasrealistic enough to be compareddirectly with experimental diagnostics.

The simulation captured with remark-able fidelity some major features ofthe experimental data, such as flame-generated outward deflection in theunburned gases, inward flow conver-gence, and a centerline flow accelera-tion in the burned gases (Figure 7).

FIGURE 6. The calculated surface of a turbu-lent premixed laboratory methane flame.

FIGURE 7. Left: A typical centerline slice of the methane concentration obtained from the simu-lation. Right: Experimentally, the instantaneous flame location is determined by using thelarge differences in Mie scattering intensities from the reactants and products to clearly out-line the flame. The wrinkling of the flame in the computation and the experiment is of similarsize and structure.

4 J. B. Bell, M. S. Day, I. G. Shepherd, M. Johnson, R. K. Cheng, J. F. Grcar, V. E. Beckner, and M. J. Lijewski, “Numerical simulation of a laboratory-scale turbulentV-flame,” Proceedings of the National Academy of Sciences 102, 10006 (2005).

SIMULATION EXPERIMENT

Research News 11

Efficient mixing is one of the basicchallenges in turbulent combustion,especially in situations where the fueland air streams are initially separated(so-called nonpremixed combustion),as in jet aircraft engines and direct-injection diesel and gasoline engines.The mixing of fuel and air moleculesis necessary for the chemical reac-tions of combustion to take place.

Turbulence increases the mixing rate,so the right amount of turbulenceimproves the efficiency of combus-tion. But too much turbulent mixingcan interrupt the chemical reactionprocess and lead to partial extinctionof the flame. If extinguished pocketsof unburned fuel–air mixture fail toreignite promptly, they may be emittedin the exhaust. Thus extinction maylead to reduced fuel efficiency,increased harmful emissions, and ifpervasive enough, destabilization orblowout of the flame—a potentiallycatastrophic effect for aircraft.

Small-scale turbulence–chemistryeffects such as extinction and reigni-tion, flow and flame unsteadiness,and differential diffusion of chemical

Combustion up closeDirect numerical simulations reveal small-scale structures and processes thatexperiments cannot measure

FIGURE 8. A simulated planar jet flame, colored by the rate of molecular mixing (scalar dissipa-tion rate), which is critical for determining the interaction between reaction and diffusion in aflame. The image shows that high scalar dissipation regions exist in thin, highly intermittentstructures aligned with principal strain directions. (Visualization created using an applicationwritten by Hongfeng Yu and Kwan-Liu Ma, UC Davis.)

The simulation results were found tomatch the experimental results withina few percent. This agreement directlyvalidated both the computationalmethod and the chemical model ofhydrocarbon reaction and transportkinetics in a turbulent flame.

The results demonstrate that it ispossible to simulate a laboratory-scale flame in three dimensions with-out having to sacrifice a realistic rep-resentation of chemical and transportprocesses. This advance has thepotential to greatly increase our

understanding of how fuels behave inthe complicated environments insideturbulent flames.

Research funding: BES, ASCR, SciDAC

Computational resources: NERSC

This article written by: John Hules, Jon Bashor

Research News12

species are difficult, if not impossi-ble, to measure experimentally.Engineering-scale combustion simula-tions, in order to be computationallyfeasible, must use a spatial grid thatis too coarse to capture these small-scale phenomena; to approximatetheir effects, mathematical modelsextrapolated from experimentalresults must be used. But the rapidgrowth of computational capabilitiesin recent years has presented newopportunities for high-resolution directnumerical simulations (DNS) of turbu-lent combustion that address unan-swered questions about turbulence–chemistry interactions.

In the 2005 INCITE project “DirectNumerical Simulation of TurbulentNonpremixed Combustion—Fundamental Insights towardsPredictive Modeling,” JacquelineChen, Evatt Hawkes, and RamananSankaran of Sandia NationalLaboratories have performed the first3D DNS simulations of a turbulentnonpremixed H2/CO–air flame withdetailed chemistry (Figure 8). The

simulations, including 11 chemicalspecies and 33 reactions, were per-formed with up to 500 million gridpoints and 100,000 time steps, andran for 2.5 million processor hours onSeaborg and 1.5 million hours on Bassi.

An initial analysis of the data shows forthe first time how detailed transport andchemistry effects can influence themixing of chemical molecules.5 (Themathematical representations of thesemolecules are called scalars.) This isan example of how DNS can be usedto assess the validity of assumptionsused in combustion models. Mixingmodels typically assume that the mix-ing rate is the same for each differentscalar and corresponds to the turbu-lence time scale. In fact, the INCITEsimulation reveals that the mixingrates of active (reacting) scalars andpassive scalars may vary by a factorof 3. Models may need to incorporatethese different mixing rates, becausea poor mixing model could incorrectlypredict a stable flame when actuallyextinction occurs. This finding andothers yet to come from further analy-

sis may lead to more accurate modelsfor use in lower-resolution simulations.

To ramp up to the large INCITE calcu-lations, Chen’s team started by run-ning many test calculations with theirhighly scalable S3D code, using differ-ent numbers of grid points on a vari-ety of large-scale computing systems.After analyzing and optimizing thecode’s performance with the help ofNERSC staff, they improved thecode’s efficiency by 45%.

The code improvements helped theresearchers achieve the highest-everReynolds number (ratio of inertial toviscous forces) in a 3D fully resolvedDNS of a nonpremixed flame withdetailed chemistry. Simulations withdifferent Reynolds numbers shed lighton a phenomenon called intermittency—intense, localized fluctuations of anyquantity in a turbulent flow—whichcan result in localized extinction andreignition of the combustion process.The results show a relationshipbetween scalar intermittency and theReynolds number (Figure 9).

5 E. R. Hawkes, R. Sankaran, J. C. Sutherland, and J. H. Chen, “Direct numerical simulation of turbulent combustion: Fundamental insights towards predictive models,” Journal of Physics Conference Series 16 (SciDAC 2005), 65 (2005).

FIGURE 9. Instantaneous isocontours of the total scalar dissipation rate field for successively higher Reynolds numbers at a time when reignitionfollowing extinction in the domain is significant. The dissipation fields are organized into thin sheet-like lamellar structures, with lengths farexceeding their thickness, consistent with experimental observations in nonreactive flows. Increasingly fine-scaled structures are observed athigher Reynolds numbers. (From E. R. Hawkes, R. Sankaran, J. C. Sutherland, and J. H. Chen, “Direct Numerical Simulation of Temporally-Evolving Plane Jet Flames with Detailed CO/H2 Kinetics,” submitted to the 31st International Symposium on Combustion, 2006.)

Research News 13

What happens when you shoot one ofthe coldest materials into the hottestenvironment on earth? The answer mayhelp solve the world’s energy crisis.

Imagine that there is a large chamberin hell that’s shaped like a doughnut,and that you’d like to shoot a seriesof hailstones into that chamber sothat as they melt, the water vaporpenetrates as deeply as possible anddisperses as evenly as possiblethroughout the chamber. Settingaside for a moment the question ofwhy you would want to do that, con-sider the multitude of how questions:Should you shoot in the hailstonesfrom outside the ring of the doughnutor from inside the hole? What sizehailstones should you use? At whatspeed and angle should they enterthe chamber?

This strange scenario is actually ananalogy for one of the questions fac-ing fusion energy researchers: how torefuel a tokamak. A tokamak is amachine that produces a toroidal(doughnut-shaped) magnetic field. In

that field, two isotopes of hydrogen—deuterium and tritium—are heated toabout 100 million degrees Celsius

(more than six times hotter than theinterior of the sun), stripping the elec-trons from the nuclei. The magnetic

This project has generated 10 tera-bytes of raw DNS data, which theyintend to share with an ongoing international collaboration of experi-mentalists and modelers called theTurbulent Nonpremixed FlameWorkshops.6 The DNS data will serveas a numerical benchmark, comple-mentary to experimental data, formodel validation and the advance-

ment of basic scientific understand-ing of turbulent combustion.

Extracting and analyzing useful infor-mation from this massive dataset isa daunting task. Chen’s group isworking with Kwan-Liu Ma of theUniversity of California at Davis todevelop automated feature extractiontools and multi-variable visualization

capabilities that will help researchersunderstand how turbulent mixinginteracts with chemical reactions.Phenomena they are working to visu-alize include flame edges and cusps,extinction pockets, and ignition kernels.

Research funding: BES, INCITE, SciDAC

Computational resources: NERSC, MSCF, NCCS

This article written by: John Hules

6 http://www.ca.sandia.gov/TNF/abstract.html

Hailstones in hellSimulations examine the behavior of frozen fuel pellets in fusion reactors

FIGURE 10. Cutaway illustration of the ITER tokamak.

Research News14

field makes the electrically chargedparticles follow spiral paths aroundthe magnetic field lines, so that theyspin around the torus in a fairly uni-form flow and interact with eachother, not with the walls of the toka-mak. When the hydrogen ions (nuclei)collide at high speeds, they fuse,releasing energy. If the fusion reac-tion can be sustained long enoughthat the amount of energy releasedexceeds the amount needed to heatthe plasma, researchers will havereached their goal: a viable energysource from abundant fuel that pro-duces no greenhouse gases and nolong-lived radioactive byproducts.

High-speed injection of frozen hydrogenpellets is an experimentally provenmethod of refueling a tokamak. Thesepellets are about the size of smallhailstones (3–6 mm) and have a tem-perature of about 10 degrees Celsiusabove absolute zero. The goal is tohave these pellets penetrate asdeeply as possible into the plasmaso that the fuel disperses evenly.

Pellet injection will be the primaryfueling method used in ITER (Latin for“the way”), a multinational tokamakexperiment to be built at Cadarachein southern France (Figure 10). ITER,one of the highest strategic prioritiesof the DOE Office of Science, isexpected to produce 500 million ther-mal watts of fusion power—10 timesmore power than is needed to heatthe plasma—when it reaches fulloperation around the year 2016. Asthe world’s first production-scalefusion reactor, ITER will help answerquestions about the most efficientways to configure and operate futurecommercial reactors.

However, designing a pellet injectionsystem that can effectively deliverfuel to the interior of ITER representsa special challenge because of itsunprecedented large size and high

temperatures. Experiments have shownthat a pellet’s penetration distance intothe plasma depends strongly on howthe injector is oriented in relation to thetorus. For example, an “inside launch”(from inside the torus ring) results inbetter fuel distribution than an “out-side launch” (from outside the ring).

In the past, progress in developing anefficient refueling strategy for ITER hasrequired lengthy and expensive experi-ments. But thanks to a three-year,SciDAC-funded collaboration betweenthe Computational Plasma PhysicsTheory Group at Princeton PlasmaPhysics Laboratory and the AdvancedNumerical Algorithms Group at LawrenceBerkeley National Laboratory, computercodes have now reproduced some keyexperimental findings, resulting in sig-nificant progress toward the scientificgoal of using simulations to predict theresults of pellet injection in tokamaks.

“To understand refueling by pelletinjection, we need to understand twophases of the physical process,” said

Ravi Samtaney, the Princeton re-searcher who is leading the codedevelopment effort. “The first phaseis the transition from a frozen pelletto gaseous hydrogen, and the secondphase is the distribution of that gasin the existing plasma.”

The first phase is fairly well under-stood from experiments and theoreti-cal studies. In this phase, called abla-tion, the outer layer of the frozenhydrogen pellet is quickly heated,transforming it from a solid into anexpanding cloud of dense hydrogengas surrounding the pellet. This gasquickly heats up, is ionized, andmerges into the plasma. As ablationcontinues, the pellet shrinks until allof it has been gasified and ionized.

The second phase—the distributionof the hydrogen gas in the plasma—is less well understood. Ideally, theinjected fuel would simply follow themagnetic field lines and the “flux sur-faces” that they define, maintaining astable and uniform plasma pressure.

FIGURE 11. Time sequence of 2D slices from a 3D simulation of the injection of a fuel pellet intoa tokamak plasma. Injection from outside the torus (bottom row, injection from right) resultsin the pellet stalling and fuel being dispersed near the plasma boundary. Injection from inside thetorus (top row, injection from left) achieves fuel distribution in the hot central region as desired.

Research News 15

But experiments have shown that thehigh-density region around the pelletquickly heats up to form a localregion of high pressure, higher thancan be stably confined by the localmagnetic field. A form of “local insta-bility” (like a mini-tornado) then devel-ops, causing the high-density regionto rapidly move across, rather thanalong, the field lines and flux surfaces—a motion referred to as “anomalous”because it deviates from the large-scale motion of the plasma.

Fortunately, researchers have discov-ered that they can use this instabilityto their advantage by injecting the pel-let from inside the torus ring, becausefrom this starting point, the anomalousmotion brings the fuel pellet closer tothe center of the plasma, where itdoes the most good. This anomalousmotion is one of the phenomena thatSamtaney and his colleagues want toquantify and examine in detail.

Figure 11 shows the fuel distributionphase as simulated by Samtaney and

The four experimental groups conduct-ing research at the Relativistic HeavyIon Collider (RHIC) at BrookhavenNational Laboratory have created a newstate of hot, dense matter out of thequarks and gluons that are the basicparticles of atomic nuclei, but it is astate quite different and even moreremarkable than had been predicted.In an announcement at the April 2005meeting of the American Physical

Society in Tampa, Florida, and in peer-reviewed papers summarizing the firstthree years of RHIC findings, the sci-entists said that instead of behavinglike a gas of free quarks and gluons,as was expected, the matter createdin RHIC’s heavy ion collisions appearsto be more like a liquid.

The papers, which the four RHIC col-laborations (BRAHMS, PHENIX, PHOBOS,

and STAR) had been working on fornearly a year, were published simulta-neously by the journal Nuclear PhysicsA8 and were also compiled in a spe-cial Brookhaven report.9 These sum-maries indicate that some of theobservations at RHIC fit with the the-oretical predictions for a quark-gluonplasma (QGP), the type of matter pos-tulated to have existed just microsec-onds after the Big Bang. Indeed, many

his colleagues in the first detailed 3Dcalculations of pellet injection.7 Theinside launch (top row) distributes thefuel in the central region of the plas-ma, as desired, while the outsidelaunch (bottom row) disperses thefuel near the plasma boundary, asshown in experiments.

Simulating pellet injection in 3D is dif-ficult because the physical processesspan several decades of time andspace scales. The large disparitybetween pellet size and tokamak size,the large density differences betweenthe pellet ablation cloud and the ambi-ent plasma, and the long-distanceeffects of electron heat transport allpose severe numerical challenges. Toovercome these difficulties, Samtaneyand his collaborators used an algo-rithmic method called adaptive meshrefinement (AMR), which incorporatesa range of scales that change dynami-cally as the calculation progresses.AMR allowed this simulation to runmore than a hundred times fasterthan a uniform-mesh simulation.

While these first calculations representan important advance in methodology,Samtaney’s work on pellet injection isonly beginning. “The results presentedin this paper did not include all thedetailed physical processes which we’restarting to incorporate, along with morerealistic physical parameters,” he said.“For example, we plan to developmodels that incorporate the perpendi-cular transport of the ablated mass.We also want to investigate otherlaunch locations. And, of course, we’llhave to validate all those resultsagainst existing experiments.”

This pellet injection model will eventu-ally become part of a comprehensivepredictive capability for ITER, whichits supporters hope will bring fusionenergy within reach as a commercialsource of electrical power.

Research funding: FES, SciDAC

Computational resources: NERSC

This article written by: John Hules

7 R. Samtaney, S. C. Jardin, P. Colella, and D. F. Martin, “3D adaptive mesh refinement simulations of pellet injection in tokamaks,” Computer PhysicsCommunications 164, 220 (2004).

8 Nuclear Physics A, Volume 757, Issues 1–2, August 8, 2005.9 BRAHMS, PHENIX, PHOBOS, and STAR Collaborations, Hunting the Quark Gluon Plasma: Results from the First 3 Years at RHIC (Upton, NY: Brookhaven National

Laboratory report No. BNL-73847-2005).

A perfect liquidThe newly created state of quark-gluon matter is more remarkable thanpredicted and raises many new questions

Research News16

theorists have concluded that RHIChas already demonstrated the cre-ation of quark-gluon plasma. However,all four collaborations note that thereare discrepancies between the experi-mental data and early theoretical pre-dictions based on simple models ofquark-gluon plasma formation.

“We know that we’ve reached thetemperature [up to 150,000 timeshotter than the center of the sun] andenergy density [energy per unit vol-ume] predicted to be necessary forforming such a plasma,” said SamAronson, Brookhaven’s AssociateLaboratory Director for High Energyand Nuclear Physics. But analysis ofRHIC data from the start of opera-tions in June 2000 through the 2003physics run reveals that the matterformed in RHIC’s head-on collisions ofgold ions (Figure 12) is more like aliquid than a gas.

That evidence comes from measure-ments of unexpected patterns in thetrajectories taken by the thousands ofparticles produced in individual colli-

sions. These measurements indicatethat the primordial particles producedin the collisions tend to move collec-tively in response to variations ofpressure across the volume formedby the colliding nuclei (Figure 13).

Scientists refer to this phenomenonas “flow,” since it is analogous to theproperties of fluid motion.

However, unlike ordinary liquids, inwhich individual molecules move

Gaseous Liquid

FIGURE 12. Two views of one of the first full-energy collisions between gold ions at Brookhaven Lab’s Relativistic Heavy Ion Collider, as captured bythe Solenoidal Tracker At RHIC (STAR) detector. The tracks indicate the paths taken by thousands of subatomic particles produced in the colli-sions as they pass through the STAR Time Projection Chamber, a large, 3D digital camera.

FIGURE 13. These images contrast the degree of interaction and collective motion, or “flow,”among quarks in the predicted gaseous quark-gluon plasma state (left) vs. the liquid statethat has been observed in gold-gold collisions at RHIC (right). The “force lines” and collectivemotion in the observed collisions show the much higher degree of interaction and flowamong the quarks in what is now being described as a nearly “perfect” liquid. An animationdemonstrating the differences between the expected gas and the observed liquid can beviewed at http://real.bnl.gov/ramgen/bnl/RHIC_animation.rm.

Research News 17

about randomly, the hot matterformed at RHIC seems to move in apattern that exhibits a high degree ofcoordination among the particles—somewhat like a school of fish thatresponds as one entity while movingthrough a changing environment.

“This is fluid motion that is nearly‘perfect,’” Aronson said, meaning itcan be explained by equations ofhydrodynamics. These equations weredeveloped to describe theoretically“perfect” fluids—those with extremelylow viscosity and the ability to reachthermal equilibrium very rapidly dueto the high degree of interactionamong the particles. While RHIC sci-entists don’t have a direct measureof viscosity, they can infer from theflow pattern that, qualitatively, the vis-cosity is very low, approaching thequantum mechanical limit.

Together, these facts present a com-pelling case. “In fact, the degree ofcollective interaction, rapid thermal-ization, and extremely low viscosity ofthe matter being formed at RHICmake this the most nearly perfect liq-uid ever observed,” Aronson said.

In results reported earlier, othermeasurements at RHIC have shown“jets” of high-energy quarks and glu-ons being dramatically slowed downas they traverse the hot fireball pro-duced in the collisions. This “jet

quenching” demonstrates that theenergy density in this new form ofmatter is extraordinarily high—muchhigher than can be explained by amedium consisting of ordinarynuclear matter.

“The current findings don’t rule outthe possibility that this new state ofmatter is in fact a form of the quark-gluon plasma, just different from whathad been theorized,” Aronson said.Many scientists believe this to be thecase, and detailed measurements arenow under way at RHIC to resolve thisquestion.

Theoretical physicists, whose stan-dard calculations cannot incorporatethe strong coupling observed betweenthe quarks and gluons at RHIC, arealso revisiting some of their earlymodels and predictions. To try toaddress these issues, they are run-ning massive numerical simulationson some of the world’s most powerfulcomputers. Others are attempting toincorporate quantitative measures ofviscosity into the equations of motionfor fluid moving at nearly the speed oflight. One subset of calculations usesthe methods of string theory to pre-dict the viscosity of the liquid beingcreated at RHIC and to explain someof the other surprising findings. Suchstudies will provide a more quantita-tive understanding of how “nearly per-fect” the liquid is.

The unexpected findings also intro-duce a wide range of opportunity fornew scientific discovery regarding theproperties of matter at extremes oftemperature and density previouslyinaccessible in a laboratory.

The STAR Collaboration, consisting of614 researchers from 52 institutions in12 countries, performs its data analysison NERSC’s PDSF system. The com-putations carried out at NERSC arefocused around analysis of the proces-sed data, comparison of the data withexperimental models, and studies ofdetector performance and accept-ance. A storage resource allocation atNERSC supports an important collab-oration between STAR, the BerkeleyLab Scientific Data ManagementGroup, and the Particle Physics DataGrid (PPDG) project, involving develop-ment and deployment of data gridtechnology to move terabytes of dataefficiently and securely.

One of STAR’s next priority goals atNERSC is to study the spin structureof the proton, focusing first on meas-urements of the gluon contribution.

Research funding (STAR only): NP, NSF, BMBF,

IN2P3, RA, RPL, EMN, EPSRC, FAPESP, RMST,

MEC, NNSFC, GACR, FOM, DAE, DST, CSIR, SNSF,

PSCSR, STAA

Computational resources: NERSC, BNL

This article written by: Karen McNulty Walsh

(Brookhaven National Laboratory), John Hules

Research News18

Neutrinos and antineutrinos are elusiveparticles. With no charge and almostno mass, they rarely react with othermatter, even as they zip through theEarth (and us) at nearly the speed oflight. Recording the traces of rareneutrino interactions requires special-ized, large-volume detectors, and find-ing those few traces among the massof collected data is a little like listen-ing for a whisper at a rock concert.

But the same qualities that makeneutrinos elusive also make themvaluable tools for probing environ-ments, like the core of the Earth, thatresist penetration by more commonobservational techniques.

So when an international group ofresearchers reported the first observa-tion of geologically produced antineu-trinos in a paper10 that was featuredon the cover of Nature (Figure 14), thegeological community responded withenthusiasm. This discovery is expect-ed to lead to better estimates of theabundance and distribution ofradioactive elements in the Earth, andof the Earth’s overall heat budget.

The geoneutrinos were detected atthe Kamioka Liquid Scintillator Anti-Neutrino Detector in Japan, betterknown as KamLAND. Located in amine cavern beneath the mountainsof Japan’s main island of Honshu,near the city of Toyama, KamLAND isthe largest low-energy antineutrinodetector ever built. It consists of aweather balloon 13 meters (43 feet)in diameter, filled with about a kiloton

of liquid scintillator, a chemical soupthat emits flashes of light when anincoming antineutrino collides with aproton. These light flashes are detectedby a surrounding array of 1,879 pho-tomultiplier light sensors which convertthe flashes into electronic signals thatcomputers can analyze.

Most of the geoneutrino data producedat KamLAND was stored on the HighPerformance Storage System (HPSS)at NERSC and analyzed using NERSC’s

PDSF cluster. Together, these systemsallowed scientists to find the scientificequivalent of a whisper at a rock con-cert.

KamLAND records data 24 hours aday, seven days a week. This data isshipped on tapes from the experimen-tal site to Berkeley Lab, where it isread off the tapes and stored in theHPSS. KamLAND records about 200GB of data each day, and HPSS cur-rently has more than 250 TB ofKamLAND data stored, makingKamLAND the second-largest user ofNERSC’s HPSS system.

During dedicated production periods atNERSC, the KamLAND data are readout of HPSS and run through recon-struction software to convert the wave-forms (essentially oscilloscope traces)from the photomultiplier tubes intophysically meaningful quantities suchas energy and position of the eventinside the detector. This reduces thedata volume by a factor of 60–100,and the reconstructed events arestored on disk for further analysis.

“The event reconstruction requires alot of computing power, and with over600 CPUs, PDSF is a great facility torun this kind of analysis,” said PatrickDecowski, a Berkeley Lab physicistwho works with NERSC staff on theproject. “PDSF has been essential forour measurements.”

With the data on disk, specializedanalysis programs run over the recon-structed events to extract the geoneu-

FIGURE 14. The July 28, 2005 issue of Naturereported that electron antineutrinos ema-nating from the earth (geoneutrinos) mayserve as a unique window into the interiorof our planet, revealing information that ishidden from other probes. The left half ofthe cover image shows the production dis-tribution for the geoneutrinos detected atKamLAND, and the right half shows thegeologic structure.

Whispers from undergroundGhostly geoneutrinos hint at Earth’s inner secrets

10 T. Araki et al., “Experimental investigation of geologically produced antineutrinos with KamLAND,” Nature 436, 499 (2005).

Research News 19

trino events and perform the finalanalysis. PDSF is also used for varioussimulation tasks in order to betterunderstand the background signals inthe detector.

“The whole analysis is like looking fora needle in a haystack—out of morethan 2 billion events, only 152 candi-dates were found,” Decowski said.“And of these, about 128 are back-ground events.”

Forty years ago, the late John Bahcallproposed the study of neutrinos comingfrom the sun to understand the fusionprocesses inside the sun. The meas-urement of a persistent deficit of theobserved neutrino flux relative toBahcall’s calculations led to the 2002Nobel Prize for Ray Davis and the dis-covery of neutrino oscillation, a quantummechanical phenomenon whereby aneutrino created with a specific lepton“flavor” (electron, muon, or tau) canlater be measured to have a differentflavor. KamLAND provided the mostdirect evidence of neutrino oscillationin 2004.

Today, antineutrinos are being used tostudy the interior of the Earth, whichis still little known. The deepest bore-hole ever drilled is less than 20 kmin depth, while the radius of the Earthis more than 6000 km. While seismicevents have been used to deduce theinterior composition of the Earth’sthree basic regions—the core, themantle and the crust—there are nodirect measurements of the chemicalmakeup of the deeper regions.

An important observation for under-standing the Earth is the measure-ment of the heat coming from within(Figure 15). These measurements showthat the Earth produces somewherebetween 30 and 45 terawatts of heat(1 TW is 1012 watts; the range dependson certain model assumptions). Twoimportant sources of heat generationare the primordial energy released fromplanetary accretion and latent heatfrom core solidification. However, it isbelieved that heat produced by radioac-tivity also plays an important role inthe Earth’s heat balance, contributingperhaps half of the total heat.

Neutrinos can help researchers under-stand the Earth’s internal structureand heat generation. Three importantisotopes that are part of currentEarth models—potassium, uranium,and thorium—produce electron anti-neutrinos (the so-called geoneutrinos)in their radioactive decay.

“KamLAND is the first detector sensi-tive enough to measure geoneutrinosproduced in the earth from the decayof uranium-238 and thorium-232,” saidStuart Freedman, a nuclear physicistwith a joint appointment at BerkeleyLab and the University of California atBerkeley, who is a co-spokesperson forthe U.S. team at KamLAND. “Since thegeoneutrinos produced from the decaychains of these isotopes have exceed-ingly small interaction cross sections,they propagate undisturbed in theEarth’s interior, and their measurementnear the Earth’s surface can be usedto gain information on their sources.”

In measuring geoneutrinos generatedin the decay of natural radioactiveelements in the earth’s interior, scien-tists believe it should be possible toget a three-dimensional picture of theearth’s composition and shell struc-ture. This could provide answers tosuch questions as how much terres-trial heat comes from radioactivedecays and how much is a primordialremnant from the birth of our planet.It might also help identify the sourceof Earth’s magnetic field and whatdrives the geodynamo.

The research is a multinational effort,as shown by the fact that the Naturearticle represented the work of 87authors from 14 institutions spreadacross four nations.

Research funding: NP, JAMSTEC

Computational resources: NERSC

This article written by: John Hules, Jon Bashor,

Lynn Yarris

0 40 60 85 120mW m–2

180 240 350

FIGURE 15. Earth’s conductive heat flow is estimated to be about 30–45 terawatts. Radioactivityis known to account for perhaps half of this heat, but there has been no accurate way tomeasure radiogenic heat production. (Image from H. N. Pollack, S. J. Hurter, and J. R.Johnson, Reviews of Geophysics 31, 267 [1993])

Research News20

Need to understand the details of howa molecule is put together? Want tosee the effects of the intricate dancethat its electrons do to make a chem-ical bond? Try blowing a molecule tobits and calculating what happens toall the pieces. That’s the approachtaken by an international group of col-laborators from the University ofCalifornia at Davis, universities inSpain and Belgium, and LawrenceBerkeley National Laboratory.

When a hydrogen molecule, H2, is hit bya photon with enough energy to sendboth its electrons flying, the two protonsleft behind—the hydrogen nuclei—repel each other in a so-called Coulombexplosion. In this event, called thedouble photoionization of H2, thepaths taken by the fleeing electronshave much to say about how closetogether the two nuclei were at themoment the photon struck, and justhow the electrons were correlated inthe molecule (Figure 16).

Correlation means that properties of theparticles like position and momentumcannot be calculated independently.When three or more particles areinvolved, calculations are notoriouslyintractable, both in classical physicsand quantum mechanics. But in theDecember 16, 2005 issue of Science,the researchers reported on the first-ever complete quantum mechanicalsolution of a system with four chargedparticles.11

The groundbreaking calculations wereinspired by earlier experiments on the

photofragmentation of deuterium(heavy hydrogen) molecules, per-formed at Berkeley Lab’s AdvancedLight Source (ALS) in 2003 by a groupof scientists from Germany, Spain,and several institutions in the UnitedStates. The experimenters were led byThorsten Weber, then with the ALS andnow at the University of Frankfurt.

“If you were trying to do this experi-ment and you didn’t have access tothe Advanced Light Source and aCOLTRIMS experimental device”—asophisticated, position-sensitivedetector for collecting electrons andions—“you’d just fire photons at arandom sample of hydrogen mole-cules and measure the electrons thatcame out,” said Thomas Rescigno ofBerkeley Lab’s Chemical SciencesDivision, one of the authors of theScience paper. “What made this exper-iment special was that they couldmeasure what happened to all fourparticles. From their precise positionsand energy they could reconstruct thestate of the molecule when it was hit.”

Weber presented early experimentaldata at a seminar attended by Rescigno,William McCurdy of the Lab’s ChemicalSciences Division, who is also a pro-fessor of chemistry at the University ofCalifornia at Davis, and Wim Vanroose,a postdoctoral fellow at Berkeley Labwho is now at the Department ofComputer Science at the KatholiekeUniversiteit Leuven in Belgium.

Said Rescigno, “Thorsten teased uswith his results, some of which were

extremely nonintuitive. What was remark-able was that very small differencesin the internuclear distance”—the dis-tance between the two protons at themoment the photon was absorbed—“made for radical differences in theways the electrons were ejected.”

“When I saw the results of the molec-ular experiments, in which smallchanges in the internuclear distanceproduced large and unexpectedchanges in the electron ejection pat-terns, it immediately occurred to methat the differences were because ofthe molecule’s effects on electroncorrelations,” McCurdy said.

McCurdy had recently been workingwith Fernando Martín, a professor ofchemistry at the UniversidadAutónoma de Madrid, merging compu-tational techniques developed byMartín with a method McCurdy,Rescigno, and others had developedfor calculating systems of threecharged particles. Martín andMcCurdy extended these methods tothe helium atom, a system that, tech-nically speaking, has four chargedparticles. But because the heliumatom’s two protons are bound togeth-er in the nucleus, the calculated dis-tribution of electrons ejected by theabsorption of an energetic photontend to be quite symmetrical aroundthe nucleus, with most pairs flying offin opposite directions.

The picture can look quite differentfor a hydrogen or deuterium molecule,in which a plot of the likelihood that

11 Wim Vanroose, Fernando Martín, Thomas N. Rescigno, and C. William McCurdy, “Complete photo-induced breakup of the H2 molecule as a probe of molecularelectron correlation,” Science 310, 1787 (2005).

Breaking up is hard to calculateBut researchers have achieved the first complete numerical solution of thefragmentation of a system with four charged particles

Research News 21

electrons will be ejected at certainangles groups into lobes that growincreasingly asymmetric as the bondlength between the two hydrogen atomsgrows longer. McCurdy read this as theeffect of the bond length on the corre-lation of the shared electrons. Indeed,this is what Weber and his colleaguesspeculated when they published theresults of their deuterium photofrag-mentation studies in Nature in 2004.

Rescigno pointed out a fly in the oint-ment, however—namely that insteadof being caused by electron correla-tions, large differences in ejectionpatterns caused by small differencesin internuclear distance “could just bekinematics.”

In other words, the scattered electronsmight be sharing some of the potentialenergy stored by the Coulomb repul-sion between the two like-chargedprotons. The closer together thesetwo nuclei are at the moment thephoton breaks up the molecule, themore energy goes into the Coulombexplosion, some of which could betransmitted to the outgoing electronsand affect their flight paths.

How to decide between kinematiceffects or electron correlations? Theexperimental results could not addressthe question, since all the data werecollected at the same photon energy;whether the electrons were acquiringadditional kinetic energy was unknown.

But, said Vanroose, “because wewere doing computations, we coulddo experiments the experimenterscouldn’t do. We had much more flexi-bility to fix the conditions.”

Using supercomputers at NERSC, atUC Berkeley, and in Belgium,Vanroose was able to rerun the hydro-gen molecule experiments “in silico,”this time with different photon energies,distributed so that the outgoing elec-

FIGURE 16. A hydrogen molecule hit by an energetic photon breaks apart. The ejected electrons,blue, take paths whose possible trajectories (here represented by net-like lobes) depend onhow far apart the hydrogen nuclei, red, are at the moment the photon strikes. The bondlength at that instant reflects how the molecule’s electrons are correlated. (Courtesy WimVanroose)

Research News22

trons always shared exactly the samekinetic energy no matter what the dis-tance between the protons at themoment of photon absorption.

The results turned out to be remarkablysimilar in all cases. Even when kineticenergy made no contribution, the elec-trons flew off in patterns determinedby the length of the bond between thenuclei (Figure 17). Therefore the dif-ferences were due almost entirely tothe way the electrons were correlatedin their orbital paths around the mole-cule’s two nuclei.

Martín of UA Madrid sees the newcalculations, which are a completenumerical solution for the Schrödingerequation of the photoionization of H2, as“just the beginning. Probing the com-plicated physics of electron correlationswill lead the way to more comprehen-sive methods combining theory andexperiment to address some of themost pressing problems in chemistry.”

Vanroose credits their success to day-in, day-out collaboration between top-notch theorists and experimenters“who are talking to each other all thetime. The ability of experimentaliststo call on the latest computationaltechniques is good for both; it’s whywe’re two years ahead of other theo-rists in this field.”

To Rescigno, the latest results showthat “what began as blue-sky physicstheory is now connecting with the nutsand bolts of practical experiment.”

Said McCurdy, “These large-scale the-oretical calculations, stimulated by theneed to interpret novel experimentsat the ALS, are already stimulatingnew experiments and establishing anew line of inquiry at Berkeley Lab.”

Research funding: BER, MECS, BSPO

Computational resources: NERSC, UCB, KUL

This article written by: Paul Preuss

A

B

FIGURE 17. Effect of the internuclear distance on the possible electron trajectories (angular dis-tributions, shown as lobes). (A) Angular distributions shown with molecular axis (yellow) 15°from polarization vector (green arrow) and fixed electron (red arrow) leaving perpendicular tothat plane with 50% of the available ejection energy. (Left) Inner vibrational turning point;(middle) equilibrium internuclear distance; (right) outer turning point. Background showsground state potential curve (blue), ground state vibrational wave function (red), and energy(green line, with black dots indicating equilibrium and inner and outer turning points). (B) Theeffect on the angular distribution of varying internuclear distance is primarily due to changesin molecular electronic correlation. The angular distributions closely resemble those in (A)when the photon energy is varied, so as to produce the same amount of final kinetic energy tobe shared by the outgoing electrons, regardless of the internuclear distance at which ioniza-tion occurs. Inset shows the corresponding atomic case (helium).

Research News 23

A reality show called “Density FunctionalTheory” or “DFT” is not going to bechallenging “American Idol” in the tel-evision ratings anytime soon. After all,DFT does not involve snarky judges andviewer voting; it is a quantum mechan-ical computational method used tocalculate quantities such as the bindingenergy of molecules. But in the righthands, DFT has something in com-mon with the popular talent contest—the ability to sort through a mob of“wannabes” and find a small numberof candidates with unusual talents.

In a series of three recent papers,12

chemical engineers Jeff Greeley andManos Mavrikakis13 of the Universityof Wisconsin-Madison reported thepromising results of their talentsearch using DFT calculations—a search for better catalysts.

Catalysts reduce the amount of energyneeded to start a chemical reaction,enabling the reaction to happen fasteror at a lower temperature. They play amajor role in the chemical industry forthe production and processing ofmaterials such as plastics, fuel, andpesticides, and in the pharmaceuticalindustry for drug synthesis.

Hydrogen catalysts are particularlyimportant to the DOE HydrogenProgram, whose mission is toresearch and develop fuel cell andhydrogen production, delivery, andstorage technologies, thus ensuringan abundant and affordable supply ofclean energy. One of the HydrogenProgram’s priority research areas is

design of catalysts at the nanoscale,where reduced size often maximizescatalytic properties. Catalysis is vitalto the success of the program becauseof its role in producing hydrogen fromwater or from carbon-containing fuelssuch as coal and biomass, and itsrole in producing electricity fromhydrogen in fuel cells.

“Anything that would reduce the costof materials in low-temperature fuelcells would certainly make the tech-nology more economically competitiveand thus bring fuel cells closer tobeing implemented in our everydaylives,” Mavrikakis said. “A major frac-tion of that cost is the cost of noblemetals, such as platinum, that areused as catalysts.” Combining plat-inum with less expensive metals isone possible way of reducing theprice of fuel cells, but only if thealloys are efficient catalysts—andeven today’s expensive catalysts arenot efficient enough to be economi-cally competitive.

So the research goal is clear: find lessexpensive, more efficient catalysts forhydrogen-related reactions. However, thediscovery of catalysts by experimentalmethods is an expensive process oftrial and error, often involving hundredsor even thousands of candidate mate-rials. But a new theoretical approach—rational design of catalysts from firstprinciples, based on the reactions wewant them to facilitate—can speedup the discovery process by eliminat-ing most of the candidates that areunlikely to succeed.

Using this approach, Greeley andMavrikakis found a new class of near-surface alloys (NSAs) that exhibitsuperior catalytic behavior for hydrogen-related reactions (Figure 18). NSAsare alloys in which the compositionnear the surface differs from the bulkcomposition. Two idealized NSA struc-tures (Figure 19) were used in thestudy: an overlayer of the solute metalon the host metal, and a subsurfacealloy, in which a layer of solute liesjust below the surface of the host.

Earlier studies had discovered a fewNSAs that had unusual catalytic prop-erties, so Greeley and Mavrikakisdeveloped a novel and systematic DFTframework for screening a large num-ber of NSA structures and compositionsto determine their stability in hydrogen-rich environments. The stable NSAswere then examined for their hydrogenbinding and dissociation energies. (Infuel cells, dissociation is the separa-tion of the hydrogen atoms’ protonsand electrons, which allows a flow ofelectrical energy to be created.)

After screening 46 different NSAs, theresearchers found a group of five thatbind atomic hydrogen (H) as weaklyas noble metals, such as gold, cop-per, and silver, while at the same timedissociating molecular hydrogen (H2)much more easily. This is an unusualcombination of properties: mostmaterials with weak H binding areinactive for H2 dissociation. “Thisunique set of properties may permitthese alloys to serve as low-tempera-ture, highly selective catalysts for

Talent scoutingVirtual “auditions” of many alloys find a few with the potential to become“superstar” catalysts

12 J. Greeley and M. Mavrikakis, “Alloy catalysts designed from first principles,” Nature Materials 3, 810 (2004); “Surface and subsurface hydrogen: Adsorptionproperties on transition metals and near-surface alloys,” Journal of Physical Chemistry B 109, 3460 (2005); “Near-surface alloys for hydrogen fuel cell applica-tions,” Catalysis Today 111, 52 (2006).

13 Mavrikakis is Associate Professor and head of the Computational Surface Science and Catalysis Group in the Department of Chemical and Biological Engineeringat UWM. Since receiving his Ph.D. from UWM, Greeley has accepted a position in the Center of Atomic-Scale Materials Physics at the Technical University of Denmark.

Research News24

pharmaceuticals production and asrobust fuel-cell anodes,” Greeley andMavrikakis wrote in their NatureMaterials article.

Another important consideration forfuel cell anodes is resistance to car-bon monoxide (CO) poisoning. Smallamounts of CO are always present inhydrogen fuel streams, and thosetraces can bind to a platinum anode,ruining its ability to act as a catalyst.However, the NSAs with weak H bind-ing also exhibit weak CO binding,making them ideal candidates for fuelcell anodes. The DFT results identi-fied the following subsurface alloysas having all of these desirable prop-erties: vanadium in platinum, tanta-lum in palladium, tungsten in plat-inum, molybdenum in platinum, andtantalum in platinum.

FIGURE 19. Two types of near-surface alloys were studied: overlayers (left) and subsurface alloys(right).

14 http://www.nano.gov/

Host Solute

FIGURE 20. Schematic representation of anNSA defect site in cross-section and topview. The gold circles denote host metalatoms, while the blue circles indicate thesubsurface solute impurity.

FIGURE 18. Illustration of bimetallic near surface alloys (NSAs) that bind H as weakly as noblemetals but activate H2 much more easily. Niels Bohr (1885–1962, left) and Paul Sabatier(1854–1941, right) represent the founders of quantum mechanics and catalysis theory.

Research News 25

How relevant are these computationalresults, based on idealized struc-tures, to NSAs in the real world? Intheir Catalysis Today paper, Greeleyand Mavrikakis admitted, “In reality, itis fairly uncommon for NSAs to havea pure solute monolayer in either thefirst or second metal layers.” But,they argued, “Calculations haveshown … that modest deviations …do not qualitatively change the unusu-al catalytic properties of NSAs.” Infact, they suggest that minor impuri-ties, or “defect sites” (Figure 20),may have unusual catalytic propertiesin their own right, combining fastkinetics for bond-breaking or bond-

making events with resistance to poi-soning by the reactants or productsof those reactions.

In any case, deviations from the idealstructure would produce quantitativechanges in NSA properties. “Thus, tofully benefit from the unique catalyticproperties of NSAs, it will be neces-sary to achieve more precise control ofmetal catalyst surface structure, per-haps by implementing improved cata-lyst nanofabrication techniques,” theresearchers wrote. Several promisingnanofabrication techniques are alreadybeing developed, and perfecting suchtechniques is part of the mission of

DOE’s new Nanoscale Science ResearchCenters, established under theNational Nanotechnology Initiative.14

One additional advantage of the “tal-ented” NSAs is that they allow easydiffusion of atomic hydrogen into thebulk of the metal, raising the possibil-ity of using them as light hydrogen-storage media. This characteristic willbe studied in more detail in futureresearch.

Research funding: BES, NSF

Computational resources: NERSC, NPACI, MSCF

This article written by: John Hules

Surface chargeElectrical charging is found to increase the reactivity of a gold nanocluster catalyst

14 http://www.nano.gov/15 B. Yoon, H. Häkkinen, U. Landman, A. S. Wörz, J.-M. Antonietti, S. Abbet, K. Judai, and U. Heiz, “Charging effects on bonding and catalyzed oxidation of CO on

Au8 clusters on MgO,” Science 307, 403 (2005)

Researchers at the Georgia Instituteof Technology and Technical UniversityMunich have discovered evidence of aphenomenon that may lead to drasti-cally lowering the cost of manufacturingof materials from plastics to fertilizers.Studying nano-sized clusters of goldon a magnesium oxide surface, scien-tists found direct evidence for electri-cal charging of a nano-sized catalyst.15

This is an important factor in increas-ing the rate of chemical reactions.

“The fabrication of most syntheticmaterials that we use involves usingcatalysts to promote reaction rates,”said Uzi Landman, director of theCenter for Computational MaterialsScience at Georgia Tech. “Designingcatalysts that are more efficient, moreselective and more specific to a cer-tain type of reaction can lead to sig-nificant savings in manufacturing

expenses. Understanding the princi-ples that govern nanocatalysis is keyto developing more effective catalysts.”

The current study builds on jointresearch done since 1999 by the twogroups that found gold, which is non-reactive in its bulk form, is a veryeffective catalyst when it is in nan-oclusters of eight to about two dozenatoms in size. Those specific sizesallow the gold clusters to take on athree-dimensional structure, which isimportant for its reactivity.

“It is possible to tune the catalyticprocess not only by changing thecomposition of the materials, but alsoby changing the cluster’s size atom byatom,” explained Ueli Heiz, professor ofchemistry at Technical University Munich.In these earlier studies of the reactionwhere carbon monoxide and molecular

oxygen combine to form carbon diox-ide, Landman’s group used computersimulations to predict that when goldnanoclusters of eight atoms are usedas the catalyst and magnesium oxideis used as the catalytic bed, reactionswould occur when the bed had defectsin the form of missing oxygen atoms,but would not occur when the magne-sium oxide was defect-free.

Heiz’s experiments confirmed thisprediction and the teams concludedthat the gold must be anchoring itselfto the defect where it picks up anelectron, giving the gold a slight nega-tive charge. Theoretical simulationsshowed that the electronic structureof the gold clusters matches up withthe oxygen and carbon monoxide(Figure 21). The charged gold transfersan electron to the reacting molecules,weakening the chemical bonds that

Research News26

keep them together. Once the bond isweak enough it breaks, allowing reac-tions to occur.

Now, in this latest study, the group hasfound direct evidence that this isindeed what is happening. Using eight-atom gold clusters as the catalyst andmagnesium oxide as the catalyticbed, the team measured and calculat-ed the strength of the bonds in thecarbon monoxide by recording the fre-quency of the molecule’s vibrations.

“If carbon monoxide is a strong bond,

then there is a certain frequency tothis vibration,” explained Landman.“If the bond of the carbon monoxidebecomes weaker, then the frequencybecomes lower. That’s exactly whatwe saw—when we had defects in themagnesium oxide, we had largershifts than when we had magnesiumoxide without defects.”

Lead author of the paper BokwonYoon, a senior research scientist inLandman’s group, commented, “Theagreement between the predicted andthe measured values of the vibrational

frequency shifts is very gratifying,confirming the charging and bondingmechanisms.”

“And all this happens at low tempera-tures,” said Heiz. Typically, reactionsrequiring catalysts need heat or pres-sure to get the reaction going, andthat adds to the cost of manufacturing,but that isn’t the case here. Sincethe properties of the catalytic bedscan increase the rate of reactions fornanocatalysts, new and better low-temperature catalysts may be found.

“We knew the specific number ofatoms in the catalyst and that defectsin the catalytic beds are important.Now we know why those defects areso essential—because they allow thecatalyst to be electrically charged. Wehope these guidelines will lead to moreresearch in search of nano-sized cata-lysts. It’s possible that at the nanoscaleyou may find catalysts that can dothings under more gentle and cheaperconditions,” said Landman.

Research funding: BES, USAF-OSR, DFG, LBW,

AVHF, JSPS, SNSF, GMODGO

Computational resources: NERSC, HPCMP

This article written by: David Terraso (Georgia Tech)

Magnetic disks in spaceA new generation of simulations and experiments aim to pinpoint the ori-gins of instability in accretion disks

(a) (b)

Auexcesscharge

chargedeficiency

Mg

CO

O2

O

(c)

FIGURE 21. Charge differences for three configurations of an eight-atom gold nanocluster on amagnesium oxide (MgO) surface: (a) gold cluster on a perfect MgO surface; (b) gold clusteranchored on an oxygen vacancy defect, showing enhanced charging of the cluster; (c) sameas (b) but with adsorbed reactants, showing charging of the carbon monoxide and oxygenmolecules.

Accretion disks—structures formed bymaterial falling toward and then orbitingaround a gravitational source—are akey part of the process by which newlyborn stars and black holes accumu-late mass and grow (Figure 22). Thegravitational energy released by accre-

tion is believed to power many of theenergetic phenomena observed in theuniverse, such as the jets streamingfrom galactic nuclei (Figure 23).

Accretion disks form because most ofthe matter being attracted toward the

gravitational well has some angularmomentum, which results in the mat-ter orbiting the proto-star or blackhole rather than falling right in. Andthat creates a problem for theorists,according to Fausto Cattaneo, aresearcher at Argonne National

Research News 27

Laboratory and the Center forMagnetic Self-Organization at theUniversity of Chicago.

“If the angular momentum is con-served,” Cattaneo said, “the materialwill happily orbit the central object andnever fall in. In order for material to fall,it must lose its angular momentum.”

In the early days of accretion disktheory, the loss of angular momentumwas not considered a problem, becausetwo obvious explanations were easilyavailable: collisional processes likefriction or viscosity, and shear-driventurbulence.

If two particles in the disk collide, theywill exchange angular momentum,and the faster-rotating one will slowdown and fall into a lower orbit. If thecollision rate is high enough, therewill be a spiral of matter flowing downinto the central gravitational well. Thishypothesis makes sense until you dothe math. “In most astrophysical situ-ations, collisional processes aremany orders of magnitude too smallto account for the observed energyrelease rates,” Cattaneo explained.Accretion disks are simply too big tobe destabilized by such processes.

Turbulence is a more promisinghypothesis, because turbulent eddies

380 Arc Seconds88,000 Light Years

1.7 Arc Seconds400 Light Years

FIGURE 22. An artist’s impression of a black hole with a closely orbiting companion star. In-falling matter forms an accretion disk, which feeds matter into the black hole. Hot gas rushesfrom the vicinity of the black hole, creating jets perpendicular to the disk. (Image courtesy ofNASA.)

FIGURE 23. The giant elliptical galaxy NGC4261, one of the 12 brightest galaxies inthe Virgo cluster. (Left) Composite ground-based optical and radio view of the galaxy.Photographed in visible light (white), thegalaxy appears as a fuzzy disk of hundredsof billions of stars. A radio image (orange)shows a pair of opposed jets emanatingfrom the nucleus. (Right) Hubble SpaceTelescope image of the core of NGC 4261:The disk is tipped enough to provide aclear view of the bright hub, which presum-ably harbors a black hole. The dark, dustydisk represents a cold outer region whichleads to an ultra-hot accretion disk withina few hundred million miles from the blackhole. (Image courtesy of NASA.)

Research News28

provide a much more efficient mecha-nism for transporting angular momen-tum. And models of accretion disksthat assume a reasonable amount ofturbulence have produced credibleaccretion rates. “The challenge hasbeen to provide an underlying physi-cal mechanism to generate the turbu-lence in the first place,” Cattaneosaid. Shear-driven instabilities, arisingfrom the varying rotational speedswithin the disk, were another attrac-tive possibility that had to be ruledout. “The problem is that the rotationprofiles—the average angular velocityas a function of radius—of typicalaccretion disks are close to followingKepler’s laws of planetary motion,and so they are very stable to infini-tesimal perturbations.”

“Two decades of accretion disk researchfailed to provide a single local insta-bility mechanism that could operatein disks and lead to the desired stateof turbulence,” Cattaneo continued.“This lamentable state of affairsimproved significantly in the early 1990swhen it was realized that the stabilityproperties of near-Keplerian disks couldbe dramatically affected by magneticfields. Disks that were hydrodynami-cally stable could become hydromag-netically unstable, and remarkably, allit took was a weak magnetic field.Since most astrophysical accretiondisks are ionized and hence goodelectrical conductors, this magneto-rotational instability, as it came to becalled, could provide the desiredmechanism for turbulence production.”

The physical origin of magneto-rota-tional instability can be illustrated bya simple analogy: Imagine two space-craft orbiting near each other at slightlydifferent altitudes. If the orbits arenearly circular, the inner spacecraftwill have higher angular velocity thanthe outer one. Now assume that alight elastic cord joins the two space-craft, providing a weak tension. The

effect of the tension on the innerspacecraft is to slow it down, i.e., toreduce its angular velocity, and there-fore to move it into a lower orbit. Incontrast, the effect of the tension onthe outer spacecraft is to accelerateit, i.e., to increase its angular velocity,thereby moving it to a higher orbit.With one spacecraft going lower andthe other going higher, it is hard topredict what will happen when theystretch their cord to its full length—will they start bouncing wildly, or willthe cord break and propel them fur-ther apart?—but clearly, the systemwill become unstable.

A magnetic field in an electrically con-ducting disk can produce magnetictension effects that are analogous tothe elastic tension in the spacecraftexample. By connecting portions ofthe disk that are rotating at differentspeeds, the magnetic field turnsangular velocity into a source of insta-bility. And because the central gravita-tional field does not play a direct rolein magneto-rotational instability, labo-ratory experiments have been devel-oped that can test many aspects ofthis mechanism on a small scale. Inthese experiments, the space betweentwo coaxial rotating cylinders is filledwith an electrically conducting fluid,typically liquid sodium or gallium. Therotation rates of the inner and outercylinders are chosen so that the rota-tion profile is hydrodynamically stable.Then a weak external magnetic fieldis applied so that the origins anddevelopment of magneto-rotationalinstability can be studied.

“Although the physical input providedby these experiments is invaluable,”Cattaneo said, “they suffer from twolimitations: First, in a flow of liquidmetal, it’s impossible to see, and dif-ficult even to detect, what is happen-ing.” The second limitation concernsthe Reynolds number, a mathematicalexpression that measures the

strength of advective effects relativeto diffusion. “In any laboratory setup,the magnetic Reynolds number of liq-uid metals is a few tens, and withextreme efforts it can be raisedslightly to exceed one hundred,” hesaid. “This should be contrasted withthe typical astrophysical situation inwhich the magnetic Reynolds numberis huge—millions to billions.”

Fortunately, this is a case in whichnumerical simulations can elucidateand reach even further than theexperiments. In an INCITE-funded project titled “Magneto-RotationalInstability and Turbulent AngularMomentum Transport,” Cattaneo andhis colleagues Paul Fischer andAleksandr Obabko have created three-dimensional numerical simulationsthat reproduce the geometry of thelaboratory experiments conducted byHantao Ji of the Princeton PlasmaPhysics Laboratory and JeremyGoodman of Princeton University. Butunlike the experiments, the simulationshave magnetic Reynolds numbersexceeding 50,000—not quite as highas in actual accretion disks, but avaluable extension of the liquid metalresults. The purpose of these simula-tions is to clarify the mechanisms ofangular momentum transport in mag-netized, rotationally constrained turbu-lence, and to apply them to the prob-lem of astrophysical accretion flows.

“If you can do the research both com-putationally and experimentally, youare much better off than just usingone approach,” Cattaneo said.“Seeing is knowing, and in numericalsimulations, you can visualize any-thing, so it is much easier to identifythe beginnings of the instability andthe structures that transport angularmomentum.”

In these simulations, as in mostexperiments to date, the magneticfield that catalyzes the magneto-rota-

Research News 29

tional instability is aligned with therotation axis. Visualizations show thatin this case, the coherent structuresthat transport the angular momentumoutward are toroidal vortex rings ema-nating from the inner cylinder (Figure24). Knowing what the structures looklike and how they behave advancesthe theoretical understanding of thisinstability and also makes possiblethe future creation of phenomenologi-cal models to describe this phenome-non. These models could then beplugged into simulations that can runon workstations instead of supercom-puters, or that can use parameters

closer to those found in real astro-physical accretion disks.

Although Cattaneo and his colleagueswill be spending the next year analyz-ing all the data from their simulations,some intriguing possibilities are alreadyemerging. “If you turn the cylinderson their side,” he said, “the transportof angular momentum looks similar tothe transport of heat in a convectingfluid, like a pot of water on a hot-plate. We’re going to examine this inmore detail to see whether some ofthe ideas of convection theory can beapplied to fluid flow in accretion disks.”

The close collaboration between thetheorists in Chicago and the experi-mentalists in Princeton facilitates theexchange of results from bothapproaches. “We used the experimentalresults to validate our computationalapproach,” Cattaneo said. “And oursimulations are giving the experimentalresearchers a better idea of what tolook for and where to look for it. Theyare already using our results to improvethe design of their experiments.”

Research funding: INCITE, HEP, NSF

Computational resources: NERSC, CMSO, BGL

This article written by: John Hules

8.8e-03 8.8e-03

–8.8e-03

fh = 0.005 fm = 0.005

–8.8e-03

8.8e-03 8.8e-03

–8.8e-03

fh = 0.005 fm = 0.005

–8.8e-03

8.8e-03 8.8e-03

–8.8e-03

fh = 0.005 fm = 0.005

–8.8e-03

FIGURE 24. Visualization of the time evolution of the outward transport of angular momentum in a magnetic fluid bounded by rotating cylinders.The two colors correspond to the transport by hydrodynamic (orange) and hydromagnetic (purple) fluctuations. (Simulations by F. Cattaneo, P.Fischer, and A. Obabko, with visualization assistance from Cristina Siegerist of the Berkeley Lab/NERSC Visualization Group. The animation canbe viewed at http://flash.uchicago.edu/~cattaneo/Animations/fh-fm_1024x768_1.mpg.)

Research News30

Proteins in motionBiochemists are systematically simulating the unfolding pathways of all knownprotein folds, hoping to discover the general principles of protein folding

Proteins do the work of life; they areessential to the structures and func-tions of all living cells. Some proteinsplay structural or mechanical roles,others catalyze chemical reactions,and still others are involved in storageor transportation or immune response.

A protein’s biological function dependson its shape. Proteins are synthesizedas long, non-branching chains of aminoacids; but in order to perform theirproper functions, each type of proteinmust assume a unique, stable, andprecisely ordered three-dimensionalfolded structure, which biologists call itsnative state. Protein folding might becompared to a shoelace tying itself,except the shapes of proteins are muchmore complicated than knots, and thefolding process is much faster, beingcompleted only microseconds to sec-onds after the protein is created.

Incorrectly folded proteins are respon-sible for many illnesses, includingAlzheimer’s disease, cystic fibrosis,Huntington’s disease, Parkinson’s dis-ease, Type II diabetes, Creutzfeldt-Jakob disease (the human counter-part of mad cow disease), and manycancers. The degradation of proteinfolding may even play a key role inaging. But despite the importance ofprotein folding, the phenomenonremains largely a mystery.

“Protein folding is one of the funda-mental unsolved problems in molecularbiology,” said Valerie Daggett, Professorof Medicinal Chemistry and AdjunctProfessor of Biochemistry, Biomedical

and Health Informatics, and Bioengin-eering at the University of Washington.“Although we know a lot about of thestructural details of the native foldedconformation of proteins, very little isknown about the actual foldingprocess. Experimental approachesonly provide limited amounts of infor-mation on the structural transitionsand interactions occurring during pro-tein folding. Given that protein foldingis of such widespread importance tohuman health, we are using computersimulation methods in an attempt todelineate the important forces actingduring this process.”

Daggett is principal investigator of anINCITE project called “MolecularDynameomics,” which was awarded 2 million processor-hours at NERSC.Just as genomics studies the com-plete genetic makeup of an organism,and proteomics studies the completeprotein structure and functioning ofan organism, dynameomics is anambitious attempt to combine molec-ular dynamics and proteomics, usingmolecular dynamics simulations tocharacterize and catalog thefolding/unfolding pathways of repre-sentative proteins from all known pro-tein folds. There are approximately1,130 non-redundant protein folds,and Daggett’s group has used theirINCITE grant to simulate proteinsfrom the 151 most common folds,which represent about 75% of allknown protein structures (Figure 25).

“Structure prediction remains one ofthe elusive goals of protein chem-

istry,” Daggett said. “In order to trans-late the current deluge of genomicinformation into knowledge about pro-tein functions, which we can then usefor drug design, we have to be able tosuccessfully predict the native statesof proteins starting with just theamino acid sequence.”

The novelty of her group’s computationalapproach, however, is that they workin the opposite direction, starting notwith a protein’s amino acid sequencebut with its folded structure, as deter-mined by NMR spectroscopy and X-raycrystallography experiments. A proteincan easily be unfolded or denatured,in both experiments and simulations,by simply raising the temperature.(Anyone who has ever cooked an egghas seen the results of denaturation:the heat causes the bonds within theegg white proteins to break, unfoldingthe individual protein molecules andallowing them to form bonds withother molecules, thus solidifying theegg white.) Daggett’s research overthe last several years suggests thatprotein unfolding follows the samepattern as folding, moving in reversethrough the same transition statesand intermediate structures. The hightemperature of the simulations, 498Kelvin (437° F), just speeds up theprocess without modifying the overallpathway, she says.

To perform these simulations, Daggett’sgroup uses a code called “in lucemMolecular Mechanics” or ilmm, whichwas developed by David Beck, DarwinAlonso, and Daggett. The code solves

Research News 31

FIGURE 25. The first 156 dynameomics simulation targets.

Research News32

Newton’s equations of motion for everyatom in the protein molecule and sur-rounding solvent environment, determin-ing the trajectories of the atoms overtime. Obtaining a reasonably good pic-ture of protein unfolding pathwaysrequires six simulations for each pro-tein: one 20 nanosecond (ns) simula-tion of the native (folded) moleculardynamics at a temperature of 298 Kor 77° F (Figure 26); and five simula-tions at 498 K, three at 2 ns durationand two at 20 ns (Figure 27).

Thus the 151 proteins studied in theINCITE project required a total of 906simulations and produced 2 terabytesof compressed data. Even with theallocation of 2 million processor-hours,the researchers had to limit the sizeof the chosen proteins to no morethan 300 amino acids. Fortunately,the larger, more complex proteins aregenerally structured as if they wereassembled from a number of smallershapes, so understanding the smallershapes should reveal principles thatare applicable to all proteins.

In fact, Daggett hopes that catalogingthe folding strategies for every type ofprotein fold will enable the discoveryof general rules of protein folding, whichso far have eluded researchers, andthat those rules will be used to improvealgorithms for predicting protein struc-ture. The results of the protein unfold-ing simulations will be made publiclyavailable on the Dynameomics web-site (www.dynameomics.org). Just asthe Protein Data Bank has becomean important repository of experimen-tal results that has enabled many fur-ther discoveries, Daggett envisionsthe Dynameomics simulation data-base as a resource for the wholeresearch community, in which datamining may produce unexpected find-ings and a dynamic description of pro-teins will aid in understanding theirbiological functions.

David Beck, a graduate student in theDaggett lab, worked with NERSC con-sultants to optimize ilmm’s performanceon Seaborg. “The INCITE award gaveus a unique opportunity to improve thesoftware, as well as do good science,”Beck said. Improvements included

load balancing, which sped up the codeby 20%, and parallel efficiency, whichreached 85% on 16-processor nodes.The INCITE award enabled the teamto do five times as many simulationsas they had previously completedusing other computing resources.

FIGURE 26. Protein target 3rub: A schematic representation of the secondary structure takenfrom a native state simulation of the enzyme RuBisCO, the most abundant protein in leavesand possibly the most abundant protein on Earth. Water, hydrogen, and protein atoms includ-ed in the simulation are not shown. The highly dynamic region (image top) is responsible forfinding the protein’s binding partner (not included in the simulation). RuBisCO catalyzes thefirst major step of carbon fixation, a process by which atmospheric carbon dioxide is madeavailable to organisms in the form of energy-rich molecules such as sucrose.

Research News 33

The researchers are now analyzing thesimulation results and comparing themto experimental results whenever pos-sible. “Our preliminary analysis of thisdata set focused on the native controlsimulations and the endpoint of thethermal unfolding simulations—thedenatured state,” Daggett said. “Inparticular, we are evaluating the inter-

actions between amino acids. We havealready found a shift in the preva-lence and types of interactions as theproteins unfold. Characterizing theinteractions in the denatured stateprovides much-needed informationabout the starting point of folding.This represents the first step towardsproviding crucial information for pre-

diction of the folding process and ulti-mately the folded, native structure.”

And, of course, there are still hun-dreds more protein folds to simulate.

Research funding: INCITE, BER

Computational resources: NERSC

This article written by: John Hules

FIGURE 27. Protein target 1imf: Schematic representation of secondary structures taken at 1 ns intervals from a thermal unfolding simulation ofinositol monophosphatase, an enzyme that may be the target for lithium therapy in the treatment of bipolar disorder.

Two NERSC users with joint appoint-ments at the University of California,Berkeley and Lawrence BerkeleyNational Laboratory were elected tothe National Academy of Sciences:Steven G. Louie, Professor of Physicsand an expert on nanotubes andfullerenes, is one of the 25 mosthighly cited authors in nanoscienceand one of the 100 most citedresearchers in all of physics. BarbaraA. Romanowicz, Director of theBerkeley Seismological Laboratory,studies deep earth structure anddynamics; she discovered that themysterious waves that constantlyreverberate through the Earth, evenon days without earthquakes, origi-nate from storm-driven sea waves.

Michael Norman, Professor of Physicsat the University of California, SanDiego, was elected a fellow of theAmerican Academy of Arts andSciences. Norman is PI of the 2006

INCITE project “Precision CosmologyUsing the Lyman Alpha Forest,” whichaims to measure the cosmologicalparameters that describe the shape,matter-energy contents, and expan-sion history of the Universe.

The American Physical Society pre-sented the David Adler LectureshipAward to James Chelikowsky, Professorof Physics, Chemical Engineering, andChemistry and Biochemistry at theUniversity of Texas, “for his creativeand outstanding research in computa-tional materials physics and for hiseffectiveness in communicatingresearch results through lectures andpublications.”

Three NERSC users were recipients ofthe Presidential Early Career Awardsfor Scientists and Engineers: Wei Cai,an Assistant Professor of MechanicalEngineering at Stanford University, washonored for work he did at Lawrence

Livermore National Laboratory indeveloping a computer model thatsimulates the dynamics of crystals asthey deform. Hong Qin, a physicist atPrinceton Plasma Physics Laboratory,was recognized for his contributions tothe physics of high-intensity particlebeams and for his work on electro-magnetic effects in magnetically con-fined plasmas. Zhangbu Xu, a physicistat Brookhaven National Laboratory,was honored for his novel researchtechniques which led to the detectionof subatomic particles termed “short-lived resonances” and “open charm”at the Relativistic Heavy Ion Collider.

Yu-Heng Tseng, a computational climateresearcher at Lawrence BerkeleyNational Laboratory and a lecturer atthe University of California, Berkeley,was honored as an OutstandingOverseas Young Scientist by theFoundation for the Advancement ofOutstanding Scholarship in Taiwan.

Research News34 Research News34

NERSC Users Honored

Top row, left to right: Louie, Romanowicz, Norman, and Chelikowsky; bottom row, left to right:Cai, Qin, Xu, and Tseng

THE NERSC CENTER

NERSC traditionally provides systems and services that maximize the scientific productivity of its

user community, and takes pride in its reputation for the expertise of its staff and the high quality of

services delivered to its users. In support of the DOE Office of Science’s mission, the NERSC Center

served 2,677 scientists throughout the United States in 2005. Appendix B provides statistical infor-

mation about NERSC’s clients, their primary affiliations, and their scientific disciplines.

Scientific productivity is the most important measure of NERSC’s success, as evidenced by more

than 1,400 scientific publications in 2005 that were based entirely or in part on calculations done at

NERSC.1 To maintain its effectiveness, NERSC proactively addresses new challenges. How the

NERSC Center is working to improve that productivity even more in the face of growing scientific

and technological challenges is discussed in the following pages.

The NERSC Center36

Science-DrivenComputingNERSC is continuously reassessingits approach to supporting high-endscientific computing, and from time totime undertakes a major reevaluationand realignment. In 2005 this involvedseveral activities: the NERSC Users’Group writing and publishing the latestDOE Greenbook, the development ofNERSC’s five-year plan for 2006– 2010,and a programmatic peer review con-ducted for the DOE. The overall themeof this evaluation and planning effortwas Science-Driven Computing.

DOE Greenbook PublishedThe DOE Greenbook: Needs andDirections in High-PerformanceComputing for the Office of Sciencewas compiled by Steve Jardin for theNERSC Users Group and published inJune 2005 (Figure 1). With contribu-tions from 37 scientists from a varietyof disciplines and organizations, thisreport documents the computationalscience being done at NERSC andother DOE computing centers andprovides examples of computationalchallenges and opportunities that willguide the evolution of these centersover the next few years.

According to the Greenbook, researchersin all of the disciplines supported bythe Office of Science are finding thatlarge-scale computational capabilitiesare now essential for the advance-ment of their research. Today’s mostpowerful computers and scientificapplication codes are being used toproduce new and more precise scien-tific results at the cutting edge ofeach discipline, and this trend is des-tined to continue for years to come.The Greenbook presents many exam-ples of the impact of large-scale com-putations on the sciences.

However, the Greenbook points out that

the current computational resourcesavailable through NERSC are saturated,and the lack of additional computingresources is becoming “a major bot-tleneck in the scientific research anddiscovery process.” The report advises,“A large increase in computer poweris needed in the near future to takethe understanding of the science tothe next level and to help secure theU.S. DOE SC leadership role in thesefundamental research areas.”

The Greenbook’s specific recommen-dations include:

• Expand the high performance computing resources available atNERSC, maintaining an appropriatesystem balance to support the widerange of large-scale applicationsinvolving production computing anddevelopment activities in the DOEOffice of Science.

• Configure the computing hardwareand queuing systems to minimizethe time-to-completion of large jobs,

as well as to maximize the overallefficiency of the hardware.

• Actively support the continuedimprovement of algorithms, soft-ware, and database technology forimproved performance on parallelplatforms.

• Significantly strengthen the compu-tational science infrastructure atNERSC that will enable the optimaluse of current and future NERSCsupercomputers.

• Carefully evaluate the requirementsof data- or I/O-intensive scientificapplications in order to support aswide a range of science as possible.

NERSC’s Five-Year PlanWith guidance from the DOE Greenbookand other interactions with the NERSCuser community, as well as monitoringof technology trends, NERSC hasdeveloped a five-year plan focusing onthree components: Science-DrivenSystems, Science-Driven Services, andScience-Driven Analytics (Figure 2).NERSC management and staff haveobserved three trends that need to beaddressed over the next several years:

• the widening gap between applica-tion performance and peak perform-ance of high-end computing systems

• the recent emergence of large, multi-disciplinary computational scienceteams in the DOE research commu-nity

• the flood of scientific data from bothsimulations and experiments, andthe convergence of computationalsimulation with experimental datacollection and analysis in complexworkflows.

NERSC’s responses to these trends arethe three components of the science-

1 A list is available at http://www.nersc.gov/news/reports/ERCAPpubs05.php.

FIGURE 1. The DOE Greenbook, prepared bythe NERSC Users Group, is available onlineat http://www.nersc.gov/news/greenbook/2005greenbook.pdf.

The NERSC Center 37

driven strategy that NERSC will imple-ment and realize in the next five years:

• Science-Driven Systems: Balancedintroduction of the best new tech-nologies for complete computationalsystems—computing, storage, net-working, visualization and analysis—coupled with the activities necessaryto engage vendors in addressing theDOE computational science require-ments in their future roadmaps.

• Science-Driven Services: The entirerange of support activities, fromhigh-quality operations and userservices to direct scientific support,that enable a broad range of scien-tists to effectively use NERSC sys-tems in their research. NERSC willconcentrate on resources neededto realize the promise of the newhighly scalable architectures for sci-entific discovery in multidisciplinarycomputational science projects.

• Science-Driven Analytics: The archi-tectural and systems enhancementsand services required to integrateNERSC’s powerful computationaland storage resources to providescientists with new tools to effec-tively manipulate, visualize, andanalyze the huge data sets derivedfrom simulations and experiments.

This balanced set of objectives will becritical for the future of the NERSCCenter and its ability to serve theDOE scientific community. Elementsof this strategy that are currentlybeing implemented are discussed inthe following pages. The full five-yearplan can be read at http://www.nersc.gov/news/reports/LBNL-57582.pdf.

DOE Review of NERSCOn May 17–19, 2005, a ProgrammaticReview of NERSC was conducted forthe Department of Energy. The peerreview committee was chaired by Frank

Williams of the Arctic Region Super-computing Center at the University ofAlaska, Fairbanks. Other memberswere Walter F. Brooks of the AdvancedSupercomputing Division at NASA AmesResearch Center, Lawrence Buja of theNational Center for AtmosphericResearch, Cray Henry of the DefenseDepartment’s High Performance Com-puting Modernization Program, RobertMeisner of the National NuclearSecurity Administration, José L. Muñozof the National Science Foundation,and Tomasz Plewa of the Center forAstrophysical Thermonuclear Flashesat the University of Chicago.

In addition to reviewing the DOEGreenbook and NERSC’s five-yearplan, the review panel heard presen-tations and engaged in conversationscovering all aspects of NERSC’s oper-ations. The DOE had requested thatthe panel address a number of spe-cific topics, but they were also giventhe freedom to look into any aspectof NERSC and to comment according-

ly. The panel responded by presentinga detailed list of findings and recom-mendations to DOE and NERSC man-agers, who are now using those find-ings to improve NERSC’s operations.

The overall conclusions of the reviewcommittee included a strong endorse-ment of NERSC’s approach toenabling computational science:

NERSC is a strong, productive, andresponsive science-driven centerthat possesses the potential tosignificantly and positively impactscientific progress by providing userswith access to high performancecomputing systems, services, andanalytics beneficial to the supportand advancement of their science….

Members of the review panel eachreport that NERSC is extremelywell run with a lean and knowledge-able staff. The panel memberssaw evidence of strong and com-mitted leadership, and staff who

FIGURE 2. Conceptual diagram of NERSC’s plan for 2006–2010.

The NERSC Center38

are capable and responsive tousers’ needs and requirements.Widespread, high regard for thecenter’s performance, reflected insuch metrics as the high numberof publications supported byNERSC, and its potential to posi-tively impact future advancementof computational science, war-rants continued support.

Organizational Changes In order to implement the new initia-tives and the changes in emphasisderived from the planning and reviewprocess, in November 2005 NERSCannounced several organizationalchanges, including two new associategeneral managers, two new teams,and a new group.

“In order to efficiently carry out ourplan and meet the expectations ofour users and sponsors, we are modi-fying the NERSC Center organization,”General Manager Bill Kramer wrote inannouncing the changes. “In additionto the Division, Department andGroup components of the organiza-tion, we will have two other compo-nents: Functional Areas and Teams.”

NERSC has created two functionalareas—Science-Driven Systems andScience-Driven Services. The majorityof the NERSC staff will work in thesetwo areas (Figure 3). The functionalareas are responsible for carrying outthe responsibilities and tasks discussedin the respective sections of NERSC’sfive-year plan. Functional areas will beled by Associate General Managers(AGMs), who are responsible for coor-dinating activities across the groupsand teams in their areas. FrancescaVerdier is associate general managerfor Science-Driven Services, andHoward Walter is associate generalmanager for Science-Driven Systems(Figure 4).

The Accounts and Allocations Team,the Analytics Team, the Open Softwareand Programming Group, and theUser Services Group will report to theScience-Driven Services AGM. TheComputational Systems Group, theComputer, Operations and ESnetSupport Group, the Mass StorageGroup, and the Networking, Securityand Servers Group will report to theScience-Driven Systems AGM.

The reorganization includes the cre-ation of one new group and two newteams. They are:

• Analytics Team: Analytics is theintersection of visualization, analysis,scientific data management, human-computer interfaces, cognitive sci-ence, statistical analysis, and rea-soning. The primary focus of theAnalytics Team is to provide visuali-zation and scientific data manage-ment solutions to the NERSC usercommunity to better understandcomplex phenomena hidden in sci-entific data. The responsibilities ofthe team span the range fromapplying off-the-shelf commercialsoftware to advanced development

FIGURE 3. NERSC’s new organization reflects new priorities and promotes coordination acrossgroups and teams.

National Energy Research Scientific Computing (NERSC) Center Division

HORST SIMON Division DirectorWILLIAM KRAMER Division Deputy

Science Driven Systems

HOWARD WALTERAssociate General Manager

Computational Systems

Mass Storage

Networking, Security, & Servers

Computer Operations & ESnet Support

User Services

Analytics

Open Software & Programming

Accounts & Allocations

Science Driven Services

FRANCESCA VERDIERAssociate General Manager

Science Driven SystemArchitecture

JOHN SHALF

JAMES CRAW

HOWARD WALTER (Acting)

BRENT DRANEY

STEVE LOWE

JONATHAN CARTER

WES BETHEL

FRANCESCA VERDIER (Acting)

CLAYTON BAGWELL

High Performance Computing Department

WILLIAM KRAMERNERSC Center

General Manager

to realizing new solutions where nonepreviously existed. The AnalyticsTeam is a natural expansion of thevisualization efforts that have beenpart of NERSC since it moved toBerkeley Lab. Wes Bethel is theteam leader (Figure 5). NERSC’sanalytics strategy is discussed inmore detail beginning on page 50.

• Open Software and Programming(OSP) Group: The growing use ofopen-source software and partiallysupported software requires achange of approach to NERSC’sneeds for the future. These areasnow are a key component of NERSC’sability to provide high-quality sys-tems and services. This group isresponsible for the support andimprovement of open-source andother partially supported software,particularly the software that NERSCuses for infrastructure, operations,and delivery of services. Key effortsinclude open-source engineering,development and support of middle-ware (Grid and Web tools), and

NERSC’s software infrastructure.Francesca Verdier is the actinggroup leader until a permanent oneis recruited.

• Science-Driven System Architecture(SDSA) Team: This team performsongoing evaluation and assessmentof technology for scientific comput-ing. The SDSA Team has expertisein benchmarking, system perform-ance evaluations, workload monitor-ing, use of application modelingtools, and future algorithm scalingand technology assessment. Usingscientific methods, the team willdevelop methods for analyzing pos-sible technical alternatives and willcreate a clear understanding of cur-rent and future NERSC workloads.The SDSA Team will engage withvendors and the general researchcommunity to advocate technologi-cal features that will enhance theeffectiveness of systems for NERSCscientists. The team is responsiblefor ongoing management of a suiteof benchmarks that NERSC and

Berkeley Lab use for architecturalevaluation and procurement. Thisincludes composite benchmarksand metrics such as SSP, ESP, vari-ation, reliability, and usability. Theteam will matrix staff from bothNERSC and Berkeley Lab’sComputational Research Divisionfor specific areas such as algorithmtracking and scaling, which aredesigned to develop and documentfuture algorithmic requirements.The scientific focus for this effortwill change periodically and willstart with applied mathematics andastrophysics. The SDSA Teamleader is John Shalf (Figure 5).Their current activities are dis-cussed beginning on page 46.

Completing the reorganization,Jonathan Carter succeeds FrancescaVerdier as leader of the User ServicesGroup, and Brent Draney succeedsHoward Walter as leader of theNetworking, Security and ServersGroup (Figure 6).

FIGURE 4. NERSC’s new associate general managers, Francesca Verdierand Howard Walter, are responsible for Science-Driven Services andScience-Driven Systems, respectively.

FIGURE 5. Wes Bethel heads NERSC’s new Analytics Team, while JohnShalf heads the new Science-Driven Systems Architecture Team.

The NERSC Center 39

Science-Driven SystemsNERSC deploys five major types ofhardware systems—computationalsystems, filesystems, storage sys-tems, network, and analytics andvisualization systems—all of whichmust be carefully designed and inte-grated to maximize productivity for theentire computational community sup-ported by the DOE Office of Science.In 2005 NERSC implemented majorupgrades or improvements in all fivecategories, as described below.

System specifications must meetrequirements that are constantlyevolving because of technologicalprogress and the changing needs ofscientists. Therefore, NERSC is con-stantly planning for the future, notjust by tracking trends in science andtechnology and planning new systemprocurements, but also by activelyinfluencing the direction of technologi-cal development through efforts suchas the Science-Driven SystemArchitecture collaborations.

Two New Clusters: Jacquard andBassi

In August 2005 NERSC accepted a722-processor Linux Networx Evolocitycluster system named “Jacquard” forfull production use (Figure 7). Theacceptance test included a 14-dayavailability test, during which a selectgroup of NERSC users were given fullaccess to the Jacquard cluster tothoroughly test the entire system inproduction operation. Jacquard had a99 percent availability uptime duringthe testing while users and scientistsran a variety of codes and jobs on thesystem.

The Jacquard system is one of thelargest production InfiniBand-basedLinux cluster systems and met rigorousacceptance criteria for performance,reliability, and functionality that areunprecedented for an InfiniBand cluster.Jacquard is the first system to deployMellanox 12x InfiniBand uplinks in itsfat-tree interconnect, reducing network

hot spots and improving reliability bydramatically reducing the number ofcables required.

The system has 640 AMD 2.2 GHzOpteron processors devoted to com-putation, with the rest used for I/O,interactive work, testing, and intercon-nect management. Jacquard has apeak performance of 2.8 teraflop/s.Storage from DataDirect Networksprovides 30 TB of globally availableformatted storage.

Following the tradition at NERSC, thesystem was named for someone whohas had an impact on science orcomputing. In 1801, Joseph-MarieJacquard invented the Jacquard loom,which was the first programmablemachine. The Jacquard loom usedpunched cards and a control unit thatallowed a skilled user to programdetailed patterns on the loom.

In January 2006, NERSC launched an888-processor IBM cluster named“Bassi” into production use (Figure 8).Earlier, during the acceptance testing,users reported that codes ran from 3 to 10 times faster on Bassi thanon NERSC’s other IBM supercomputer,Seaborg, leading one tester to call thesystem the “best machine I have seen.”

Bassi is an IBM p575 POWER5 sys-tem, and each processor has a theo-retical peak performance of 7.6gigaflop/s. The processors are distrib-uted among 111 compute nodes witheight processors per node. Processorson each node have a shared memorypool of 32 GB. A Bassi node is anexample of a shared memory proces-sor, or SMP.

The compute nodes are connected toeach other with a high-bandwidth, low-latency switching network. Each noderuns its own full instance of the stan-dard AIX operating system. The diskstorage system is a distributed, paral-

FIGURE 6. Promoted to group leaders were Brent Draney of the Networking, Security andServers Group and Jonathan Carter of the User Services Group.

The NERSC Center 41

lel I/O system called GPFS (IBM’sGeneral Parallel File System). Addi-tional nodes serve exclusively asGPFS servers. Bassi’s network switchis the IBM “Federation” HPS switch,which is connected to a two-link net-work adapter on each node.

One of the test users for NERSC’s twonew clusters was Robert Duke of theUniversity of North Carolina, ChapelHill, the author of the PMEMD code,which is the parallel workhorse inmodern versions of the popular chem-istry code AMBER. PMEMD is widelyused for molecular dynamics simula-tions and is also part of NERSC’sbenchmark applications suite. Dukehas worked with NERSC’s David Skinnerto port and improve the performanceof PMEMD on NERSC systems.

“I have to say that both of thesemachines are really nothing short of

fabulous,” Duke wrote to Skinner.“While Jacquard is perhaps the best-performing commodity cluster I haveseen, Bassi is the best machine Ihave seen, period.”

Other early users during the accept-ance testing included the INCITE proj-ect team “Direct Numerical Simulationof Turbulent Nonpremixed Combustion”(see page 11). “Our project requireda very long stretch of using a largefraction of Bassi processors—512processors for essentially an entiremonth,” recounted Evatt Hawkes.“During this period we experiencedonly a few minor problems, which isexceptional for a pre-productionmachine, and enabled us to completeour project against a tight deadline.We were very impressed with the reli-ability of the machine.”

Hawkes noted that their code alsoported quickly to Bassi, starting withFIGURE 7. Jacquard is a 722-processor Linux Networx Evolocity cluster system with a theoretical

peak performance of 2.8 teraflop/s.

FIGURE 8. Bassi is an 888-processor IBM p575 POWER5 system with a theoretical peak per-formance of 6.7 teraflop/s.

The NERSC Center42

a code already ported to Seaborg’sarchitecture. “Bassi performs verywell for our code. With Bassi’s fasterprocessors we were able to run on farfewer processors (512 on Bassi asopposed to 4,096 on Seaborg) andstill complete the simulations morerapidly,” Hawkes added. “Based onscalar tests, it is approximately 7times faster than Seaborg and 11/2

times faster than a 2.0 GHz Opteronprocessor. Also, the parallel efficiencyis very good. In a weak scaling test,we obtained approximately 78 per-cent parallel efficiency using 768processors, compared with about 70percent on Seaborg.”

The machine is named in honor ofLaura Bassi, a noted Newtonianphysicist of the eighteenth century.Appointed a professor at the Universityof Bologna in 1731, Bassi was thefirst woman to officially teach at aEuropean university.

New Visual Analytics Server:DaVinciIn mid-August, NERSC put into produc-tion a new server specifically tailoredto data-intensive visualization andanalysis. The 32-processor SGI Altix,called DaVinci (Figure 9), offers inter-active access to large amounts oflarge memory and high performanceI/O capabilities well suited for analyz-ing large-scale data produced by theNERSC high performance computingsystems (Bassi, Jacquard, andSeaborg).

With its 192 gigabytes (GB) of RAMand 25 terabytes (TB) of disk,DaVinci’s system balance is biasedtoward memory and I/O, which is dif-ferent from the other systems atNERSC. This balance favors data-intensive analysis and interactivevisualization. DaVinci has 6 GB ofmemory per processor, compared to

4 GB per processor on Jacquard andBassi and 1 GB on Seaborg.

Users can obtain interactive accessto 80 GB of memory from a singleapplication (or all 192 GB of memoryby prior arrangement), whereas theinteractive limits on productionNERSC supercomputing systemsrestrict interactive tasks to a smalleramount of memory (256 MB on loginnodes). While DaVinci is available pri-marily for interactive use, the systemis also configured to run batch jobs,especially those jobs that are dataintensive.

The new server runs a number ofvisualization, statistics, and mathe-matics applications, including IDL,AVS/Express, CEI Ensight, VisIT (aparallel visualization application from

Lawrence Livermore NationalLaboratory), Maple, Mathematica, andMatLab. Many users depend on IDLand MatLab to process or reorganizedata in preparation for visualization.The large memory is particularly ben-eficial for these types of jobs.

DaVinci is connected to the NERSCGlobal Filesystem (see below), HighPerformance Storage System (HPSS),and ESnet networks by two independ-ent 10 gigabit Ethernet connections.

With DaVinci now in production, NERSChas retired the previous visualizationserver, Escher, and the math server,Newton.

NERSC Global Filesystem In early 2006, NERSC deployed theNERSC Global Filesystem (NGF) intoproduction, providing seamless dataaccess from all of the Center’s com-putational and analysis resources.NGF is intended to facilitate sharing ofdata between users and/or machines.For example, if a project has multipleusers who must all access a commonset of data files, NGF provides a com-mon area for those files. Alternatively,when sharing data between machines,NGF eliminates the need to copylarge datasets from one machine toanother. For example, because NGFhas a single unified namespace, auser can run a highly parallel simula-tion on Seaborg, followed by a serialor modestly parallel post-processingstep on Jacquard, and then perform adata analysis or visualization step onmove a single data file.

NGF’s single unified namespace makesit easier for users to manage theirdata across multiple systems (Figure10). Users no longer need to keeptrack of multiple copies of programsand data, and they no longer need tocopy data between NERSC systems forpre- and post-processing. NGF provides

FIGURE 9. DaVinci is a 32-processor SGI Altixwith 6 GB of memory per processor and25 TB of disk memory, a configurationdesigned for data-intensive analysis andinteractive visualization.

The NERSC Center 43

several other benefits as well: storageutilization is more efficient becauseof decreased fragmentation; computa-tional resource utilization is more effi-cient because users can more easilyrun jobs on an appropriate resource;NGF provides improved methods ofbacking up user data; and NGFimproves system security by eliminat-ing the need for collaborators to use“group” or “world” permissions.

“NGF stitches all of our systemstogether,” said Greg Butler, leader of theNGF project. “When you go from sys-tem to system, your data is just there.Users don’t have to manually movetheir data or keep track of it. They cannow see their data simultaneouslyand access the data simultaneously.”

NERSC staff began adding NGF tocomputing systems in October 2005,starting with the DaVinci visualizationcluster and finishing with the Seaborgsystem in December. To help test thesystem before it entered production,a number of NERSC users were givenpreproduction access to NGF. Earlyusers helped identify problems withNGF so they could be addressedbefore the filesystem was made avail-able to the general user community.

“I have been using the NGF for sometime now, and it’s made my work a loteasier on the NERSC systems,” saidMartin White, a physicist at BerkeleyLab. “I have at times accessed fileson NGF from all three compute plat-forms (Seaborg, Jacquard, and Bassi)semi-simultaneously.”

NGF also makes it easier for mem-bers of collaborative groups to accessdata, as well as ensure data consis-tency by eliminating multiple copiesof critical data. Christian Ott, a Ph.D.student and member of a team study-ing core-collapse supernovae, wrotethat “the project directories make ourcollaboration much more efficient. We

can now easily look at the output ofthe runs managed by other teammembers and monitor their progress.We are also sharing standard inputdata for our simulations.”

NERSC General Manager Bill Kramersaid that as far as he knows, NGF isthe first production global filesystemspanning five platforms (Seaborg,Bassi, Jacquard, DaVinci, and PDSF),three architectures, and four differentvendors. While other centers and dis-tributed computing projects such asthe National Science Foundation’sTeraGrid may also have sharedfilesystems, Butler said he thinksNGF is unique in its heterogeneity.

The heterogeneous approach of NGFis a key component of NERSC’s five-year plan. This approach is importantbecause NERSC typically procures amajor new computational systemevery three years, then operates it forfive years to support DOE research.Consequently, NERSC operates in aheterogeneous environment with sys-tems from multiple vendors, multipleplatforms, different system architec-tures, and multiple operating systems.The deployed filesystem must operate

in the same heterogeneous clientenvironment throughout its lifetime.

Butler noted that the project, which isbased on IBM’s proven GPFS technol-ogy (in which NERSC was a researchpartner), started about five years ago.While the computing systems, stor-age, and interconnects were mostly inplace, deploying a shared filesystemamong all the resources was a majorstep beyond a parallel filesystem. Inaddition to the different system archi-tectures, there were also differentoperating systems to contend with.However, the last servers and storagehave now been deployed. To keepeverything running and ensure agraceful shutdown in the event of apower outage, a large uninterruptiblepower supply has been installed inthe basement of the OaklandScientific Facility.

While NGF is a significant change forNERSC users, it also “fundamentallychanges the Center in terms of ourperspective,” Butler said. For example,when the staff needs to do mainte-nance on the filesystem, the variousgroups need to coordinate their effortsand take all the systems down at once.

NGF

FIOURE 10. NGF is the first production global filesystem spanning five platforms (Seaborg,Jacquard, PDSF, DaVinci and Bassi), three architectures, and four different vendors.

The NERSC Center44

Storage servers, accessing the con-solidated storage using the shared-disk filesystems, provide hierarchicalstorage management, backup, andarchival services. The first phase ofNGF is focused on function and notraw performance, but in order to beeffective, NGF has to have perform-ance comparable to native clusterfilesystems. The current capacity ofNGF is approximately 70 TB of user-accessible storage and 50 millioninodes (the data structures for indi-vidual files). Default project quotasare 1 TB and 250,000 inodes. Thesystem has a sustainable bandwidthof 3 GB/sec bandwidth for streamingI/O, although actual performance foruser applications will depend on avariety of factors. Because NGF is adistributed network filesystem, per-formance will be only slightly lessthan that of filesystems that are localto NERSC compute platforms. Thisshould only be an issue for applica-tions whose performance is I/Obound.

NGF will grow in both capacity andbandwidth over the next several years,eventually replacing or dwarfing theamount of local storage on systems.NERSC is also working to seamlesslyintegrate NGF with the HPSS dataarchive to create much larger “virtual”data storage for projects. Once NGFis completely operational within theNERSC facility, Butler said, users atother centers, such as the NationalCenter for Atmospheric Research andNASA Ames Research Center, couldbe allowed to remotely access theNERSC filesystem, allowing users toread and visualize data without havingto execute file transfers. Eventually,the same capability could be extend-ed to experimental research sites,such as accelerator labs.

NGF was made possible by IBM’sdecision to make its GPFS softwareavailable across mixed-vendor super-

computing systems. This strategy wasa direct result of IBM’s collaborationwith NERSC. “Thank you for driving usin this direction,” wrote IBM FederalClient Executive Mike Henesy toNERSC General Manager Bill Kramerwhen IBM announced the project inDecember 2005. “It’s quite clear wewould never have reached this pointwithout your leadership!”

NERSC’s Mass Storage Group collab-orated with IBM and the San DiegoSupercomputer Center to develop aHierarchical Storage Manager (HSM)that can be used with IBM’s GPFS.The HSM capability with GPFS pro-vides a recoverable GPFS filesystemthat is transparent to users and fullybacked up and recoverable fromNERSC’s multi-petabyte archive onHPSS. GPFS and HPSS are both clus-ter storage software: GPFS is ashared disk filesystem, while HPSSsupports both disk and tape, movingless-used data to tape while keepingcurrent data on disk.

One of the key capabilities of theGPFS/HPSS HSM is that users’ filesare automatically backed up on HPSSas they are created. Additionally, fileson the GPFS which have not beenaccessed for a specified period oftime are automatically migrated fromonline resources as space is neededby users for files currently in use.Since the purged files are alreadybacked up on HPSS, they can easilybe automatically retrieved by userswhen needed, and the users do notneed to know where the files arestored to access them. “This givesthe user the appearance of almostunlimited disk storage space withoutthe cost,” said NERSC’s MassStorage Group Leader Nancy Meyer.

This capability was demonstrated inthe Berkeley Lab and IBM booths atthe SC05 conference. Bob Coyne ofIBM, the industry co-chair of the HPSS

executive committee, said, “There areat least ten institutions at SC05 whoare both HPSS and GPFS users, manywith over a petabyte of data, who haveexpressed interest in this capability.HPSS/GPFS will not only serve theseexisting users but will be an impor-tant step in simplifying the storagetools of the largest supercomputercenters and making them available toresearch institutions, universities, andcommercial users.”

“Globally accessible data is becomingthe most important part of Grid com-puting,” said Phil Andrews of the SanDiego Supercomputer Center. “Theimmense quantity of informationdemands full vertical integration froma transparent user interface via a highperformance filesystem to an enor-mously capable archival manager. Theintegration of HPSS and GPFS closesthe gap between the long-term archivalstorage and the ultra high perform-ance user access mechanisms.”

The GPFS/HPSS HSM will be includedin the release of HPSS 6.2 in spring2006.

Integrating HPSS into GridsNERSC’s Mass Storage Group is cur-rently involved in another develop-ment collaboration, this one withArgonne National Laboratory and IBM,to integrate HPSS accessibility into theGlobus Toolkit for Grid applications.

At Argonne, researchers are addingfunctionality to the Grid file transferdaemon2 so that the appropriateclass of service can be requestedfrom HPSS. IBM is contributing thedevelopment of an easy-to-call libraryof parallel I/O routines that work withHPSS structures and are also easy tointegrate into the file transfer dea-mon. This library will ensure that Gridfile transfer requests to HPSS moversare handled correctly.

2 A daemon is a program that runs continuously in the background until it is activated by a particular event. The word daemon is Greek for “an attendant power or spirit.”

The NERSC Center 45

NERSC is providing the HPSS plat-form and testbed system for IBM andArgonne to do their respective devel-opment projects. As pieces are com-pleted, NERSC tests the componentsand works with the developers to helpidentify and resolve problems.

The public release of this capability isscheduled with HPSS 6.2, as well asfuture releases of the Globus Toolkit.

Bay Area MAN InauguratedOn August 23, 2005, the NERSCCenter became the first of six DOEresearch sites to go into full produc-tion on the Energy Science Network’s(ESnet’s) new San Francisco Bay AreaMetropolitan Area Network (MAN).The new MAN provides dual connec-tivity at 20 to 30 gigabits per second(10 to 50 times the previous sitebandwidths, depending on the site

using the ring) while significantlyreducing the overall cost.

The connection to NERSC consists oftwo 10-gigabit Ethernet links. One linkis used for production scientific com-puting traffic, while the second is dedicated to special networkingneeds, such as moving terabyte-scaledatasets between research sites ortransferring large datasets which arenot TCP-friendly.

“What this means is that NERSC isnow connected to ESnet at the samespeed as ESnet’s backbone network,”said ESnet engineer Eli Dart.

The new architecture is designed tomeet the increasing demand for net-work bandwidth and advanced networkservices as next-generation scientific

instruments and supercomputers comeon line. Through a contract with QwestCommunications, the San FranciscoBay Area MAN provides dual connec-tivity to six DOE sites—the StanfordLinear Accelerator Center, LawrenceBerkeley National Laboratory, the JointGenome Institute, NERSC, LawrenceLivermore National Laboratory, andSandia National Laboratories/California(Figure 11). The MAN also provideshigh-speed access to Califor-nia’shigher education network (CENIC),NASA’s Ames Research Center, andDOE’s R&D network, Ultra Science Net.The Bay Area MAN connects to boththe existing ESnet production back-bone and the first segments of the newScience Data Network backbone.

The connection between the MAN andNERSC was formally inaugurated on

BERKELEY

OAKLAND

PALO ALTO

WALNUTCREEK

LIVERMORE

CONNECTION TO EL PASO

CONNECTIONTO CHICAGO

SUNNYVALE

NASAAmes

FIGURE 11. ESnet’s new San Francisco Bay Area Metropolitan Area Network provides dual con-nectivity at 20 to 30 gigabits per second to six DOE sites and NASA Ames Research Center.

FIGURE 12. DOE Office of Science DirectorRaymond Orbach (left) and Berkeley LabDirector Steven Chu made the ceremonialconnection between NERSC and ESnet inJune. After testing, the full production con-nection was launched in August.

The NERSC Center46

June 24 by DOE Office of ScienceDirector Raymond Orbach and BerkeleyLab Director Steven Chu (Figure 12).

Another Checkpoint/RestartMilestone

On the weekend of June 11 and 12,2005, IBM personnel used NERSC’sSeaborg supercomputer for dedicatedtesting of IBM’s latest HPC SoftwareStack, a set of tools for high perform-ance computing. To maximize systemutilization for NERSC users, insteadof “draining” the system (letting run-ning jobs continue to completion)before starting this dedicated testing,NERSC staff checkpointed all runningjobs at the start of the testing period.“Checkpointing” means stopping aprogram in progress and saving thecurrent state of the program and itsdata—in effect, “bookmarking” wherethe program left off so it can start uplater in exactly the same place.

This is believed to be the first full-scale use of the checkpoint/restartsoftware with an actual productionworkload on an IBM SP, as well as thefirst checkpoint/restart on a systemwith more than 2,000 processors. Itis the culmination of a collaborativeeffort between NERSC and IBM thatbegan in 1999. Of the 44 jobs thatwere checkpointed, approximately65% checkpointed successfully. Ofthe 15 jobs that did not checkpointsuccessfully, only 7 jobs were deletedfrom the queuing system, while therest were requeued to run again at alater time. This test enabled NERSCand IBM staff to identify some previ-ously undetected problems with thecheckpoint/restart software, and theyare now working to fix those problems.

In 1997 NERSC made history by beingthe first computing center to achievesuccessful checkpoint/restart on amassively parallel system, the Cray T3E.

Science-Driven SystemArchitecture

The creation of NERSC’s Science-Driven System Architecture (SDSA)Team formalizes an ongoing effort tomonitor and influence the direction oftechnology development for the bene-fit of computational science. NERSCstaff are collaborating with scientistsand computer vendors to refine com-puter systems under current or futuredevelopment so that they will provideexcellent sustained performance perdollar for the broadest possible rangeof large-scale scientific applications.

While the goal of SDSA may seemambitious, the actual work that pro-motes that goal deals with the nitty-gritty of scientific computing—forexample, why does a particular algo-rithm perform well on one system butpoorly on another—at a level of detailthat some people might find tediousor overwhelming, but which the SDSAteam finds fascinating and challenging.

“All of our architectural problems wouldbe solvable if money were no object,”said SDSA Team Leader John Shalf,“but that’s never the case, so we haveto collaborate with the vendors in acontinuous, iterative fashion to worktowards more efficient and cost-effec-tive solutions. We’re not improvingperformance for its own sake, but weare improving system effectiveness.”

Much of the SDSA work involves per-formance analysis: how fast do variousscientific codes run on different sys-tems, how well do they scale to hun-dreds or thousands of processors, whatkinds of bottlenecks can slow themdown, and how can performance beimproved through hardware develop-ment. A solid base of performance datais particularly useful when combinedwith workload analysis, which considerswhat codes and algorithms are common

to NERSC’s diverse scientific workload.These two sets of data lay a founda-tion for assessing how that workloadwould perform on alternative systemarchitectures. Current architecturesmay be directly analyzed, while futurearchitectures may be tested throughsimulations or predictive models.

The SDSA Team is investigating anumber of different performance mod-eling frameworks, such as the SanDiego Supercomputer Center’s MemoryAccess Pattern Signature (MAPS), inorder to assess their accuracy in pre-dicting performance for the NERSCworkload. SDSA team members areworking closely with San Diego’sPerformance Modeling and Characteri-zation Laboratory to model the per-formance of the NERSC-5 SSP bench-marks and compare the performancepredictions to the benchmark resultscollected on existing and proposedHPC systems.

Another important part of the SDSAteam’s work is sharing performance andworkload data, along with benchmarkingand performance monitoring codes, withothers in the HPC community. Bench-marking suites, containing applicationcodes or their algorithmic kernels, arewidely used for system assessment andprocurement. NERSC has recentlyshared its SSP benchmarking suitewith National Science Foundation (NSF)computer centers. With the DefenseDepartment’s HPC ModernizationProgram, NERSC has shared bench-marks and jointly developed a new one.

Seemingly mundane activities like thesecan have an important cumulativeimpact: as more research institutionsset specific goals for application per-formance in their system procurementspecifications, HPC vendors have torespond by offering systems that arespecifically designed and tuned tomeet the needs of scientists andengineers, rather than proposing strictly

The NERSC Center 47

off-the-shelf systems. By workingtogether and sharing performance datawith NERSC and other computer cen-ters, the vendors can improve theircompetitive position in future HPCprocurements, refining their systemdesigns to redress any architecturalbottlenecks discovered through theiterative process of benchmarkingand performance modeling. The endresult is systems better suited for sci-entific applications and a better-defined niche market for scientificcomputing that is distinct from thebusiness and commercial HPC market.

The SDSA Team also collaborates onresearch projects in HPC architecture.One key project, in which NERSC iscollaborating with Berkeley Lab’sComputational Research Division andcomputer vendors, is ViVA, or VirtualVector Architecture. The ViVA conceptinvolves hardware and softwareenhancements that would coordinate aset of commodity scalar processors tofunction like a single, more powerfulvector processor. ViVA would enablemuch faster performance for certaintypes of widely used scientific algo-rithms, but without the high cost ofspecialized processors. The research isproceeding in phases. ViVA-1 is focusedon a fast synchronization register tocoordinate processors on a node ormulticore chip. ViVA-2 is investigatinga vector register set that hides latencyto memory using vector-like semantics.Benchmark scientific kernels are beingrun on an architectural simulator withViVA enhancements to assess theeffectiveness of those enhancements.

Perhaps the most ambitious HPCresearch project currently under wayis the Defense Advanced ResearchProjects Agency’s (DARPA’s) HighProductivity Computer Systems(HPCS) program. HPCS aims to devel-op a new generation of hardware andsoftware technologies that will takesupercomputing to the petascale

level and increase overall system pro-ductivity ten-fold by the end of thisdecade. NERSC is one of several“mission partners” participating inthe review of proposals and mile-stones for this project.

Proposals for New SystemEvaluatedAs part of NERSC’s regular computa-tional system acquisition cycle, theNERSC-5 procurement team wasformed in October 2004 to developan acquisition plan, select and testbenchmarks, and prepare a requestfor proposals (RFP). The RFP wasreleased in September 2005; propos-als were submitted in November andare currently being evaluated. TheRFP set the following general goalsfor the NERSC-5 system:

• Support the entire NERSC workload,specifically addressing the DOEGreenbook recommendations (seepage 36).

• Integrate with the NERSC environ-ment, including the NERSC GlobalFilesystem, HPSS, Grid software,security and networking systems, andthe user environment (software tools).

• Provide the optimal balance of thefollowing system components:

o computational: CPU speed, memorybandwidth, and latency

o memory: aggregate and per paralleltask

o global disk storage: capacity andbandwidth

o interconnect: bandwidth, latency,and scaling

o external network bandwidth.

The RFP also stated specific goals forperformance, disk storage, space andpower requirements, software, etc.NERSC-5 is expected to significantly

increase computational time forNERSC users beginning in the 2007allocation year.

Two recent reports3,4 on high-endcomputing recommended interagencycollaboration on system procure-ments. The National Research Councilreport stated, “Joint planning andcoordination of acquisitions willincrease the efficiency of the procure-ment processes from the governmentviewpoint and will decrease variabilityand uncertainty from the vendor view-point.”5 NERSC-5 is possibly the firstprocurement involving collaborationwith other government agencies. Thiscollaboration includes the sharing ofbenchmarks with DOD and NSF asmentioned above. In addition, fourorganizations—the DOD HPC Modern-ization Program, the National Centerfor Supercomputing Applications, thePittsburgh Supercomputing Center,and Louisiana State University—sentrepresentatives to observe NERSC’sBest Value Source Selection process.The NSF centers have adopted severalof NERSC’s procurement practices.

SC05 Conference HonorIn November 2005, NERSC demon-strated a production version of NGF atthe SC05 conference, using it for theStorCloud Challenge. StorCloud was aspecial initiative for building a high per-formance computing storage capabilityshowcasing HPC storage technologies(topologies, devices, interconnects)and applications. The StorCloudChallenge invited applicants from sci-ence and engineering communities touse the unique StorCloud infrastruc-ture to demonstrate emerging tech-niques or applications, many of whichconsume enormous amounts of net-work and storage resources.

A NERSC/Berkeley Lab team, led byWill Baird of NERSC’s ComputationalSystems Group, won an award for

3 Federal Plan for High-End Computing: Report of the High-End Computing Revitalization Task Force (HECRTF). Washington, D.C.: National Coordination Office forInformation Technology Research and Development, May 10, 2004.

4 Susan L. Graham, Marc Snir, and Cynthia A. Patterson, eds., Getting Up to Speed: The Future of Supercomputing, Committee on the Future of Supercomputing,National Research Council. Washington, D.C.: The National Academies Press, 2005.

5 Ibid., p. 171.

The NERSC Center48

“Best Deployment of a Prototype for aScientific Application.” The team, whichalso included Jonathan Carter and TaviaStone of NERSC and Michael Wehner,Cristina Siegerist and Wes Bethel ofBerkeley Lab’s Computational ResearchDivision, used the StorCloud infra-structure “to test the wide-area deploy-ment of an unprecedented system insupport of a groundbreaking climatemodeling application,” according tothe award. The application wasfvCAM—or Finite Volume CommunityAtmospheric Model—which is beingused to predict hurricane formation.

Baird designed the TRI Data Stormprototype around the concept of usingan integrated, multisystem filesystemto improve the analysis of results pro-duced by the demanding HPC applica-tion—the Community AtmosphericModel (CAM).

“Our aim was to take the output ofCAM from a high-resolution grid, filterout the data of interest, and visualizethe formation of storms in the NorthAtlantic basin,” according to Baird.“This tool will be used in a studycomparing real hurricane data withsimulations. While this is a fairlygeneric workflow that could hold truefor virtually any HPC application, theunique aspect to our approach is thatthere is a single high-performanceparallel filesystem serving all of thedifferent systems and applications.”

For TRI Data Storm, the team used anexternalized GPFS system, shared outby a dedicated cluster and mountedon all of the different computationalresources used by their tool: the IBMPOWER5 cluster Bassi and the PDSFLinux cluster at NERSC, the GPFSservers and storage at the confer-ence, and an SGI Altix cluster in theBerkeley Lab booth at SC05.

“All the communication between thesystems was simply through the

filesystem, not through anythingelse,” Baird said.

Science-Driven ServicesNERSC’s Science-Driven Services aredesigned to strike a balance betweenmeeting the special needs of leading-edge computational research projectsand providing responsive, comprehen-sive services for routine operations.Major issues addressed in NERSC’sfive-year plan include working withusers to lower the gap between peakperformance of terascale computingsystems and the performance real-ized by scientific applications; provid-ing special support for large-scaleprojects while at the same time main-taining the high level of support for allusers; and helping users make pro-ductive use of the flood of scientificdata from simulations and experiments.

Aspects of this strategy discussedbelow include support for large-scaleprojects, development of new datastorage and retrieval strategies, andongoing improvements in NERSC’severyday operations and servicesbased on feedback from clients.

Support for Large-Scale ProjectsNERSC works directly with scientists onmajor projects that require extensivescientific computing capabilities, suchas the SciDAC and INCITE collabora-tions. These projects are often charac-terized by large collaborations, thedevelopment of community codes, andthe involvement of computer scientistsand applied mathematicians. In addi-tion to high-end computing, theselarge projects handle issues in datamanagement, data analysis, and datavisualization, as well as automationfeatures for resource management.

NERSC provides its highest level ofsupport to these researchers, including

special service coordination for queues,throughput, increased limits, etc.; andspecialized consulting support, whichmay include algorithmic code restruc-turing to increase performance, I/Ooptimization, visualization support—whatever it takes to make the compu-tation scientifically productive. Thethree INCITE projects for 2005 aregood examples of this kind of support.

The INCITE project “Magneto-Rotational Instability and TurbulentAngular Momentum Transport,” led byFausto Cattaneo of the University ofChicago, is attempting to understandthe forces that help newly born starsand black holes increase in size bysimulating laboratory experimentsthat study magnetically caused insta-bility (see page 26). “With the help ofNERSC staff, we were able to tuneour software for Seaborg’s hardwareand realize performance improve-ments that made additional simula-tions possible,” Catteneo said.NERSC also provided crucial help increating animated visualizations ofthe simulation results, which involvethe formation of complex, three-dimensional structures that need tobe seen to be understood.

For the INCITE project “Direct Numer-ical Simulation of Turbulent Nonpre-mixed Combustion,” Jacqueline Chen,Evatt Hawkes, and Ramanan Sankaranof Sandia National Laboratories haveperformed the first 3D direct numeri-cal simulations of a turbulent non-premixed flame with detailed chemistry(see page 11). After analyzing andoptimizing the code’s performancewith the help of NERSC staff, theresearchers improved the code’s effi-ciency by 45%. The simulations gener-ated 10 TB of raw data, and NERSCconsultants helped the researchersfigure out the best strategy for effi-ciently transferring all that data fromNERSC systems to the researchers’local cluster. “The assistance we

The NERSC Center 49

received from the NERSC computingstaff in optimizing our code and withterascale data movement has beeninvaluable,” Chen said. “The INCITEaward has enabled us to extend ourcomputations to three dimensions sothat we may investigate interactionsbetween turbulence, mixing, and finite-rate detailed chemistry in combustion.”

The “Molecular Dynameomics” INCITEproject, led by Valerie Daggett of theUniversity of Washington, is an ambi-tious attempt to use molecular dynam-ics simulations to characterize andcatalog the folding/unfolding pathwaysof representative proteins from allknown protein folds (see page 30).David Beck, a graduate student in theDaggett lab, worked with NERSC con-sultants to optimize the performanceof the group’s code on Seaborg. “TheINCITE award gave us a unique oppor-tunity to improve the software, as wellas do good science,” Beck said.Improvements included load balanc-ing, which sped up the code by 20%,and parallel efficiency, which reached85% on 16-processor nodes. TheINCITE award enabled the team to dofive times as many simulations asthey had previously completed usingother computing resources. “We arequite satisfied with our experience atNERSC,” Daggett commented.

Archiving Strategies for GenomeResearchers

When researchers at the ProductionGenome Facility of DOE’s JointGenome Institute (JGI) found theywere generating data faster than they could find somewhere to storethe files, a collaboration with NERSC’sMass Storage Group developedstrategies for improving the reliabilityof data storage while also makingretrieval easier.

JGI is one of the world’s leading facili-

ties in the scientific quest to unravelthe genetic data that make up livingthings. With advances in automaticsequencing of genomic information,scientists at the JGI’s ProductionGenome Facility (PGF) found them-selves overrun with sequence data, astheir production capacity had grown sorapidly that data had overflowed theexisting storage capacity (Figure 13).Since the resulting data are used byresearchers around the world, PGF hasto ensure that the data are reliablyarchived as well as easily retrievable.

As one of the world’s largest publicDNA sequencing facilities, the PGFproduces 2 million files per month oftrace data (25 to 100 KB each), 100assembled projects per month (50MB to 250 MB), and several verylarge assembled projects per year(~50 GB). In aggregate, this averagesabout 2,000 GB per month.

In addition to the amount of data, amajor challenge is the way the dataare produced. Data from the sequenc-ing of many different organisms areproduced in parallel each day, result-ing in a daily archive that spreads thedata for a particular organism overmany tapes.

DNA sequences are considered thefundamental building blocks for therapidly expanding field of genomics.Constructing a genomic sequence isan iterative process. The trace frag-ments are assembled, and then thesequence is refined by comparing itwith other sequences to confirm theassembly. Once the sequence isassembled, information about itsfunction is gleaned by comparing andcontrasting the sequence with othersequences from both the same organ-ism and other organisms. Currentsequencing methods generate a large

FIGURE 13. JGI’s automated sequencing facilities threatened to produce more genomic datathan they could store or manage.

The NERSC Center50

volume of trace files that have to bemanaged—typically 100,000 files ormore. And to check for errors in thesequence or make detailed compar-isons with other sequences, researchersoften need to refer back to thesetraces. Unfortun-ately, these tracesare usually provided as a group of fileswith no information as to where thetraces occur in the sequence, makingthe researchers’ job more difficult.

This problem was compounded by thePGF’s lack of sufficient online storage,which made organization (and subse-quent retrieval) of the data difficultand led to unnecessary replication offiles. This situation required signifi-cant staff time to move files and reor-ganize filesystems to find sufficientspace for ongoing production needs;and it required auxiliary tape storagethat was not particularly reliable.

Staff from NERSC’s Mass StorageGroup and the PGF agreed to worktogether to address two key issues

facing the genome researchers. Themost immediate goal was for NERSC’sHPSS to become the archive for theJGI data, replacing the less-reliablelocal tape operation and freeing updisk space at the PGF for more imme-diate production needs (Figure 14).The second goal was to collaboratewith JGI to improve the data handlingcapabilities of the genome sequenc-ing and data distribution processes.

NERSC storage systems are robustand available 24 hours a day, sevendays a week, as well as highly scalableand configurable. NERSC has high-quality, high-bandwidth connectivity tothe other DOE laboratories and majoruniversities provided by ESnet.

Most of the low-level data producedby the PGF are now routinely archivedat NERSC, with ~50 GB of raw tracedata being transferred from JGI toNERSC each night.

The techniques used in developing the

archiving system allow it to be scaledup over time as the amount of datacontinues to increase—up to billions offiles can be handled with these tech-niques. The data have been aggregatedinto larger collections which hold tensof thousands of files in a single file inthe NERSC storage system. This datacan now be accessed as one large file,or each individual file can be accessedwithout retrieving the whole aggregate.

Not only will the new techniques be ableto handle future data, they also helpedwhen the PGF staff discovered raw datathat had been previously processed bysoftware that had an undetected bug.The staff were able to retrieve the rawdata from NERSC and reprocess it inabout a month and a half, rather thango back to the sequencing machinesand produce the data all over again—which would have taken about sixmonths. In addition to saving time,this also saved money—a rough esti-mate is that the original data collectioncomprised up to 100,000 files per dayat a cost of $1 per file, which addedup to $1.2 million for processing sixmonths’ worth of data. Comparing thisfigure to the cost of a month and ahalf of staff time, the estimated sav-ings are about $1 million—and theend result is a more reliable archive.

User Survey Provides ValuableFeedbackThe results from the 2005 user surveyshow generally high satisfaction withNERSC’s systems and support. Areaswith the highest user satisfactioninclude account support services, thereliability and uptime of the HPSSmass storage system, and HPC con-sulting. The largest increases in satis-faction over last year’s survey includethe NERSC CVS server, the Seaborgbatch queue structure, PDSF compilers,Seaborg uptime, available computinghardware, and network connectivity.

FIGURE 14. NERSC’s High-Performance Storage System (HPSS) is capable of storing and retriev-ing all of JGI’s genomic data efficiently.

The NERSC Center 51

Areas with the lowest user satisfac-tion include batch wait times on bothSeaborg and Jacquard, Seaborg’squeue structure, PDSF disk stability,and Jacquard’s performance anddebugging tools. Only three areas wererated significantly lower this year:PDSF overall satisfaction and uptime,and the amount of time taken toresolve consulting issues. The intro-duction of three major systems in thelast year combined with a reduction inconsulting staff explain the latter.

Eighty-two users answered the ques-tion “What does NERSC do well?”Forty-seven respondents stated thatNERSC gives them access to powerfulcomputing resources without whichthey could not do their science; 32mentioned excellent support servicesand NERSC’s responsive staff; 30pointed to very reliable and well man-aged hardware; and 11 said“Everything.”

Sixty-five users responded to “Whatshould NERSC do differently?” Theareas of greatest concern are theinterrelated issues of queue turn-around times (24 comments), jobscheduling and resource allocationpolicies (22 comments), and the needfor more or different computationalresources (17 comments). Users alsovoiced concerns about data manage-ment, software, group accounts,staffing, and allocations.

As in the past, comments from theprevious survey led to changes in2005, including a restructuring ofSeaborg’s queuing polices, the addi-tion of the new Jacquard and Bassiclusters, the upgrade of ESnet’s con-nectivity to NERSC to 10 gigabits persecond, and the installation of addi-tional visualization software.

The complete survey results can befound at https://www.nersc.gov/news/survey/2005/.

Science-Driven AnalyticsSimulations and experiments are gen-erating data faster than it can be ana-lyzed and understood. Addressing thisbottleneck in the scientific discoveryprocess is the emerging discipline ofanalytics, which has the simple goalof understanding data.

The term analytics refers to a set ofinterrelated technologies and intellec-tual disciplines that combine to pro-duce insight and understanding fromlarge, complex, disparate, and some-times conflicting datasets. Thesetechnologies and disciplines includedata management, visualization,analysis, and discourse aimed at pro-ducing specific types of understand-ing. These in turn rely on the compu-tational infrastructure, expertise inusing that infrastructure, and closecooperation between domain scien-tists, computational scientists, andcomputer scientists.

More specifically, the term visual ana-lytics is the science of analytic rea-soning facilitated by interactive visualinterfaces. Its objective is to enableanalysis of overwhelming amounts ofinformation, and it requires humanjudgment to make the best possibleevaluation of incomplete, inconsistent,and potentially erroneous information.

NERSC’s analytics strategy builds ontwo of the Center’s existing strengths:(1) proven expertise in effectivelymanaging large, complex computing,infrastructure, and data storage sys-tems to solve scientific problems ofscale; and (2) exemplary user servic-es, consulting, and domain scientificknowledge that help the NERSC usercommunity effectively employ theCenter’s resources to solve challeng-ing scientific problems. On this foun-dation, NERSC’s analytics strategyadds an increased emphasis on facili-ties, infrastructure, expertise, and

alliances that can be used to realizeanalytics solutions.

With the establishment of its newAnalytics Team, NERSC is realigning itsresources to support analytics activi-ties. The NERSC Center’s infrastructureis being broadened to include ele-ments such as database deploymentand support, with an increased focuson data analysis and scientific datamanagement to support analytics. Theexisting visualization program is beingexpanded to include information visu-alization and integrated data manage-ment, analysis, and distributed com-puting. The goal is a well-roundedservice and technology portfolio thatis responsive to the analytics needsof NERSC’s user community.

NERSC’s analytics strategy includesfive elements:

1. Taking a proactive role in deployingemerging technologies. NERSC willincreasingly become a conduit forprototype technologies that emergefrom the DOE computer scienceresearch community. Analytics willrequire adapting and deployingtechnologies from several differentareas—data management, analy-sis, visualization, dissemination—into a unified workflow that func-tions effectively in a time-criticalproduction environment. The roleof NERSC staff will include deploy-ing new system and support soft-ware, helping applications softwareengineers effectively use NERSCresources, and playing a proactiverole in providing feedback to theoriginal computer science researchersand developers to address securityor performance concerns.

2. Enhancing NERSC’s data manage-ment infrastructure. The NERSCGlobal Filesystem offers increasedperformance for all applications,including data-intensive analytics

The NERSC Center52

tasks. It also helps streamline dis-tributed workflows and provideshigh I/O rates, which are importantfor large datasets. NERSC alsoplans to increase its archival stor-age to nearly 40 PB over the nextfive years. In the near term,NERSC will evaluate and deploysoftware that provides distributed,file-level data management.

3. Expanding NERSC’s visualizationand analysis capabilities. One ofthe most significant activities per-formed by the NERSC visualizationstaff is in-depth, one-on-one con-sulting services, such as thoseprovided to INCITE and other largeprojects. These activities typicallyinvolve finding or engineering solu-tions where none exist off theshelf. In addition, visualizationstaff will evaluate new visualizationhardware and software technolo-gies to determine which are benefi-cial to the user community. Thesetechnologies may include informa-tion visualization, which differsfrom the better-known scientificvisualization in that the underlyingdata does not readily lend itself to

spatial mapping—for example,comparing the results of genomealignment across multiple species.As data size and complexity grow,it will become increasingly crucialto use analysis technologies toreduce the processing load throughthe computational and visualizationpipelines, as well as to reduce the“scientific processing load” on thehumans who must interpret andunderstand the results. A portfolioof commercial, production, open-source, and research-grade tech-nologies is expected to be mosteffective in meeting users’ scientif-ic needs.

4. Enhancing NERSC’s distributedcomputing infrastructure. NERSC’sstrategy for supporting distributedcomputing will be tailored to provideservices that have the broadestpossible benefit and that conformto security requirements. In additionto providing low-level infrastructuresuch as the Open Grid ServicesArchitecture (OGSA) and similartechnologies that provide authenti-cation and secure data movementacross the network, NERSC will

investigate and deploy higher-levelapplications and services thatemerge from research and applica-tions communities like the ParticlePhysics Data Grid and the EarthSystem Grid. Both of those proj-ects rely on standard services forbrokering access to data and toolsthat serve large, distributed usercommunities. NERSC will workclosely with the user community toprovide the documentation andassistance they need to constructanalytics workflows.

5. Understanding the analytics needsof the user community. To be effec-tive, NERSC’s new program focuson Science-Driven Analytics willrequire additional information fromthe user community. To that end,the entire user community was sur-veyed in early 2006 to identifytheir most pressing analyticsneeds. The findings from this sur-vey have been instrumental inshaping and prioritizing the emerg-ing analytics effort. NERSC willcontinue soliciting input from usersas well as tracking analytics trendsin the larger scientific community.

The NERSC Center 53

APPENDIX ANERSC Policy Board

Daniel A. Reed (Chair)University of North Carolina, Chapel Hill

David Dean(ex officio, NERSC Users Group Chair)Oak Ridge National Laboratory

Robert J. GoldstonPrinceton Plasma Physics Laboratory

Tony HeyMicrosoft Corporation

Sidney KarinUniversity of California, San Diego

Pier OddoneFermi National Accelerator Laboratory

Tetsuya SatoEarth Simulator Center/Japan Marine Science andTechnology Center

Stephen L. SquiresHewlett-Packard Laboratories

54 Appendices

APPENDIX BNERSC Client Statistics

NERSC served 2,677 scientists throughout the UnitedStates in 2005. These researchers work in DOE laborato-ries, universities, industry, and other Federal agencies. Figure1 shows the proportion of NERSC usage by each type ofinstitution, while Figures 2 and 3 show laboratory, university,

and other organizations that used large allocations of com-puter time. Computational science conducted at NERSC cov-ers the entire range of scientific disciplines, but is focusedon research that supports the DOE’s mission and scientificgoals, as shown in Figure 4.

55Appendices

Universities47%

Other Labs4%

DOE Labs44%

Industries5%

Georgia Institute of Technology997,043

General Atomics 965,661

Massachusetts Institute of Technology1,064,604

University of California, Santa Cruz1,906,033

University of Arizona1,479,758

University of California, Berkeley1,356,929

University of California, Los Angeles1,341,278

University of Chicago2,551,429

University of Washington2,245,556

Auburn University1,893,349

65 Others6,501,906

Yale University513,769

University of Kentucky1,205,832

University of Michigan 710,978

Harvard University 710,379

University of Wisconsin, Madison 667,556

J. Craig Venter Institute 558,298

Argonne1,670,798

National Renewable Energy886,224

Princeton Plasma Physics5,031,477

Oak Ridge3,486,528

Lawrence Livermore3,672,455

Lawrence Berkeley6,744,963

Seven Others152,874

Sandia2,093,356

National Center for Atmospheric Research1,672,339

Stanford Linear Accelerator Center 477,387Brookhaven 366,550

Los Alamos 164,879Pacific Northwest 129,107Ames 122,864

Nuclear and High Energy Physics2%

Geoscience and Engineering2%

Fusion Energy29%

Chemistry13%Astrophysics

12%

Materials Science9%

Life Sciences8%

Lattice QCD8%

Climate andEnvironmental Science

8%

Accelerator Physics5%

Applied Math and Computer Science4%

FIGURE 1. NERSC MPP usage by institution type, 2005. FIGURE 2. DOE and other Federal laboratory usage at NERSC, 2005 (MPP hours).

FIGURE 3. Academic and private laboratory usage at NERSC, 2005 (MPP hours). FIGURE 4. NERSC usage by scientific discipline, 2005.

APPENDIX CNERSC Users Group Executive Committee

Members at LargeYuen-Dat Chan, Lawrence Berkeley National Laboratory

Ricky A. Kendall, Ames Laboratory, Iowa State University

Donald A. Spong, Oak Ridge National Laboratory

Douglas Swesty, State University of New York at Stony Brook

Office of Advanced Scientific Computing ResearchDavid Bernholdt, Oak Ridge National Laboratory

David Keyes, Old Dominion University

David Turner, Ames Laboratory

Office of Basic Energy SciencesFeng Liu, University of Utah

G. Malcolm Stocks, Oak Ridge National Laboratory

Theresa L. Windus, Pacific Northwest National Laboratory

Office of Biological and Environmental ResearchMichael Colvin, University of California, Merced, andLawrence Livermore National Laboratory

Brian Hingerty, Oak Ridge National Laboratory

Vincent Wayland, National Center for Atmospheric Research

Office of Fusion Energy SciencesStephane Ethier (Vice Chair), Princeton Plasma PhysicsLaboratory

Alex Friedman, Lawrence Livermore and Lawrence BerkeleyNational Laboratories

Carl Sovinec, University of Wisconsin, Madison

Office of High Energy PhysicsOlga Barranikova, Purdue University

Warren Mori, University of California, Los Angeles

Donald Sinclair, Argonne National Laboratory

Office of Nuclear PhysicsDavid Dean (Chair), Oak Ridge National Laboratory

Patrick Decowski, Lawrence Berkeley National Laboratory

Douglas L. Olson, Lawrence Berkeley National Laboratory

Appendices56

APPENDIX DSupercomputing Allocation Commitee

The Supercomputing Allocation Committee (SAC) is responsi-ble for setting the policies associated with the utilization ofcomputing, data storage, and other auxiliary services avail-able to DOE Office of Science (SC) researchers and other-wise coordinating SC’s computational projects. TheCommittee sets the distribution of NERSC and other avail-able Office of Advanced Scientific Computing Researchresources for scientific computing applications every year.

Committee ChairBarbara Helland

Office of Advanced Scientific Computing ResearchBarbara Helland

Office of Basic Energy SciencesRichard Hilderbrandt

Office of Biological and Environmental ResearchMichael Riches

Office of Fusion Energy SciencesJohn Mandrekas

Office of High Energy Physics Glen Crawford

Office of Nuclear PhysicsSidney Coon

Appendices 57

APPENDIX EOffice of Advanced Scientific Computing Research

Mathematical, Information, and Computational SciencesDivisionThe primary mission of the Advanced Scientific ComputingResearch (ASCR) program, which is carried out by theMathematical, Information, and Computational Sciences(MICS) subprogram, is to discover, develop, and deploy thecomputational and networking tools that enable researchersin the scientific disciplines to analyze, model, simulate, andpredict complex phenomena important to the Department ofEnergy. To accomplish this mission, the program fosters andsupports fundamental research in advanced scientific com-puting—applied mathematics, computer science, and net-working—and operates supercomputer, networking, and relat-ed facilities. In fulfilling this primary mission, the ASCR pro-gram supports the Office of Science Strategic Plan’s goal ofproviding extraordinary tools for extraordinary science as wellas building the foundation for the research in support of theother goals of the strategic plan. In the course of accom-plishing this mission, the research programs of ASCR haveplayed a critical role in the evolution of high performancecomputing and networks. Berkeley Lab thanks the programmanagers with direct responsibility for the NERSC programand the MICS research projects described in this report:

Michael R. StrayerAssociate Director, ASCR, and Acting Director, MICS

Walter M. PolanskyASCR Senior Technical Advisor for Project Management

Anil DeaneProgram Manager for Applied Mathematics

Barbara HellandProgram Manager for NERSC and Leadership ComputingFacility

Daniel HitchcockProgram Manager for ESnet

Frederick C. JohnsonProgram Manager for Computer Research

Thomas D. Ndousse-FetterProgram Manager for Networking

Mary Anne ScottProgram Manager for Collaboratory Research

Yukiko SekineProgram Manager for Scientific Data Management andVisualization

George R. SeweryniakProgram Manager

Appendices58

APPENDIX FAdvanced Scientific Computing Advisory Committee

The Advanced Scientific Computing Advisory Committee(ASCAC) provides valuable, independent advice to theDepartment of Energy on a variety of complex scientific andtechnical issues related to its Advanced Scientific ComputingResearch program. ASCAC’s recommendations includeadvice on long-range plans, priorities, and strategies toaddress more effectively the scientific aspects of advancedscientific computing including the relationship of advancedscientific computing to other scientific disciplines, and main-taining appropriate balance among elements of the program.The Committee formally reports to the Director, Office ofScience. The Committee primarily includes representatives ofuniversities, national laboratories, and industries involved inadvanced computing research. Particular attention is paid toobtaining a diverse membership with a balance among scien-tific disciplines, institutions, and geographic regions.

Jill P. Dahlburg, ChairNaval Research Laboratory

John W. Connolly, Vice ChairUniversity of Kentucky

F. Ronald BaileyNASA Ames Research Center (retired)

Gordon BellMicrosoft Bay Area Research Center

David J. GalasBattelle Memorial Institute

Roscoe C. GilesBoston University

James J. HackNational Center for Atmospheric Research

Thomas A. ManteuffelUniversity of Colorado at Boulder

Ellen B. StechelSandia National Laboratories

Virginia TorczonCollege of William and Mary

Stephen S. WolffCisco Systems

Appendices 59

APPENDIX GAcronyms and Abbreviations

AGM . . . . . Associate General Manager

AIX . . . . . . Advanced InteractiveeXecutive (an IBM operat-ing system)

ALS. . . . . . Advanced Light Source atBerkeley Lab

AMBER . . . Assisted Model Buildingwith Energy Refinement

AMR . . . . . Adaptive mesh refinement

ASC . . . . . Advanced Simulation andComputing Program,National Nuclear SecurityAdministration (DOE)

ASCR-MICS Office of AdvancedScientific ComputingResearch, Mathematical,Information, and Com-putational SciencesDivision (DOE)

AVHF . . . . . Alexander von HumboldtFoundation (Germany)

BER . . . . . Office of Biological andEnvironmental Research(DOE)

BES . . . . . Office of Basic EnergySciences (DOE)

BGL . . . . . Blue Gene/L, ArgonneNational Laboratory

BMBF . . . . Bundesministerium fürBildung und Forschung(Germany)

BNL . . . . . Brookhaven NationalLaboratory

BRAHMS . . Broad Range HadronMagnetic Spectrometer atBrookhaven NationalLaboratory

BSPO . . . . Belgian Science Policy Office

CAM . . . . . Community AtmosphericModel

CCSE . . . . Center for ComputationalSciences and Engineering(Berkeley Lab)

CCSM . . . . Community ClimateSystem Model

CENIC . . . . Corporation for EducationNetwork Initiatives inCalifornia

CMSO . . . . Center for Magnetic Self-Organization at theUniversity of Chicago

COLTRIMS . Cold target recoil ionmomentum spectroscopy

CPU . . . . . Central processing unit

CSIR . . . . . Council of Scientific andIndustrial Research (India)

CVS . . . . . Concurrent VersionsSystem

DAE . . . . . Department of AtomicEnergy (India)

DARPA . . . Defense AdvancedResearch Projects Agency

DEFRA. . . . Department for Environ-ment, Food and RuralAffairs (UK)

DFG . . . . . Deutsche Forschungs-gemeinschaft (Germany)

DFT . . . . . Density functional theory

DNS . . . . . Direct numerical simul-ation

DOE . . . . . U.S. Department of Energy

DST . . . . . Department of Scienceand Technology (India)

EPSRC . . . Engineering and PhysicalSciences ResearchCouncil (UK)

ES . . . . . . Earth Simulator (Japan)

ESnet . . . . Energy Sciences Network(DOE)

FAPESP . . . Fundação de Amparo àPesquisa do Estado deSão Paulo (Brazil)

FES. . . . . . Office of Fusion EnergySciences (DOE)

FOM . . . . . Fundamenteel Onderzoekder Materie (Netherlands)

GACR . . . . Grant Agency of the CzechRepublic

GB . . . . . . Gigabyte

GMODGO. . GraduiertenkollegMolekulare Organisationund Dynamik an Grenz- undOberflächen (Germany)

GMRDP . . . Government Meteorologi-cal Research and Devel-opment Programme (UK)

GPFS . . . . General Parallel FileSystem (IBM)

HadCM3 . . Hadley Centre CoupledModel, version 3

HEP . . . . . Office of High EnergyPhysics (DOE)

HPC . . . . . High performance comput-ing

HPCMP . . . High Performance Comput-ing Modernization Program(U.S. Department ofDefense)

HPCS . . . . High Productivity Com-puter Systems

HPSS . . . . High Performance StorageSystem

HSM . . . . Hierarchical Storage Manager

ilmm . . . . . in lucem MolecularMechanics

IN2P3 . . . . Institut National de laPhysique Nucléaire et dela Physique des Particules(France)

INCITE. . . . Innovative and NovelComputational Impact onTheory and Experiment(DOE)

I/O . . . . . . Input/output

Appendices60

ITER . . . . . A multinational tokamakexperiment to be built inFrance (Latin for “the way”)

JGI . . . . . . Joint Genome Institute(DOE)

JSPS . . . . . Japan Society for thePromotion of Science

KamLAND . Kamioka Liquid ScintillatorAnti-Neutrino Detector(Japan)

KUL . . . . . Katholieke UniversiteitLeuven (Belgium)

LANL. . . . . Los Alamos NationalLaboratory

LBNL. . . . . Lawrence BerkeleyNational Laboratory

LBW . . . . . Landesstiftung Baden-Württemberg (Germany)

MAN . . . . . Metropolitan area network

MAPS . . . . Memory Access PatternSignature

MB . . . . . . Megabyte

MEC . . . . . Ministry of Education ofChina

MECS . . . . Ministerio de Educación yCiencia (Spain)

MEXT . . . . Ministry of Education,Culture, Sports, Scienceand Technology (Japan)

MPP . . . . . Massively parallel pro-cessing

MSCF . . . . Molecular ScienceComputing Facility atPacific Northwest NationalLaboratory

NASA . . . . National Aeronautics andSpace Administration

NCAR . . . . National Center forAtmospheric Research

NCCS . . . . National Center forComputational Sciencesat Oak Ridge NationalLaboratory

NERSC . . . National Energy ResearchScientific ComputingCenter

NGF . . . . . NERSC Global Filesystem

NMR . . . . . Nuclear magnetic reso-nance

NNSFC . . . National Natural ScienceFoundation of China

NOAA . . . . National Oceanographicand Atmospheric Admin-istration

NP . . . . . . Office of Nuclear Physics(DOE)

NPACI . . . . National Partnership forAdvanced ComputationalInfrastructure

NSA . . . . . Near-surface alloy

NSF . . . . . National ScienceFoundation

OGSA . . . . Open Grid ServicesArchitecture

ORNL . . . . Oak Ridge NationalLaboratory

OSP . . . . . Open Software andProgramming Group(NERSC)

PB . . . . . . Petabyte

PCM . . . . . Parallel Climate Model

PDSF . . . . Parallel DistributedSystems Facility (NERSC)

PGF . . . . . Production GenomeFacility (JGI)

PHENIX . . . Pioneering High EnergyNuclear InteractionExperiment at BrookhavenNational Laboratory

PHOBOS . . A detector experiment atBrookhaven NationalLaboratory

PMEMD. . . Particle mesh Ewaldmolecular dynamics

PPDG . . . . Particle Physics Data Grid

PSCSR . . . Polish State Committeefor Scientific Research

QGP . . . . . Quark-gluon plasma

RFP. . . . . . Request for proposals

RHIC . . . . Relativistic Heavy IonCollider at BrookhavenNational Laboratory

RMST . . . . Russian Ministry ofScience and Technology

SC . . . . . . Office of Science (DOE)

SciDAC . . . Scientific Discoverythrough AdvancedComputing (DOE)

SDSA . . . . Science-Driven SystemArchitecture (NERSC)

SNSF . . . . Swiss National ScienceFoundation

SSP . . . . . Sustained SystemPerformance benchmark

STAA . . . . . Science and TechnologyAssistance Agency(Slovakia)

STAR . . . . Solenoidal Tracker at RHIC

TB. . . . . . . Terabyte

TCP. . . . . . Transmission ControlProtocol

UC . . . . . . University of California

UCB . . . . . University of California,Berkeley

USAF-OSR. . U.S. Air Force Office ofScientific Research

UWM. . . . . University of Wisconsin-Madison

ViVA . . . . . Virtual Vector Architecture

WHOI . . . . Woods Hole OceanographicInstitution

Appendices 61

Jon BashorNERSC CommunicationsBerkeley Lab, MS 50B42301 Cyclotron RoadBerkeley, CA 94720-8148email: jbashor@lbl.govphone: (510) 486-5849fax: (510) 486-4300

NERSC’s web site: http://www.nersc.gov/

Published by the Berkeley Lab Creative Services Office in collaboration withNERSC researchers and staff. JO#11554

Editor and principal writer: John Hules.

Contributing writers: Jon Bashor, Lynn Yarris, and Paul Preuss (Berkeley Lab);Robert Sanders (UC Berkeley); Mario Aguilera and Cindy Clark (ScrippsInstitution of Oceanography); Karen McNulty Walsh (Brookhaven NationalLaboratory); David Terraso (Georgia Institute of Technology); NCAR MediaRelations Office.

FOR MORE INFORMATIONContact

DISCLAIMER This document was prepared as an account of work sponsored by the United StatesGovernment. While this document is believed to contain correct information, neither theUnited States Government nor any agency thereof, nor The Regents of the University ofCalifornia, nor any of their employees, makes any warranty, express or implied, or assumesany legal responsibility for the accuracy, completeness, or usefulness of any information,apparatus, product, or process disclosed, or represents that its use would not infringe private-ly owned rights. Reference herein to any specific commercial product, process, or service byits trade name, trademark, manufacturer, or otherwise, does not necessarily constitute orimply its endorsement, recommendation, or favoring by the United States Government or anyagency thereof, or The Regents of the University of California. The views and opinions ofauthors expressed herein do not necessarily state or reflect those of the United StatesGovernment or any agency thereof, or The Regents of the University of California.

Ernest Orlando Lawrence Berkeley National Laboratory is an equal opportunity employer.

Lawrence Berkeley National Laboratory / 1 Cyclotron Road / Berkeley CA / 94720

LBNL-60296