MARCH/APRIL 2000 21

Safety and reliability are paramount con-cerns in rocket motor design because ofthe enormous cost of typical payloads and,in the case of the Space Shuttle and other

manned vehicles, the crew’s safety. In the springof 1999, for example, a series of three consecutivelaunch failures collectively cost more than US $3.5billion. The most notorious launch failure, ofcourse, was the tragic loss of the Space ShuttleChallenger and its seven crew members. Thus,there is ample motivation for improving our un-derstanding of solid rocket motors (SRMs) and thematerials and processes on which they are based,as well as the methodology for designing and man-ufacturing them.

The use of detailed computational simulationin the virtual prototyping of products and deviceshas heavily influenced some industries—for ex-ample, in automobile and aircraft design—but todate, it hasn’t made significant inroads in rocketmotor design. Reasons for this include the mar-ket’s relatively small size and the lack of sufficientcomputational capacity. Traditional design prac-tices in the rocket industry primarily use top-down, often 1D modeling of components and sys-tems based on gross thermomechanical and

chemical properties, combined with engineeringjudgment based on many years of experience,rather than detailed, bottom-up modeling fromfirst principles. Moreover, there has been a ten-dency to study individual components in isolation,with relatively little emphasis on the often intimatecoupling between the various components. For ex-ample, SPP1—an industry-standard code for an-alyzing solid propulsion systems—includes a fairlydetailed model of the propellant thermochemistrybut no structural analysis and no detailed modelof internal flow.

One of our primary goals at the Center for Sim-ulation of Advanced Rockets (CSAR) is to developa virtual prototyping tool for SRMs based on de-tailed modeling and simulation of their principalcomponents and the dynamic interactions amongthem. Given a design specification—geometry,materials, and so on—we hope to predict the en-tire system’s resulting collective behavior with suf-ficient fidelity to determine both nominal perfor-mance characteristics and potential weaknesses orfailures. Such a “response tool” could explore thespace of design parameters much more quickly,cheaply, and safely than traditional build-and-testmethods. Of course, we must validate such a ca-pability through rigorous and extensive compari-son with data for known situations to have confi-dence in its predictions for unknown situations.Although it is unlikely that simulation will evertotally replace empirical methods, it can poten-tially dramatically reduce the cost of those meth-

VIRTUAL PROTOTYPING OF SOLIDPROPELLANT ROCKETS

Researchers seek a detailed, whole-system simulation of solid propellant rockets undernormal and abnormal operating conditions. A virtual-prototyping tool for solid propellantrocket motors based on first principles models of rocket components and their dynamicinteractions meets this goal.

MICHAEL T. HEATH AND WILLIAM A. DICK

Center for Simulation of Advanced Rockets, UIUC

1521-9615/00/$10.00 © 2000 IEEE

A S C I A L L I A N C E C E N T E R S

22 COMPUTING IN SCIENCE & ENGINEERING

ods by identifying the most promising approachesbefore building actual hardware.

Challenges in rocket simulation

Solid propellant boosters are the “heavy lifters”of the space launch industry. Most of the world’slarge, multistage launch vehicles—including the Ar-iane, Delta, Titan, and Space Shuttle—employ twoor more SRBs in the initial stage to provide 80% ormore of the immense thrust needed to lift a pay-load in excess of 10,000 pounds off the launch padand propel it the first few tens of miles above Earth.Beyond this point, subsequent stages—typically liq-uid-fueled—take over into orbit and beyond.

SRMs are notably simpler than liquid rocketengines.2 The latter have far more moving parts(pumps, valves, and so on) and require storingand handling of liquids that might be cryogenicor potentially hazardous. SRMs, though, have al-most no moving parts (often only a gimballednozzle for thrust vector control), and the com-posite solid propellant (containing both fuel andoxidizer) forms the combustion chamber. Themain disadvantage of SRMs is that once ignited,combustion is essentially uncontrollable: the pro-pellant burns at maximum rate until exhausted.Thus, solid motors are ideal for the initial stagesof flight, when raw power is more important thanfinesse, and then liquid-propellant rockets takeover for the portions of flight requiring more del-icate maneuvering. Despite their relative sim-plicity, SRMs are still fiendishly complex in termsof the chemical and thermomechanical processesthat take place during firing, as well as the designand manufacturing processes required to makethem reliable, safe, and effective.

Figure 1 shows a schematic drawing of a typicalSRM—the major parts are indicated along withthe types of mathematical models that might beused to represent them. Reality, of course, is con-siderably more complex than this 2D picture, and

we note the following major challenges in achiev-ing our goals.

• The complex behavior of SRMs requires fully3D modeling to capture the essential physicsadequately. Examples include the combustionof composite energetic materials; the turbu-lent, reactive, multiphase fluid flows in thecore and nozzle; the global structural responseof the propellant, case, liner, and nozzle; andpotential accident scenarios such as pressur-ized crack propagation, slag ejection, and pro-pellant detonation.

• The coupling between components is strongand nonlinear. For example, the loading dueto fluid pressure deforms the solid propellant,which changes the geometry of the fluid flow,which in turn affects pressure, and so on. Sim-ilarly, the burn rate increases with pressureand vice versa.

• The geometry is complex and changes dy-namically as the rocket consumes propellant.The inclusion of slots and fins, which formsa star-shaped cross section, enhances theamount of burning surface area. Whateverits initial shape, the propellant surface re-gresses at a pressure-dependent rate as thepropellant burns, and discrete representa-tions of the solid and fluid components, aswell as the interface between them, mustadapt accordingly.

• The spatial and temporal scales are ex-tremely diverse. For example, processes suchas combustion and crack propagation occuron micron-length scales and microsecondtime scales, or less, which are entirely infea-sible to treat a two-minute burn of a 125-foot-long rocket.

• Manufacturing and transportation constraintsnecessitate the use of numerous joints, in-cluding field joints where motor segments areassembled at the launch site. This significantly

Combustion interface

Combustion-injectionboundary layer model

Compressible-turbulence LES model

Solid propellant Interior cavity

Igniter Insulation

Case Nozzle Exhaustplume

Thermomechanicalfoam model

Thermoviscoelasticmodel

Thermoelastic model

Elasticity/ablation model

LESmodel

Figure 1. Ide-alized solidpropellantrocket. Majorcomponentsare along thetop, our initialsimulationmodels alongthe bottom.

MARCH/APRIL 2000 23

complicates the geometry and structural re-sponse of the motor and introduces potentialpoints of failure.

• Modeling and simulating each componentis challenging both methodologically andcomputationally. Although there is consid-erable experience in the field in modelingthe various rocket motor components, amore fundamental understanding of theconstitutive and energetic properties of ma-terials and of the processes they undergo re-quires much greater detail along with teras-cale computational capacity.

• Modeling and simulating component cou-pling is even more demanding because it notonly requires still greater computational ca-pacity, but also demands that the correspond-ing software modules interact in a mannerthat is physically, mathematically, and nu-merically correct and consistent. When dataare transferred between components, theymust honor physical conservation laws, mu-tually satisfy mathematical boundary condi-tions, and preserve numerical accuracy, eventhough the corresponding meshes might dif-fer in structure, resolution, and discretizationmethodology.

• Integrated, whole-system SRM simulationrequires enormous computational capacity,currently available only through massivelyparallel systems that have thousands ofprocessors. Thus, the software integrationframework, mesh generation, numerical al-gorithms, input/output, and visualizationtools necessary to support such simulationsmust be scalable to thousands of processors.

In September 1997, CSAR embarked on an am-bitious plan to tackle these daunting challengesand produce a virtual-prototyping tool for SRMs.3

This article is a progress report almost two yearsinto our five-year plan.

Our initial plans seemed audacious, but thesubstantial resources our sponsor (the US De-partment of Energy’s Accelerated StrategicComputing Initiative program) provided let usassemble a team of over 100 researchers, includ-ing roughly 40 faculty, 40 graduate students, and20 staff (research scientists, programmers, andpostdoctoral associates), that represents 10 de-partments across our university. This diversegroup provides the broad expertise needed incombustion, fluid dynamics, structural mechan-ics, and computer science, but it also presentsthe additional challenge of coordinating a large

collaborative project that cuts across traditionaldepartmental boundaries. We organized our ef-fort along basic disciplinary lines without regardto the academic departments of the participants.This has had the salutary effect of inducing col-laboration among faculty and students in a givendiscipline, such as fluid dynamics, regardless ofwhich department they might occupy. Cross-cutting teams—such as System Integration andValidation and Specification—draw membersfrom all four disciplinary groups and require anadditional level of collaboration.

Staged approach

We realized from the outset that a project ofthis complexity would require a staged approach:we would need to learn to walk before we couldrun (much less fly). The primary axes of com-plexity in our problem are physical and geometric(see Figure 2). Physical complexity refers to thedetail and sophistication of physical models em-ployed and the degree of coupling among them.Geometric complexity refers to the dimension ofthe problem and the degree of detail and fidelityin representing a real SRM. In essence, we wishto move along the diagonal of this diagram overtime. In this spirit, we defined three successivegenerations of integrated rocket simulation codes:

• GEN0: A 2D ideal rocket with steady-stateburning at chamber pressure, power law forpropellant regression, Euler equations forfluid flow, a rigid case, linearly elastic propel-lant, and one-way coupling from fluid tosolid. We based its physical parameters onthe Space Shuttle reusable solid rocket mo-tor (see sidebar). GEN0 was intended pri-marily as a warm-up exercise.

• GEN1: A fully 3D whole-system simulationcode using relatively simple component

Weaklycoupled

Accidents

Joints

Star grain

3D

2D

1D

Fullycoupled

GEN0

GEN1family

GEN2family

Detailed

Physical complexity

Geo

met

rical

com

plex

ity

Figure 2. Ourcode develop-ment followsthis stagedapproach withincreasingcomplexity incomponentmodels andcoupling.

24 COMPUTING IN SCIENCE & ENGINEERING

models, two-way coupling, and reasonablyrealistic geometry approximating that of theSpace Shuttle RSRM. The star grain of theShuttle RSRM is included but not joints, in-hibitors, or cracks. Solid components includeviscoelastic propellant and linearly elasticcase. The fluid component is an unsteady,viscous, compressible flow, with a large-eddysimulation turbulence model but with noparticles, radiation, or chemical reactions inthe flow. The combustion model assumes ho-mogeneous surface burning and a pressure-dependent regression rate. There is full, two-way aeroelastic coupling between the fluidand solid components. The development ofGEN1 was expected to span the first threeyears of the five-year project.

• GEN2: A fully capable rocket simulation toolwith detailed component models, complexcomponent interactions, and support for sub-scale simulations of accident scenarios such as

pressurized crack propagation, slag accumu-lation and ejection, and potential propellantdetonation. GEN2 includes more detailedgeometric features, such as joints and in-hibitors, and also includes more detailed andaccurate models for materials and processesbased on separate subscale simulations.GEN2 was expected to span the last threeyears of the five-year project, overlappingwith the final year of GEN1.

Progress to date

We assembled the integrated GEN0 code fromexisting in-house modules for fluids, solids, andcombustion components, and we completed it inMay 1998. We ran it with modest levels of paral-lelism on a shared-memory SGI Power Challenge.The computed results agreed reasonably well withpredictions of classical 1D theory, but we didn’textensively validate it because we never intended

We chose the Space Shuttle Reusable Solid Rocket Motor as ourprimary simulation target for a variety of reasons, including its na-tional importance, its public visibility, its fairly typical design, andthe availability of detailed specifications and extensive test data.We outline here the basic technical facts about the Space ShuttleRSRM.1,2 Figure A shows a composite drawing of the RSRM.

Height: 126.11 ft.Diameter: 12.16 ft.Weight: 149,275 lb., empty; 1,255,415 lb. full Case: High-strength D6AC steel alloy, 0.479 in. to 0.506 in.

thick Nozzle: Aluminum nose-inlet housing and steel exit cone,

with carbon-cloth phenolic ablative liners and glass-clothphenolic insulators. The nozzle is partially submerged and ismovable for thrust vector control.

Insulation: Asbestos-silica-filled nitrile butadiene rubber Propellant (material percent by weight):

Ammonium perchlorate oxidizer: 70Powdered aluminum fuel: 16Polybutadiene polymer (PBAN) binder: 12Epoxy curative agent: 2Ferric oxide burn rate catalyst: trace

Propellant grain: 11-point star-shaped perforation in head end of forward segment, aft-tapered cylindrical perfor-ation in remaining segments. Liquid and solid ingredientsare first thoroughly mixed into a thick paste; then curativeagent is added before the mixture is vacuum-cast into amold and then cured in a “slow” oven for several days.Consistency of the resulting composite solid is similar to

that of a pencil eraser. Igniter: Solid rocket pyrogen igniter mounted in forward end,

47.5-in. long and containing 134 lb. of TP-H1178 propellantTotal launch weight: 4.5 million lb. (including two SRBs, exter-

nal tank, orbiter, and payload)Maximum thrust: 3,320,000 lb. force (each SRB)Acceleration: Lift-off 1.6 g (maximum 3.0 g)Launch timeline:

Liquid engines fire: –6.0 secSRB igniter initiated: 0.0 secLift-off pressure: 564 psia reached at 0.23 secAll exposed propellant ignited: 0.3 secMaximum operating pressure: 914 psia reached at 0.6 secRoll program begins: 10 secStar grain burnout: 21 secLiquid engines throttled down: 30 secMach 1 reached: 40 secSolid propellant burnout: 111 secSRB separation: 126 sec

Velocity at separation: 3,100 mphAltitude at separation: 25 nmi

References1. Design Data Book for Space Shuttle Reusable Solid Rocket Motor, Publication

No. 930480, TRW-16881, Revision A, Thiokol Space Operations, Brigham

City, Utah, 1997.

2. A.J. McDonald, “Return to Flight with the Redesigned Solid Rocket Motor,”

Proc. AIAA/ASME/SAE/ASEE 25th Joint Propulsion Conf., AIAA Paper No. 89-

2404, AIAA Press, Reston, Va., 1989, pp. 1–15.

Space Shuttle Reusable Solid Rocket Motor

MARCH/APRIL 2000 25

GEN0 as a realistic, high-fidelity simulation; wesaw it simply as a start-up system integration ex-ercise to get our team accustomed to working to-gether. Visit www.csar.uiuc.edu to view animationsof our results.

We based the subsequent GEN1 code on newlywritten or substantially modified in-house codesfor the various modules. We completed a simpli-fied, serial, but full 3D version of it in October1998. Its principal simplifications included our useof a strictly cylindrical geometry (no star grain,which is a slot-and-fin geometry used to increasethe propellant’s initial burning surface area), nosupport for interface regression due to burning(not a significant factor for short burn times, butobviously necessary for longer runs), the require-ment that the solid and fluid meshes must match atthe interface to simplify data transfer, and our useof a linearly elastic (rather than viscoelastic) modelfor the propellant.

Computed results without the star grain in the

propellant gave a head-end pressure at the onsetof steady burning of roughly half the empiricallymeasured value for the Space Shuttle RSRM.This was not surprising, as the whole point of thestar grain is to increase the exposed surface areaof propellant early in the burn, which increasespressure and thrust accordingly. Thus, imple-menting the star grain became a high priority.Another high priority was a fully parallel imple-mentation, not only because this was an impor-tant general goal, but also for the practical rea-son that we could not otherwise make runs ofsignificant duration at a reasonable resolution.By May 1999, we had completed a fully parallelimplementation of GEN1.4 With the star graingeometry, computed head-end pressure was nowwithin a few percentage points of the Space Shut-tle RSRM’s measured value.

Recent efforts have focused on implementinginterface regression in both fluid and solid mod-ules and on allowing for nonmatching meshes at

Figure A. The Reusable Solid Rocket Motor (RSRM) is a primary booster for NASA’s Space Transportation System (STS). Section A-A shows11-point slot and fin star grain structure in the RSRM’s forward segment. Propellant in forward-center and aft-center segments form straight-walled cylinders; aft-segment propellant tapers outward to a submerged nozzle. Inhibitors between segments are asbestos-filled carboxyl-termi-nated polybutadiene used to tailor burning surface to meet the motor’s thrust requirements.

26 COMPUTING IN SCIENCE & ENGINEERING

the interface. Both fluid and solid modules nowsupport interface regression using an ALE (Ar-bitrary Lagrangian-Eulerian) approach in whichthe mesh moves to allow for the dynamicallychanging geometry. We also devised generalmesh-association and conservative data-inter-polation schemes that permit accurate datatransfer between components with nonmatch-ing meshes at the interface.

The main features of the solids codes (Roc-solid) include finite elements, unstructured hexa-hedral meshes, linear elastodynamics, ALEtreatment of regression, implicit time integra-tions, multigrid equation solvers, and Fortran 90with MPI parallelism. The main features of thefluids codes (Rocflo5) include finite volume;block-structured meshes; unsteady, viscous,compressible flow; ALE treatment of movingboundaries; second-order upwind total variationdiminishing schemes; explicit time integrations;and Fortran 90 with MPI parallelism. Figure 3shows how the two compare.

Parallel scalability of Rocsolid, Rocflo, andthe GEN1 code that integrates them has beenexcellent. We’ve run Rocflo on up to 2,048processors, and we’ve run Rocsolid and GEN1on up to 512 processors on a variety of plat-forms, including the SGI Origin at NCSA, theCray T3E at the Pittsburgh SupercomputerCenter, and all three ASCI platforms at the USDepartment of Energy laboratories. Figure 4shows “scaled speedup,” meaning that the prob-lem size per processor is constant. The largestmesh sizes have about four million elements forthe solid and seven million zones for the fluid.Figures 5 and 6 show visualizations of somecomputational results.

The features we still have to implement tocomplete the full GEN1 code include vis-coelasticity, large deformations, and thermal ef-fects in the solid; a large-eddy simulation tur-bulence model in the fluid; a more detailedmodel of combustion and interface regression;and a flame-spreading model to capture ignitiontransients.

Rocsolid

• Finite element

• Linear elastodynamics

• Unstructured hexahedral meshes

• ALE treatment of interface regression

• Implicit time integration

• Multigrid equation solver

• F90, MPI parallelism

Rocflo

• Finite volume

• Unsteady, viscous, compressible flow

• Block-structured meshes

• ALE moving boundaries

• Explicit time integration

• 2nd order upwind TVD

• F90, MPI parallelism

Figure 3. Solids and fluids codeshave different approaches tocomponent simulation.

(a)1

1,000

100

10

110 100 1,000

Processors

T3E

O2K

SP2

Sca

led

spee

dup

(b)1

1,000

100

10

110 100 1,000

Processors

Sca

led

spee

dup

(c)1

1,000

100

10

110 100 1,000

Processors

Sca

led

spee

dup

T3E

O2K

CLU

T3E

O2K

Figure 4. Graphs show scaled speedup of separate (a) Rocsolid and(b) Rocflo modules and of (c) integrated GEN1 code for the SpaceShuttle RSRM. The problem size is fixed at 14,625 fluid grid pointsand 8,192 solid elements per processor as the number of processorsgrows. Computers used include a Cray T3E, SGI Origin 2000 (02K),IBM SP2, and Intel Pentium cluster (CLU).

MARCH/APRIL 2000 27

System integration issues

A number of technical issues arise in buildingan integrated, multicomponent code such asGEN1. First is the overall integration strategy,where the fundamental choice is between modularand monolithic approaches. In building theGEN1 code, we chose a modular or partioned ap-proach in that we used separately developed com-ponent modules and created an interface to tiethem together. In such an approach, separatelycomputed component solutions might requiresubiterations back and forth between componentsto attain self-consistency. This approach contrastswith a more monolithic strategy in which all thephysical components are incorporated into a singlesystem of equations and all the relevant variablesare updated at the same time, thereby obviatingthe need to iterate to self-consistency. Although ithas some theoretical advantages, a monolithic ap-proach impedes separate development and main-tenance of individual component modules by spe-cialists in the respective areas. The modularapproach not only expedites separate developmentand maintenance, it also allows swapping of indi-vidual modules without replacing the entire codeor even affecting the other modules. The modularstrategy seemed to offer clear practical advantagesin our somewhat dispersed organizational setting,as well as potentially letting users include com-

mercial modules when appropriate.However, even less coupled approaches are

commonly used in practice, in which entirely in-dependent codes interact only offline (often withhuman intervention), perhaps through exchangeof input and output data files. By contrast, in ourmodular GEN1 code, the component modules arecompiled into a single executable code and theyexchange data throughout a run with subroutinecalls and interprocessor communication.

Another important issue is the physical, mathe-matical, and geometric description of the interface

Figure 6. Gas temperature computed by Rocflo in star grain regionof the Space Shuttle RSRM near the onset of steady burning, visual-ized by Rocketeer. Values range from 3,364 K (magenta) to 3,392 K(red). Temperature is represented as (a) tint on the interior surfaceof propellant and as (b) a series of translucent colored isosurfacesin the interior at slightly later time. The rocket is cut in half alonglateral axis to improve visibility.

Figure 5. Fully coupled 3D simulation of star grain in theSpace Shuttle RSRM showing stress in propellant and gaspressure isosurfaces in slots and core region. Executed on a256-processor SGI Origin 2000, visualized with Rocketeer,our in-house visualization tool.

28 COMPUTING IN SCIENCE & ENGINEERING

between components, which in our case includesthe jump conditions combustion induces. A care-ful formulation of the interface boundary condi-tions is necessary to satisfy the relevant conserva-tion laws for mass and linear momentum, as wellas the laws of thermodynamics.

Time-stepping procedures are another signifi-cant issue in component integration. Here, the timesteps for the fluid are significantly smaller thanthose for the solid. We employ a predictor–correc-tor approach in which the fluid is explicitly steppedforward by several (say, 10) time steps, based on thecurrent geometry the solid determines. The result-ing estimate of fluid pressure at the future time isthen available for taking an implicit time step forthe solid. However, the resulting deformation andvelocity of the solid change the fluid’s geometry, sothe time-stepping of the fluid repeats and so on, un-til we attain convergence, which usually requiresonly a few subiterations. Unless iterated until con-vergence, this scheme is only first-order accurate,and it is “serial” in that only one component com-putes at a time. Parallel time-stepping schemes thatare second-order accurate without subiterations arepossible,6 and we plan to investigate these. But be-cause we map both fluid and solid components ontoeach processor, our current scheme does not pre-vent us from utilizing all processors.

As we mentioned briefly earlier, data transferbetween disparate meshes of different componentsis another highly nontrivial issue in component in-tegration. In our approach, we let the meshes dif-fer at the interface in structure, resolution, and dis-cretization methodology, and indeed this is thecase in GEN1, because the fluid mesh is block-structured, relatively fine, and based on cell-centered finite volumes, whereas the solid mesh isunstructured, relatively coarse, and based on node-centered finite elements. Although in principle thetwo interface meshes should abut because they dis-cretize the same surface, in practice we can’t as-sume this because of discretization or roundingerrors. Thus, we have developed general mesh-association algorithms that efficiently determinewhich fluid points are associated with each ele-ment (facet) of the solid interface mesh;7 we thenuse the local coordinates of the associated elementto interpolate relevant field values in a physicallyconservative manner.

Yet another thorny issue in component inte-gration is partitioning the component meshesfor parallel implementation in distributed mem-ory. The block-structured fluid mesh is relativelyeasy to partition in a highly regular manner, butthe unstructured solid mesh is partitioned by a

heuristic approach, currently using Metis, whichoften yields irregular partitions (visit www-users.cs.umn.edu/~karypis/metis for further in-formation). Moreover, because we partition thetwo component meshes separately, there is noway to maintain locality at the interface—adja-cent partitions across the interface may not beplaced on the same or nearby processors. In ourcurrent approach, this effect complicates thecommunication pattern and might increase com-munication overhead, but it has not been a seri-ous drag on parallel efficiency so far. Neverthe-less, we plan to explore more global, coordinatedpartitioning strategies that will preserve localityand perhaps simplify communication patterns.

Software integration framework

Our overarching goal in CSAR is not only todevelop a virtual prototyping tool for SRMs butalso to develop a general software framework andinfrastructure to make such integrated, multi-component simulations much easier. Toward thisend, we have initiated research and developmentefforts in several relevant areas of computer sci-ence, including parallel programming environ-ments, performance monitoring and evaluation,parallel input/output, linear solvers, mesh genera-tion and adaptation, and visualization.

Our work in parallel programming environ-ments has focused on creating an adaptive softwareframework for component integration based ondecomposition and encapsulation through objects.This environment provides automatic adaptiveload balancing in response to dynamic change orrefinement, as well as high-level control of parallelcomponents. It is purposely designed to maintaincompatibility with component modules in con-ventional languages such as Fortran 90, and it pro-vides an automated migration path for the existingparallel MPI code base. This work is in part an ex-tension of the previously developed Charm++ sys-tem, which has successfully built a large parallelcode for molecular dynamics, NAMD.8 Althoughthis framework is not yet mature enough to serve asthe basis for the current GEN1 code, results forpilot implementations of the GEN1 componentmodules using the framework show great promise.

In performance evaluation, we are leveragingongoing development of the Pablo performanceenvironment,9 which provides capabilities for dy-namic, multilevel monitoring and measurement,real-time adaptive resource control, and intelli-gent performance visualization and steering of dis-tributed, possibly geographically dispersed, appli-

MARCH/APRIL 2000 29

cations. We used these tools to instrument theGEN1 code, view the resulting performance data,and relate it back to the application code’s callgraph to identify performance bottlenecks. Wealso updated the popular ParaGraph performancevisualization tool10 to display the behavior and per-formance of parallel programs using MPI, and weused it to analyze the GEN1 component modules.

Parallel I/O is often a bottleneck in large-scalesimulations, both in terms of performance and ofthe programming effort required to manage I/Oexplicitly. For large-scale rocket simulations, weneed good performance for collective I/O acrossmany processors for purposes such as periodicallytaking snapshots of data arrays for visualization orcheckpoint and restart. Because we use geograph-ically dispersed machines, we also need efficientand painless data migration between platforms andback to our home site. Panda11 provides all theseservices—it runs on top of MPI and uses the stan-dard file systems provided with the platforms weuse. Its library offers server-driven collective I/O,

support for various types of data distributions, self-tuning of performance, and integrated support fordata migration that exploits internal parallelism.We’ve already incorporated Panda into Rocflo andare doing the same in Rocsolid. Again, early re-sults are promising, and we plan to use Panda tohandle parallel I/O within our main visualizationtool, Rocketeer (see sidebar).

Validation

Comparison of simulation results with knowntest cases and laboratory data is essential to estab-lish this approach’s validity for science and engi-neering. With our GEN1 integrated code rapidlymaturing, we have begun an aggressive series ofcomputational experiments to verify and validateits efficacy and fidelity. Fluid-solid interactionproblems we use for validation include flow overan elastic panel, flow over a wing (Agard Wing445.6), and a model of inhibitor deformation in anSRM. We hope also to be able to make compar-

Rocketeerby Robert A. Fiedler and John Norris

We need a powerful scientific visualization tool to analyze thelarge, complex 3D data sets our whole system and subscalerocket simulations generate. After an extensive review of ex-isting packages, we decided to develop our own tool, whichwe call Rocketeer. (Visit www.csar.uiuc.edu/F_software/rocketeer to download a user guide and software.) The toolhas a number of features that make it ideal for visualizing datafrom multicomponent simulations, including its support forboth structured and unstructured grids, cell-centered andnode-centered data, ghost cells, seamless merging of multi-ple data files, automated animation, and a smart reader forHDF (hierarchical data format). A particularly useful feature forvisualizing field data in the interior of an SRM is Rocketeer’sability to depict translucent isosurfaces, which lets us view iso-surfaces of temperature or pressure, for example, withoutblocking the view of other isosurfaces deeper inside (see Fig-ure B). Although our need to visualize data from SRM simula-tions specifically motivated many of these features, Rocketeeris a broadly useful, general-purpose visualization tool.

Rocketeer is based on the Visualization Toolkit (VTK),1

which is in turn based on OpenGL to take advantage ofgraphics hardware acceleration. It currently runs on Micro-soft Windows and Unix/Linux. Planned enhancementsinclude implementing a client–server model so that we canperform most compute-intensive operations (in parallel) ona remote supercomputer while interactive control and ren-dering are performed locally on a graphics workstation.

Reference1. W. Schroeder, K. Martin, and W.E. Lorensen, The Visualization Toolkit: An

Object-Oriented Approach to 3D Graphics, Prentice Hall, New York, 1997.

Figure B. Rocketeer visualization of temperature profile computed byRocflo in the star grain region of the RSRM at the onset of steadyburning. Values range from 3,364 K (magenta) to 3,392 K (red).Temperature is represented by the tint on the propellant fin’s surfaceand by a series of translucent colored isosurfaces inside the slot. The im-age is cut in half along the rocket’s axis to enhance visibility. The hottestpoint in this imaged flow is in the stagnation region in the middle of therocket’s head end; the coolest point is in the middle of the star grain re-gion’s open end.

30 COMPUTING IN SCIENCE & ENGINEERING

isons with test data for small laboratory-scale rock-ets through collaboration with various govern-ment laboratories. A larger-scale test we are pur-suing is to try to predict the propellant “slumping”that led to failure for the Titan IV SRB’s originaldesign. Our ultimate test will be comparison withthe immense amount of test data for the SpaceShuttle RSRM taken during static firing tests afterits redesign. These data include literally thousandsof readings from strain gauges, pressure curves,and so on for a liberally instrumented test versionof the Shuttle RSRM.

New research directions

Our second-generation rocket simulation code,GEN2, will require significantly more detailedand sophisticated component models than thosein GEN1. Moreover, GEN2 will also support ac-cident scenarios that will require even greater de-tail and finer resolution. To have these new mod-els ready by the time we need them in GEN2, wehave already begun extensive research into thesemodeling issues, some of which we outline here.

Heterogeneous propellant flamesThe propellant in the Shuttle SRM consists of a

high-density packing of ammonium perchlorate(AP) oxidizer particles embedded in a matrix ofpowdered aluminum fuel and polymeric binder.The aluminum particles burn in the gas-phase prod-ucts of AP-binder combustion. The propellant usesan initial bimodal distribution of AP particle sizes of20 and 200 microns, and the primary combustionfield is located within a few tens of microns of thepropellant surface. Flow disturbances originating in

the chamber due to acoustic-wave or mean-flow in-teractions and turbulence can affect this field, lead-ing to what is known as erosive burning.

Some of the heat generated in the combustionfield is conducted to the propellant surface. Thisheat is responsible for the surface regression andthe conversion of solid propellant to combustiblegases. The resulting burning surface is not flatbecause the instantaneous regression rates of APand binder differ, and if cracks form in the pro-pellant, the increase in propellant surface canlead to a sharp increase in combustion intensity.

To describe the surface regression and to gen-erate boundary conditions for chamber flow, wemust resolve the 3D combustion field and coupleit with physical processes such as heat conductionin a thin surface layer in the propellant and allowfor pressure and thermal feedback from the cham-ber flow (see Figure 7).

Crack propagation in solid propellantThe initiation and propagation of one or more

cracks in the solid propellant or along the grain-case interface can dramatically affect rocket per-formance. Cracks create additional burning sur-faces in the SP, so the propagation of cracks cangreatly affect the pressure history in the rocketchamber, leading in some cases to the rocket’s de-struction. Modeling crack propagation in burningSP is quite complex, because the problem is char-acterized by a tight coupling between structural dy-namics, combustion, and fluid mechanics, alongwith a rapidly evolving geometry. Using a fullycoupled explicit aeroelastic finite-element, finite-volume code, we are investigating potential acci-dent scenarios associated with the presence of pre-

Figure 7. (a) The 2D propellant surface model employs two sizes of ammonium perchlorate particlesimbedded in a fuel/binder matrix. (b) 3D flames supported by AP and binder decomposition productgases for configuration appear at midline region of (a).

MARCH/APRIL 2000 31

existing cracks at various locations in the solidbooster, with special emphasis on propellant-linerinterfacial failures. We’re using a novel cohesive-volumetric finite-element scheme to capture thespontaneous motion of the crack, allowing for thepossibility of crack arrest or branching. As the crackpropagates and the reacting gas pressurizes newlycreated crack faces, the fluid domain undergoescomplex changes that an adaptive unstructured fi-nite-volume scheme can capture. As illustrated inFigure 8, the reactive gas flow emanating from apreexisting radial crack in the SP interacts with thecore flow in the rocket motor. This generates a re-gion of high pressure on the leeward face of thecrack and a region of low pressure in the down-stream vicinity of the crack that lead to substantialdeformation of the propellant grain.

Aluminum particle combustionThe solid propellants in modern rockets use alu-

minum (Al) particles as fuel. As combustion pro-ceeds, these particles in the propellant melt and ag-glomerate on the combustion interface. A complexprocess follows as Al droplets detach from the sur-face and are injected into the core flow. Thedroplets, whose initial size varies from 20 to 300 mi-crons, are injected into a strong cross-flow, and theyprovide a significant source of heat as they burn toform Al oxides. Near-wall turbulence plays an im-portant role in the dispersion of Al droplets, and asa result, heat release is volumetrically distributed,although dominant mainly in the near-wall region.Al2O3 is the primary product of combustion, and itappears either as fine powder of micron size or aslarger residual particles. The deposition of Al2O3in the form of slag in the submerged nozzle can ad-versely affect motor performance.

The combustion of Al droplets strongly cou-ples the dispersed phase (droplets and Al2O3 par-ticles) with the continuous phase (the surroundingcore flow). We expect the GEN2 version ofRocflo to include an Eulerian implementation ofthe core flow and a Lagrangian implementationof the Al droplets and the larger oxide particles.The simulation will introduce tens of millions ofAl droplets into the flow at the combustion inter-face according to a specified probability distribu-tion and local mass injection rate. The position,velocity, temperature, and species concentrationof each droplet will be tracked in the simulationover time by solving a set of ordinary differentialequations. The effect of the surrounding flow willbe parameterized in terms of lift and drag coeffi-cients, heat and mass transfer coefficients, anddroplet burn rate. So far we have developed a de-

tailed time-dependent 3D subsystem simulationof the flow, evaporation, and burning of an iso-lated Al droplet to obtain accurate state-of-the-art parameterizations (see Figure 9).

The individual problems just discussedare themselves examples of fluid−solidinteractions, albeit on finer scales, forwhich we will be able to use the same

software integration framework as for the globalsimulation of the SRM. In this way, we hope toleverage much of our current software develop-ment effort in component integration, and to beable to spin off these subscale simulations from thelarger-scale system simulation in a highly com-patible manner.

AcknowledgmentsWe thank our many colleagues at CSAR for their researchcontributions to this article. This program is truly acollaborative effort based on the technical strengths ofmany people. We thank Amit Acharya, Prosenjit Bagchi, S.Balachandar, Dinshaw Balsara, John Buckmaster, PhilippeGeubelle, Changyu Hwang, Thomas L. Jackson, and Biing-Horng Liou for their contributions to the “New researchdirections” section. The CSAR research program issupported by the US Department of Energy through theUniversity of California under subcontract B341494.

Figure 8. Theeffect of a pre-existing radialcrack on themotor’s core flow:Colored contoursdenote pressure inthe core flow andhydrostatic pres-sure in the solidpropellant. Notethe region of highpressure (red) inthe deformedcrack and of lowpressure (blue)downstream fromthe crack.

32 COMPUTING IN SCIENCE & ENGINEERING

References1. G.R. Nickerson et al., The Solid Propellant Rocket Motor Perfor-

mance Prediction Computer Program (SPP), Version 6.0, Tech. Re-port AFAL-TR-87-078, US Air Force Materials Lab., Edwards AirForce Base, Calif., 1987.

2. G.P. Sutton, Rocket Propulsion Elements, 6th ed., John Wiley &Sons, New York, 1992.

3. M.T. Heath and W.A. Dick, “Virtual Rocketry: Rocket ScienceMeets Computer Science,” IEEE Computational Science & Eng.,Vol. 5, No. 1, Jan.–Mar. 1998, pp. 16–26.

4. I.D. Parsons et al., “Coupled Multi-Physics Simulations of SolidRocket Motors,” Parallel and Distributed Processing Techniquesand Applications Conf., Vol. VI, CSREA Press, 1999.

5. P.V.S. Alavilli, D. Tafti, and F. Najjar, “The Development of anAdvanced Solid-Rocket Flow Simulation Program ROCFLO,” Proc.38th AIAA Aerospace Sciences Meeting and Exhibit, AIAA Press, Re-ston, Va., 2000.

6. C. Farhat and M. Lesoinne, “Two Efficient Staggered Proceduresfor the Serial and Parallel Solution of Three-Dimensional Nonlin-ear Transient Aeroelastic Problems,” Computer Methods in Ap-plied Mechanics and Eng., Vol. 182, Nos. 3 and 4, 2000.

7. X. Jiao, H. Edelsbrunner, and M.T. Heath, “Mesh Association:Formulation and Algorithms,” Proc. Eighth Int’l Meshing Round-table, Tech. Report 99-2288, Sandia Nat’l Labs., Albuquerque,N.M., 1999, pp. 75–82.

8. L. Kale et al., “NAMD2: Greater Scalability for Parallel MolecularDynamics,” J. Computational Physics, Vol. 151, No. 1, May 1999,pp. 283–312.

9. L. DeRose et al., “An Approach to Immersive Performance Visu-alization of Parallel and Wide-Area Distributed Applications,”Proc. Eighth IEEE Symp. High-Performance Distributed Computing,IEEE Computer Soc. Press, Los Alamitos, Calif., 1999, pp.247–254.

10. M.T. Heath and J.A. Etheridge, “Visualizing the Performance ofParallel Programs,” IEEE Software, Vol. 8, No. 5, Sept. 1991, pp.29–39.

11. Y. Cho et al., “Parallel I/O for Scientific Applications on Hetero-geneous Clusters: A Resource-Utilization Approach,” Proc. 13thACM Int’l Conf. Supercomputing, ACM Press, New York, 1999,pp. 253–259.

Michael T. Heath is the director of the Center for Sim-ulation of Advanced Rockets at the University of Illinois,Urbana-Champaign. He is also a professor in the De-partment of Computer Science, the director of theComputational Science and Engineering Program, anda senior research scientist at the National Center forSupercomputing Applications at the university. His re-search interests are in numerical analysis—particularlynumerical linear algebra and optimization—and in par-allel computing. He wrote Scientific Computing: An In-troductory Survey (McGraw-Hill, 1997) and has servedas editor of several journals in scientific and high-per-formance computing. He received a BA in mathemat-ics from the University of Kentucky, an MS in mathe-matics from the University of Tennessee, and a PhD incomputer science from Stanford University. Contacthim at CSAR, 2262 Digital Computer Lab., 1304 WestSpringfield Ave., Urbana, IL 61801; m-heath@uiuc.edu;www.csar.uiuc.edu.

William A. Dick is the managing director of the Centerfor Simulation of Advanced Rockets at the Universityof Illinois, Urbana-Champaign. He received a BS in me-chanical engineering from the University of Delawareand an MBA from the University of Illinois. Prior tocoming to the University of Illinois, he was a deputydirector and composites engineer in the National Sci-ence Foundation’s Engineering Research Center forComposites Manufacturing Science and Engineering.Contact him at CSAR, 2266 Digital Computer Lab.,1304 West Springfield Ave., Urbana, IL 61801; w-dick@uiuc.edu; www.csar.uiuc.edu.

Figure 9. Results obtained from a time-dependent 3D simulation offlow and heat transfer from a spherical droplet in cross-flow. Thesimulation employs a resolution of 81 × 96 × 32 points along the ra-dial, circumferential, and azimuthal directions at Reynolds numberRe = U∞D/ν = 350, based on free stream velocity (U∞) and droplet di-ameter (D). At this Reynolds number, flow is unsteady with time-periodic vortex shedding. (a) Azimuthal velocity contours at an in-stant in time, where we can see an imprint of vortex shedding. (b)Temperature contours, where the approaching flow is hotter (red)than the droplet (blue), which is considered isothermal.