Alumni Project
Discretization Tools for Terascale Computational Science from the Terascale Simulation Tools and Technology (TSTT) Center
PIs: J. Glimm1,2, D. Brown3, L. Freitag4, Co-PIs: E. D’Azevedo5, P. Fischer6, P. Knupp4, X.L. Li2, M. Shephard7, H. Trease8, Affiliated Researchers: J. Drake5 (Climate), K. Ko9 (Accelerators), S. Jardin10 (CEMM), T. Mezzacappa5 (TSI), R. Rosner11 (MRC/FLASH), D. Quinlan3 (PERC ISIC)
1Brookhaven National Laboratory, 2State University of New York at Stony Brook, 3Lawrence Livermore National Laboratory, 4Sandia National Laboratory, 5Oak Ridge National Laboratory, 6Argonne National Laboratory, 7Rensselear Polytechnic Institute, 8Pacific Northwest National Laboratory 9Stanford Linear Accelerator, 10Princeton Plasma Physics Laboratory, 11University of Chicago.
Summary
A major goal of the TSTT Center is to deliver interoperable discretization software for PDE-based terascale scientific simulation. Through both algorithm and software development, we are delivering high-order time and space discretizations and boundary conditions to application scientists for use in mesh-based simulation codes. SciDAC-enabled collaborations with the astrophysics, fusion, high-energy accelerator modeling, and climate modeling communities provide an early insertion path for our technology and assure the relevance of our interoperable discretization tool development.
Vision. Many simulation projects in the Department of Energy use the solutions of partial differential equations (PDEs) to model physical behavior of interest. Solving these PDEs on terascale computers first requires the representation of the equations and their solution at discrete points in space and time. A collection of such discrete points is called a computational mesh. The process of constructing the discrete representation is called discretization. TSTT discretization technology is being used to impact SciDAC application efforts through direct insertion of currently available TSTT algorithmic and software technology into the application solution process and through the development of new techniques that will impact future software development. SciDAC is revolutionizing the way DOE science is done by bringing together TSTT mathematicians and computer scientists with domain scientists in accelerator design, astrophysics, climate and fusion, enabling scientific discovery through the application of state-of-the-art discretization technology.
Astrophysics. Simulations performed at the two astrophysics SciDAC centers, the Terascale Supernova Initiative (TSI) and the Magenetic Reconnection Center, employ discretization strategies for various aspects of the solution process including the transport of neutrinos and the simulation of hydrodynamic behavior. The primary focus of our collaboration with these two centers is to explore high resolution discontinuous Galerkin discretization strategies for application to these areas. In addition, for the TSI Center, we are exploring a 3D spatial computer CPU caching strategy to reduce costly evaluations of the scattering kernels for Boltzman transport. During the next year, we will continue to test these novel discretization strategies on increasingly more difficult test problems, and work to integrate the successful approaches into astrophysics simulation codes.
High-energy Accelerator Modeling. Essential to the design of advanced high-energy physics accelerators is the ability to simulate and model the electromagnetic wave-guide cavity structures that steer and focus a beam of high-energy charged particles through the accelerator. Performing this task requires the computer solution of the time-dependent Maxwell’s equations of electromagnetics. In collaboration with scientists at SLAC, we are analyzing and developing new discretization schemes for solving these equations in order to increase the robustness of their simulation and design tools.
This figure shows the time evolution of a magnetic reconnection instability computed using discontinuous Galerkin methods. The two oppositely directed currents are unstable in the presence a constant magnetic field in the horizontal plane, which forces them to rotate to a horizontal configuration.
The objective is to design schemes that are uniformly stable on many classes of computational meshes. The result will be to give the scientists more options in the choice of mesh type, and hence more flexibility in the design process. We are investigating the addition of spatial filtering and dissipation techniques to the current schemes, as well as pursuing the redesign of the discretization schemes altogether. The ultimate goal is to develop approaches for making the current simulation tools more robust without requiring major restructuring or rewriting of code.
Climate Modeling. Climate and weather models have traditionally been based on spectral discretization methods that exploit the underlying symmetries of spherical geometry by representing the solution as a tensor product of global basis functions. Such methods provide a high degree of accuracy per grid point and make the implicit steps of the underlying PDEs easy to solve. Unfortunately, the transforms needed for both operator evaluation and inversion, require all-to-all communication on modern multi-processor terascale computers, often resulting in a bottleneck that destroys the performance of the algorithms. The TSTT Center is investigating the application of spectral element methods, which employ local spectral expansions, hence offering the high accuracy of the traditional spectral methods, but avoiding the communications bottleneck on high-end computers. A new preconditioner for spectral element simulations has been developed through collaborative work between TSTT and climate scientists that shows significant promise when applied to a standard test problem for climate simulations. Since spectral element methods are well suited for use together with a lightweight adaptive meshing strategy, we will pursue the combination of these two technologies during the coming year.
Fusion. Our work with the fusion community has also targeted the use of high-order adaptive discretization technologies. In collaboration with fusion scientists, we are testing discontinuous Galerkin discretization technology on problems of relevance to MHD fusion. During the coming year, based on the outcome of these tests, we will determine the most suitable forms of adaptivity for insertion into fusion applications.
Development of interoperable discretization tools. Within the TSTT center, we are leveraging our broad expertise in finite difference, finite element, and spectral element discretization methods for partial differential equations (PDEs) to provide interoperable discretization tools that will enable the rapid development of new software applications for scientific discovery. In support of this long-term goal, researchers in the TSTT center have been separating and re-implementing low level discretization operators from their respective frameworks in preparation for insertion into an interoperable discretization library. Research in collaboration with the PERC ISIC will lead to automatic performance optimizations
Further information: http://www.tstt-scidac.org
James Glimm, Brookhaven National Laboratory,
Phone: 631-333-8155, glimm@bnl.gov
David L. Brown, Lawrence Livermore National Laboratory,
Phone: 925-424-3557, dbl@llnl.gov
Lori Freitag Diachin, Sandia National Laboratories,
Phone: 505-284-9711, ladiach@sandia.gov
back to project page
