Archive of SciDAC Discovery Highlights

SciDAC Team Develops Petascale-Ready Version of CCSM’s Atmospheric Model


Cubed-sphere spectral element grid

The SciDAC project “Modeling the Earth System” is focused on creating a first-generation Earth system model based on the Community Climate System Model (CCSM). As these improvements will require petascale computing resources, the project is also working to ensure that CCSM is ready to fully utilize DOE’s upcoming petascale platforms. The main bottleneck to petascale performance in Earth system models is the scalability of the atmospheric dynamical core. Team members at Sandia, ORNL and NCAR have thus been focusing on the integration and evaluation of new, more scalable, dynamical cores (based on cubed-sphere grids) into the atmospheric component of the CCSM. The first model successfully integrated uses a new formulation of the spectral element method that locally conserves both mass and energy and has positive preserving advection.

This dynamical core allows the CCSM atmospheric component to use true two-dimensional domain decomposition for the first time, leading to unprecedented scalability demonstrated on LLNL’s BG/L system. The model scales well out to 96,000 processors with an average grid spacing of 25 km. Even better scalability will be possible when computing with a global resolution of 10 km, DOE’s long term goal (DOE ScaLeS Report, 2004). As part of the project’s model verification work, a record-setting one-year simulation was just completed on 64,000 processors of BG/L. This initial simulation was obtained using prescribed surface temperatures and without the CCSM land and ice models. Coupling with the other CCSM component models is the team’s current focus.

Unified Programming Environment for Quantum Chromodynamics

In this snapshot of gluon fields from a supercomputer QCD simulation, the gluon fields are started in a nonuniform, chaotic state (left), and quickly diffuse into the full volume of space simulated on the computer (middle and right).

According to the Standard Model of Particles and Interactions, the fundamental constituents of subatomic particles, such as protons and neutrons, are quarks and gluons. The equations governing the forces among quarks have been known for decades. These forces are mediated by particles called gluons, in much the same way that electromagnetic forces are mediated by photons. However, unlike the forces of electricity and magnetism, they become stronger as quarks are pulled apart; this remarkable behavior, which is responsible for the permanent confinement of quarks, is not captured by other force or field theories. The part of the Standard Model that describes this strong interaction, or color force, between quarks and gluons is called Quantum ChromoDynamics (QCD). Only large scale numerical simulations have allowed us to calculate, to high precision, QCD quantities such as the masses and lifetimes of particles containing quarks (i.e. protons, neutrons, etc.). In QCD, quark and gluon fields are defined on a four-dimensional space-time grid called a lattice. The quantum fluctuations of these fields are calculated by Monte Carlo methods. Under its SciDAC grants the U.S. QCD Collaboration (www.usqcd.org) has created a unified programming environment (www.usqcd.org/software.html) for large scale simulations of lattice QCD. With it, they have performed a wide variety of calculations. These include investigations at unprecedented precision of the properties of strongly interacting matter at high temperatures and densities, investigations of the structure and interactions of hadrons, and determinations of the fundamental parameters of the Standard Model, which encompasses our current knowledge of the forces of nature.

New Methods for Accelerating Molecular Dynamics Simulations

Silver (100) surface at T=600K in contact with a Lennard-Jones model liquid

Using an extension of the parallel-replica dynamics method, the surface diffusion of the silver adatom was accurately accelerated by a factor of 6.5 on 8 processors. Using this same approach with more processors will give much larger boost factors for systems at lower temperature

The main challenge in the SciDAC project "Hierarchical Petascale Simulation Framework for Stress Corrosion Cracking" is developing a computational methodology that can simultaneously treat the vast range of scales in time (picoseconds to seconds and beyond) and length (angstroms to millimeters) necessary for accurately simulating the technologically critical process of stress corrosion cracking. As part of this multi-institution project (involving University of Southern California, Harvard, Purdue, California State University at Northridge, and Los Alamos and Lawrence Livermore national laboratories), researchers at Los Alamos National Laboratory are developing a method for accelerating molecular dynamics simulations at the solid-liquid interface.

In the parallel-replica dynamics method, time is parallelized to achieve longer simulations for infrequent-event processes, such as the diffusion of atoms on a surface, or, as is relevant to this project, the activated processes that advance a stress-loaded crack tip. Because stress corrosion cracking often involves a liquid phase in contact with the crack tip, the parallel-replica dynamics method is being extended so that it can be used to accelerate the dynamics at a solid-liquid interface. Initial results look promising for obtaining significant parallel speedup in time for this much more complex system, which heretofore was limited to time scales accessible to direct molecular dynamics.

Topological Analysis Provides Deeper Insight into Hydrodynamic Instabilities


different stages of the mixing process [animation (mov)]

 

Valerio Pascucci of Lawrence Livermore National Laboratory, working with members of the SciDAC Visualization and Analytics Center for Enabling Technology (VACET), has developed the first feature-based analysis of extremely high-resolution simulations of turbulent mixing. The focus is on Rayleigh-Taylor (RT) instabilities, which are created when a heavy fluid is placed above a light fluid and tiny vertical perturbations in the interface create a characteristic structure of rising bubbles and falling spikes. RT instabilities have received much attention because of their importance in understanding many phenomena, ranging from the rate of formation of heavy elements in supernovae to the design of capsules for inertial confinement fusion. However, systematic, detailed analysis has been difficult due to the extremely complicated features found in the mixing region. This novel approach, based on robust Morse theoretical techniques, systematically segments the envelope of the mixing interface into bubble structures and represents them with a new multi-resolution model, allowing a multi-scale quantitative analysis of the rate of mixing based on bubble count. This analysis enabled new insights and deeper understanding of this fundamental phenomenon by highlighting and providing precise measures for four fundamental stages in the turbulent mixing process that scientists could previously only observe qualitatively. more

Code Shows Radio Waves Are Hot Enough for ITER

 
AORSA simulation of 3D radio-wave electric field
propagating in the ITER plasma

Using his AORSA application on the Cray XT4 Jaguar supercomputer, ORNL physicist Fritz Jaeger has performed 3D simulations of radio-wave heating in fusion reactors. The simulations demonstrate that radio-wave heating should work effectively for both present experiments and the multibillion-dollar ITER fusion reactor. ITER is being developed as a cooperative effort between nations in Europe and Asia, as well as the United States, to demonstrate the scientific and technological feasibility of fusion power. The reactor will use radio waves to heat the ionized gas (plasma) ten times hotter than the sun, thereby causing atoms in the gas to fuse and release energy. Analytical theory, one- and two-dimensional simulations, and experiments have provided an understanding of the relative success of radio-wave heating on medium-scale experiments and its relative inefficiency on smaller experiments. Jaeger’s simulations verified that radio waves tended to heat the edge of the plasma instead of the center on smaller experiments. However, he also demonstrated that radio-wave heating should work efficiently on the larger ITER reactor, which measures more than 12 meters across and will hold more than 840 cubic meters of plasma.
SciDAC project page

Materials by Design Takes a Closer Look at the Structure of Water


Electronic Density of Water in a Carbon Nanotube

  
Although the structure of water has been probed for more than 100 years, scientists still do not agree on its electronic structure and detailed atomic structure. Understanding water's structure could lead to revolutionary new insights in biology, chemistry, and other fields. Using an INCITE allocation on the Blue Gene/L at the Argonne Leadership Computing Facility, members of the Quantum Simulations of Materials and Nanostructures (Q-SIMAN) SciDAC project are using first principles to examine the structure of water in confined and compressed states. Led by Giullia Galli of the University of California, Davis, the Q-SIMAN project is using the Qbox code to study the behaviour of water molecules in nanotubes and between graphene sheets. As Prof. Galli commented in a 2006 Post-Gazette article, "interactions with the wall of the [nano]tube might change water's structure."

The team is also studying hydration with benzene and hexafluorobenzene. Their 2007 paper in J. Phys. Chem. B reports that the electronic structure of interfacial water molecules differs from that of bulk water, as a result of the interaction with the aromatic solute. These results indicate that the solvation of aromatic species is determined by subtle but important charge transfer and dipole redistribution effects, and cast some doubts on the validity of nonpolarizable models for the study of these systems. These findings also indicate that electronic structure information, as contained in ab initio MD simulations, is an important component in a microscopic description of aromatic hydration.

read the Hydrophobic Hydration article in J. Phys. Chem. B
read the PNAS Hydrophobicity commentary by Giullia Galli

SciDAC Results in Nature Letters: Pulsar spins from an instability in the accretion shock of supernovae

   

Researchers John M. Blondin and Anthony Mezzacappa of the Terascale Supernova Initiatve (Alumni SciDAC project: Shedding New Light on Exploding Stars) have a Letter published in the January 2007 issue of Nature. They report a robust instability of the stalled accretion shock in core-collapse supernovae that is able to generate a strong rotational flow in the vicinity of the accreting proto-neutron star (PNS).
read the nature article (pdf)

The flow vectors highlight two strong rotational flows. On the right the flow is moving clockwise along with the shock pattern, whereas at the bottom left the post-shock flow is being diverted into a narrow stream moving anticlockwise, fueling the accretion of angular momentum onto the PNS.


New Project added to SciDAC-2

   
A new project--"Building a universal energy density functional"--has been added to SciDAC-2. It will bring advanced computing resources to bear on the physics of atomic nuclei. Knowledge in this area is needed for pure science as well as engineering.

Scientists need a better theory understanding astrophysical processes, particularly the creation of elements in the stars. Engineering applications include the design of next-generation power reactors, reactors to burn nuclear waste, and simulations to obviate the need for nuclear weapons testing. The theoretical methods to be applied will make extensive use of "density functional theory", a tool that has been spectacularly successful in chemistry and in materials science for predicting the properties of molecules and material systems. Because of the many computational challenges to construct the theory, the project calls for a collaborative effort between computer scientists and nuclear physicists. It is anticipated that this 5-year project will produce a theory and codes that will dramatically improve the accuracy and reliability of predictions of nuclear properties.

Professor Bertsch Aurel Bulgac
The project team is a consortium of 8 universities and 6 national laboratories with funding of $15M. It is led by Professors George Bertsch (far right) and Aurel Bulgac (near right) in the Institute for Nuclear Theory and the Department of Physics at the University of Washington.
more about the project


SciDAC Project Pushing the Frontiers of Chemistry

 
In most first-year chemistry classes at universities, students begin by learning how atoms of carbon and other elements form bonds with other atoms to form molecules, which in turn combine to form ourselves, our planet and our universe.

Understanding the properties and behavior of molecules, or better yet, being able to predict the behavior, is the driving force behind modern chemistry. Theoretically, quantum mechanics means that all the properties of molecules could be predicted. The problem is that the equations are too complex to actually solve, even using the most powerful supercomputers. Predicting the behavior of just one molecule with one electron requires 1,000,000 calculations, while doing the same for an atom with 20 electrons would require 1, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000, 000 calculations. And since scientists are typically interested in atomic systems with at least a few hundred electrons, a better method is needed.

To meet this requirement, the Advanced Methods for Electronic Structure: Local Coupled Cluster Theory project was designed to develop new methods which strike novel compromises between accuracy and feasibility.

SciDAC Tools Fuel Groundbreaking Combustion Research

Using methods developed by SciDAC’s Algorithmic and Software Framework for Applied Partial Differential Equations (APDEC), computational and combustion scientists at the U.S. Department of Energy’s Lawrence Berkeley National Laboratory have created an unparalleled computer simulation of turbulent flames. The research was featured on the cover article of the July 19, 2005 Proceedings of the National Academy of Sciences. The research led to a three-dimensional combustion simulation of unmatched accuracy, a simulation that closely matches conditions found in laboratory combustion experiment. The code allows the researchers to model a flame about 12 cm in height and consisting of 80 chemical species and more than 300 chemical processes. (MORE) - July 2005

Simulations were computed on the IBM SP at NERSC


Better Understanding of Ignition Front Propagation May Lead to Cleaner Engines

Researchers from the Terascale High-Fidelity Simulations of Turbulent Combustion with Detailed Chemistry (TSTC) project are seeking better understanding of inhomogeneous autoignition. Numerical experiments on the effect of thermal stratification on controlling burn rate,under homogeneous charge compression ignition (HCCI) engine conditions, show that increasing thermal stratification promotes more flame-like structures and the zonal model deteriorates with increased stratification. MORE - May 2005


Terascale Supernova Initiative Discovers a New Way Neutron Stars May Be Spun Up

A series of 3D hydrodynamic simulations show the flow in a stellar explosion developing into a strong, stable, rotational flow (streamlines wrapped around the inner core). The flow deposits enough angular momentum on the inner core to produce a core spinning with a period of only a few milliseconds. (MORE) - May 2005

Simulations were computed on the Cray X1 in the Leadership Computing Facility at ORNL.


 


Home  |  ASCR  |  Contact Us  |  DOE disclaimer