TOPS Scalable Eigensolvers for Accelerator Design and Thermochemistry

PIs: W.G. Gao ² , P. Husbands ² , S. Li ^2,3 , M. Minkoff ¹ , E.G. Ng ² , C. Yang ²
Principal affiliates: K. Ko (AST), L.Q. Li (AST), A. Wagner (ASCTKD) ¹ Argonne National Laboratory, ² Lawrence Berkeley National Lab, ³ U. California-Berkeley

Summary

In support of the design of next-generation accelerators and molecular quantum mechanical analysis, the Terascale Optimal PDE Simulations (TOPS) project is designing next-generation large-scale eigenanalysis software.

The Next Linear Collider (NLC) represents a major investment by DOE in high-energy physics. Because of the costs involved in building the accelerating structure, reliable and accurate numerical simulation of its characteristics before construction is essential. To this end, researchers at the Stanford Linear Accelerator Center (SLAC) have developed several codes that simulate different aspects of the structure.

Figure 1. Model of a 47-cell section of the 206-cell Next Linear Collider structure

One such code, Omega3P, is used in accelerator cavity design. This code calculates cavity mode frequencies and field vectors by solving a generalized (real and symmetric) eigenvalue problem arising from a finite element discretization of Maxwell's equations. The eigenvalues of interest in this problem are in the interior of the spectrum and are often tightly clustered. To confirm their results and improve the current software, TOPS has been working on alternative eigensolvers, including implementations of the Exact Shift-Invert Lanczos (ESIL) and the Automated Multilevel Substructuring Method (AMLS) methods.

The ESIL method is known to be very reliable, but presents several engineering challenges when applied to problems of the scale needed by SLAC. At the heart of the ESIL eigensolver are the parallel factorization and triangular solution of sparse matrices. These operations are implemented in TOPS' SuperLU package, a leading high-performance scalable solver for sparse linear systems. The software has been carefully designed for optimal performance on modern architectures. The sequential version of the code has achieved up to 40% of the peak megaflop rate on hierarchical memory machines. The parallel version has also achieved high performance, up to a hundred-fold speedup on large matrices.

SuperLU is paired with PARPACK, a parallel implementation of the Implicitly Restarted Arnoldi Method to obtain a parallel implementation of the ESIL algorithm. Depending on the size of the structure under study, the discretization of the equations, and the order of the elements used, the matrices involved can be very large, with up to 300 million nonzeros. On the IBM SP at NERSC, we have tested many problems in Omega3P, the largest being the 47-cell structure shown in Figure 1. A sample matrix (of order 1.3 million) is shown in Figure 2. Over the past year TOPS has been working to scale up the ESIL method and has been able to find the interior eigenvalues of a matrix of order 7.5 million with over 300 million nonzero entries. The running time for this problem is also faster than with the Filtering Algorithm, a technique that combines an Inexact Shift-Invert Method with a correction step.

Figure 2. City plot of the matrix arising from the 47-cell structure.

Our ESIL code is currently integrated as a run-time option in Omega3P and more comparisons on larger problems are in progress. Our method takes advantage not only of the high floating-point execution rate of modern parallel machines but also of the tremendous amounts of available memory. SuperLU has been modified to execute more effectively on supercomputers made up of clusters of SMPs such as the IBM SP. This optimization can be used for all problems, not just the application to finding interior eigenvalues for the accelerator design.

Future work on SuperLU includes improving the performance of the triangular solution phase. This is important in contexts similar to ours where only one factorization is performed, but many triangular solutions are needed. We also plan to leverage the memory hierarchy tuning techniques being developed in TOPS.

Work is also progressing on an implementation and further analysis of the Automated Multilevel Substructuring Method (AMLS). This technique has shown great promise in structural engineering problems and we are investigating its application to accelerator design. A serial implementation is complete and parallelization efforts are under way.

Additionally, TOPS is supporting the SciDAC project “Advanced Software for the Calculation of Thermochemistry, Kinetics, and Dynamics” with the development of a multigrid-like preconditioner for extending the Davidson iterative eigenanalysis method. This approach, referred to as Subspace Projected Approximated Methods (SPAM), generates a sequence of approximating preconditioners. It is being applied to molecular quantum vibrational analysis and applications for CI-calculations (via the COLUMBUS software library). SPAM is being extended to treat linear systems for use in Green's function calculations.

To present a common, easy-to-use interface to our software we are also working on the integration of our libraries into the Common Component Architecture (CCA) framework. In addition to ensuring that our code will work with the TOPS SIDL interface we are also investigating the programmability of alternative eigensolver interfaces.

The TOPS project webpage may be found at http://www.tops-scidac.org .

For further information on this subject contact:
Prof. David E. Keyes, Project Lead
Columbia University
Phone: 212-854-1120
david.keyes@columbia.edu

back to project page