BLUE WATER

The Blue Waters supercomputer started being built at the National Center for Supercomputing Applications (NCSA) back in January. When completed, it will be the most powerful supercomputer in the world and able to achieve sustained performance above 1 petaflop. Getting to that point requires a lot of hardware be installed, though. The good news for those wanting to take advantage of all that performance is that Blue Waters doesnt need to be fully installed before it goes online. In fact, Cray has managed to bring the supercomputer online at just 15 percent of its computational power. That first 15 percent is made up of 48 Cray XE6 cabinets and is called the Blue Waters Early Science System. Even at this early stage the 48 cabinets offer up the most computational power the National Science Foundation has to offer.

The ability to use Blue Waters comes down to whether your project has been awarded Petascale Computing Resource Allocations (PRAC). So far, 24+ projects have received PRACs, but only 6 of those are allowed to start using Blue Waters in its current state.

Those 6 projects use Blue Waters for the following purposes:

Modeling high-temperature plasmas to help understand what impact solar flares and solar wind has on our atmosphere. Simulating the formation of very early galaxies that appeared soon after the Big Bang and classed as the Milky Ways ancestors. The volume of the simulation is greater than 200 megaparsecs, where as previous projects only managed 1 megaparsec. Simulating the protein capsid of a HIV-1 genome using 12.5 million atoms to better understand how infection occurs. A lattice quantum chromodynamics study into exchange particles called gluons and the charmonium spectrum. High resolution adaptive mesh refinement simulation of a Type Ia supernova. High resolution time slice simulation of the end of the 20th and 21st centuries to better understand extreme environmental events such as tropical cyclones.

As you can see, even at just 15% of its potential, the Blue Waters system is offering the opportunity to run simulations at a scale and resolution not possible before.

About the Blue Waters project

The Blue Waters project will deliver a supercomputer capable of sustained performance of 1 petaflop on a range of real-world science and engineering applications. It is expected to be one of the most powerful supercomputers in the world. Scientists will create breakthroughs in nearly all fields of science using Blue Waters. They will predict the behavior of complex biological systems, understand how the cosmos evolved after the Big Bang, design new materials at the atomic level, predict the behavior of hurricanes and tornadoes, and simulate complex engineered systems like the power distribution system and airplanes and automobiles. Blue Waters will be composed of more than 235 Cray XE6 cabinets based on the recently announced AMD Opteron 6200 Series processor (formerly code-named "Interlagos") and more than 30 cabinets of a future version of the recently announced Cray XK6 supercomputer with NVIDIA Tesla GPU computing capability. Blue Waters is supported by the National Science Foundation and the University of Illinois. The Blue Waters project also includes a far-reaching educational and workforce development program. It will impact students from K-12 through postgraduate education, reaching out to geographical areas and communities that have been historically underrepresented in supercomputing. At the undergraduate level, the program will educate the next generation of graduate students, K-12 teachers, future technical staff, and the informed public. At the graduate and postgraduate levels, the program will educate and train the next generation of researchers. An expanded industrial partner program is an integral part of the Blue Waters project. Members of the Great Lakes Consortium for Petascale Computation will work with their business and industry partners to introduce them to the world of petascale computing, giving industrial outreach a truly national scale.

Understanding space weather with Blue Waters

Earth is embedded in the sun's atmosphere, so we are exposed to storms on the sunso-called space weather. Because this space weather affects the Earth and our technological systemsuch as communications satellites, GPS signals, power grids, etc.there is an urgent need to develop accurate forecasting capabilities. The dominant mechanism that triggers storms on the sun and the process that enables the radiation from the sun to enter the Earth's magnetosphere is magnetic reconnection. So better understanding of magnetic reconnection can lead to better understanding of and prediction of disruptive space weather.

Magnetic reconnection is initiated on the electron scale but leads to global rearrangement of magnetic fields. This multi-scale structure of reconnection has posed a major computational challenge. Using petascale computers, we are, for the first time, able to conduct 3D large-scale kinetic simulations that include the important electron physics. Blue Waters will be one of the fastest computers in the world, and with this added computational power, we will be able to address new aspects of reconnection that would be out of reach otherwise.

This the compute-intensive, memory-intensive, and data-intensive supercomputer will have over 17,000 disks and a bandwidth to near-line storage of 100 GB/s. The full specifications of Blue Waters supercomputer are robust to say the least:

Cray XE6 cabinets: >235 Cray XK6 cabinets: >30 Total cabinets, including storage and server cabinets: >300 Compute nodes: >25,000 Usable storage bandwidth: >1 TB/s

Aggregate system memory: >1.5 PB Memory per core: 4 GB Gemini network cables: Over 9,000 (~4,500km of single wires) Interconnect topology: 3D Torus Number of disks: >17,000 Number of memory DIMMS: >190,000 Usable storage: >25 PB Peak performance: >11.5 PF Number of AMD processors: >49,000 Number of AMD x86 cores: >380,000 Number of NVIDIA GPUs: >3,000 External network bandwidth: 100 Gb/s scaling to 300 Gb/s Integrated Near Line Environment: Scaling to 500 petabytes Bandwidth to near-line storage: 100 GB/s

NVIDIAs Tesla GPUs will accelerate some of those compute-intensive applications in conjunction with the large number of Cray systems general purpose CPUs. The Blue Waters system will be a powerful hybrid supercomputer with more than 235 Cray XE6 cabinets, and more than 30 cabinets of a future version of the recently announced Cray XK6 supercomputer, which includes next-generation NVIDIA Tesla GPUs based on the Kepler architecture. The Blue Waters project focuses on sustained petascale performance for full-form science and engineering challenges. NCSA will be working closely with NVIDIA and Cray to expand the efficient use of this part of the Blue Waters architecture for real applications. Tesla GPUs are massively parallel accelerators based on the CUDA parallel computing architecture. Application developers for the Blue Waters supercomputer can accelerate applications in C, C++, or Fortran using simple compiler directives, or with NVIDIAs CUDA tools. Compiler directives are a rapidly evolving approach that allows developers to simply augment their code with a few hints that direct the compiler on how to automatically parallelize the application.
Submitted by: Ajay Dabas (208/co/09) references: http://www.ncsa.illinois.edu