The National Institute for Computational Sciences

Researcher Question and Answer

Wladimir Lyra remembers vividly when he decided to become an astronomer. "I was five years old," he says. "I happened to come across a sketch of the Solar System in a school science book. It was a simple drawing. I recognized the Earth, the Sun, and the Moon, but I had never heard of the other stuff. The sudden realization that the Earth was just one among a family of worlds hit me with a force that is hard to describe. I was hooked. I wanted to know everything about it. For a year, I didn't talk about anything else. It became an obsession, then a lifelong interest, a career goal, and now, indeed, my profession."

Today, Lyra is an assistant professor at California State University at Northridge, Department of Physics and Astronomy. He is also a research associate at NASA’s Jet Propulsion Laboratory and a visitor at Caltech’s Division of Geological and Planetary Sciences. A native of Rio de Janeiro, Brazil, he received his Ph.D. in astronomy in February 2009 from Uppsala University, Sweden.

Lyra is a recipient of the Sagan Fellowship, the prestigious postdoctoral research grant named after the famous astronomer and science communicator Carl Sagan. The highly competitive grant (usually oversubscribed 15 to 1) is given to support independent research that is broadly related to the science goals of NASA's Exoplanet Exploration program, the primary goal of which is to discover and characterize planetary systems and Earth-like planets around nearby stars. The Sagan Fellowship provided $309,000. Lyra has also been awarded a National Science Foundation (NSF) grant (award number 1009802) in the sum of $460,911.

He says NSF's eXtreme Science and Engineering Discovery Environment (XSEDE) has played an essential role in moving his research forward. XSEDE is a single virtual system that scientists can use to interactively share computing resources, data, and expertise. People around the world use these resources and services—things like supercomputers, collections of data and new tools—to improve our planet.

So far in his young career, Lyra has published 32 scientific articles, 22 of those as a first or second author. At the time of this writing, his articles have been cited 900 times. In one of the papers, published in the journal Nature, he and co-researchers discuss how they uncovered a mechanism to produce the puzzling rings seen in debris disks even without planets, a result that has exciting implications for the interpretation of observations that do not see the planet and use these rings to infer properties of the hypothetical planet. The Kraken supercomputer (now decommissioned), managed by the National Institute for Computational Sciences (NICS), provided the advanced computing support for the research detailed in the Nature article.

Lyra's research has the broader goal of developing the physics of gas in evolved disks, which as long been overlooked. The assumption has been that these evolved disks had no gas, and all the dynamics was due simply to gravity. The work of Lyra and his colleagues is showing that hydrodynamics is important; the next step is to develop the theory.

After Kraken was decommissioned, Lyra migrated to the Stampede system at the Texas Advanced Computing Center. A list of peer-reviewed articles that Lyra has co-authored describing research that received the support of XSEDE advanced computing resources is provided at the end of this Question and Answer piece.

Lyra has also been a co-developer of the Pencil Code, an open-source and free software for solving partial differential equations designed for efficient computation with massive parallelization. The code is versatile and applicable to a variety of astrophysical problems.

Published applications of the code include turbulence, magnetic dynamos, solar physics, stellar explosions (supernovae), and astrobiology. Industry applies the code as well, in combustion research, where it is valued for being open source and free. The code is maintained by an international collaboration of about 10 core developers, including Lyra. His edits currently account for 15 percent of the nearly 250,000 lines in the Pencil source code.

Popular science articles about Lyra's work have been written for American and international publications (in Sweden, France, and Brazil), such as Sky & Telescope, New Scientist, and Time magazine.

Lyra's current research focuses on computer simulations of circumstellar disks, with emphasis on the processes that led to planet formation, such as magnetorotational turbulence, streaming instability, and disk vortices. A goal of the research is to establish a model that combines all the necessary physics to simulate the formation of planetary systems and enable comparisons with the astronomical observations.

What is the context for your research with respect to the benefits to society?

Lyra: My current research interests revolve around the theory of planet formation, a very ancient question: How did the Earth come to be? Virtually every society in recorded history, every culture known to anthropology, has tried at some point to answer this question. In modern times, the quest has been stirred up by the discovery of the roughly 2,000 planets around other stars (known as extrasolar planets or exoplanets). The vast majority of these exoplanets lie in planetary systems remarkably different from our own. A major goal of my research is to understand this diversity, at the same time tracing back our own origins.

I do it by reconstructing the conditions that existed in the primordial nebula around the young Sun that engendered the Solar System. The planets orbit the Sun in the same plane, and all in the same direction. The simplest way to obtain this configuration is to suppose that the planets were formed in a disk that orbited the young Sun. The fact that Jupiter and Saturn are mostly composed of gas indicates that this disk was a gas disk. Theoretical research on planet formation thus involves an accurate understanding of the behavior of this disk of gas. Basically, one needs to solve the equations of hydrodynamics in a gravitational field.

However, there is a problem: hydrodynamics is hard. There is no known mathematical method to solve its central equation, the Navier-Stokes equation, without making approximations. Because of this, we do not have a consistent theory yet to describe turbulence, a phenomenon of immense importance in physics and engineering. In fact, the Clay Mathematics Institute lists the Navier-Stokes equation as one of its “Millenium Prize Problems" and offers a $1,000,000 prize to the first person who can provide a solution to it.

A hundred years ago, there was not much to do concerning this problem. Today, however, we have a powerful tool: computers. Instead of trying to solve the intractable equation with pen and paper, we write a computer program that solves it. We program an initial state, and this initial state determines the subsequent time evolution of the system.

We are living in a special time when computers have become powerful enough to solve these equations—not in a full solution to claim the Millenium Prize, but accurately enough to understand the behavior of physical systems. Today, we perform computer simulations of the disk whence the Earth was formed. Finally, for the first time in history, we are building a scientific answer to the age-old question of our origins. In my computer simulations, in addition to hydrodynamics, I include physical processes such as magnetic fields and turbulence, thermodynamics and radiation, dust, and aerodynamics. By doing this, I construct, harnessing the full power of modern computers, a digital laboratory to study the processes taking place during the formation of our planet.

What challenges are you addressing using the XSEDE advanced computing resources?

Lyra: The main challenge we have is the multiscale nature of the problem at hand. We need to go to high enough resolution to resolve the spatial scales of the processes, but at the same time model the global evolution of the circumstellar disk. Moreover, the dynamical time scale is short compared with the lifetime of these same disks. These simulations include not only hydrodynamics but also magnetic fields, which are ubiquitous in astrophysics. In fact, the main process leading to structure in these disks is hydromagnetic.

Also, the code we use is particle-mesh, meaning that we have a two-phase calculation, of gas and solids, where the gas is solved in a mesh, where the grains that will collapse gravitationally to form planets are treated as numerical particles. To parallelize this code efficiently, we need to use special algorithms to achieve load balance across processors. In addition, since we want to follow the gravitational collapse of a swarm of pebbles and boulders into a planet, we need to include gravity in the calculations as well. The challenge of parallelizing a gravity code efficiently over tens of thousands of processors is far from trivial.

Fast Fourier transforms are usually used for solving the equation that describes gravity, but their applicability is restricted to particular grid geometries; also they do not scale too well over processor counts higher than 1000s. Instead, we moved to use what are known as hierarchical particle trees, which we are still in the process of testing.

We published results on the excitation of planet-forming whirlpools in protoplanetary disks. That took 2 million computer hours on 18,000 processors with Kraken, before it was decommissioned. Back then, it was the highest processor count used in a published astrophysical simulation. Now, one of my projects is to model the formation of Super-Earths close to their stars—the main type of planet discovered by Kepler [the space observatory launched by NASA to discover Earth-like planets orbiting other stars]. I estimate that one such simulation will take about 3 million computer hours on Stampede.

This is an artist's impression of a young star surrounded by a protoplanetary disk in which planets are forming. Using the 15-kilometer baseline of the Atacama Large Millimeter Array (ALMA), astronomers were able to make the first detailed image of a protoplanetary disk, which revealed the complex structure of the disk. Concentric rings of gas, with gaps indicating planet formation, are visible in this picture and were predicted by computer simulations. Now these structures have been observed by ALMA for the first time. Note that the planets are not shown to scale. [Image credit: European Southern Observatory/L. Calçada]

What successes have you had in your research?

Lyra: I uncovered a new route for planet formation, via vortices that exist in the gas disk, like hurricanes on Earth’s atmosphere. In the computer simulations, these vortices trap rocks of pebble and boulder size, concentrating them so intensely that their combined gravity makes them glue together. The formed objects are of the size of the Moon or Mars, that subsequently merge to form bigger planets. Evidence of this process has recently been discovered through observations using the Atacama Large Millimeter Array (ALMA), a $1.4-billion array of antennas built in Chile with American participation. ALMA is the largest and most expensive astronomical instrument in operation on Earth. My work has been instrumental in interpreting these observations, and I am periodically invited to review publication works in the field, as well as to give talks about it at international conferences.

This artist's impression shows the dust trap in the system Oph-IRS 48. The dust trap provides a safe haven for the tiny rocks in the disc, allowing them to clump together and grow to sizes that allow them to survive on their own. [Image credit: European Southern Observatory/L. Calçada]

Have you taken advantage of XSEDE's Extended Collaborative Support Services program?

Lyra: No, I haven't. As I mentioned previously, I work building a 'community code.' Our group is composed of about 20 people all over the world checking in improvements on a daily basis. And we hold annual meetings, so we have our own internal support system. But the Extended Collaborative Support Services program, being specific to XSEDE, sounds very useful, and I would like to try it.

This simulation video, created with the support of the Kraken supercomputer (now decommissioned) as part of research published in the journal Nature in 2013, shows sharp and eccentric rings in debris disks. These disks are usually taken as evidence for the gravitational influence of unseen planets, as they are mostly gas-free systems of dust and leftover planetesimals, objects formed from dust, rock, or other material. Simulation: Wladimir Lyra.

Listen to a podcast interview with Wladimir Lyra here.

Lyra-co-authored Publications Acknowledging the Use of XSEDE

  • On shocks driven by high-mass planets in radiatively inefficient disks. II. Three-dimensional global disk simulations. Lyra, W., Richert, A.J.W., Boley, A., Turner, N., Mac Low, M.-M., Okuzumi, S., & Flock, M. 2015, ApJ, in press.
  • On shocks driven by high-mass planets in radiatively inefficient disks. I. Two-dimensional global disk simulations. Richert, A.J.W., Lyra, W., Boley, A.C., Mac Low, M.-M. & Turner, N. 2015, ApJ, 804, 95.
  • Particle trapping and streaming instability in vortices in protoplanetary disks. Raettig, N., Klahr H., & Lyra, W. 2015, ApJ, 804, 35.
  • Rossby wave instability does not require sharp resistivity gradients. Lyra, W., Turner, N.J., McNally, C.P. 2015, A&A, 574, A10.
  • Convective overstability in accretion disks: Three-dimensional linear analysis and nonlinear saturation. Lyra, W. 2014, ApJ, 789, 77.
  • Formation of sharp eccentric rings in debris disks with gas but without planets. Lyra, W. & Kuchner, M.J. 2013, Nature, 499, 184.
  • A parameter study for baroclinic vortex amplification. Raettig, N., Lyra, W., & Klahr, H. 2013, ApJ, 765, 115.
  • Rossby wave instability at dead zone boundaries in 3D resistive magnetohydrodynamical global models of protoplanetary disks. Lyra, W. & Mac Low, M.-M. 2012, ApJ, 756, 62.
  • A well-posed Kelvin-Helmholtz instability test and comparison. McNally, C., Lyra, W., & Passy, J.-C. 2012, ApJS, 201, 18.

Article posting date: 14 December 2015

About JICS and NICS: The Joint Institute for Computational Sciences (JICS) was established by the University of Tennessee and Oak Ridge National Laboratory (ORNL) to advance scientific discovery and leading-edge engineering, and to further knowledge of computational modeling and simulation. JICS realizes its vision by taking full advantage of petascale-and-beyond computers housed at ORNL and by educating a new generation of scientists and engineers to be well versed in the application of computational modeling and simulation for solving the most challenging scientific and engineering problems. JICS operates the National Institute for Computational Sciences, NICS, one of the nation's leading advanced computing centers. NICS is co-located on the UT Knoxville campus and ORNL, home of the world's most powerful computing complex. The center's mission is to expand the boundaries of human understanding while ensuring the United States' continued leadership in science, technology, engineering, and mathematics. NICS is a major partner in the National Science Foundation's eXtreme Science and Engineering Discovery Environment (XSEDE).