Opportunities in AI/ML for the Rubin LSST Dark Energy Science Collaboration

The Big Picture

Starting in 2025, a camera the size of a small car began scanning the entire southern sky every few nights from a mountaintop in Chile. The Vera C. Rubin Observatory’s Legacy Survey of Space and Time (LSST) will spend a decade photographing 20 billion galaxies, generating roughly 20 terabytes of data every single night. That’s like downloading the entire Library of Congress twice, daily, for ten years.

The goal: understanding the two biggest mysteries in physics. Dark energy, the unknown force accelerating the universe’s expansion. Dark matter, the invisible substance shaping how galaxies cluster and move. But traditional analysis tools will buckle under LSST’s data deluge. A new white paper from the LSST Dark Energy Science Collaboration (DESC), with over 70 scientists from four continents, lays out a roadmap for how AI and machine learning must evolve to meet this challenge.

Key Insight: The same core AI/ML challenges — uncertain predictions, distributional shift, and lack of interpretability — appear across every major dark energy measurement technique. Focused investment in a few methodological areas could produce gains across the entire field at once.

How It Works

The DESC isn’t starting from scratch. AI and machine learning are already woven into their pipelines.

Photometric redshifts (estimating galaxy distances from the colors of their light, rather than through slow spectroscopic observations) already use neural networks. Transient classification, deciding in real time whether a new light source is a supernova, asteroid, or active galactic nucleus, relies on ML classifiers processing millions of nightly alerts. Weak gravitational lensing, which maps dark matter by measuring how it subtly distorts background galaxy shapes, increasingly uses deep learning for shape measurement.

The white paper identifies a gap, though: most current AI tools aren’t ready for precision cosmology. Cosmology demands error bars you can actually trust. If a neural network predicts “this galaxy is at redshift 0.8 with 5% uncertainty,” it needs to be right 95% of the time. Not just on the training set, but on real LSST data that will look slightly different from any simulation used to train it. This is covariate shift, and it’s everywhere: simulations never perfectly match reality, and instruments drift as they age.

The collaboration identifies four methodological priorities that cut across all their science cases:

• Bayesian inference at scale: Methods like simulation-based inference (SBI), which learns statistical patterns by comparing thousands of simulations to observations rather than solving equations directly, and normalizing flows, algorithms that map out the full probability distribution of any unknown quantity, can replace unreliable point estimates. Scaling these to LSST’s volume requires new algorithmic approaches.
• Physics-informed methods: Embedding known physical laws directly into neural network architectures (how gravity bends light, how galaxies cluster) reduces training data requirements and improves reliability when data falls outside the training distribution.
• Validation frameworks: Protocols to test whether AI systems fail gracefully or catastrophically when faced with unexpected inputs, before such failures corrupt a decade of measurements.
• Active learning for discovery: Intelligently selecting which objects warrant expensive spectroscopic follow-up, rather than random sampling, could multiply the scientific yield of complementary surveys.

The white paper also looks ahead to foundation models, massive AI systems pre-trained on enormous datasets and then fine-tuned for many tasks, and agentic AI systems. The authors draw an explicit parallel to how large language models transformed natural language processing: a single model trained on broad astronomical data might work as a universal backbone for dozens of downstream analyses.

Early examples like AstroCLIP and UniverseNet have already shown promise. LLM-driven agents that write code, run analyses, interpret results, and iterate autonomously might eventually handle entire pipeline segments, from raw images to cosmological parameters.

The white paper tempers this optimism, however. Foundation models trained on biased simulations could propagate subtle systematic errors across every downstream analysis that depends on them. The authors call for serious evaluation and governance before deploying such systems in production cosmology pipelines.

Why It Matters

The questions DESC is trying to answer sit among the deepest in fundamental physics: whether dark energy is Einstein’s cosmological constant (a fixed background energy woven into the fabric of spacetime) or something stranger, and whether general relativity holds on cosmic scales. Getting the right answer requires not just better telescopes but better statistical machinery.

The AI methods being developed for LSST push hard on uncertainty quantification (measuring how confident a prediction actually is), domain adaptation (making models work reliably on data that differs from their training set), and physics-constrained machine learning. Drug discovery, climate modeling, and particle physics all face the same problems.

What sets this white paper apart is its collaborative scope. Rather than advocating for a single lab’s approach, it synthesizes the priorities of an entire international community. The authors flag the need for open-source tools, shared validation datasets, and reproducible pipelines. They also raise the underappreciated challenge of compute equity: as analyses grow more computationally intensive, smaller institutions risk being locked out of frontier science. And they stress the need to train scientists who can speak both physics and machine learning fluently.

Bottom Line: LSST will overwhelm conventional analysis methods with data. This roadmap shows how AI, built on trustworthy uncertainty quantification, physics-informed design, and careful validation, can turn that flood into the most precise measurements of dark energy ever made.