Anomaly Detection and Approximate Similarity Searches of Transients in Real-time Data Streams

The Big Picture

Imagine trying to find a single misprinted page inside every book ever published, while new books arrive by the thousands every night. That’s roughly the challenge facing astronomers searching for rare cosmic explosions in modern sky surveys.

The Zwicky Transient Facility (ZTF) is a wide-field camera in California that scans the entire visible sky every few nights, generating roughly a million alerts per night. Each alert flags a point of light that changed: something brightened, faded, or flickered where it wasn’t before. Hidden in that flood are supernovae, stars being torn apart by black holes, and phenomena we haven’t yet named.

The problem isn’t finding things that go bump in the night. The problem is finding the right bumps.

A team of researchers built a system called LAISS (Lightcurve Anomaly Identification and Similarity Search) to do exactly that, automatically, in real time, every single night. Astronomers call these short-lived cosmic events “transients”: objects that appear, change dramatically, and fade over days to months. Instead of asking scientists to sift through thousands of candidates, LAISS delivers a curated list of roughly 1 to 5 genuinely unusual objects per night, ready for telescope follow-up.

Key Insight: LAISS uses machine learning anomaly detection and millisecond-speed similarity search to systematically hunt rare cosmic transients in live data streams, and it’s already found hundreds of objects that nobody knew existed.

How It Works

The pipeline runs two major engines in tandem. The first is an anomaly detection model, a random forest classifier (many decision trees that vote together) trained to flag transients that look unusual compared to the broad population of known objects. The second is an approximate similarity search that finds the closest known analogs to any flagged event, in milliseconds, across a database of thousands of light curves.

LAISS extracts two broad categories of features for each transient. Statistical light-curve features (a light curve is a graph of brightness over time) capture the shape of how a source brightens and fades: rise time, peak magnitude, color evolution, variability statistics. Contextual host-galaxy features describe the environment: stellar mass, star formation rate, galaxy morphology. Together, these paint a portrait of both the event and its neighborhood.

The classifier sorts anomalies into three types:

• Spectroscopic anomalies — rare-class transients whose chemical fingerprints look unusual, such as superluminous supernovae or tidal disruption events
• Contextual anomalies — events in unexpected environments, like a core-collapse supernova in a supposedly “dead” elliptical galaxy
• Behavioral anomalies — objects with peculiar light-curve shapes, including those powered by collisions between ejected material and surrounding gas shells, producing multiple peaks or extended plateaus

Once LAISS flags a candidate, approximate nearest-neighbor algorithms (techniques that trade a small amount of exactness for enormous speed) return a ranked list of historical analog transients. Astronomers get an instant prior on what they’re looking at before any spectrum is taken.

LAISS runs on the nightly ZTF Alert Stream through the ANTARES broker, an intermediary that receives ZTF’s million-per-night alerts and applies filters before they reach scientists. By focusing on a curated stream of extragalactic transients, the system keeps the false positive rate low enough that only a manageable handful of candidates surface each night.

Why It Matters

The scale problem LAISS is solving will only intensify. The Vera C. Rubin Observatory, currently being commissioned in Chile, will survey the entire southern sky every few nights and generate roughly ten million transient alerts per year.

Astronomers estimate only about 1% will ever receive spectroscopic follow-up, the gold standard for classifying what an explosion actually is. That makes automated systems like LAISS not a convenience but a necessity. Without them, genuinely novel events will slip through unnoticed, filed as ordinary supernovae or simply ignored.

The retrospective results from the paper make this concrete. Running LAISS on historical ZTF data from 2018 to 2021, the team discovered 325 transients that had never appeared in any public catalog, roughly 1% of all ZTF reports to the Transient Name Server over that period. They also recovered about 50 previously identified rare transients that standard pipelines had missed or misclassified: superluminous supernovae, tidal disruption events, and several exotic subtypes driven by ejecta–circumstellar matter collisions.

The similarity search is particularly valuable for what comes next. When an entirely new type of transient appears, something that matches no known class, astronomers can still instantly retrieve the closest known objects and use them as a guide for follow-up strategy. It’s a data-driven way to bootstrap knowledge about the unknown.

Bottom Line: LAISS found 325 previously uncatalogued transients and ~50 rare events that automated pipelines had missed, and it runs every night in real time, already positioned to tackle the coming flood of data from the Rubin Observatory.