An Introduction to Time Based Transformers for Anomaly Detection

In the previous post I laid out why satellite telemetry breaks the assumptions most anomaly detection tooling is built on: scale, multi-channel relational faults, irregular sampling, mixed channel types, and a noise floor that turns architectural sophistication into operator fatigue. I closed by naming transformers as a promising but unvalidated generalization strategy, and promised a follow-up comparing them against hand-tuned LSTM baselines across missions.

That comparison is coming, but it needs a foundation first. Before asking if a transformer generalizes better, it's worth being precise about what a transformer even is once you take away the thing it was invented for: text. Everyone's mental model of a transformer is shaped by GPT-style language models: tokens, a vocabulary, next-token prediction, a causal mask. Almost none of that survives the move to continuous, multivariate, irregularly-sampled telemetry. This post is about what does and doesn't carry over, using a small from-scratch implementation I built and trained on real ESA telemetry as a running example.

What a text transformer looks like

Strip a GPT-style transformer down to its essentials and you get: a fixed vocabulary of discrete tokens, an embedding table that looks each token up, a positional encoding added on top so the model knows where each token sits in the sequence, a stack of self-attention layers with a causal mask so token i can only see tokens before it, and a softmax head that predicts a probability distribution over the next token in the vocabulary. The training signal is cross-entropy: was the predicted next token correct.

Every one of those five pieces assumes something that satellite telemetry doesn't have.

What changes for telemetry

There is no vocabulary. A telemetry reading (temperature, voltage, speed) is a continuous value, not a symbol drawn from a finite set. You can't build a lookup table for every possible float. Instead of an embedding table, you need a projection: a linear layer that maps a vector of channel readings at one instant into the model's internal dimension. Every timestep's reading across all channels gets projected into the same embedding space a token embedding would occupy, but there's no lookup, no vocabulary, and no bound on the output space.

Position is a clock. A language model's positional encoding only needs to know that token 5 comes after token 4. Telemetry isn't evenly spaced: there are gaps, dropped packets. If you encode position as an integer index, a 10-second gap and a 10-minute gap in the data both just look like the next step, and the model has no way to tell them apart. The fix is to encode elapsed time instead of position: take the real timestamp deltas and feed those into the same sinusoidal encoding scheme transformers already use, so the frequency components correspond to actual seconds elapsed, not sequence slots. It's a small change to the formula and a large change to what the model can represent: irregular sampling stops being invisible to it.

The fixed formula above isn't the only option. It uses the same sinusoids as the original Transformer paper, with frequencies chosen by a fixed geometric formula and never touched by training. Time2Vec (Kazemi et al., 2019) takes the same periodic structure and makes the frequencies and phases learnable parameters instead, so the model can find which timescales actually matter in the data rather than having them fixed in advance, plus a separate linear term that can capture non-periodic progression. But for learning purposes, fixing the formula is the better way to understand how to codify the time.

The encoder doesn't need a causal mask. Autoregressive language models mask attention so a token can't see the future. That's what makes predicting the next token a well-posed training objective, and it's also what makes generation possible one token at a time at inference. Anomaly detection over a fixed telemetry window isn't trying to generate anything; it's trying to characterize a window that already fully exists. So the natural choice is a plain bidirectional encoder. Every timestep attends to every other timestep in the window, both past and future.

That bidirectionality carries a hidden assumption: it only makes sense because both training and evaluation here run over data that has already been fully collected. A window sliding over the train values treats the future relative to any point as just a later row already sitting in the same array. A genuinely real-time pipeline, scoring telemetry as it arrives, doesn't have that. The attention mechanism itself doesn't break (nothing forces a timestep to look forward, it's only permitted to), but whatever detection quality came from mixing in real future context isn't available for the newest point in a live stream. That leaves two honest options at deployment: accept a fixed detection latency, waiting for a full window's worth of future samples to arrive before scoring a point the way it was scored during evaluation, or run the model causally at inference, scoring each point on only what's arrived so far, which behaves like a causally-masked model in practice, just without a mask written into the architecture. Which of those is acceptable depends on the constraints of the problem.

The training objective becomes reconstruction. Without a vocabulary there's no softmax and no cross-entropy. What replaces the prediction the next token is usually reconstructing the input: run the window through the encoder, project back out to the original number of channels with a linear head, and minimize mean squared error against the real values. The model is being asked to compress and rebuild normal behavior, not to generate anything new. This reframes the whole point of training: a language model is trained to be a good generator, and this is trained to be a good summarizer of normal, with anomalies expected to show up as the summarization failing.

Detection becomes a side effect of failure, not a classifier's output. Because there's no label being predicted, flagging as anomalous isn't a direct model output but derived after the fact. The reconstruction error at each point becomes an anomaly score, and a threshold (commonly a percentile of the error distribution on held-out normal data) turns that score into a decision. This is elegant in that it requires no anomaly labels during training but it also means detection quality depends entirely on a hope that hasn't been guaranteed anywhere in the architecture: that the model reconstructs normal behavior well and anomalous behavior poorly. Nothing forces that gap to exist. More on that in a moment.

Other architectures being researched

The five changes above describe the minimum viable time-series transformer: an encoder-only reconstruction autoencoder. It's also close to the simplest thing you could build, and the research landscape has mostly moved past it.

Anomaly Transformer replaces plain reconstruction error with something called association discrepancy: the idea that normal points can form meaningful attention associations across the whole series, while anomalies, being rare and unrepresentative, can only attend locally. It scores anomalies by how much a point's attention pattern deviates from a learned prior, trained with a minimax scheme, rather than by how badly the point gets reconstructed.

TranAD uses a transformer encoder-decoder with a self-conditioning and adversarial training setup, explicitly optimized for low inference latency, a real constraint if you're running detection onboard or near-real-time rather than as a ground-based batch job.

MEMTO targets a specific failure mode in Anomaly Transformer: overgeneralization, where an encoder-decoder with enough capacity ends up reconstructing anomalous windows just as well as normal ones, because nothing in the objective stops it. MEMTO bolts a gated memory module onto a transformer encoder-decoder, storing a small set of normal-behavior prototypes that get updated incrementally during training rather than fixed in advance. Anomaly scoring then comes from a bi-dimensional deviation criterion, measured in both the input space and the latent space, instead of reconstruction error alone. It's the same instinct as association discrepancy in Anomaly Transformer, giving the model something other than raw reconstruction error to separate normal from anomalous, but implemented as an explicit memory of "normal" rather than a learned attention prior.

PatchTrAD is closer in spirit to the minimal architecture above, but reconstructs patches of consecutive timesteps rather than individual points. It's a useful reminder that even within "reconstruction-based transformer," the granularity of what gets reconstructed is a real design axis.

MTAD-GAT (Zhao et al., 2020) is worth naming even though it's not strictly a transformer: it's a graph attention network, with separate feature-wise and temporal attention modules, trained on a joint reconstruction-and-forecast objective. It gets grouped with attention-era methods in most surveys, and the "attend across channels, not just across time" idea it introduces is one that pure sequence transformers don't get for free.

On the aerospace side specifically, results are thinner than the text-transformer hype would suggest. The clearest results I found are in Fejjari et al. (2025), which benchmarked TranAD, Anomaly Transformer, and MEMTO against each other, and against supervised baselines (FCNN, XGBOD) and GAN-based ones (MO-GAAL, SO-GAAL), on ESA's OPS-SAT telemetry benchmark. Anomaly Transformer and MEMTO came out essentially tied for best (F1 ≈ 98.7% and 98.96%), well ahead of TranAD (F1 ≈ 79.8%, dragged down by poor recall on the two shortest, lowest-sample channels) and of every supervised baseline tested. That's the same OPS-SAT-AD release the experiment below uses, and it's scored per channel the same way — but the comparison still isn't apples-to-apples: different channel, different train/test split, F1 at a fixed operating point rather than ROC-AUC/PR-AUC, so it's a reason to expect these architectures do better here, not a benchmark for the numbers that follow.

The experiment: OPS-SAT as a working signal

Before pointing this architecture at something bigger, I wanted a small, well-understood benchmark to validate the five changes above actually work, rather than just sounding reasonable on paper. OPS-SAT-AD (Ruszczak et al., 2024), a single-channel anomaly detection release from ESA's OPS-SAT CubeSat mission, is that benchmark. It's already segmented and labeled channel by channel: each labeled unit is a short, self-contained stretch of one telemetry channel, 14 to 687 samples long, sampled every one to five seconds, with gaps between segments.

That structure is a good match for the elapsed-time positional encoding described above. Instead of chopping the data into fixed windows, each segment goes through the model whole: because the positional encoding runs on real elapsed seconds rather than array index, sequence length doesn't need to be fixed in advance. I trained a small version of the same model (two encoder layers) purely on 267 nominal segments, thresholded on held-out validation error, and scored 132 held-out segments (24% anomalous) by mean reconstruction error.

ROC-AUC came out to 0.78, PR-AUC to 0.59, comfortably above chance, and the reconstruction error genuinely separates the two classes without needing point-adjustment to make it look that way. The mechanism is visible directly in the reconstruction itself:

On the nominal segment (left), the reconstruction tracks the raw signal closely enough that the two lines are barely distinguishable. On the anomalous segment (right), two sharp spikes in the raw reading appear where the reconstruction doesn't follow. It was never trained to expect them, so it smooths toward what normal looks like instead. That gap between raw and reconstruction is the anomaly score, and here it's a real one: 0.0235, roughly 18× the detection threshold, against 0.0003 for the nominal segment.

That's a small, well-behaved benchmark by design: one channel, short self-contained segments, sharp spike-shaped anomalies. The next post in this series takes the same architecture to Mission 2 of the ESA Anomaly Detection Benchmark (Kotowski et al., 2024).

The open question from the last post, whether a transformer generalizes better across missions than a hand-tuned baseline like Telemanom-ESA, is still open after this one. OPS-SAT shows the mechanism is sound: a plain reconstruction transformer can produce a genuine anomaly signal from real telemetry. Whether that signal survives the move to a larger, multivariate mission is next. If you're working on telemetry or ground segment systems in the space sector, I'd like to hear from you.

The code for the models and experiments described here is on GitHub: higim/time-based-transformers-for-anomaly-detection.

References

1. Kazemi, S. M., et al. (2019). Time2Vec: Learning a Vector Representation of Time. arXiv:1907.05321
2. Xu, J., et al. (2022). Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy. ICLR 2022 (Spotlight). arXiv:2110.02642
3. Tuli, S., Casale, G., & Jennings, N. R. (2022). TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data. VLDB 2022. arXiv:2201.07284
4. Song, J., Kim, K., Oh, J., & Cho, S. (2023). MEMTO: Memory-Guided Transformer for Multivariate Time Series Anomaly Detection. NeurIPS 2023. OpenReview:UFW67uduJd
5. Zhao, H., et al. (2020). Multivariate Time-Series Anomaly Detection via Graph Attention Network. arXiv:2009.02040
6. Vilhes, S.-M., Gasso, G., & Alaya, M. Z. (2025). PatchTrAD: A Patch-Based Transformer Focusing on Patch-Wise Reconstruction Error for Time Series Anomaly Detection. arXiv:2504.08827
7. Fejjari, A., Delavault, A., Camilleri, R., & Valentino, G. (2025). Transformer-based anomaly detection for satellite telemetry data. Acta Astronautica 238, 739-745. doi/pii S0094576525006095
8. Kotowski, K., et al. (2024). European Space Agency Benchmark for Anomaly Detection in Satellite Telemetry. arXiv:2406.17826
9. Ruszczak, B., Kotowski, K., Evans, D., & Nalepa, J. (2024). The OPS-SAT benchmark for detecting anomalies in satellite telemetry. Scientific Data. arXiv:2407.04730