Network traffic is not text. It has no sentences, no grammar, no vocabulary in any human sense. And yet it has all of these things — just written in a protocol language that a transformer can learn to read, if you show it enough examples.
Look at a packet capture and ask yourself: is this sequential? Yes. Each packet follows the previous one in time. Does it have repeating structural patterns? Yes — protocol behaviors, handshake sequences, request-response pairs, keepalives. Are there statistical regularities between elements? Absolutely — a SYN is almost always followed by a SYN-ACK, which is followed by an ACK. An HTTP request has a highly predictable structure. A DNS query has an expected response shape.
In other words: network traffic has grammar. Not the grammar of English or French. The grammar of TCP/IP. And a transformer doesn't care which grammar it's learning.
Normal network behavior is predictable. A model trained on enough normal traffic learns what sequences of events are grammatically valid — just as a language model learns what word sequences are grammatically valid. Anomalous traffic is traffic that violates the learned grammar. It produces high prediction error — a low-probability token in context — and that signal is the detection.
The first design decision is: what is the atom? What becomes a token?
There are several valid approaches, each with different tradeoffs:
Each packet header becomes one token. You encode the relevant fields — source IP class, destination port, protocol, flags, payload length, inter-arrival time — into a fixed-size vector. That vector is the token. A sequence of packets becomes a sequence of tokens. The model learns: given these packets, what does the next packet look like?
Instead of individual packets, you aggregate packets into flows — all packets sharing the same five-tuple (source IP, destination IP, source port, destination port, protocol). Each flow becomes one token, encoded with aggregate statistics: packet count, byte count, duration, flag distribution, average inter-arrival time. A session becomes a sequence of flow tokens.
Higher still — security events, log entries, or network observations become tokens. A failed login attempt is a token. A DNS query to an external resolver is a token. A new outbound connection on port 443 is a token. The sequence is a timeline of what happened on the network, and the model learns the grammar of normal operational behavior.
This is not hypothetical. NetBERT, Whisper-Net, and several commercial NDR (Network Detection and Response) platforms use transformer-based models trained on packet and flow sequences. The vocabulary sizes are small compared to language models — a few thousand distinct flow types instead of 50,000 words — but the architecture is essentially identical.
During training on months of normal network captures, the model learns things like:
None of these rules were written explicitly. The model extracted them from statistical patterns, the same way a language model extracts grammar from text.
During inference, the model scores live traffic. For each incoming event, it predicts what the next event should look like — and computes how surprised it is by what actually happened. A low-probability prediction is a signal that something unusual occurred.
A workstation initiating an outbound connection on port 4444 at 3 AM — when it normally only talks to a domain controller and a file server — would produce high prediction error. Not because anyone wrote a rule saying "alert on port 4444." But because this event is statistically inconsistent with everything the model learned about this device's normal behavior.
This is the fundamental advantage over rule-based detection: the model can flag things no one thought to write a rule for. It detects deviation from normal, not just matches against known-bad signatures.
"The cat sat on the mat" — grammatically valid, high probability.
"The cat sat on the Tuesday" — grammatically suspect, low probability.
"Workstation → DNS query → HTTPS to CDN" — behaviorally normal, high probability.
"Workstation → DNS query → raw TCP to foreign IP on port 9001" — behaviorally anomalous, low probability.
Same detection mechanism. Different domain.
The attention mechanism adds something rule-based systems can't easily replicate: the ability to relate events that are far apart in time.
A lateral movement attack might begin with a credential phishing event, followed hours later by a quiet internal reconnaissance sweep, followed days later by a slow exfiltration over HTTPS to a cloud storage endpoint. Each event in isolation looks benign. The sequence, across a long time window, is the signal.
Attention can directly connect the exfiltration event to the credential event from two days earlier — assigning that early event high attention weight when evaluating the later one — because the model learned that this sequence pattern has statistical significance. A rule-based system would need someone to explicitly write: "if event A then watch for event B within N days." The transformer discovers these temporal correlations from data.
One genuine challenge in network language models is vocabulary construction. In text, words are well-defined and relatively stable. In network traffic, the "vocabulary" — the set of distinct token types — must be carefully designed.
Raw IP addresses make poor tokens: there are too many of them, they change meaning constantly, and the specific address matters less than its behavioral role (internal server, external CDN, known-bad IP). Good network tokenization usually involves abstraction — encoding IP addresses as roles or subnet classes rather than literal values, normalizing port numbers into service categories, treating payload sizes as bucketed ranges rather than exact values.
This vocabulary design is the domain expertise that distinguishes a useful model from a useless one. The architecture is generic. The tokenization is where network knowledge matters.