When the labels come from an LLM
Research notes on WAF alert triage when the analyst verdicts in a security operations center are produced by language models. Work in progress, October 2026.
The problem
A web application firewall (WAF) sends a large number of alerts to a security operations center (SOC). Many of these alerts are false positives: legitimate requests that match an attack rule. Analysts cannot examine every alert, so real attacks can stay hidden in the queue.
A model can sort the queue, but a supervised model needs reliable labels. The usual source of labels is the analysts' own verdicts: this alert was an attack, that one was not. My thesis started from that assumption. The assumption did not hold.
What I observed
In the production SOC that I studied, analysts now give much of the alert investigation to LLMs. The verdict in the ticket is often the output of a model, accepted by a person. This has three consequences for anyone who wants to learn from those verdicts.
- The labels are not independent. A triage model trained on recent verdicts learns to imitate the LLM, not to recognize the attack.
- The evaluation is not independent. If you measure a new model against the same verdicts, you measure agreement with the LLM, not correctness.
- The clean history is incomplete. Older analyst resolutions, from before LLM use, are probably reliable. But the alert events for that period are not retained, so the verdicts cannot be joined to the data.
This is not a criticism of the analysts. Alert volume is high and the tools are useful. The problem is structural: as AI systems take part in security work, the ground truth that we use to train and evaluate the next system is produced, in part, by the previous one.
The approach: weak supervision
If no single source of labels is reliable, one option is to combine several imperfect sources and model their agreement and their errors. This is weak supervision, in the line of data programming and Snorkel. In my thesis the sources include:
- the WAF rule engine itself (OWASP Core Rule Set), at different paranoia levels;
- an independent scan of the request content;
- an LLM, as one labeling source among several, not as the label.
The method only works if the sources are reasonably independent. Vendor fields that are calculated from each other, for example a severity rating derived from the same rule matches, must not count as separate votes. Otherwise the combination gives the same error more weight, not less.
To measure the method honestly, I evaluate it on public labeled datasets of web attacks, where the ground truth does not come from an LLM. Production data is used only offline, and never in public outputs.
The lab
The demonstrator is a small, realistic stack: Nginx with ModSecurity and the OWASP Core Rule Set in front of a test web application, a triage model after the WAF, and a Wazuh SIEM that collects all events. Two dashboards read the same index: one sorted by the WAF's own signals, one sorted by the model. Labeled attack traffic and normal traffic are replayed against the stack.
The lab already gave three design lessons.
- A blocking WAF hides its own false positives. If the WAF blocks a request inline, a model after it never sees that request. It can only find missed attacks. To study false positives, the WAF must run in detection mode, and a separate proxy must act on the model's decision: release, block, or quarantine. On a timeout, the proxy must not release the request.
- The replay can create its own alerts. Rewritten Host headers, a single proxy source address, and replay timestamps can trigger rules that the original traffic would not trigger. These alerts are lab artifacts, and they must be separated before false-positive rates are calculated.
- Some false positives repeat. For example, a higher-paranoia command-injection rule matches the User-Agent of a common SSL monitoring tool. The paranoia level is an experimental variable in its own right: the same traffic gives different false-positive rates at different levels.
A dataset lesson
Popular intrusion-detection datasets in CSV form contain network flow statistics, not HTTP request content. A WAF examines request content, so these files alone cannot feed it. Engelen and colleagues showed that a widely used benchmark also contains labeling and flow-extraction errors that change the measured performance of detectors. The same lesson applies here as in production: when the labels are wrong, the evaluation is wrong.
Open questions
- Can we detect, from the data alone, which historical verdicts were produced by an LLM?
- How much does a triage model trained on LLM verdicts inherit the LLM's blind spots, and how can we measure that without clean labels?
- When an LLM is one labeling source among several, how should its weight change as attackers learn to evade it?
- Where must a person stay in the decision, so that the system keeps an independent signal at all?
Status
This is my engineering thesis at Universidad Católica “Nuestra Señora de la Asunción” in Paraguay, with a planned defense in March 2027. With my advisors, I plan to submit a paper on this work to CLEI. Numerical results will follow when the experiments are complete. I will not publish results that I have not measured.