Building a network anomaly detector with Scapy and scikit-learn

One of my projects was a small anomaly detection system for network traffic: capture live packets, turn them into features a model can use, flag what looks unusual, and show it on a dashboard. Here’s how the pipeline fits together.

1. Packets into flows

Single packets don’t say much. Grouping them into flows (source, destination, ports, protocol) gives you something with shape. Scapy keeps the capture side short:

from collections import defaultdict
import time
from scapy.all import sniff, IP, TCP

flows = defaultdict(lambda: {"pkts": 0, "bytes": 0, "syn": 0,
                             "fin": 0, "rst": 0, "first": None, "last": None})

def on_packet(p):
    if IP not in p:
        return
    key = (p[IP].src, p[IP].dst,
           getattr(p, "sport", 0), getattr(p, "dport", 0), p[IP].proto)
    f = flows[key]
    now = time.time()
    f["first"] = f["first"] or now
    f["last"] = now
    f["pkts"] += 1
    f["bytes"] += len(p)
    if TCP in p:
        flags = int(p[TCP].flags)
        f["syn"] += bool(flags & 0x02)
        f["fin"] += bool(flags & 0x01)
        f["rst"] += bool(flags & 0x04)

sniff(prn=on_packet, store=False, timeout=60)

2. Features that mean something

From each flow I derived duration, packet and byte counts, mean packet size, packets per second, and the share of SYN, FIN and RST flags. Those cover a lot of common oddities: a scan shows up as many short flows heavy on SYNs, a flood as a huge packet rate, a broken connection as a burst of resets.

Before training, the data needed cleaning: dropping zero-duration flows (they break the rate features), capping extreme outliers, and making sure the same flow wasn’t counted twice across capture windows.

3. The model

A tree-based classifier from scikit-learn worked well here: it copes with features on very different scales, and it’s easy to see which features drive a decision.

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)

clf = RandomForestClassifier(n_estimators=200, class_weight="balanced",
                             random_state=42)
clf.fit(X_train, y_train)
print(classification_report(y_test, clf.predict(X_test)))

4. Keeping false positives down

Anomalies are rare, so a model can look accurate while being useless, and a detector that cries wolf gets ignored. Two things helped:

• Look at precision and recall, not accuracy. Class weights make the model care about the rare class.
• Tune the threshold. Instead of the default 0.5, pick the probability cut-off from the precision-recall curve that gives an alert rate people will actually read.

5. Making it visible

The last step was a live dashboard with logging, so flagged flows show up alongside the features that triggered them. A detector nobody looks at doesn’t detect anything.

What I’d do differently

Start with a simple rule-based baseline (thresholds on packet rate and flag ratios) before reaching for a model. It gives you something to beat and a quick way to label data.