Server and Network Anomaly Detection

Your monitoring tools alert on thresholds: CPU above 90%, response time above 500ms, disk above 80%. But some problems look normal on any single metric and only stand out when you look at everything together, such as an unusual combination of request rate, wait time and bytes sent. Isolation forest scores each record by how unlike the rest it is, so you can review the most unusual rows first. This page shows a simulated example with the code and the printed output, then explains how to read the results.

Worked Example (Simulated): Flagging Unusual Log Rows

Question: Which made-up server log rows differ from the usual pattern?

Inputs: Simulated data, seed 417. The run has 120 rows and three fields: requests per minute, wait time in milliseconds, and bytes sent. Rows 7, 38 and 91 are set to odd values by hand; the other 117 rows are random draws around usual values.

Method: Isolation Forest, a model that spots records unlike most, with 200 trees and a contamination setting of 0.05. That setting tells the model to flag 5 percent of the rows, whatever the data look like.

import numpy as np
from sklearn.ensemble import IsolationForest

seed = 417
rng = np.random.default_rng(seed)
rows = 120
ids = np.arange(rows)
req_per_min = rng.normal(40, 5, rows).clip(20, None)
latency_ms = rng.normal(120, 18, rows).clip(60, None)
bytes_sent = rng.normal(900, 140, rows).clip(400, None)
# Simulated log records with deliberately unusual combinations.
req_per_min[[7, 38, 91]] = [110, 18, 105]
latency_ms[[7, 38, 91]] = [310, 360, 115]
bytes_sent[[7, 38, 91]] = [2400, 420, 2800]
X = np.column_stack([req_per_min, latency_ms, bytes_sent])
model = IsolationForest(n_estimators=200, contamination=0.05, random_state=seed)
labels = model.fit_predict(X)
scores = model.decision_function(X)
flagged = np.flatnonzero(labels == -1)
print(f"Simulated data, seed {seed}")
print(f"Rows: {rows}; input fields: req_per_min, latency_ms, bytes_sent")
print("Method: IsolationForest(n_estimators=200, contamination=0.05, "
      "random_state=417)")
print(f"Flagged row IDs ({len(flagged)}): " + ", ".join(map(str, flagged)))
print("Lowest-score flagged records (lower means more unusual):")
for i in sorted(flagged, key=lambda i: scores[i])[:5]:
    print(f"row={i}, req_per_min={req_per_min[i]:.1f}, "
          f"latency_ms={latency_ms[i]:.1f}, bytes_sent={bytes_sent[i]:.1f}, "
          f"score={scores[i]:.4f}")

Printed output:

Simulated data, seed 417
Rows: 120; input fields: req_per_min, latency_ms, bytes_sent
Method: IsolationForest(n_estimators=200, contamination=0.05, random_state=417)
Flagged row IDs (6): 7, 12, 30, 38, 63, 91
Lowest-score flagged records (lower means more unusual):
row=7, req_per_min=110.0, latency_ms=310.0, bytes_sent=2400.0, score=-0.2886
row=91, req_per_min=105.0, latency_ms=115.0, bytes_sent=2800.0, score=-0.2763
row=38, req_per_min=18.0, latency_ms=360.0, bytes_sent=420.0, score=-0.2709
row=30, req_per_min=41.3, latency_ms=81.6, bytes_sent=1232.4, score=-0.0359
row=12, req_per_min=27.4, latency_ms=113.6, bytes_sent=1232.4, score=-0.0275

Result: The model marked six rows as unusual, and six is set by the contamination setting of 0.05, which asks for 5 percent of 120 rows. Three rows (7, 38, 91) were made odd on purpose and got the lowest scores. The other three flagged rows were not planted; they are ordinary rows flagged to fill that quota. Rows 30 and 12 have scores near zero (-0.036 and -0.028), which are borderline. The scores alone do not show why a row stands out, so check flagged rows in the source logs.

Limit: The script adds odd values to made-up records. This result does not show that a record is an attack or that this method will find issues in real logs.

Sources

• Liu F, Ting K, Zhou Z (2008). Isolation Forest. 2008 Eighth IEEE International Conference on Data Mining. doi 10.1109/ICDM.2008.17. Cited for who published the method and when.

Why Review Unusual Rows?

Threshold-based alerting tells you when a single metric has already crossed a line. A score built from several fields at once can point you to rows worth a look before then, such as a server that is within limits on every metric but behaving unlike its peers. Isolation forest can flag records that differ from the data it sees. A flag is a reason to review a record. It does not show that the record is an attack.

To find the rows in your own data that do not fit the usual pattern, try the free anomaly detection tool.

What This Analysis Does

This analysis uses isolation forest, an unsupervised machine learning method for finding unusual records in multi-dimensional data (Liu, Ting and Zhou published it in 2008; see Sources). Unlike a single-metric threshold, it scores every record using all the fields you give it (packet size, protocol, source IP, response time, bytes transferred) at once. You still choose how many rows get flagged, through the contamination setting.

Here, the model gave the lowest scores to rows 7, 91, and 38. Check flagged rows in the source logs. A score alone does not show why a row stands out. In the example above, IsolationForest used 200 trees and marked six rows as unusual.

Scoring several fields together can point to cases that single-metric alerts do not express:

• Multi-dimensional anomalies: a packet that's normal in size, normal in protocol, but from an unusual source at an unusual time. Each metric is fine alone; the combination is suspicious.
• Records unlike the rest: signature-based tools look for known patterns. Isolation forest can flag records that differ from the data it sees, whether or not a rule exists for them. A flag is a reason to review a record, not a finding that it is an attack.
• Drift from peers: a server that is gradually moving away from its peers may show up as a low score before any single threshold is crossed.
• Misconfigured services: a load balancer sending traffic asymmetrically, a DNS server resolving to unexpected destinations, a firewall rule allowing traffic it shouldn't.

When to Use This Analysis

• Post-incident investigation: after an outage, upload the logs from the period before it. Review the flagged rows for early warning signs and ideas for better monitoring.
• Periodic log review: export a period of firewall or network traffic logs and run the analysis. Review the top-scored rows.
• Cloud cost anomalies: upload your cloud usage metrics and review the outlier resources or time periods.
• Supplementing existing monitoring: you may already have threshold alerts in a monitoring tool. This analysis looks at several fields together, which single-metric thresholds do not.

What Data Do You Need?

A CSV export from any system that produces timestamped event data with numeric and categorical fields.

Common sources

• Network traffic: Wireshark/tcpdump captures exported to CSV: Time, Source, Destination, Protocol, Length, Info
• Server access logs: Apache/Nginx access logs converted to CSV: timestamp, source_ip, request_path, status_code, response_time, bytes_sent
• Firewall logs: export from your firewall management console: timestamp, source_ip, destination_ip, port, protocol, action (allow/deny)
• Cloud service logs: AWS CloudTrail, Azure Activity Log, or GCP Audit Log exported as CSV
• Application event logs: any system exporting timestamped events with numeric measures

What makes a good dataset

• At least one numeric column: packet size, response time, byte count, request count, or any measurable quantity
• Categorical context: protocol, source IP, event type, status code. The algorithm encodes these automatically and uses them to find anomalous combinations.

A single column of response times is little more than a threshold with extra steps. Several columns (response time, request size, status code, source, and endpoint) give the method context to score combinations.

How to Read the Report

Anomaly score distribution: a histogram showing scores for every record. Normal records cluster at low scores; anomalies sit in the right tail. A clear gap between the lowest scores and the rest suggests distinct rows, as in the example above. A gradual tail with no clear gap means the boundary is fuzzy: review the top-scored items but don't treat the flag as absolute. Scores near zero are borderline.

Feature importance: ranks which dimensions contributed most to the anomaly scores. If "packet_length" dominates, your anomalies are primarily about unusual sizes. If multiple features contribute roughly equally, the anomalies are multi-dimensional. They look unusual across several metrics simultaneously, which often indicates more interesting patterns.

Top anomalies table: the action list. Each row shows the anomaly score and the actual feature values for the flagged record. Start with the highest-scored items and work down. The raw values help you see how a record differs, such as a very large packet, an error status from a source that normally gets successes, or a request to a destination no other client contacts. The score alone does not show why a row stands out.

Feature space scatter plot: shows anomalies highlighted against normal data in two dimensions. If the anomalies form a cluster, you may have a systematic pattern (a batch of bad requests, a compromised host, a misconfigured service). If they scatter, each anomaly may have a different cause.

Normal vs. anomaly comparison: side-by-side statistics for flagged vs. normal records on every dimension. This is where you can see which fields differ most between flagged and normal rows.

What to Do With the Results

Immediate

• Investigate top 10 anomalies: look up the flagged IPs, timestamps, and protocols in your existing logs. Are they expected (a backup job, a deployment) or genuinely suspicious?
• Cross-reference with known events: match anomaly timestamps to your incident log. Anomalies during a known deployment are expected; anomalies during a quiet period need investigation.
• Build new alert rules: patterns you confirm become candidates for new threshold alerts.

Strategic

• Run weekly or after incidents: make this part of your ops review cadence. Consistent scanning catches slow-moving patterns that a single analysis would miss.
• Pair with traffic analysis: run the traffic analysis module first to understand baseline patterns (protocol distribution, top sources, temporal patterns), then run anomaly detection to find what breaks from that baseline.
• Feed into SIEM rules, if you have Splunk or ELK, use the feature importance results to prioritize which log fields to index and which combinations to alert on.

When to Use Something Else

• Single-metric monitoring: If you only care about one number (response time, CPU, error rate), a simple threshold or Z-score approach is simpler and more interpretable. Isolation forest is meant for several fields at once.
• Real-time alerting: This analysis runs on a CSV export, not a live stream. For real-time anomaly detection, you need a streaming platform like Datadog, Grafana, or CloudWatch. Use this tool for periodic reviews and post-incident investigation.
• Known attack signatures: If you're looking for specific known vulnerabilities (SQL injection patterns, known malware signatures), signature-based IDS tools like Snort or Suricata are purpose-built. Isolation forest can flag records unlike the rest, with no signature. It's complementary, not a replacement.
• Time series anomaly detection: If your data is a single metric over time (like server response time by hour), a time series forecast will detect temporal anomalies better than isolation forest, which doesn't inherently understand time ordering.