Security alert triage: who reads it at 3 a.m.?

Sophos analysed 661 incident response cases across 70 countries and found a pattern that has nothing to do with technology: 88% of ransomware payloads are deployed outside business hours, and 79% of data exfiltration actions likewise. Attackers do not wait for the night out of superstition. They wait for the moment when security alert triage stops, because nobody is watching any more.

In a company with no security operations centre, the problem is rarely that it sees nothing. The signals are already there: sshd, nginx, the firewall, an antivirus sending emails. The problem is triage — who separates, and how fast, the three things that matter from the three hundred that do not. In many companies, the honest answer to "how fast" is "Monday morning".

Why security alert triage is the bottleneck, not detection

Why you have the alerts in the first place is a separate discussion, covered in our article on internet-exposed servers. Here we start after that point: the alert already exists, it is sitting in the queue, and somebody has to decide what to do with it.

How big the pile is depends on what you call an "alert". Vectra AI, in a study of 1,450 practitioners, reports an average of 2,992 alerts a day — counted raw, across the entire tool estate; 69% of respondents run more than ten detection and response tools, 39% more than twenty. Prophet Security, in a study of 250 respondents, reports a median volume of around 100 alerts a day — but there the count is what actually reaches an analyst's queue, after filtering. The thirtyfold difference is not a measurement error, it is a different definition of the word "alert". Both are studies paid for by vendors selling the solution to the problem being measured; that is precisely why we quote both.

What is better established is how much of the pile is empty. Omdia research commissioned by Microsoft — 300 SOC practitioners in organisations of more than 750 employees — found 46% false positives and analysts switching between 10.9 consoles on average, while 66% of teams lose a fifth of their week just aggregating data. The SANS Detection & Response Survey 2025 puts false positives first among detection problems for 73% of teams, and the "very frequent" category climbed from 13% to 20% in a single year.

Every alert costs human time, and not a little of it. Prophet measures a median investigation time of roughly 45 minutes (mean around 75), with 64% of teams above 30 minutes, plus another 23 minutes median spent simply waiting in the queue before anyone looks. Multiply that by your real volume and you get the number of people you would need for nothing to slip through. Almost no mid-sized company has that number.

The consequence is measured by everyone, with results that do not line up: 42% of alerts "uninvestigated" at Omdia/Microsoft, 28% on average (22% median) at Prophet, 63% "unaddressed" at Vectra. "Nobody looked" and "nothing was done" are not the same thing, and the populations studied differ. The order of magnitude holds regardless of whom you believe: somewhere between a quarter and two thirds of alerts go nowhere. Prophet adds the uncomfortable detail — 60% of organisations have had ignored alerts that later turned out to be real incidents.

• The serious alert arrives at 2:40 on a Saturday and is read at 9:15 on Monday, on the same screen as three hundred other routine notifications.
• Nobody can say quickly whether the alert is relevant here: a real campaign against a product the company does not run looks, in a list, exactly like one that matters.
• The person who could decide is the same one who looks after the printers, the backups and the migration away from the mail provider.
• After the third false alarm from the same source, the rule is disabled "temporarily" — and stays disabled.
• When something does happen, there is no record anywhere of the decisions: who looked, what they saw, on what basis they closed it.

In Romania, a simple staffing arithmetic sits on top of all this. A report by DNSC, the national cybersecurity authority, shows the country has under 3% ICT specialists in its employed workforce against a European average of 5%, with the shortage of qualified staff and insufficient funding as the main obstacles in most of the sectors assessed. ISC2 finds 33% of organisations without the resources to staff their security team adequately and 29% unable to afford people with the skills they need. The same DNSC report also shows that almost 70% of the organisations assessed rate their own security as "good" or "very good" — confidence does not follow the same curve as headcount.

The two clocks: how long it really takes before you find out

The question "how long before you find out" gets two answers that look like they contradict each other. IBM, in the Cost of a Data Breach Report 2026 (Ponemon research, 602 organisations, breaches occurring between March 2025 and February 2026), measures 247 days on average to identify and contain a breach — up from 241 in the previous edition, reversing several years of improvement. Sophos, across its 661 incident response cases, measures a median dwell time of 3 days.

They do not contradict each other. They measure opposite ends of the same distribution: in the cases that reach a response team the median is days, because there the intrusion was detected; in the statistics of reported breaches it is months. The distance between those two populations is exactly what you are buying when you buy triage.

The useful window, though, is far smaller than either figure. Sophos measures a median of 3.4 hours before the attacker reaches the Active Directory server. CrowdStrike describes an intrusion in which data exfiltration began 4 minutes after initial access. At 3.4 hours, an alert that waits 23 minutes in the queue and then needs 45 minutes of investigation can still arrive in time. At 4 minutes, it cannot.

The distance between "it happened" and "somebody decided it was worth reporting" shows up most clearly in national data. Romania's DNSC sensors collected over 25 million cybersecurity events in 2025, of which around 5.88 million brute-force attempts. In the same year, 47 brute-force incidents were reported to DNSC. The two figures are not comparable — one is sensor telemetry, the other is incident reporting — and that is exactly why they are useful together: triage sits between them. And the reporting obligation does not wait for it: under NIS2 the clock starts with an initial warning at 24 hours, set out in detail in our reporting readiness guide. You cannot report within 24 hours something you find out about three weeks later.

Detection is cheap and can run on its own, around the clock. Judgement is expensive and sleeps at night. That is where the bottleneck is, not at the sensors.

What a model can and cannot do in triage

This is where AI comes in, and also where the exaggeration starts. The market data is unusually clear about where the gain sits. Prophet reports that 72% of those using AI in triage see investigation time fall by more than 25%, with an average around a third, and 18% report more than 50%. The SANS Institute, in a survey of 536 practitioners sponsored by a consortium of mutually competing vendors, shows AI adoption in security climbing from 50% to 78% in a year — but also 63% of practitioners reporting significant gaps when AI detects or responds to threats, up from 45%. Only 27% describe their implementation as "mature".

The two results do not cancel each other out. "It saves me time" and "it sometimes misleads me" can both be true for the same person in the same week. The failures reported to SANS cluster around false positives, difficulty in identifying novel threats, and output phrased with confidence but wrong. That is an accurate description of what a language model is: good at explaining and prioritising, bad at guaranteeing. When the same respondents list the most effective controls against AI-enabled threats, the top places go to behavioural detection (45%), awareness training (45%) and review by a human analyst (39%).

Practice confirms the caution. Torq asked 450 CISOs and SOC leaders: 97% are convinced AI can do triage, but only 35% actually use it for that, and 9 out of 10 want to see how the model reaches a decision before extending its autonomy — 46% put transparency and explainability first among the factors that would raise their trust. In Prophet's distribution, 57% require human review before an alert is closed, 44% let the model recommend and a human execute, 30% allow automatic execution only for low-risk actions — and 0% grant full, unsupervised autonomy. Zero, in a survey run by a vendor with every interest in a different number.

There is another, less frequently discussed reason why a model has no business holding execution rights on a production server. Prompt injection is risk number one in the OWASP Top 10 for LLM Applications, 2025 edition, its second consecutive year in first place, and the cause is structural: the model receives instructions and data over the same channel, with no separation. Now consider what a log line actually is: text written by the attacker, in a field the attacker controls. If that text reaches a model that can execute commands, you have not built an automated analyst — you have built the LLM01 scenario itself. The mitigations on the list are boring and effective: explicitly segregating external content, restricting privileges, validating input, and keeping a human in the loop for sensitive operations.

What triage that holds up at 3 a.m. looks like

Nothing that follows assumes a SIEM or a team on shifts. It only assumes that the order of operations is right.

1. 1Write down what "serious" means before you have an incident, by category: successful access versus attempt, privileged account versus application account, exposed service versus internal service. Without that threshold, everything after it is opinion.
2. 2Cut the noise before the queue, not in the queue. An alert filtered out by a human has already cost human time; one rejected by a deterministic rule cost nothing.
3. 3Explicitly rule out what cannot touch you. A real, persistent campaign against a product you do not run stays background noise — and it is the category easiest to automate correctly.
4. 4Keep the blocking decision deterministic: thresholds, lists, rules you can read, test and explain to an auditor. The model explains and prioritises; the rule decides.
5. 5Put in place a channel that reaches a human outside working hours, not a dashboard opened on Monday morning. 88% of ransomware payloads are deployed, according to Sophos, outside business hours.
6. 6Prioritise fixes by real exposure, not by raw score: a CVSS 6.5 on the CISA list of actively exploited vulnerabilities matters more than a 9.8 nobody is exploiting. Recurring vulnerability scanning keeps the list current without depending on anyone's memory.
7. 7Keep what was decided, not just what happened. A record of triage decisions is the only thing that makes reporting within 24 hours possible, and the only thing that gets you out of the "who closed that alert" conversation.
8. 8If you have nobody to put on shift, outsource the judgement, not just the tooling — a cybersecurity partner with a written and rehearsed incident response plan.

What we built — and what it still does not do

Sentinel is our answer to exactly the problem above, and the most honest way to describe it is through what it does not do. Detection, scoring and the decision run deterministically, around the clock, at zero cost. The model is called only for incidents that pass the severity threshold: it reads what happened and says in plain language what is real, what is noise and what you need to do, within two minutes of the incident opening, then drafts the remediation procedure. It installs on the client's own infrastructure and is handed over with documentation. A single machine or a fleet, no SIEM, no external agent, no data leaving the server.

The model's four limits are public, because they are, in fact, the architecture. It does not decide blocks. It executes nothing — the plan it writes goes through a deterministic validator that refuses it if it is not safe; the model drafts, the validator decides. It is not on the critical path: if the API is down or the budget is exhausted, detection, blocking and alerting carry on, marked "AI analysis unavailable". And it does not exceed the budget — the cap is checked before every call.

To the OWASP problem above, the answer is as boring as it needs to be: log lines and HTTP paths are treated as untrusted data, with explicit delimiters, and the transport that reads files from the server runs in plan mode — structurally incapable of changing anything. There is a test that plants IGNORE PREVIOUS INSTRUCTIONS in an access log and verifies that nothing was executed. On the blocking side, the list of protected addresses is hard-coded in the code, not in configuration and not in the database; the base rule is policy accept, that is, a deny-lister rather than a firewall; and a PANIC file clears every block in under 60 seconds. Auto-block ships off: for the first 72 hours you only see what it would have blocked.

The figures that follow come from a single server of ours, over 24 hours — a production install on AlmaLinux 9 with 14 monitored applications. They are not a sample and they are not "typical": 1,055 unique attackers, 2,765 hostile events out of 7,031 in total, 554 incidents opened of which 13 serious, 5 addresses blocked. Noise is rejected before the database — around 96,000 engine diagnostics a day never get there — and the model is called only for the incidents that pass the threshold, typically a few dozen a day. A concrete triage example from the same dashboard: "active campaign against GLPI, 560 probes". Real, persistent, targeted. The server does not run GLPI, so it is background noise — and that needs saying plainly, not left to occupy first place on a dashboard.

What it does not do yet is equally public: container logs are not collected, there is no dedicated file integrity monitor beyond auditd, and the Debian install is covered by tests but not yet verified on a real Debian host. Licensing is per server, and the initial configuration plus the 72-hour calibration are included in every tier; the architecture, the limits and the platforms covered are described on the product page.

Conclusion

Detection has become cheap to the point of banality: any modern server already generates more signals than anyone can read. What has not become cheap is judgement — who looks, how fast, and on what basis they decide that something matters. The difference between a company that finds out in three days and one that finds out months later is not in the tooling, it is in the triage, and triage can be designed rather than merely hoped for. AI helps with the explaining and the prioritising, where it is measurably good; the blocking decision stays where it can be read and tested. If you would like us to look together at what your servers are generating right now and how much of it actually reaches a human, let us talk for 30 minutes.

Sources

• Sophos — "Active Adversary Report 2026", 661 IR and MDR cases, November 2024 – October 2025 (sophos.com)
• Microsoft Security Blog — "Unify now or pay later", Omdia research commissioned by Microsoft, N=300 SOC practitioners, February 2026 (microsoft.com/security/blog)
• SANS Institute — Detection & Response Survey 2025 (reported by stamus-networks.com) and "2026 SANS AI Survey", N=536, July 2026 (sans.org)
• Prophet Security / ViB — "State of AI in the SOC 2026", N=250, August 2026 (prophetsecurity.ai) — "AI SOC" vendor study
• Vectra AI — "State of Threat Detection and Response 2026", N=1,450 (vectra.ai) — vendor study
• Torq — "2026 AI SOC Leadership Report", fieldwork by Sapio Research, 450 CISOs and SOC leaders, March 2026 (torq.io) — vendor study
• IBM — "Cost of a Data Breach Report 2026", Ponemon Institute research, 602 organisations, July 2026 (ibm.com/reports)
• CrowdStrike — "2026 Global Threat Report", own telemetry, February 2026 (crowdstrike.com)
• DNSC, Romania's national cybersecurity directorate — 2025 annual activity report, approved by CSAT Decision no. 117 of 7 August 2026 (reported by agerpres.ro and business24.ro); DNSC report on the shortage of specialists (reported by europafm.ro)
• ISC2 — Cybersecurity Workforce Study 2025, December 2025 (isc2.org)
• OWASP GenAI Security Project — Top 10 for Large Language Model Applications 2025, LLM01 Prompt Injection (owasp.org)
• CryptoITData — Sentinel, product documentation and data from a single production install (cryptoitdata.eu/sentinel)