Introduction #
A vulnerability lands with a name, a logo and a CVSS of 10. Within an hour someone senior asks whether you are affected.
The honest answer, almost always, is that you do not know yet — and the work of finding out is the expensive part. One federal cabinet department told the US Cyber Safety Review Board it had spent roughly 33,000 hours on its Log4j response.1 The Board does not split the hours. But its account of why the response cost so much begins with organisations that did not understand their own use of Log4j.
Lenny Zeltser’s Vulnerability Investigation Brief is the right container for that work, and I would not try to improve on it.2 What follows is about what goes in the evidence box, and — the part nobody writes down — what each piece of evidence fails to prove.
Your scanner will not answer today #
The instinct is to run a scan. The NCSC is refreshingly blunt about why that fails at the start of an event: “It may take a few hours or more for your scanner or provider to include detections.”3 By the time your tooling has a plugin, you have already been asked twice.
Worse, some vulnerabilities are structurally invisible to scanning. The Board described Log4j as “routinely embedded in other software components, often unknown at later levels of integration or system operation, making it a difficult vulnerability to identify using common scanning approaches”.
And then there is the finding from that report that gets quoted less. The Board spoke to representative groups for organisations using software bills of materials, and reported that none “reported having leveraged them to identify vulnerable deployments of Log4j”. The artefact that exists precisely to answer “are we affected” went unused, as far as the Board could find, in the event most often cited to justify it.
Cheap evidence beats complete evidence #
The NCSC’s guidance contains the most useful surge trick I have read, precisely because it does not involve security tooling at all. To work out whether you even run the affected product, search your mail gateway for correspondence from the vendor’s domain, grep your perimeter proxy logs for connections to the vendor’s site, and check your own disclosure inbox.
None of that is authoritative. All of it is available immediately, and it is often enough to move you from “no idea” to “almost certainly not us” while the scanner vendor is still writing a plugin.
What each answer does not prove #
Every claim you make during a surge will be repeated by someone else, to a customer or a regulator, several weeks later. So it is worth being precise about what your evidence actually establishes.
| The claim | Cheapest sufficient evidence | What it does not prove |
|---|---|---|
| We do not run it | Procurement records, mail-gateway and proxy-log search, asset inventory | Embedded or vendored copies inside other products — the Log4j failure mode |
| We run it, but not the vulnerable configuration | Configuration check against a sampled set of hosts | That configuration has not drifted, or that non-default deployments exist |
| We patched it | Package or build version on the host | That the running process was restarted — the patched code may not be loaded4 |
| We rebuilt the image | New image digest in the registry | That anything is running it; pull policy and rollout state decide that5 |
| We mitigated it | The WAF rule or config change is in place | Very little, and CISA says so plainly (below) |
That last row deserves its own sentence, because mitigation is where surge reporting is least honest. CISA’s own Log4j guidance says it: “Deploying a WAF is an important, but incomplete, solution. While threat actors will be able to bypass this mitigation, the reduction in alerting will allow an agency SOC to focus on a smaller set of alerts.”6 A mitigation buys attention, not immunity. Report it as what it is.
The normal change, run faster #
NIST’s guidance on configuration management accepts that unscheduled changes get reviewed after the fact rather than before, and asks that they be handled under procedures written in advance and reviewed as soon as is practical.7 The NCSC is more concrete still: route the emergency through the P1 IT incident process you already have, “often with retroactive change requests”.
The useful move isn’t to design a war room. It is to decide, before the event, who is allowed to authorise skipping a control, and to write that name down. During the event is the wrong time to discover that nobody knows.
The CSRB makes the same point about the fix itself: deploying the fixes “was itself a risk decision, forcing a tradeoff between possible operational disruption and timeliness, completeness, and compensating controls”.
Choose the compensating control and it has a tail: the WAF rule added at 2am, the feature disabled, the port blocked — these outlive the emergency unless somebody scheduled their removal.
NIST’s patch management guidance puts it in one line: “Schedules should be set and enforced for both patch deployment and mitigation removal.”8 The second half of that sentence is the one programmes skip, and it is how a compensating control quietly becomes permanent architecture that nobody remembers choosing.
How you will know it is over #
Almost every article about high-profile vulnerabilities covers the first hour. Very few cover the last one.
CISA’s Log4j emergency directive is a good model, because it published its own termination condition up front — it remained in effect until CISA determined that all agencies had performed the required actions — and was then formally closed, with the residual work handed to the KEV directive then in force.9 Its reporting template is worth stealing outright for its vocabulary: assets are recorded as, for example, updated, mitigated, or removed from the agency network. Three example states, and no ambiguity about which one you are claiming.
A stand-down is a decision with a name on it, not a gradual loss of interest.
The unsatisfying conclusion #
The Board’s own retrospective judgement on Log4j is worth quoting at length, because half of it gets quoted constantly and the other half almost never. Exploitation “occurred at lower levels than many experts predicted”, and the Board was “not aware of any significant Log4j-based attacks on critical infrastructure systems” — and, in the same paragraph, the admission that “no authoritative source exists to understand exploitation trends”. Two sentences later: “Most importantly, however, the Log4j event is not over.”
So the largest vulnerability event in recent memory may have been less damaging than feared, and we can’t actually prove it either way. That isn’t a reason to respond less seriously. It is a reason to write down what you checked, what it proved, and what it did not — because in the absence of authoritative exploitation data, your own evidence is the only thing that will still be legible in six months.
-
Cyber Safety Review Board, Review of the December 2021 Log4j Event, July 2022. https://www.cisa.gov/sites/default/files/publications/CSRB-Report-on-Log4-July-11-2022_508.pdf ↩︎
-
Lenny Zeltser, “Handling High-Profile Vulnerabilities.” https://zeltser.com/high-profile-vulnerabilities ↩︎
-
NCSC (UK), “Responding to active exploitation of vulnerabilities.” https://www.ncsc.gov.uk/collection/vulnerability-management/guidance/responding-to-active-exploitation ↩︎
-
needrestart(1) manual page, Debian. https://manpages.debian.org/testing/needrestart/needrestart.1.en.html ↩︎
-
Kubernetes documentation, “Images.” https://kubernetes.io/docs/concepts/containers/images/ ↩︎
-
CISA, Log4j (CVE-2021-44228) Vulnerability Guidance. https://github.com/cisagov/log4j-affected-db ↩︎
-
NIST, SP 800-128: Guide for Security-Focused Configuration Management of Information Systems, §3.3.2. https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-128.pdf ↩︎
-
NIST, SP 800-40 Rev. 4: Guide to Enterprise Patch Management Planning, §3.5.3. https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-40r4.pdf ↩︎
-
CISA, Emergency Directive 22-02: Mitigate Apache Log4j Vulnerability (Closed). https://www.cisa.gov/news-events/directives/ed-22-02-mitigate-apache-log4j-vulnerability-closed ↩︎