The CrowdStrike outage: designing processes for the day something fails

On Friday 19 July, at 04:09 UTC, CrowdStrike pushed out a configuration update for its Falcon sensor on Windows. It contained a logic error. By 05:27 UTC it had been reverted, but in those 78 minutes millions of machines fell into a blue-screen loop. The company itself explained it in detail and stressed that it was not a cyberattack.

Microsoft estimated that the fault affected 8.5 million Windows devices, less than 1% of the total. That was enough to bring airlines, banks, hospitals and television networks to a halt. The BBC described it as possibly the worst IT incident in history.

What happened in Spain

In Spain, Aena activated its contingency systems and, according to its statement that same day, the airports remained operational at all times, with some processes handled manually. There were queues, delays and cancellations, but the system did not stop.

That phrase, “some processes handled manually”, is the one that interests me most. Behind it are people who knew how to check passengers in or board them without the usual system, written procedures, and someone who had the paper forms to hand.

The automation paradox

I build automations and bespoke tools for a living, and I believe in them. But there is a paradox worth keeping in mind: the better an automated process works, the more people forget how to do it by hand. After two years, nobody on the team remembers which steps the system performed, in what order or with what exceptions.

With AI, that paradox becomes sharper, because much of what it automates is judgement: classifying, prioritising, drafting. If the service goes down, “going back to Excel” is not enough; you need to know what decisions the tool was making.

Four design habits

What I try to apply on every project:

1. Document the process in human language, not just in code. One page that anyone on the team can understand: what goes in, what comes out, what the exceptions are.
2. Define the degraded mode. What do we do if the tool is unavailable for an hour? For a day? The answer doesn’t have to be perfect, but it has to exist.
3. Avoid single points of failure where the cost justifies it. Not everything needs redundancy, but whatever invoices or serves customers probably does.
4. Rehearse. Once a year, switch the automation off for a morning and check that the team can carry on. It’s uncomfortable, and it’s very revealing.

CrowdStrike was not an AI failure, but it was a reminder of how dependent we are on software we don’t control. The question is not whether something will fail, but what your team will do when it does.