On Friday 19 July 2024, the UK awoke to news of a fast-spreading IT outage, seemingly global in its nature, affecting hundreds – if not thousands – of organisations.
The disruption began in the early hours of Friday morning in Australia, before spreading quickly across Asia, Europe and the Americas, with the travel industry among the most widely affected.
The outage was quickly tracked to cyber security firm CrowdStrike, which is already engaged in incident response amid the chaos. Keep on top of this developing incident over the coming days and weeks with our Essential Guide.
What does CrowdStrike do?
CrowdStrike is one of the world’s most prominent cyber security companies, with thousands of customers all over the world. Based in Texas, it employs more than 8,000 people and books about $3bn in revenues per annum. It has been around since 2011.
The organisation bills itself thus: “CrowdStrike has redefined security with the world’s most advanced cloud-native platform that protects and enables the people, processes and technologies that drive modern enterprise. CrowdStrike secures the most critical areas of risk – endpoints and cloud workloads, identity, and data – to keep customers ahead of today’s adversaries and stop breaches.”
CrowdStrike will be unfamiliar to most people not steeped in the technology industry, although Formula 1 fans will be aware of it thanks to its headline sponsorship of the Mercedes AMG Petronas team – its branding appears on the halo safety device and is clearly seen on onboard footage from Lewis Hamilton’s car.
Security practitioners will know CrowdStrike from its frequent contributions to major incident investigations, including the Sony Pictures hack, the WannaCry crisis, and the 2016 hack of the Democratic National Committee by Russia.
What happened during the CrowdStrike outage?
The disruption at first manifested in the form of the infamous blue screen of death – which signals a fatal system error – on Windows PCs.
Given the disruption appeared to be a Microsoft problem to begin with, it was Redmond that first responded, confirming just before 8am BST that it was investigating problems affecting cloud services in the US.
It quickly became apparent that the issue was not down to Microsoft itself, but rather a faulty channel file rolled out to CrowdStrike’s Falcon sensor product.
Falcon is a solution designed to prevent cyber attacks by unifying next-gen antivirus, endpoint detection and response (EDR), threat intelligence and threat hunting, and security hygiene. This is all managed and delivered through a lightweight, cloud-delivered and -managed sensor, which seems to be whence the issue arose.
The botched roll-out effectively caused what is known as a boot loop. This is a situation that occurs when a Windows device restarts without warning during its startup process – meaning the machine cannot finish a complete and stable boot cycle and, therefore, won’t turn on.
At the time of writing, the full facts of the incident have not been fully established, and an investigation will likely take some time.
However, such issues will in general occur either due to inadequate testing across various desktop and server environments, or due to a lack of proper sandboxing and rollback mechanisms for updates that involve a kernel-level interaction.
Is there a cyber security threat from the CrowdStrike outage?
Though similar in its effect and origins to a supply chain attack, it is important to note that the CrowdStrike outage is not a cyber security incident and nobody is known to be under attack as a result of it.
However, as it affects a cyber security product threat actors will take advantage of the downtime caused and any gaps in coverage arising. This has already started to happen, within hours of the incident unfolding CrowdStrike itself said it identified a malicious ZIP archive circulating which purported to contain a utility to help automate recovery, but was in fact a so-called remote access Trojan (RAT).
Multiple national cyber security agencies, including the UK's National Cyber Security Centre (NCSC) and partners in Australia, Singapore and the US, have also issued cyber alerts and advisories in the wake of the outage.
The coming days and weeks will see threat actors exploiting the incident in phishing and social engineering attacks as they attempt to lure new victims. Potential lures could include offers of technical support or bogus CrowdStrike updates, and the consequences could include data exfiltration, ransomware deployment and extortion.
Security and IT leaders and admins would be well-advised to communicate the potential follow-on dangers to their users.
Who was affected by the CrowdStrike outage?
According to Microsoft, the incident affected approximately 8.5 million Windows devices worldwide, making up less than 1% of the entire estate. Redmond said that while this was a tiny number, all things considered, the widespread economic and societal impacts of the incident reflected the use of CrowdStrike by many organisations that run critical public-facing services.
The full number of organisations affected by the outage is not known for now. However, those that are known to have, or have confirmed they have, experienced some impact include:
- Airlines including American Airlines, Delta, KLM, Lufthansa, Ryanair, SAS and United;
- Airports including Gatwick, Luton, Stansted and Schiphol;
- Financial organisations including the London Stock Exchange, Lloyds Bank and Visa;
- Healthcare including most GP surgeries and many independent pharmacies;
- Media organisations including MTV, VH1, Sky and some BBC channels;
- Retailers, leisure and hospitality organisations including Gail’s Bakery, Ladbrokes, Morrisons, Tesco and Sainsbury’s;
- Sporting bodies including F1 teams Aston Martin Aramco, Mercedes AMG Petronas and Williams Racing, which were preparing to compete at the Hungarian Grand Prix at the time, and the Paris 2024 Organising Committee for the Olympic and Paralympic Games, which begin in a few days;
- Train operating companies (TOCs) such as Avanti West Coast, Merseyrail, Southern and Transport for Wales.
What is CrowdStrike saying about the outage?
In an initial statement, CrowdStrike CEO George Kurtz said: “CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This is not a security incident or cyber attack.
“The issue has been identified, isolated and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website.
“We further recommend organisations ensure they’re communicating with CrowdStrike representatives through official channels.Our team is fully mobilised to ensure the security and stability of CrowdStrike customers.”
In a breakfast TV interview with NBC in the US on 19 July, Kurtz added: “We’re deeply sorry for the impact that we’ve caused to customers, to travellers, to anyone affected by this, including our companies.”
What has been Microsoft's response
Microsoft has been working extensively alongside CrowdStrike to automate work on developing and pushing a fix, and in the wake of the outages hundreds of its engineers and software experts were deployed to work directly with customers on service restoration.
Microsoft has also been collaborating with cloud providers such as Google Cloud and Amazon Web Services (AWS) to share awareness on the impacts seen, and better inform ongoing dialogue with customers and CrowdStrike itself.
Can I fix the CrowdStrike problem myself?
CrowdStrike has rolled back the changes to the affected product automatically, but hosts may continue to crash or be unable to stay online to receive the remedial update.
The short answer to the question is yes, but unfortunately, such issues can be daunting to fix, requiring IT teams to put in a lot of work. It may be days, or even longer, before all the affected devices can be reached.
System administrators are advised to take the following steps:
- Boot Windows into safe mode, or the Windows Recovery Environment;
- Navigate to C:\Windows\System32\drivers\CrowdStrike directory;
- Locate the file matching “C-00000291*.sys”. Delete this file;
- Boot normally.
CrowdStrike customers can access more information by logging into its support portal.
How can I avoid similar problems in the future?
Security firms such as CrowdStrike are under a great deal of pressure when it comes to product development and updates, which must be done frequently as they strive to keep their customers protected from new zero-days, ransomware and the like.
This pressure also trickles down to customers themselves, who will understandably often want to take advantage of settings to allow their security tools to update automatically.
To avoid falling victim to this kind of problem going forward, IT teams should consider taking a phased approach to software updates – particularly if they pertain to security solutions – and test them in a sandbox environment, or on a limited set of devices, prior to full deployment.
It is also wise to have some level of system redundancy built in to properly isolate and manage fault domains, particularly when running critical infrastructure.
Computer Weekly and TechTarget coverage of the CrowdStrike incident
- July 19 2024: An update to CrowdStrike’s Falcon service has led to many Windows users being unable to work this morning. Microsoft 365 is also affected.
- The Emis Web IT system used by more than half of GP practices in the UK is down, following the worldwide Microsoft outage.
- The global outage of Microsoft is rapidly sending shockwaves across all sectors, demonstrating the risk of having a single point of failure.
- A CrowdStrike update with a faulty sensor file has global implications for Windows systems. But competitors need to limit the finger-pointing in case it happens to them.
- As organisations recover from today’s outages, the cyber security industry will need to develop new security software evaluation criteria and requirements and learn to parlay risks.
- July 22: About 8.5 million devices globally were hit by the botched CrowdStrike update, with a significant number now back online and operational.
