CrowdStrike Update Causes Global Windows Crash: Microsoft's Response
Microsoft Blue Screen Crisis: A global IT outage on July 19 due to an antivirus update, affecting flights, banking, and supermarket systems worldwide.
On July 19, local time, some Microsoft applications and services experienced delays, partial functionality, or were completely inaccessible.
Microsoft’s blue screen error became a trending topic. Multiple users shared images on social media showing blue screens on company computers with the message “Your device ran into a problem and needs to restart.”
This was a global IT system outage. Flight systems, banking systems, and supermarket systems in several countries and regions were affected.
Here is a summary of the affected countries and regions:
Hong Kong, China: The airport system was impacted, requiring manual check-ins. Cathay Pacific’s website couldn’t provide flight booking services.
New Zealand: Windows systems experienced blue screens.
Australia: Flights, banks, government networks, businesses, and supermarket self-checkouts were affected.
Japan: West Japan Railway Company (JR West) couldn’t retrieve train position information due to Windows system failure.
India: Flights were affected.
USA: San Francisco International Airport flights were canceled. Major airlines, including Delta, United, and American Airlines, halted flights due to communication issues.
Singapore: Due to Microsoft system failures, some airlines at Changi Airport required manual check-ins.
UK: Flights were affected.
Germany: Flights were affected.
Spain: Flights were affected.
Microsoft stated, “We are aware of an issue affecting virtual machines running Windows clients and Windows Server, which are running CrowdStrike’s Falcon software. These VMs may experience a bug check (BSOD) and get stuck in a reboot loop. We estimate the issue started around July 18 at 19:00 UTC.”
Microsoft added, “We can confirm that the update causing the issue has been rolled back by CrowdStrike. Customers still facing issues should contact CrowdStrike for further assistance.”
Previously, Microsoft reported that its cloud services had recovered after an outage affecting its Azure and Microsoft 365 applications in the Central US region.
A Microsoft spokesperson told the media that these are two different and unrelated issues: one concerning Microsoft’s Azure cloud and the other involving cybersecurity company CrowdStrike.
Currently, Microsoft’s pre-market stock drop widened to nearly 2%, and CrowdStrike’s pre-market drop exceeded 20%.
My Experience
The cause was this terrible CrowdStrike software.
I was the first one in our company to get a blue screen!
Around 1 PM, likely one of the earliest globally, I had a local minidump saved, so I pinpointed it right away. Our IT initially thought it was a hard drive issue.
I was lucky to only get one blue screen, while others couldn’t even start their computers afterward.
CrowdStrike claims to be a security software protecting against ransomware, but it performs terribly. On Windows, even with no activity, it uses about 20% CPU, worse than Kaspersky. It imposes many restrictions on your system, making software development very unfriendly.
For example, if you write a simple Windows program in Visual Studio, it gets killed right after compiling. Even some Python scripts with download functions are flagged as viruses. It follows a “better safe than sorry” approach. Functionally, it’s much worse than Microsoft Security Essentials.
Using this in a software development company is disastrous.
Combined with domain management and policies, you can’t turn it off, it forces updates, and even restarts your system to update.
I don’t know how this company manages its public relations, but installing it has become standard for Western companies, especially in the US. This company’s market value is extremely high ($80 billion). I seriously doubt whether its market value matches its technical capabilities. I even suspect it might have achieved its market size through bribery.
Subsequent Impact
The severe impact of this incident is that many devices cannot start up, requiring manual on-site fixes. Labor costs are always high, especially in Western countries. IT staff have to fix each computer individually, as they are now bricked and cannot be repaired remotely.
If the devices are in remote areas, maintenance costs skyrocket.
To fix the affected devices, IT staff must visit the site, switch to safe mode, uninstall the faulty software, or manually apply a patch. However, many Windows devices have BitLocker enabled by default. The BitLocker keys are stored on Windows machines, which have also blue-screened, causing a deadlock.
BitLocker combined with CrowdStrike is the biggest challenge.
I believe there will be lawsuits against this company, and its stock price will likely plummet, possibly even leading to bankruptcy. Though, I hope it doesn’t, as a worse supplier might take its place.
Solution
Based on online information, this incident was caused by a series of errors. CrowdStrike obtained configurations and updates through Azure cloud services. Azure experienced a minor fault, causing CrowdStrike to receive incorrect configuration files, leading to a faulty update that caused the blue screens.
The biggest lesson for software development is that testing usually focuses on normal code paths, rarely addressing error handling. Many software disasters result from inadequate error handling.
If security software is poorly implemented, it becomes insecure. Security software that is hard to stop or uninstall is even less secure.
This incident had a massive global impact.
Official responses have been unhelpful.
Comments suggest this company has ties to the U.S. government. Thus, in security-related fields, Western companies are almost forced to install this software.
The circulated online fixes may not work:
To delete or rename the problematic file, you must enter safe mode. If BitLocker is enabled, you need the key. Many enterprise keys are centrally managed, and the management computers are also blue-screened.
Most companies enforce BitLocker on mobile devices, making recovery difficult.
Deleting the C-000029*.SYS file may not help. I tried this on a device without BitLocker, but it auto-updated and restored the file.
Blue screen dump analysis: (Note: I cannot provide the original dump/SYS file as the computer is shut down and left at the office. IT advised not to turn it on unless necessary).
For analysis of this file, run !analyze -v
nt!KeBugCheckEx:
fffff805`77017f80 48894c2408 mov qword ptr [rsp+8],rcx ss:0018:fffffd89`395719c0=0000000000000050
3: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
PAGE_FAULT_IN_NONPAGED_AREA (50)
Invalid system memory was referenced. This cannot be protected by try-except.
Typically the address is just plain bad or it is pointing at freed memory.
Arguments:
Arg1: ffffe400000000a8, memory referenced.
Arg2: 0000000000000000, X64: bit 0 set if the fault was due to a not-present PTE.
bit 1 is set if the fault was due to a write, clear if a read.
bit 3 is set if the processor decided the fault was due to a corrupted PTE.
bit 4 is set if the fault was due to attempted execute of a no-execute PTE.
- ARM64: bit 1 is set if the fault was due to a write, clear if a read.
bit 3 is set if the fault was due to attempted execute of a no-execute PTE.
Arg3: fffff805893414ed, If non-zero, the instruction address which referenced the bad memory
address.
Arg4: 0000000000000002, (reserved)
Debugging Details:
------------------
KEY_VALUES_STRING: 1
Key : AV.Type
Value: Read
Key : Analysis.CPU.mSec
Value: 1468
Key : Analysis.Elapsed.mSec
Value: 1506
Key : Analysis.IO.Other.Mb
Value: 12
Key : Analysis.IO.Read.Mb
Value: 0
Key : Analysis.IO.Write.Mb
Value: 30
Key : Analysis.Init.CPU.mSec
Value: 483
Key : Analysis.Init.Elapsed.mSec
Value: 27920
Key : Analysis.Memory.CommitPeak.Mb
Value: 96
Key : Bugcheck.Code.LegacyAPI
Value: 0x50
Key : Bugcheck.Code.TargetModel
Value: 0x50
Key : Dump.Attributes.AsUlong
Value: 100c
Key : Dump.Attributes.DiagDataWrittenToHeader
Value: 1
Key : Dump.Attributes.ErrorCode
Value: 0
Key : Dump.Attributes.InsufficientDumpfileSize
Value: 1
Key : Dump.Attributes.KernelGeneratedTriageDump
Value: 1
Key : Dump.Attributes.LastLine
Value: Dump completed successfully.
Key : Dump.Attributes.ProgressPercentage
Value: 0
Key : Dump.Attributes.RequiredDumpfileSize
Value: 0x180243647
Key : Failure.Bucket
Value: AV_R_(null)_csagent!unknown_function
Key : Failure.Hash
Value: {0a7d0180-2167-fed0-1865-46fb6bb26816}
BUGCHECK_CODE: 50
BUGCHECK_P1: ffffe400000000a8
BUGCHECK_P2: 0
BUGCHECK_P3: fffff805893414ed
BUGCHECK_P4: 2
FILE_IN_CAB: 071924-35578-01.dmp
DUMP_FILE_ATTRIBUTES: 0x100c
Insufficient Dumpfile Size
Kernel Generated Triage Dump
READ_ADDRESS: fffff8057791c4a8: Unable to get MiVisibleState
Unable to get NonPagedPoolStart
Unable to get NonPagedPoolEnd
Unable to get PagedPoolStart
Unable to get PagedPoolEnd
unable to get nt!MmSpecialPagesInUse
ffffe400000000a8
MM_INTERNAL_CODE: 2
BLACKBOXBSD: 1 (!blackboxbsd)
BLACKBOXNTFS: 1 (!blackboxntfs)
BLACKBOXPNP: 1 (!blackboxpnp)
BLACKBOXWINLOGON: 1
CUSTOMER_CRASH_COUNT: 1
PROCESS_NAME: System
TRAP_FRAME: fffffd8939571be0 -- (.trap 0xfffffd8939571be0)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=fffffd89395721c0 rbx=0000000000000000 rcx=0000000000000003
rdx=fffffd8939572190 rsi=0000000000000000 rdi=0000000000000000
rip=fffff805893414ed rsp=fffffd8939571d70 rbp=fffffd8939571e70
r8=ffffe400000000a8 r9=0000000000000000 r10=0000000000000000
r11=0000000000000014 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei ng nz na pe nc
csagent+0xe14ed:
fffff805`893414ed 458b08 mov r9d,dword ptr [r8] ds:ffffe400`000000a8=????????
Resetting default scope
STACK_TEXT:
fffffd89`395719b8 fffff805`770b1fb0 : 00000000`00000050 ffffe400`000000a8 00000000`00000000 fffffd89`39571be0 : nt!KeBugCheckEx
fffffd89`395719c0 fffff805`76e3014c : ffffe400`ebeecac8 00000000`00000000 00000000`00000000 ffffe400`000000a8 : nt!MiSystemFault+0x1b9d60
fffffd89`39571ac0 fffff805`7702917e : ffffc084`d2694868 00000000`00000001 fffffd89`39571fe0 fffffd89`39571fe0 : nt!MmAccessFault+0x29c
fffffd89`39571be0 fffff805`893414ed : 00000000`00000000 fffffd89`39571e70 ffffe401`0a0d600c ffffe401`0a0ce05c : nt!KiPageFault+0x37e
fffffd89`39571d70 00000000`00000000 : fffffd89`39571e70 ffffe401`0a0d600c ffffe401`0a0ce05c ffffe400`e7000140 : csagent+0xe14ed
SYMBOL_NAME: csagent+e14ed
MODULE_NAME: csagent
IMAGE_NAME: csagent.sys
STACK_COMMAND: .cxr; .ecxr ; kb
BUCKET_ID_FUNC_OFFSET: e14ed
FAILURE_BUCKET_ID: AV_R_(null)_csagent!unknown_function
OSPLATFORM_TYPE: x64
OSNAME: Windows 10
FAILURE_ID_HASH: {0a7d0180-2167-fed0-1865-46fb6bb26816}
Followup: MachineOwnerUninstalling CrowdStrike: Taking back control from software that turns protection into a peril.
Top 6 Python Libraries for Web Scraping You Need to Know
There are many powerful libraries in Python for web scraping and parsing. Here are six of the most common ones.