How a cheap barcode scanner helped fix CrowdStrike'd Windows PCs in a flash
(2024/07/25)
- Reference: 1721874553
- News link: https://www.theregister.co.uk/2024/07/25/crowdstrike_remediation_with_barcode_scanner/
- Source link:
Not long after Windows PCs and servers at the Australian limb of audit and tax advisory Grant Thornton started BSODing last Friday, senior systems engineer Rob Woltz remembered a small but important fact: When PCs boot, they consider barcode scanners no differently to keyboards.
That knowledge nugget became important as the firm tried to figure out how to respond to the mess CrowdStrike created, which at Grant Thornton Australia threw hundreds of PCs and no fewer than 100 servers into the doomloop that CrowdStrike's [1]shoddy testing software made possible.
All of Grant Thornton's machines were encrypted with Microsoft's BitLocker tool, which meant that recovery upon restart required CrowdStrike's [2]multi-step fix and entry of a 48-character BitLocker key.
[3]
The firm prioritized recovery for its servers, and tackled that task manually. But infrastructure manager Ben Watson and Woltz felt the sheer number of PCs at the firm meant an automated response would be required.
[4]
[5]
That response could not, however, involve distributing BitLocker keys – doing so was just too risky to contemplate.
So was reading keys to workers over the phone or in person. "It felt like a bad idea to read a 48-character key to people who were already stressed out," Woltz told The Register .
[6]
Which was when his memory about barcode scanners came into play. The firm had the BitLocker keys for all its PCs, so Woltz and colleagues wrote a script that turned them into barcodes that were displayed on a locked-down management server's desktop. The script would be given a hostname and generate the necessary barcode and LAPS password to restore the machine.
Woltz went to an office supplies store and acquired an off-the-shelf barcode scanner for AU$55 ($36).
At the point when rebooting PCs asked for a BitLocker key, pointing the scanner at the barcode on the server's screen made the machines treat the input exactly as if the key was being typed. That's a lot easier than typing it out every time, and the server's desktop could be accessed via a laptop for convenience.
[7]
Woltz, Watson, and the team scaled the solution – which meant buying more scanners at more office supplies stores around Australia.
[8]Uncle Sam opens probe into CrowdStrike turbulence at Delta Air Lines
[9]How did a CrowdStrike file crash millions of Windows computers? We take a closer look at the code
[10]Windows Patch Tuesday update might send a user to the BitLocker recovery screen
[11]EU gave CrowdStrike the keys to the Windows kernel, claims Microsoft
On Monday, remote staff were told to come to the office with their PCs and visit IT to connect to a barcode scanner. All PCs in the firm's Australian fleet were fixed by lunchtime – taking only three to five minutes for each machine.
Watson told us manually fixing servers needed about 20 minutes per machine.
[12]
A Grant Thornton Australia IT worker scanning a laptop displaying the necessary BitLocker barcodes ... Click to enlarge
Woltz is pleased that his idea translated into a swift recovery, but also a little regretful he didn't think of using QR codes – they could have encoded sufficient data to automate the entire remediation process.
Watson thinks Woltz did more than enough. On LinkedIn he [13]hailed the effort of Woltz and other team members as "remarkable innovation in streamlining workstation recovery."
Woltz told The Register he and the team are chuffed that they were able to help, and also that some of them feature as hand models wielding barcode scanners in Watson's LinkedIn post... ®
Get our [14]Tech Resources
[1] https://www.theregister.com/2024/07/24/crowdstrike_validator_failure/
[2] https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZqHN6qVzwanoOwwWGPXjYgAAAEw&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZqHN6qVzwanoOwwWGPXjYgAAAEw&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZqHN6qVzwanoOwwWGPXjYgAAAEw&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZqHN6qVzwanoOwwWGPXjYgAAAEw&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZqHN6qVzwanoOwwWGPXjYgAAAEw&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2024/07/24/transport_department_delta_probe/
[9] https://www.theregister.com/2024/07/23/crowdstrike_failure_shows_need_for/
[10] https://www.theregister.com/2024/07/24/windows_update_bitlocker/
[11] https://www.theregister.com/2024/07/22/windows_crowdstrike_kernel_eu/
[12] https://regmedia.co.uk/2024/07/25/supplied_grant_thornton_crowdstrike_bitlocker_barcode_scanner.jpg
[13] https://www.linkedin.com/posts/ben-watson-792b092b_teamwork-crowdstrike-proud-activity-7221053667418783744-c11K
[14] https://whitepapers.theregister.com/
That knowledge nugget became important as the firm tried to figure out how to respond to the mess CrowdStrike created, which at Grant Thornton Australia threw hundreds of PCs and no fewer than 100 servers into the doomloop that CrowdStrike's [1]shoddy testing software made possible.
All of Grant Thornton's machines were encrypted with Microsoft's BitLocker tool, which meant that recovery upon restart required CrowdStrike's [2]multi-step fix and entry of a 48-character BitLocker key.
[3]
The firm prioritized recovery for its servers, and tackled that task manually. But infrastructure manager Ben Watson and Woltz felt the sheer number of PCs at the firm meant an automated response would be required.
[4]
[5]
That response could not, however, involve distributing BitLocker keys – doing so was just too risky to contemplate.
So was reading keys to workers over the phone or in person. "It felt like a bad idea to read a 48-character key to people who were already stressed out," Woltz told The Register .
[6]
Which was when his memory about barcode scanners came into play. The firm had the BitLocker keys for all its PCs, so Woltz and colleagues wrote a script that turned them into barcodes that were displayed on a locked-down management server's desktop. The script would be given a hostname and generate the necessary barcode and LAPS password to restore the machine.
Woltz went to an office supplies store and acquired an off-the-shelf barcode scanner for AU$55 ($36).
At the point when rebooting PCs asked for a BitLocker key, pointing the scanner at the barcode on the server's screen made the machines treat the input exactly as if the key was being typed. That's a lot easier than typing it out every time, and the server's desktop could be accessed via a laptop for convenience.
[7]
Woltz, Watson, and the team scaled the solution – which meant buying more scanners at more office supplies stores around Australia.
[8]Uncle Sam opens probe into CrowdStrike turbulence at Delta Air Lines
[9]How did a CrowdStrike file crash millions of Windows computers? We take a closer look at the code
[10]Windows Patch Tuesday update might send a user to the BitLocker recovery screen
[11]EU gave CrowdStrike the keys to the Windows kernel, claims Microsoft
On Monday, remote staff were told to come to the office with their PCs and visit IT to connect to a barcode scanner. All PCs in the firm's Australian fleet were fixed by lunchtime – taking only three to five minutes for each machine.
Watson told us manually fixing servers needed about 20 minutes per machine.
[12]
A Grant Thornton Australia IT worker scanning a laptop displaying the necessary BitLocker barcodes ... Click to enlarge
Woltz is pleased that his idea translated into a swift recovery, but also a little regretful he didn't think of using QR codes – they could have encoded sufficient data to automate the entire remediation process.
Watson thinks Woltz did more than enough. On LinkedIn he [13]hailed the effort of Woltz and other team members as "remarkable innovation in streamlining workstation recovery."
Woltz told The Register he and the team are chuffed that they were able to help, and also that some of them feature as hand models wielding barcode scanners in Watson's LinkedIn post... ®
Get our [14]Tech Resources
[1] https://www.theregister.com/2024/07/24/crowdstrike_validator_failure/
[2] https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZqHN6qVzwanoOwwWGPXjYgAAAEw&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZqHN6qVzwanoOwwWGPXjYgAAAEw&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZqHN6qVzwanoOwwWGPXjYgAAAEw&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZqHN6qVzwanoOwwWGPXjYgAAAEw&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZqHN6qVzwanoOwwWGPXjYgAAAEw&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2024/07/24/transport_department_delta_probe/
[9] https://www.theregister.com/2024/07/23/crowdstrike_failure_shows_need_for/
[10] https://www.theregister.com/2024/07/24/windows_update_bitlocker/
[11] https://www.theregister.com/2024/07/22/windows_crowdstrike_kernel_eu/
[12] https://regmedia.co.uk/2024/07/25/supplied_grant_thornton_crowdstrike_bitlocker_barcode_scanner.jpg
[13] https://www.linkedin.com/posts/ben-watson-792b092b_teamwork-crowdstrike-proud-activity-7221053667418783744-c11K
[14] https://whitepapers.theregister.com/
Brilliant concept!
anothercynic
Bravo! Someone who thought outside the box, and I'm actually surprised that they were allowing USB barcode scanners to be plugged into the PCs! Imagine if the USB ports had been disabled by policy for anything but mouse and keyboard (although some scanners actually identify as a keyboard).
Nonetheless, kudos and applause (and a beer)
Re: Brilliant concept!
Nick Stallman
Barcode scanners actually typically appear as keyboards. They scan the number or text in a range of different supported formats and just type it in and hit enter at the end of the barcode automatically.
Zero drivers, zero fuss. They are awesome things.
An alternative would have been to use a 'rubber duck' - though this might have taken longer to procure.