Veteran Microsoft engineer shares some enterprise support tips
(2024/12/06)
- Reference: 1733477534
- News link: https://www.theregister.co.uk/2024/12/06/raymond_chen_support_desk_advice/
- Source link:
Microsoft veteran Raymond Chen revealed another product support trick from within the corridors of Microsoft. This time, it's not about blowing on connectors but about avoiding casting some embarrassing shade on a customer's purchasing decisions.
When something goes wrong in the Enterprise, it isn't [1]always DNS . It might have something to do with something a customer has installed - perhaps a driver, or maybe that anti-malware software that has latched on to something deep in the internals of Windows and is now wreaking havoc.
Chen [2]explained , "It can happen that investigating a problem reveals that a problem occurred when calling a function that has been patched or hooked."
[3]
He added, "In the case of enterprise customers, the offender is typically some 'advanced anti-malware software' that they paid a lot of money for."
[4]
[5]
"The code running in the hook ends up does something sketchy, the most common example of which is hooking a low-level function and then having the hook call a higher-level function, resulting in a deadlock."
Chen used the example of hooking HeapAlloc , a low-level function, and calling MessageBox , a high-level user interface function. He called his example "ridiculous," although this writer would counter with "let those who have not done something similarly naughty in the name of debugging cast the first stone."
[6]
The problem is working out a way to let the customer know that it is the vendor's software that has caused the problem. "After all, they paid a lot of money for that anti-malware software, and a recommendation of the form 'throw away a lot of money you already spent' is not going to land well," said Chen.
Instead, the trick is to persuade the customer to turn off the offending anti-malware software to… er… better debug the issue. In other words, you tell the user, "We're not smart enough to debug the problem. Can you help us?"
"I'm told that what usually happens is that the customer, for some mysterious reason, is unable to get the problem to occur when the anti-malware software is disabled. 'Wow, that's weird'."
[7]PEBCAK problem transformed young techie into grizzled cynical sysadmin
[8]VirusTotal: We're sorry someone fat-fingered and exposed 5,600 users
[9]Errors logged as 'nut loose on the keyboard' were – ahem – not a hardware problem
[10]That hardware will be more reliable if you stop stabbing it all day
Chen said, "Sometimes the customer gets the hint and opens a support ticket with the anti-malware vendor.
"Sometimes we have to suggest to them, 'Why don't you check if there’s an update available for your anti-malware software?'"
[11]
And the blowing on the connector trick? Twenty years ago, Chen explained that enterprise support engineers would ask customers to blow dust out of connectors rather than suggesting that perhaps, just perhaps, they'd forgotten to plug something in or plugged it into the wrong port.
Now, just like then, the move allowed the customer to save face while also fixing the problem, which some might say is the true craft of a quick-witted support engineer. ®
Get our [12]Tech Resources
[1] https://www.cyberciti.biz/humour/a-haiku-about-dns/
[2] https://devblogs.microsoft.com/oldnewthing/20241203-00/?p=110601
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Z1LZVop0bT2mC0zlRIe7yQAAAE0&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z1LZVop0bT2mC0zlRIe7yQAAAE0&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z1LZVop0bT2mC0zlRIe7yQAAAE0&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z1LZVop0bT2mC0zlRIe7yQAAAE0&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2023/09/08/on_call/
[8] https://www.theregister.com/2023/07/21/virustotal_data_exposure_apology/
[9] https://www.theregister.com/2023/03/24/on_call/
[10] https://www.theregister.com/2024/11/15/on_call/
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z1LZVop0bT2mC0zlRIe7yQAAAE0&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[12] https://whitepapers.theregister.com/
When something goes wrong in the Enterprise, it isn't [1]always DNS . It might have something to do with something a customer has installed - perhaps a driver, or maybe that anti-malware software that has latched on to something deep in the internals of Windows and is now wreaking havoc.
Chen [2]explained , "It can happen that investigating a problem reveals that a problem occurred when calling a function that has been patched or hooked."
[3]
He added, "In the case of enterprise customers, the offender is typically some 'advanced anti-malware software' that they paid a lot of money for."
[4]
[5]
"The code running in the hook ends up does something sketchy, the most common example of which is hooking a low-level function and then having the hook call a higher-level function, resulting in a deadlock."
Chen used the example of hooking HeapAlloc , a low-level function, and calling MessageBox , a high-level user interface function. He called his example "ridiculous," although this writer would counter with "let those who have not done something similarly naughty in the name of debugging cast the first stone."
[6]
The problem is working out a way to let the customer know that it is the vendor's software that has caused the problem. "After all, they paid a lot of money for that anti-malware software, and a recommendation of the form 'throw away a lot of money you already spent' is not going to land well," said Chen.
Instead, the trick is to persuade the customer to turn off the offending anti-malware software to… er… better debug the issue. In other words, you tell the user, "We're not smart enough to debug the problem. Can you help us?"
"I'm told that what usually happens is that the customer, for some mysterious reason, is unable to get the problem to occur when the anti-malware software is disabled. 'Wow, that's weird'."
[7]PEBCAK problem transformed young techie into grizzled cynical sysadmin
[8]VirusTotal: We're sorry someone fat-fingered and exposed 5,600 users
[9]Errors logged as 'nut loose on the keyboard' were – ahem – not a hardware problem
[10]That hardware will be more reliable if you stop stabbing it all day
Chen said, "Sometimes the customer gets the hint and opens a support ticket with the anti-malware vendor.
"Sometimes we have to suggest to them, 'Why don't you check if there’s an update available for your anti-malware software?'"
[11]
And the blowing on the connector trick? Twenty years ago, Chen explained that enterprise support engineers would ask customers to blow dust out of connectors rather than suggesting that perhaps, just perhaps, they'd forgotten to plug something in or plugged it into the wrong port.
Now, just like then, the move allowed the customer to save face while also fixing the problem, which some might say is the true craft of a quick-witted support engineer. ®
Get our [12]Tech Resources
[1] https://www.cyberciti.biz/humour/a-haiku-about-dns/
[2] https://devblogs.microsoft.com/oldnewthing/20241203-00/?p=110601
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Z1LZVop0bT2mC0zlRIe7yQAAAE0&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z1LZVop0bT2mC0zlRIe7yQAAAE0&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z1LZVop0bT2mC0zlRIe7yQAAAE0&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z1LZVop0bT2mC0zlRIe7yQAAAE0&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2023/09/08/on_call/
[8] https://www.theregister.com/2023/07/21/virustotal_data_exposure_apology/
[9] https://www.theregister.com/2023/03/24/on_call/
[10] https://www.theregister.com/2024/11/15/on_call/
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z1LZVop0bT2mC0zlRIe7yQAAAE0&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[12] https://whitepapers.theregister.com/
I once had to ask someone if they'd forgotten to plug their computer in. Which was a bit embarrassing. Particularly as I didn't think to suggest a clever way to avoid the shame - say, please unplug from the wall and switch off the plug socket to... erm [emergency excuse generator]... erm... re-polarise the plug socket...
To be fair, she knew how little she knew about computers - so just being polite solved the problem without any denials or grumpiness. She was checking her emails on her laptop every day, then turning it off. It was plugged in. To an extension lead to reach her dining table. The other end of which had been unplugged, in order to do the hooverying. Probably a week before, and the laptop had slowly drained its battery.
During the pandemic I went to the hospital for her, to pick up new batteries for her hearing aid. Not sure why I bothered though, given she never changed those either. Perhaps she just didn't believe in electricity?
On the subject of the excuse generator - I came across something this morning on the Mayak (Kyshtym) nuclear accident from the 50s. To cover up for all the radioactive dust in the air, after the explosion, the Soviet authorities told people that there was an unusually strong aurora borealis that was interacting with the atmosphere and causing it to turn yellow. The BOfH would have been proud.