Here's why your Samsung Blu-ray player bricked itself: It downloaded an XML config file that broke the firmware
- Reference: 1595066765
- News link: https://www.theregister.co.uk/2020/07/18/samsung_bluray_mass_dieoff_explained/
- Source link:
In the days that followed, [1]complaints about devices caught in an endless startup boot loop began to appear on various internet discussion boards, and [2]videos documenting the device failure appeared on YouTube.
To fix the issue, Samsung eventually advised customers to return their inoperable video players for repairs. There is no software fix.
"We are aware of the boot loop issue that appeared on certain 2015 Samsung Blu-Ray players and are offering free mail-in repairs to customers who have been impacted," a representative of the mega-manufacturer [3]said in a Samsung forum post.
It was speculated by netizens and some media reports that a HTTPS certificate error was to blame. However, it's been suggested to The Register that the cause of the failure was an XML file downloaded by the network-connected devices from Samsung servers during periodic logging policy checks.
This file, when fetched and saved to the device's flash storage and processed by the equipment, crashed the system software and force a reboot. Upon reboot, the player parsed the XML file again from its flash storage, crashed and rebooted again. And so on, and so on, and so on. Crucially, the XML file would be parsed before a new one could be fetched from the internet, so once the bad configuration file was fetched and stored by these particular Samsung Blu-ray players in the field, they were bricked.
Drilling in
A Register reader who is savvy with low-level hardware, and asked to be identified simply as Gray, provided us a detailed technical analysis of Samsung's blunder.
One thing you have to understand is that these internet-connected Blu-ray players in question are programmed to log their activities and send copies of this information to Samsung. This telemetry is sent to the tech giant's servers when the player's firmware is told to check for a software update. These logs include things like when you opened, say, the Netflix app and when you closed it on the player.
Exactly how much the device should log and transmit back to HQ is defined by Samsung in an XML logging policy file regularly fetched from this URL: https://configprd.samsungcloudsolution.net/openapi/dict/logpolicyconfig
The affected Blu-ray players, we're told, do not transmit their logs until a privacy notice has been accepted by the user. The privacy notice comes into play when the customer connects the device to the internet and tries to use a network service like Netflix. After that notice is accepted, these Blu-ray players no longer bug their users with privacy notices, and simply quietly send the telemetry while the system checks for software updates.
And even if you don't use something like Netflix, or don't accept the privacy notice nor download a software update, but do connect the device to the internet, your player will still routinely fetch the logging policy file. That's why, our source tells us, Samsung Blu-ray players that have never connected to the internet were not affected by the flawed file. Also, it explains why players that never performed a software update nor used a network service, and were simply connected to the internet, were bricked. The firmware routinely automatically fetches, stores, and parses the logging policy file regardless of anything else.
"Players were bricked even though the users never performed a network update. It was enough for the player to be connected to the internet. Samsung never asked the user if it was OK to download the bomb," said Gray, referring to the dodgy XML policy file.
The problem with the XML file, sent out on June 18, 2020, is that it wasn't formatted in a way compatible with the device's code. Though a valid XML file, it contained an empty list element, we're told:
"Unfortunately for Samsung, the code which handles the processing of the log policy XML file has not been tested for such an empty <list/> element and causes a crash," Gray explained. The device code appears not to have been designed to handle that possibility because the empty list produces an invalid memory reference in the device's main program, called bdpprog , which causes the kernel to terminate it.
Crashing the main program results in a reboot, but since the logging policy XML file is always processed early on after startup, the crash simply reoccurs before a fixed version of the file can be fetched. Gray suggested this XML file was sent to the Blu-ray players without proper verification.
"After the crash, the main program, bdpprog, is terminated by the kernel," said Gray. "Since bdpprog is the main program, its termination results in a reboot by init. Even less fortunately for Samsung, the code for parsing the logging policy XML file is hard-coded to run at every boot. The result is that the player is stuck in a permanent boot loop as has recently been experienced by thousands of users worldwide."
Gray continued:
Because of the monumentally stupid idea of parsing a downloaded XML file unconditionally at every boot, there seems to be no way to recover the devices from the boot loop using normal means – such as a USB stick, CD or network – because the crash happens too early in the boot sequence.
The only ways to revive the player are: erase the invalid policy file from the flash partition, or update the firmware of the player with a version in which the XML parse bug has been corrected. At the time of writing this, no such updated firmware exists.
Unfortunately, both of these fixes require low-level access to the serial debug port of the player, soldering wires to the motherboard, proprietary hardware and software tools as well as deep knowledge of the player’s architecture. This is not something that an average user can do. Hence, the best solution that Samsung can and is offering its customers is a prepaid label for sending the player to an authorized repair center.
Samsung, we're told, replaced the file on its servers on June 27, 2020, thereby preventing the problem from affecting Blu-ray players that hadn't already ingested it. But a server-side fix does nothing for devices already locked in an endless reboot loop. Hence the mail-in repair program.
The Register asked Samsung whether it could confirm Gray's claims and provide more specific details about the number of devices ultimately affected by the snafu. A spokesperson acknowledged the request, and promised a reply if the manufacturer has anything to share. We've not yet received any response beyond that. ®
Bootnote
If you try to fetch the policy file from Samsung's services, you'll probably run into a certificate error. But that's because it's using a Samsung-issued HTTPS certificate your browser or operating system doesn't trust. However, the Blu-ray players do trust the certs, which expire in 2069. "This problem has nothing to do with expiring SSL certificates as has been speculated," noted Gray.
PS: Got some inside analysis or tips, too? Send them to us via strong encryption: details and PGP key [4]here .
Get our [5]Tech Resources
[1] https://us.community.samsung.com/t5/Home-Theater/Blu-ray-player-power-cycling-whenever-plugged-in/td-p/1278935/highlight/true
[2] https://www.youtube.com/watch?v=ICbm5eAF3NI
[3] https://us.community.samsung.com/t5/Home-Theater/Blu-ray-player-power-cycling-whenever-plugged-in/m-p/1285080#M14987
[4] https://www.theregister.com/about/company/contact/
[5] https://whitepapers.theregister.com/
Rookie mistake
No code should ever trust any input, whether it's user input, a remote or local file, or from a driver or anything else. The only way to safely parse is to assume that one is being fed garbage data and to defensively program to deal with such junk.
Good to know that Samsung's QA department runs such poisoned input tests against their device firmware before releasing them to customers.
Thanks Gray!
I have now added samsungcloudsolution.net to my router's blacklist. I have a Samsung TV and you can never be too sure.
Why...? Just Why?
Does a standalone device such as this need to phone home (and all the rest)?
I see this incident as just the start of planned borkage by makers. In this case, Samsung got found out others will be a lot sneakier. It is so simple to send a [redacted] config file and then tell the poor sods that now have useless devices that there is no fix apart from buying a new one that will probably suffer the same fate the day after the legally mandated warranty runs out.
My home is an IoT free zone and none of my electronic kit other than my computers/phone/tablet will ever be even indirectly connected to the internet.
11yr old Samsung TV
Glad it doesn't have any kind of connections to the intertubes. Not looking forward replacing it either.
Samsung quality software
Why is the XML parser not a separate sandboxed process called by the main process and if it dies the main process deletes the XML file and sends a message back to the mothership saying it wasn't processed because it was bad.
Not Surprised
I feel somewhat smug now for NOT connecting any of our Samsung ‘smart’ devices, TV, Blu-Ray, Freesat box, etc., to the Internet.
I wanted these devices and what they could do, but not the endless snooping that Sammy wanted.
I do feel really sad for the poor sods caught out by this. I try not to buy Sammy stuff these days.
Because of the above, but also...
I used to be a VAR for Samsung.
I specialised in combo Sammy DVD/VHS player/recorders and HDD recorders.
After I had a number (Dozens!) of them returned for repair because the customers dared to turn of the unit at the mains switch when not in use!
Repeated ‘inrush’ current spikes destroyed a part of the power supply and these needed fixing.
Cheers Sammy, you never did pay me for the extra transport costs on those units from the customer to me and back, nor the petrol to and from your dingy repair centre (then) in Brentwood - cost me a good £500 in losses.
Pissed-off customers is great for return business.
I stopped selling Sammy gear after that.
Never regretted it.