Chinese TV uses AI to translate broadcasts into sign language. It’s not going well
- Reference: 1752121961
- News link: https://www.theregister.co.uk/2025/07/10/china_ai_sign_language_translation/
- Source link:
Writing at an outlet called Sixth Tone, professor Zheng Xuan notes that China is home to 20.5 million people with hearing disabilities, and that the nation’s government has encouraged use of avatars and virtual presenters to provide real-time translation of some television programs.
Techno-optimists believe these flaws will be solved, but we shouldn’t ignore the ethical harm
Those efforts started at the 2022 Beijing Winter Olympic Games, which Zheng studied to assess the quality of the sign language translations – and came away unimpressed.
“We transcribed and back-translated the sign language created by the avatars, then compared the results with the original audio, finding that a significant amount of key information was lost or distorted in the AI-generated version,” she wrote.
Deaf people struggled to understand the AI-generated sign language.
[1]
“On closer inspection, the movements of the avatars differed considerably from everyday sign language in terms of hand shape, position, direction, and movement,” she wrote. “Other issues were even more prominent — the avatars’ facial expressions and body language were off, and their mouth movements were distorted.”
[2]
[3]
In interviews, viewers reported “they generally couldn’t understand the avatars’ movements and noted that they seemed to have a limited vocabulary, while struggling to handle words with multiple meanings.”
Zheng thinks the AIs did badly because “Chinese words cannot be found for the meanings expressed by 50 percent of gestures in Chinese sign language.”
[4]
Developers overlooked “the difference between signed and spoken language. In particular, many perceive sign language as an accessory to spoken language, or else believe that translating between the two is similar to translating between two spoken languages.”
“But the modalities of spoken language and sign language are quite different,” she wrote. “The former is an oral-auditory language, while the latter is a visual-gestural or visual-spatial language. The term ‘gestural’ is a relatively broad concept that includes not just hand movements, but also facial expressions and body language. Full utilization of the body in space allows sign language users to express the meaning of an entire sentence – such as ‘a person walks into a room’ – with just one action.”
The diversity of Chinese sign language is also problematic. Zheng wrote that Chinese sign language includes “‘natural sign language’,” which originates from the daily lives of Deaf people, and ‘signed Chinese language’, which is an expression of Chinese characters using signs.”
[5]
“The language used by most Chinese Deaf people lies somewhere on a spectrum between the two,” she wrote, adding that dialects of sign language are another complication for developers.
Building a translation avatar to handle all those nuances is hard and made more difficult by the scanty supply of useful data to inform AIs.
[6]Top AI models - even American ones - parrot Chinese propaganda, report finds
[7]AI-hosted infomercial shifts $7.5 million worth of product in China
[8]Canada orders Chinese CCTV biz Hikvision to quit the country ASAP
[9]China spawns an x86 supercomputing monster, with an AMD connection
Zheng also criticized Chinese technology companies, which she said “have not involved sign language linguists or Deaf people in any great depth. Even in the cases where sign language teachers or interpreters are included, developers often only slot them into supporting roles, instead of taking the opinions of Deaf users as the final arbiter of their products’ effectiveness.”
That’s not just a jab: Zheng worked as consultant for a team working on translation avatars and found them poorly equipped for the task.
“They seemed to underestimate the difficulty involved, overestimated the power of tech to solve problems, and lacked the necessary experience, resources, and ability to judge the quality of work done by third-party companies,” she wrote. “By the time I joined the project, these shortcomings had already become apparent. Although the development team welcomed my participation, I felt that this respect was more for my technical knowledge as a university professor rather than my identity as a Deaf person.”
Zheng said that when she pointed out the product wouldn’t satisfy users, “my feedback was not fully embraced, as the developers seemed unable to fully empathize with my frustrations.”
She now feels there are “fundamental issues with the way tech approaches the problem of sign language translation” because “Tech companies are used to first launching a version that has a lot of bugs, then optimizing it through a large amount of user feedback.”
The products she’s seen were so bad, she fears they harm the deaf community’s faith in technological solutions.
“That’s not to mention the fact that some companies mislead users by promoting their products using real humans rather than avatars, and then release an immature generative AI version,” she wrote. “Techno-optimists may believe that these flaws will all be solved with time, but we shouldn’t ignore the irreversible ethical harm: If the real needs of Deaf users are not responded to, they’ll feel that they’re being treated as guinea pigs.” ®
Get our [10]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2aG-POFSstHF5tDqbwiS8ZgAAAJg&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aG-POFSstHF5tDqbwiS8ZgAAAJg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aG-POFSstHF5tDqbwiS8ZgAAAJg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aG-POFSstHF5tDqbwiS8ZgAAAJg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aG-POFSstHF5tDqbwiS8ZgAAAJg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2025/06/26/top_ai_models_parrot_chinese/
[7] https://www.theregister.com/2025/06/23/asia_tech_news_in_brief/
[8] https://www.theregister.com/2025/06/30/asia_tech_news_roundup/
[9] https://www.theregister.com/2025/05/27/hygon_sugon_china_x86_supercomputing/
[10] https://whitepapers.theregister.com/
Why are they using sign language
Rather than captioning? If it is AI based it still has the same "did it understand the spoken words correctly?" issue, but at least you don't add the extra layer of doing a poor job of representing the hand signs.
AI speech to text at least for English isn't perfect but it is leaps and bounds better than it used to be. It isn't ready to wholesale replace human captioners but their days are clearly numbered.
Re: Why are they using sign language
Captioning is already omniprésent on Chinese TV, as it is needed for understanding by all viewers who have a wide range of dialect.
The article hints at it, I'd say the need is one of practice to standardize on a single sign language as much as possible. Otherwise, every locale will develop its own variations.
Re: Why are they using sign language
the need is one of practice to standardize on a single sign language
I think you're making the same mistake that's being complained of. Sign languages are not an assistive construct and do not necessarily map to the vocabulary or grammatic constructs of the spoken language(s) that share their geographical locus - they are languages in their own right and develop and change like every other living language. Leave deaf children from different backgrounds together with no formal instruction and they will develop a common sign language quite quickly.
It's rather like suggesting that we should all speak English identically. Though, ironically, if broadcast signing were more common, presumably it would lead to a gradual homogenisation of the various dialects much in the same way that spoken English dialects and accents are becoming less divergent.
Re: Why are they using sign language
Yes. Sign Language has similarities to Linux distros. Most are based on two roots (like RHEL & Debian). British & French Sign Language which are very distinct from each other. American (ASL) is based on French for historic educational reasons. Most of the former empire is based on BSL. In Ireland the people from the north mostly use BSL while Irish Sign Language in the South is based on FSL.
Hence there is a discontinuity between the type of Sign Language used and Spoken Language is the USA & Ireland (both sign the French way but speak English). You can't assume one from the other. The downstream distro languages have developed their own adaptations and flavours.
As you point out even signers from the two different strands which developed independently can quickly tune in to the other which contrasts with many of us English speakers who, even after years of school French - find it impossible to follow a true French speaker.
Finally captioning is akin to SSE (Sign Supported English) in roughly following spoken words and BSL (British Sign Language) which uses completely different spacial and grammar paradigms to spoken English even if they share many signs. Fuller explanation here: [1]https://bda.org.uk/the-difference-between-bsl-sse/ .
[1] https://bda.org.uk/the-difference-between-bsl-sse/
Re: Why are they using sign language
Try turning off the sound and then putting up captions in your second language?
Sign Language is a different language, and if it's your first language (as it is for many Deaf people) then any captions are necessarily in a second language.
What’s the sign for
Cartoon bear standing in front of tanks in Tiananmen Square?
Re: What’s the sign for
That's where "a significant amount of key information was lost or distorted" becomes a particularly pertinent phrase.
Re: What’s the sign for
A talk by Elon Musk, signed by Grok. Probably wouldn't need many gestures
A little puzzled
It seemed to me that the process was transcribing the speakers words into a chinese text representation (speech recognition) then using AI to convert that captured text into sign language.
If the speech recognition was doing its job reasonably well, why wasn't the speaker just subtitled with the text ?
Translating from the spoken language to sign language was always going to be a challenge for any technology and likely a bridge too far for AI/LLM.
Pretty much the case for professional interpreters who must not only translate what is said but also attempt to convey what was meant from a speaker in one language to the listener(s) in another language. (And in both directions.)
Re: A little puzzled
Because captioning in someone's second language isn't the route to best accessibility.
"a significant amount of key information was lost or distorted"
Yup. Sounds like AI all right.