Ideally, a public service media would provide sign-language interpretation for all programs, enabling hearing-impaired viewers to enjoy every aspect of the content, but securing sufficient sign-language interpreters for all programs would be impractical. This led us to carry out R&D on automatically generated sign-language computer graphics (CG) animation. In particular, providing sign-language CG animation in live programs is a challenging task.
NHK STRL is currently developing technologies that will make it easier for hearing-impaired viewers to enjoy live sports programs with sign language using the game data provided in real time by the event organizers or other entities.
Please take a look at the video below for a glimpse of the service provided using our prototype.
Once the system receives sports game data, it extracts data such as player names, elapsed time, scores, and penalties, and inserts them into the prepared templates*1 of sign language sentences. Then, sign-language animation is generated from the completed sentences with pre-recorded sign-language motion capture data (Fig. 1). This is done in real time to convey the game status and progress. The animation is combined with the broadcast TV video and then sent through the internet to be presented on a PC web browser or a tablet app (Fig. 2). Both video and sign language are presented on the same screen together with the associated game data for better visibility of the relevant information at a glance.
We conducted trials on ice hockey and curling matches and confirmed that the automated sign-language CG allowed native signers to understand the progression of games. We will continue to improve the system to make the sign-language CG easier to understand.
In the future
We are trying to improve the CG avatar quality and expressive power to provide more natural and “humane” sign language. In sign language, movements other than the fingers, such as slight movements of the eyes, eyebrows, and facial expressions around the mouth, also play an important role in expression. The video below shows a sign-language CG avatar that has been improved by synthesizing slight movements and the newly captured skin texture. At this point, real-time rendering of this quality is not possible, but we expect to provide this quality in live programs in the future.