Sign Language CG Generation from Japanese News Scripts

NHK is researching technology for generating sign language computer graphics (CG).

Here, we exhibit a system that generates sign language CG from input Japanese news sentences.

Objective

Our goal is to provide easy-to-understand information using sign language CG for hearing-impaired individuals whose native language is sign language.

Challenge

The challenge is reproducing the essential linguistic elements of sign language in CG animation, including rhythmic factors such as pauses and speed, mouth movements, and use of space.

Features

• We are translating Japanese into sign language word sequences using neural machine translation.
• The translation AI is trained using a parallel corpus of approximately 150,000 sentence pairs of Japanese and sign language created based on the NHK sign language news program.
• The system smoothly concatenates individual words recorded by motion capture according to the translated sign language word sequences.