Diverse Vision

Enhanced Viewing Experience Using AR Technology

Tele-experience by synchronized presentation of UHDTV and free-viewpoint AR

NHK STRL has prototyped a system demonstrating a new viewing experience, where viewers watch a TV program on television while simultaneously experiencing free-viewpoint 3D AR content on their own mobile devices. For example, characters or actors in a program are displayed as life-size 3D objects that can be integrated into the viewer’s living room when viewed through a smartphone or tablet camera (Fig. 1). This makes the viewers feel as though the characters or actors have stepped out of the television screen. A video depicting this is also shown below.

Fig. 1: Examples of enhanced content viewing

Technical Features

In order to provide such an experience for a sports program or another live content, it is necessary to achieve the real-time transmission of 3D object data of changing shapes as the program progresses. As shown in Fig. 2, video and audio for TV are efficiently delivered via broadcast, while 3D objects are delivered through broadband internet. Video and audio over broadcast and 3D objects over broadband internet have different transmission delays. To synchronize the timing during playback, timing information is applied to the content as it is transmitted in real time. Therefore, the prototype system achieves

  • real-time streaming of geometry-based dynamic 3D objects over a broadband network,
  • UTC-based inter-terminal media synchronization mechanisms in combination with MMT (MPEG Media Transport)-based UHDTV broadcasting.
Fig. 2: Distribution model of synchronized UHDTV and AR content

Since the 3D AR objects are interactively viewed or viewed differently from one viewer to another, an object may be out of the user’s viewport or the object can be viewed with a small/large size (low/high resolution) depending on the individual user. Therefore, the prototype system adopts

  • an object-based multiplexing scheme that enables various functions such as object filtering according to individual viewpoints (Fig. 3).

To reduce the network load,

  • object filtering can be processed not only in a terminal device but also upstream of the network such as in a home gateway (Fig. 3).
Fig. 3: Object-based multiplexing and object filtering

NHK will continue the development of 3D media for true tele-experience, aiming to achieve this around 2030 to 2040. For more details, please visit the IBC conference program site; an on-demand presentation and technical paper with the title “Toward Tele-experience: Enhanced viewing experience by synchronized UHDTV and free-viewpoint AR” are available at https://www.ibc.org/technical-papers/ibc2020-technical-papers-xr/6690.article

See also