NHK STRL has prototyped a system demonstrating a new viewing experience, where viewers watch a TV program on television while simultaneously experiencing free-viewpoint 3D AR content on their own mobile devices. For example, characters or actors in a program are displayed as life-size 3D objects that can be integrated into the viewer’s living room when viewed through a smartphone or tablet camera (Fig. 1). This makes the viewers feel as though the characters or actors have stepped out of the television screen. A video depicting this is also shown below.
In order to provide such an experience for a sports program or another live content, it is necessary to achieve the real-time transmission of 3D object data of changing shapes as the program progresses. As shown in Fig. 2, video and audio for TV are efficiently delivered via broadcast, while 3D objects are delivered through broadband internet. Video and audio over broadcast and 3D objects over broadband internet have different transmission delays. To synchronize the timing during playback, timing information is applied to the content as it is transmitted in real time. Therefore, the prototype system achieves
- real-time streaming of geometry-based dynamic 3D objects over a broadband network,
- UTC-based inter-terminal media synchronization mechanisms in combination with MMT (MPEG Media Transport)-based UHDTV broadcasting.
Since the 3D AR objects are interactively viewed or viewed differently from one viewer to another, an object may be out of the user’s viewport or the object can be viewed with a small/large size (low/high resolution) depending on the individual user. Therefore, the prototype system adopts
- an object-based multiplexing scheme that enables various functions such as object filtering according to individual viewpoints (Fig. 3).
To reduce the network load,
- object filtering can be processed not only in a terminal device but also upstream of the network such as in a home gateway (Fig. 3).
NHK will continue the development of 3D media for true tele-experience, aiming to achieve this around 2030 to 2040. For more details, please visit the IBC conference program site; an on-demand presentation and technical paper with the title “Toward Tele-experience: Enhanced viewing experience by synchronized UHDTV and free-viewpoint AR” are available at https://www.ibc.org/technical-papers/ibc2020-technical-papers-xr/6690.article
- Y. Kawamura, T. Kusunoki, Y. Yamakami, H. Nagata, and K. Imamura, “Toward Tele-experience: Enhanced viewing experience by synchronized UHDTV and free-viewpoint AR,” IBC2020 Conference Proceedings.
- Y. Kawamura, “Real-time Transmission of Free-viewpoint AR Content,” NHK STRL Bulletin, Broadcast Technology, No. 80, Spring 2020.