1.2 AR (Augmented reality) / VR (Virtual reality)
We studied the concept of services that will offer new user experiences to viewers by using augmented reality (AR) and virtual reality (VR) technologies from two different approaches: “By AR/VR,” which provides new viewing experiences by combining existing technologies, and “For AR/VR,” which implements technologies that have yet to be introduced and newly developed technologies. We also developed a delivery technology for volumetric data to realize AR/VR content delivery using the convergence of broadcasting and telecommunications.
“By AR/VR” approach
As the concept of a new TV viewing service utilizing AR technology, we investigated a space sharing service that allows the viewer to enjoy watching TV together with family and friends at distant locations while feeling the presence of TV performers nearby as if they have come out of the TV screen. In this service, 3D images of TV performers, family and friends at distant locations and the viewer himself/herself in the past are synthesized and displayed in their actual size in a space viewed through AR glasses or a tablet terminal. We proposed the viewing style in which the viewer watches TV while sharing the space with TV performers, family and friends beyond space and time and demonstrated the service at exhibitions such as the NHK STRL Open House 2019 and IBC 2019 (Figure 1-10).
While the space sharing service offers a viewing style in which TV performers and others come to the viewer’s living room, we also proposed another viewing style in which the viewer and family or friends in the same space go to a virtual space and share 360 images. We prototyped a system that allows the viewer to enjoy 360 images while interacting with persons and objects nearby (Figure 1-11) using extended VR technology(1). This technology displays the persons and objects near the viewer by cutting out their images captured with a stereo camera installed on the viewer’s head-mounted display (HMD) while simultaneously displaying 360 images in the remaining area.
To realize the space sharing service, we conducted a survey of actual viewing conditions regarding viewing positions and the sense of space sharing to see where virtual persons should be displayed for effective communication. The results suggested that the front-back relationship of viewing positions may exercise an effect on the perception of viewing images together(2).
“For AR/VR” approach
We investigated a viewing style that utilizes high-resolution 360 images with the goal of realizing a viewing experience that allows the viewer to view images of any directions beyond the limits of a TV screen and enjoy an excellent sense of immersiveness and presence that conventional TVs cannot provide. At the NHK STRL Open House 2019, we produced 180 images with a high resolution of about 12K, exceeding 8K, by integrating (“stitching”) images captured with three 8K cameras arranged radially. Using eight 4K projectors (Figure 1-12), we projected the high-resolution VR images on a large 180-degree cylindrical screen (Figure 1-13), which was enjoyed by many visitors(3). We also reproduced highly immersive sound suitable for 180 images using multiple loudspeakers installed above and below the cylindrical screen for sound reproduction. In addition, we exhibited mock-ups of viewing styles with a high-resolution HMD and a dome-type display for personal viewing (Figure 1-14).
Transport technology for volumetric data
One example of AR/VR content taking advantage of integrated broadcast-broadband may be a service that allows the viewer to view an object in TV images from a free angle using AR technology by delivering the volumetric data of the same object to AR glasses or a tablet terminal through broadband networks. To realize such services, we are researching a transport technology for volumetric data.
We newly developed a real-time transmission technology for live-action volumetric data of human objects(4)(5) that could be used for live sports coverage and other live programs in the future. Since this technology can transmit live-action volumetric data of 30 frames per second at about 100 Mbps in real time to present AR content, the viewer does not need to download the volumetric data to his/her viewing terminal in advance. To prevent the gap in presentation timing between TV images and AR content due to the transmission latency difference between broadcasting and broadband, we employed a mechanism for adding an absolute time stamp to each frame of volumetric data before transmission and referring to the time stamp at the time of presentation, in a similar way to MPEG Media Transport (MMT) used for the advanced satellite broadcasting for 4K/8K. This enabled a service that ensures high-accuracy synchronization between TV images and AR content (Figure 1-15).
We produced AR content provided by integrated broadcast-broadband system (Figure 1-16) by capturing the 4K video and volumetric data of a TV performer simultaneously and offered visitors an opportunity to experience content viewing with free-viewpoint AR at the NHK STRL Open House 2019. This content presents the AR images of the performer wearing different colorful outfits in various settings in her real size in front of the TV screen. In addition to the NHK STRL Open House, we exhibited the content at the NHK Yamagata Station Open House in October, the NHK Showcase in ABU Tokyo 2019 General Assembly & Associated Meetings in November and a program session at ITE Winter Annual Convention in December, and offered many visitors the experience of a new viewing style combining TV and AR technology.
