Space-Sharing Content Viewing System Using VR and AR

NHK STRL has developed a space-sharing viewing system utilizing virtual reality (VR) and augmented reality (AR) technologies that enables viewers to view and share contents as if a person in a distant location is together with them in the room. A video of how it works and the experience is shown below.

A viewer, for example, can experience VR images of an aquarium with the feeling that a friend, who is actually in a distant location, is sitting beside them (Fig. 1). The 3D images of the friend in the distant location are composed with the aquarium VR images and projected onto the viewer’s head-mounted display (HMD), creating an experience that feels like the friends are together at the aquarium.

The 3D images of the friend in the distant location are captured in real time using a camera with depth sensor and transmitted to the viewer in the form of point clouds (Fig. 2). Similarly, 3D images of the viewer are also captured, sent to the friend, composed with the aquarium VR images, and displayed. Thus, they can view contents while communicating with each other by conversation, hand gestures, and body language.

AR technology combined with the use of a camera-equipped head-mounted display can be used to compose 3D images of performers as if they are in the viewer’s own room (Fig. 3). These 3D images can also be viewed while communicating with a friend at a distant location.

We will continue R&D on technologies for presenting content so that viewers in distant locations can view it together using this system, with the goal of realizing new viewing styles for the future.

See also

• K. Yoshino, “AR/VR-based space-sharing services: Watching programs together with performers and distant friends and family members,” NHK STRL Bulletin, Broadcast Technology, No.81, Summer 2020.