Space-sharing Content Viewing System

A viewing experience sharing a space with someone at a remote location through VR/AR technology

We are conducting research on services that provide experiences beyond the framework of television, and connect people more deeply through content. Here, we present a system that uses augmented reality (AR) and virtual reality (VR) technologies to make a user to feel they are experiencing content in the same space together with another person who is at a different location.

The framework of the Space-Sharing Content Viewing System

Composing 3D images of each other in a space with AR/VR content

AR and VR content video is shown in the user’s own head-mounted display (HMD) (left in the figure), composed with a life-sized, 3D image of their “friend” at a different location (right in the figure). At the same time, a 3D image of the user is also shown in the friend’s HMD, composed with the same AR/VR content. The 3D images of each user are captured in real time using cameras with depth sensors and transmitted as a point cloud* to the other user through broadband communication.

Providing interactive experiences

With this system, we have implemented a mechanism that enables people at different locations to manipulate the same content interactively, stimulating communication and enabling them to enjoy the content more deeply. Controller data from each user is sent to the other, and the content they are viewing is synchronized, creating an environment in which they can share and manipulate the same content.

* Point cloud: A data set of 3D point coordinates (x, y, z) and color data (R, G, B).

The system was proposed as a “Living room of the future,” and creates an environment in which the person at the distant location is displayed in life-sized 3D images beside the viewer and the viewers can talk and work together. For this exhibit, we have created content that enables viewers to experience vicariously, what the presenters experienced in the program, “Fresh Encounters with our Cultural Heritage” broadcasted in NHK's 8K Satellite Channel. Viewers can collaborate with their friends or family at distant locations to learn more about the “Goggle-eyed dogu (clay figurine) ,” by shining a flashlight on what they want to look at or exploring inside the figurine together.