We are researching technologies for making use of the internet to provide more user-friendly and convenient broadcasting services. In our research on a media-unifying platform, in FY 2018, we developed a system that can link multiple presentation devices and a system that can present not only the main part of a program but also program-related information such as digest video and program website, and demonstrated their effectiveness. We also studied a technology for increasing responsiveness to achieve diverse viewing styles and comfortable viewing in video delivery. Moreover, we contributed to the domestic and international standardization of technical specifications for the advanced satellite broadcasting for 4K/8K launched in December 2018.

■Media-unifying platform

  Against the backdrop of the diversification of program-viewing methods including video delivery on the internet, we are researching a "media-unifying platform" that automatically selects appropriate media for viewing programs in accordance with the user situation. In FY 2018, we developed a system to realize the efficient selection of a program to be viewed according to the practical delivery environment and to meet viewers’ diverse intentions for content viewing.
  The media-unifying platform consists of servers that manage the content distribution status of each medium and media-unifying engines that automatically select appropriate media on the terminal side. For the content distribution status management server, we demonstrated that it is possible to select from programs of multiple operators and allow more flexible program selection by appropriately supporting content with different provision ranges such as home-use recorders and internet redistribution. We exhibited implementation examples at the NHK STRL Open House 2018 (Figure 3-1).
  On the basis of the functions of the media-unifying platform, in FY 2018, we developed a system that realizes a new viewing style by linking multiple devices that can be used for program viewing, such as program viewing enabled by a linkage between a smartphone, TV and smart speaker⁽¹⁾. We also extended the functions of the content distribution status management server, which previously allowed the selection of only the main parts of programs, to enable the selection of program-related information including digest video of programs, text content such as program websites and related SNS information. This achieved the provision of flexible content access according to the viewer situation including available viewing hours and degree of interest⁽²⁾ (Figure 3-2).
  We prototyped a system that links multiple viewing devices for program viewing and a system that allows the viewer to select program-related information in addition to the main part of a program and conducted evaluation experiments. Prior to the experiments, we asked 400 participants to respond to an online questionnaire for deciding evaluation items and collected data on the genres and related content of video they frequently view and their viewing behaviors. On the basis of the results, we decided the program genres to be used for evaluation and the scenarios of content viewing. The results of evaluation experiments participated by 50 general users showed that presenting program-related information increases viewers’ appetite for and interest in program viewing⁽³⁾ and that presentation using device linkage can offer a viewing style matching viewer preferences⁽⁴⁾, demonstrating the effectiveness of the prototype system.
  To promote the development and practical use of the media-unifying platform, we prepared a specification document describing cases of use, input/output specifications between functional elements and other information.

■Adaptive-switching shaping technology

  As a way of achieving both the stabilization of viewing quality in internet video delivery and high-speed response to viewing operations such as seeks, we developed a technology for adaptive switching of shaping (a technique that delays communication packets exceeding the speed limit to smooth the communication speed).
  The technology that we developed performs bandwidth control using shaping for viewing terminal communications on a traffic control device to prevent packet loss, thus stabilizing the viewing quality. It also cancels shaping temporarily for communications of a viewing terminal on which a user performs a viewing operation to deliver video data packets preferentially to the terminal. This reduces the delay before the start of reproduction (response time) and increases the response speed.
  We implemented this system into our traffic control device and video player and conducted delivery experiments. In the experiments, ten viewing terminals simultaneously accessed the distribution server connected with the traffic control device to view video and a viewer performed a seek operation during viewing on one of the terminals. We measured the viewing quality during simultaneous viewing by the multiple terminals and the response time to a seek operation for each of the developed method (adaptive switching of shaping) and the conventional method (constant application of shaping). The results of the experiments using shaping parameters set according to conditional expressions based on video bit rate and the number of viewing terminals showed that the developed method reduced the response time to about 1/10 to 1/9 that of the conventional method without deteriorating the viewing quality⁽⁵⁾.

■Smooth switching of program sound according to user operation

  We developed a video delivery technology that allows users to switch program sound smoothly according to their preferences for new user experiences while viewing video.
  The technology produces a video stream and multiple types of audio streams with different combinations of audio sources or different level balances in MPEG-DASH (Dynamic Adaptive Streaming over HTTP) format on the distribution side, and efficiently delivers the streams only selected by user operation on the reception terminal. We prototyped a distribution system and video player equipped with this technology and verified their operation on a Hybridcast-enabled receiver (Figure 3-3). The results of experiments on the switching of program sound by remote controller operation using an audio segment length (a unit of stream distribution) of 0.5 seconds demonstrated that the switching took about 1.2 seconds. We also confirmed that the stable and smooth switching can be achieved without any sound interruption or noise⁽⁶⁾.

■Server-side rendering technology

  We made progress in our development of a server-side rendering technology that enables the viewing of program-linked content that involves advanced graphics processing such as 3D CGs and 360-degree images on terminals with a limited CPU and memory such as TVs. This technology enables interactive viewing by performing image processing in accordance with the user operation on a rendering server on the cloud and transmitting the rendered images to the terminal at low latency using web real-time communication (WebRTC) technology.
  In FY 2018, we studied a function to automatically adjust the number of rendering servers to operate in accordance with the number of connected terminals with the aim of reducing the cost of content delivery servers using this technology. As a basic verification, we implemented an experimental system using this technology on a commercial cloud and conducted delivery experiments in which multiple terminals connected to a single rendering server simultaneously to view content including video. We equipped the rendering server with a graphics processing unit (GPU) and measured the usage rates of the CPU and the graphics processing part and decoding part in the GPU in accordance with the number of terminals. The results showed a tendency of the usage rates of the CPU and decoding part being proportional to the number of connected terminals, demonstrating that they are effective for estimating the upper limit of the number of terminals connectable to a single rendering server⁽⁷⁾.

■Promotion of video distribution for TV

  We continued with our work to promote video distribution for TV by using Hybridcast and other means. We previously developed a video player that operates stably on a TV receiver and made it available to the members of the MPEG-DASH Interoperability Study Group at the IPTV Forum under the name of "dashNX." The player has been utilized in 4K delivery experiments and other efforts involving many broadcasters. We also contributed to the establishment of specifications for 2K-enabled "Hybridcast video" and 4K-enabled "Hybridcast 4K video," which are the video distribution services for TV specified at the Forum, and the development of verification test for determining the conformity of receivers with the specifications. This effort led to the announcement of logos for indicating receivers and programs compliant with these specifications in May 2018⁽⁸⁾.
  In FY 2018, we developed a low-cost MPEG-DASH-compatible 4K live encoder which uses a High Efficiency Video Coding (HEVC) codec board and is based on a general-purpose PC to promote video distribution service using 4K live delivery. This encoder is capable of flexibly adding and changing functions such as adaptive bit rate and digital rights management (DRM) in accordance with PC specifications and service requirements for video distribution.
  Using this encoder, we prototyped a sports viewing application for 4K live delivery equipped with device linkage functionality (Figure 3-4). We also conducted 4K live delivery to a Hybridcast-compatible receiver by using this application at the NHK STRL Open House 2018 and demonstrated the feasibility of stable viewing with adaptive bit rate and smooth fast-reversing to a certain scene by operation from a remote controller and a smartphone linked with the receiver⁽⁹⁾. The development of this prototype and the 4K live delivery were conducted in cooperation with Asahi Television Broadcasting Corporation.

■Standardization activities for 4K/8K multimedia broadcasting and hybrid systems

  To expand and enhance the advanced satellite broadcasting for 4K/8K launched on December 1, 2018, we contributed to the revision of ARIB standards regarding device linkage functionality in data service and compression formats for the transmission of closed captions and superimposed characters⁽¹⁰⁾.
  In our international standardization effort, we prepared the additional description of device linkage functionality for ITU-R Recommendation BT.2075, which describes Integrated broadcast-broadband systems. We also contributed to the revision of ITU-R Report BT.2342, which describes closed-captioning schemes, to add descriptions on the use of ideographic variants and closed-caption file exchange formats. At the technical committee of the Asia-Pacific Broadcasting Union (ABU), we reported on domestic and international trends of the multimedia schemes, CAS technology and security in broadcasting to promote Japan’s broadcasting systems and Hybridcast.

 

[References]
(1)	H. Endo, S. Taguchi, S. Takeuchi, K. Fujisawa and K. Kai: "Multi-Device Media-Unifying Viewing System," ITE Annual Convention, 14D-3 (2018) (in Japanese)
(2)	S. Taguchi, H. Endo, S. Takeuchi, K. Fujisawa and K. Kai: "System Model of Media-Unifying Platform for Various Types of Content," ITE Annual Convention, 14D-4 (2018) (in Japanese)
(3)	H. Endo, S. Taguchi, S. Takeuchi and K. Fujisawa: "Evaluation of Multi-Device Cooperation System for Content Viewing Independent of Distribution Media," IPSJ SIG Technical Report, Vol.2019-HCI-182, No.24 (2019) (in Japanese)
(4)	S. Taguchi, H. Endo, S. Takeuchi and K. Fujisawa: "Evaluation of the System Providing Alternative or Related Content of TV Program," IEICE General Conference, D-21-7 (2019) (in Japanese)
(5)	M. Kurozumi, S. Nishimura and M. Yamamoto: "A Study of Response Improvement in Video Delivery by Traffic Shaping Policy According to User Operation," ITE Annual Convention, 14D-2 (2018) (in Japanese)
(6)	S. Mori, H. Kubo, S. Oode, S. Takeuchi and S. Nishimura: "A Study on a Smooth Switch Method for User-Selected Sound Sources with Video Streaming for TV Receivers," ITE Winter Annual Convention,13C-2 (2018) (in Japanese)
(7)	M. Ohnishi, S. Nishimura and K. Fujisawa: "Evaluation of content delivery system for HTML5 compliant TV receiver using cloud rendering," ITE Winter Annual Convention, 13C-1 (2018) (in Japanese)
(8)	S. Nishimura: "R&D Trends in Video Streaming for TV," NHK STRL R&D, No.171, pp.20-31 (2018) (in Japanese)
(9)	T. Nakai, K. Hiramatsu, H. Terada, Y. Yamano, and S. Nishimura: "Development of a Prototype Application Using 4K Live Streaming," ITE Annual Convention, 32D-1 (2018) (in Japanese)
(10)	S. Takeuchi and M. Ohnishi: "A Study on Compression Format of Closed-caption Data in UHDTV Satellite Broadcasting Service," ITE Winter Annual Convention, 13C-3 (2018) (in Japanese)