Research Area

1.6  Video coding

  We are researching video coding techniques for full-featured 8K SHV and SHV terrestrial broadcasting.


8K 120-Hz HEVC encoder

  We are developing an encoder that supports 8K/120-Hz video (Figure 1-8). The encoder, which consists of twelve 4K/60-Hz encoding units, is capable of real-time coding of 8K/120-Hz input video by parallel processing. This system conforms to the Main 10 profile using the HEVC/H.265 scheme and supports 4:2:0 10-bit coding of 8K/120-Hz video.
  Its bitstream is compliant with ARIB standard STD-B32 Version 3.9, which allows not only an 8K/120-Hz decoder but also a decoder for 8K test broadcasting (an 8K/60-Hz decoder compliant with ARIB standard STD-B32 Version 3.9) to partially decode the 60-Hz sub-bitstream. The system also supports video usability information (VUI) parameters for HDR that are compliant with the above standard.
  Since the encoder divides the video frame into four vertical slices for parallel processing, a degradation in image quality tends to appear around the boundaries between divided slices of video, especially when the video has vertical motion. To suppress the degradation, we employed a design that reduces 8K/120-Hz video to 4K/60-Hz video and analyzes the reduced video in advance to control the entire coding. We developed technologies for increasing the image quality, which include a technique to control the quantization value of the boundary areas based on the amount of motion predicted by preliminary analysis. We conducted subjective evaluations using a software simulator to verify the effectiveness of these technologies and confirm the coding quality(1). This research was conducted in cooperation with Fujitsu Laboratories Ltd.
  As part of R&D on 120-Hz video coding, we produced evaluation images that are helpful for evaluating the performance of the processing of fast-moving images and coding control in cooperation with NTT (Nippon Telegraph and Telephone Corporation).



Figure 1-8. Appearance of 8K/120-Hz encoder

8K 120-Hz HEVC decoder

  We are developing a decoder in parallel with the encoder. Our decoder consists of a software decoder that operates on a general-purpose workstation and an interface converter. In FY 2016, we implemented a video decoder unit of the software decoder. In FY 2017, we implemented its audio decoder unit and TS input unit. This enabled the real-time decoding of TS signals of 8K/120-Hz video and 22.2 ch sound. Decoded 8K/120-Hz video is generated from eight spatiotemporally divided DisplayPort outputs and converted by the interface converter to signals for a single U-SDI. Decoded audio signals are multiplexed with video signals for output.


Development and standardization of next-generation video coding technologies

  We are developing advanced high-efficiency video coding technologies for next-generation terrestrial broadcasting. For intra-frame prediction technology, we developed a method for the high-precision prediction of chroma signals using decoded luma samples in intra prediction and a method for improving entropy coding for chroma intra prediction modes(2). For inter-frame prediction technology, we developed a motion compensation method considering the continuity with the motion vector of neighboring blocks and a method for predicting the motion vector adaptively according to the shape of partitioned coding blocks. We also developed a way to improve the entropy coding of transform coefficients by estimating the residual signal energy and a deblocking filter control method that reduces significant coding degradation in HDR-format video. We confirmed that these technologies improve coding efficiency and proposed some of them to an international standardization conference on next-generation video coding as prospective elemental technologies for advanced video coding.
  To promote the performance improvement of future video coding schemes for HDR video, we provided the JCT-VC and JVET international standardization conferences with test sequences of the HLG format and also jointly proposed efficient coding settings for using HEVC as comparative criteria with the BBC. These settings were adopted as the criteria for the performance comparison of future video coding schemes. They were also reflected in HEVC video coding guidelines for HDR coding (ISO/IEC TR 23008-15 | ITU-T H. Sup.18)(3)-(5).
  As an overseas research effort, we developed HDR tone-mapping nonlinear functions suited for video coding in cooperation with Universitat Pompeu Fabra and demonstrated the improvement in coding efficiency(6).


Application of machine learning to coding tools

  As an initial study to explore the feasibility of applying machine-learning-based coding tools to video coding, we conducted a basic evaluation on post-filters and intra prediction. We built a post-filter using a machine learning method based on convolutional neural networks and demonstrated that it can reduce mosquito noise and improve the PSNR. We also developed an intra prediction tool based on a multilayer perceptron, which is a type of neural network. The results of training and evaluation showed that it is possible to develop a single predictor that behaves as if it is equipped with multidirectional and planar prediction modes, indicating the feasibility of a new, efficient intra prediction tool.
  Also, in cooperative research with Meiji University, we developed an intra prediction process using a neural network consisting of two convolutional layers and two fully connected layers and confirmed that it can increase the speed of a prediction mode that uses prediction samples and adjacent reference samples as an input(7).


Enhancement of super-resolution technology and its application to video coding

  In our research for enhancing super-resolution technologies, we developed a technique for super-resolution reconstruction from 2K to 8K that uses an alignment and assignment method considering the frequency band by a registration process between wavelet multiscale components. The new technique achieved a higher speed and higher image quality than conventional methods(8)(9).
  We are also studying a way of applying super-resolution reconstruction to video coding technology. As inter-frame prediction images, we newly introduced blurred prediction images and super-resolved prediction images that use registration super-resolution between wavelet multiscale components and observed improvements(10).


Noise reduction and band limitation equipment

  We developed noise reduction and band limitation equipment that performs a pre-coding process to increase coding efficiency. The equipment applies shrinkage functions in each element position after the wavelet-packet decomposition of each frame and controls the amount of shrinkage according to the band limitation frequency and the pixel level, enabling a high-precision noise reduction and band limitation process(11). This research was conducted as a government-commissioned project from the Ministry of Internal Affairs and Communications titled “R&D on Advanced Technologies for Terrestrial Television Broadcasting.”


[References]
(1) S. Iwasaki, Y. Sugito, K. Chida, K. Iguchi, K. Kanda et al.: “Subjective Evaluation of 8K120Hz Encoder Simulator,” Proceedings of the 2017 IEICE General Conference, D-11-7 (2018) (in Japanese)
(2) S. Iwamura, S. Nemoto and A. Ichigaya: “Redundant flag removal on chroma intra mode coding,” JVET-H0071 (2017)
(3) S. Iwamura, S. Nemoto, A. Ichigaya and M. Naccari: “Analysis of 4K Hybrid Log-Gamma test sequences,” JVET-F0094 (2017)
(4) S. Iwamura, S. Nemoto and A. Ichigaya: “Candidate rate points of HLG material for anchor generation,” JVET-G0103 (2017)
(5) S. Iwamura, S. Nemoto, A. Ichigaya and M. Naccari: “On the need of luma delta QP for BT.2100 HLG content,” JVET-G0059 (2017)
(6) Y. Sugito et al.: “Improved High Dynamic Range Video Coding with a Nonlinearity based on Natural Image Statistics,” International Journal of Signal Processing Systems, Vol.5, No.3, pp.100-105 (2017)
(7) T. Toyozaki, Y. Shishikui and S. Iwamura: “A Study on intra prediction mode decision method using deep learning,” Proceedings of the 2017 IEICE General Conference, D-11-56 (2017) (in Japanese)
(8) Y. Matsuo and S. Sakaida: “Super-Resolution for 2K/8K Television by Wavelet-Based Image Registration,” Proceedings of IEEE GlobalSIP, GS IVM-P.1.4, pp. 378-382 (2017)
(9) Y. Matsuo, A. Ichigaya and K. Kanda: “Super-Resolution from 2K to 8K by Registration of Wavelet Multi-Scale Components,” Picture Coding Symposium of Japan 2017 (PCSJ2017), P-2-15, pp. 86-87, (2017) (in Japanese)
(10) Y. Matsuo and S. Sakaida: “Coding Efficiency Improvement by Wavelet Super-Resolution Restoration for 8K UHDTV Broadcasting,” Proceedings of IEEE ISSPIT (2017)
(11) Y. Matsuo, K. Iguchi and K. Kanda: “Coding Efficiency Improvement by Band-Limitation Equipment for Advanced Digital Terrestrial TV Broadcasting System,” Proceedings of the 2017 ITE Winter Convention, 14C-4, (2017) (in Japanese)