Archive Search System Using Image Analysis Technology

2.1 Automatic object recognition technology

The automatic object recognition technology automatically determines what is depicted in the video by image analysis. It was introduced in our verification-test with the intention of resolving the issues discussed in Section 1, namely, the high cost of applying keywords, the inaccuracy of keywords taken from captions, and the variation in expressions used for keywords.

One approach to automatic object recognition is to prepare a set of example and counter-example^*3 images of each object, and then use a recognition model (such as a support vector machine) that learns to identify objects from the differences in image characteristics in the images (Fig. 3). For image features, sets of local image features, such as scaleinvariant feature transforms (SIFTs)²⁾, expressed as a “bag of features” (BoF)³⁾, are widely recognized as an effective approach.

One local feature method that has attracted attention recently is called geometric phrase pooling (GPP)⁴⁾. It can compute more robust and expressive image features than earlier methods by considering brightness and edge^*4 gradients in the target area and surrounding areas. However, with GPP, issues remain in that broad color and texture^*5 characteristics are not represented.

For our verification-test, we used an approach of combining local features based on GPP and broad features such as color and texture⁵⁾⁶⁾. Figure 4 shows how image features are computed by our method. In the figure, locality-constrained linear coding (LLC) quantization is a feature vector quantization method that uses local constraints^*6 in the feature vector space, and the color moment feature uses the average, variance and covariance of each color component. The Haar wavelet feature is a texture feature based on the brightness in neighboring rectangular regions, and the local binary pattern (LBP) feature is a texture feature based on the relative magnitudes of each pixel to its neighboring pixels. Our method computes an overall feature vector for the image using local feature values computed by GPP from feature points and the surrounding region, as well as broad features computed from wider areas. These are then integrated by spatial pyramid matching (SPM)^7)*7 (Fig. 5).

Figure 6 shows examples of object recognition using these image features applied to the data set from TRECVID⁸⁾, which is a large-scale international workshop on video analysis. A support vector machine, as described above, was used as the recognition model. The top 20 images (in order from highest probability, as in Fig. 3) for “flag” in Fig. 6 (a) and “boat” in Fig. 6 (b) are shown, and those marked with a red circle were judged to be correctly recognized. In Fig. 6 (a) and 6 (b), there are 19 and 17 correct images, respectively, demonstrating high recognition accuracy. Details regarding this technology can be found in the reference documents⁹⁾.

2.2 Visually-similar-image search technology

As discussed in Section 1, very little content has associated text information at the level of individual shots, so most cuts cannot be found using a keyword search. This issue can be mitigated by using the automatic object recognition technology discussed in Section 2.1, but there is limited diversity in the object keywords that can be applied. Thus, a visually-similar-image search technology, which can find cut images that are similar in color, texture or overall layout without referring to keywords, is also useful.

Program producers, who are the users of archive searches, tend to focus their searches on the objects portrayed as well as the layout. However, earlier techniques¹⁰⁾¹¹⁾ of dividing images into a grid of blocks and comparing image features (Fig. 7) are not capable of absorbing small differences in layout, so search results do not always meet with the user’s intentions. In the example in Fig. 7, the user was searching for a layout and an object (apple), but the earlier method judged the image of a lemon to be the most similar.

In our verification-test, we place some blocks over the object region rather than on the region grid, as shown in Fig. 8, and increase the weight given to these blocks when computing the image similarity¹²⁾¹³⁾. Object regions are estimated using a saliency map¹⁴⁾, which indicates how much regions in the image stand out. Our method can absorb small differences in layout and preserve the overall similarity of images while emphasizing the similarity of the objects in the images.

2.3 High-speed search for visually-similar-images

We demonstrate searches on large-scale image data sets containing millions to 10 millions of images. For each visually-similar-image search, the similarity of feature vectors between images must be computed, but with 10 million images, the search process could take up to 30 minutes, so ways to increase this speed were needed. One approach is to load the feature vectors for all images into memory when the search engine starts up, but this would make the startup time long and require a large amount of memory.

We adopted a method¹⁵⁾ in which the images being searched are first classified hierarchically by their overall composition and color characteristics. Then, as shown in Fig. 9, only the feature vectors for sets of images in groups that have similarity to the query image^*8 above a certain threshold are loaded into memory when performing a search.

This enabled us to reduce the search space, increase the speed and also reduce the memory required. With this technique, approximately 10 million images could be searched in five to ten seconds. An overview of the classification process in Fig. 9 is given below.

First, composition data for all images to be searched is computed using the classification of block regions based on their color features, as in Fig. 10 (a). Then, the images are classified into composition groups based on the similarity of composition data to approximately 80 composition templates. These composition templates were prepared manually, considering composition variations used in broadcast video. Examples of composition templates are shown in Fig. 10 (b).

Then, within each of the composition groups generated, images are classified into color groups based on the similarity of overall color features. The color feature used here is the concatenation of color feature vectors of all blocks in the image. The k-means method^*9 was used for this classification, with the number of groups being set in the range of 20 to 200 depending on the number of images in the composition group.

3.1 System overview

An overview of the system used in our verification-test is shown in Fig. 11. The new search system applying the technologies introduced in Section 2 (hereinafter called the “image search system”) runs on a separate server from the archive information system, and a separate screen is launched using a link placed on the archive information system screen. Operation in this manner avoids the possibility of affecting the archive information system, even if a fault should occur in the image search system. There is also a mechanism for showing bookmarks registered^*10 in the image search system on the archive information system, and video can be played back on the image search system. This enables users to use the image search system in their real work.

For this operation, the image search was applied only to content in the archive information system that had an associated structure table giving timing information for each shot, that had images for each shot, and that can be displayed to all users.

The system is expected to be used in real work, so it is important that the data be “fresh”. Thus, the latest content was added each month, as shown in Table 1, and by the end of the verification-test, the system was able to search approximately 100,000 content items and 10 million shot images.

Processes that enable the search, including computing the image features, generating metadata (keywords, etc., associated with images), and creating indices^*11, were performed automatically by a metadata generation system based on the Metadata Production Framework (MPF)^16)*12.

3.2 Functional overview

In addition to a conventional search by program title or keyword, the top screen of the image search system [Fig. 12 (a)] provides the results of a search by a similar shot image by the upload of a query image using the technology discussed in Section 2.2. The availability of an additional screen allows a search for shots by selecting terms from among 50 object keywords that have been automatically applied using the technology discussed in Section 2.1. The archive information system structure display screen [Fig. 12 (b)] also allows a similar shot image search using the technology described in Section 2.2, where shot images in the structure table are utilized as query images. Selecting search results displays a screen with details of the content containing the shot [Fig. 12 (c)]. This screen enables a search of the content for shot images that are similar to the specified shot using the technology described in Section 2.2.

4.1 Usage status

During the verification-test period (late January to early May 2015), the image search system was accessed effectively (resulting in search operations) 1,538 times by 909 people, generating a total of 2,103 search operations. 278 people were “repeaters” accessing the system and conducting searches multiple times. Users were from all NHK stations, totaling over 40 stations, not just the NHK Broadcast Center in Shibuya, Tokyo, so the system was used more widely than expected. By function, searches by uploading an image and by selecting a shot from the structure table on the archive information system had the highest usage rates. This shows that in addition to the conventional keyword search, there is demand for visually-similar-image search technology that does not use keywords.

4.2 User Evaluation and Discussion

The use of the image search system by real users was monitored as follows to evaluate the effectiveness of the system.

• Evaluation period: April 13 to 23, 2015
• Monitored users: Five directors of general programming and news, five program production staff members specializing in searches
• Method: Each person was interviewed for about 30 minutes while they were conducting real search operations

The main comments received in the interviews are shown in Table 2. Overall, users indicated that there was a strong need for this technology. We now discuss the effects of the new technology in light of the interview results.

The main issue raised regarding the current archive information system was the difficulty of conducting the keyword search for shots (support for “overall image”, unreliability of keywords, etc.). In our verification-test, we introduced a visually-similar-image search for shots, and we received comments praising its suitability for image shot searches and its effectiveness for searching content with no attached text information, so it may be an effective technology for resolving the above issues. However, there were requests to place even more emphasis on object similarity, so one future issue will be to make the technique of positioning blocks over object areas even more effective.

We also introduced technology to apply keywords to shot images automatically using object recognition, and we received comments that it was useful for finding many shot images of the same type. Thus, this technology also appears to mitigate some issues, but it will be necessary to add more variation to the object keywords used.

There was also a comment that the current search speed of the visually-similar-image search was adequate, suggesting that introducing this technology was effective. However, users could be frustrated with the current search speed when the system can’t find their desired images. Thus, it will be necessary to improve the search accuracy.