NHK Laboratories Note No. 467

Turning Raw Information into TV programs with TVML

by
Masaki Hayashi
(Multimedia Services Research Division)


ABSTRACT
    We have been researching and developing a mechanism for producing television programs automatically by computer since we proposed a specially designed language that was TVML (TV program Making Language) in 1996. TVML is a language to describe full TV program and a TVML Player interprets a script written using TVML and generates a TV program in realtime. This paper describes attempts to present information that individual has in a TV program style using TVML. As one of the examples, we introduce automatic talk show generation from given dialogs. We also introduce a function of external control mode of the TVML Player that allows external applications to control the running TVML Player. Some examples using this function are shown.
1. INTRODUCTION
    Since the TV program making language TVML was first proposed in 1996, significant progress has been made in researching and developing a robust system for automatically producing TV programs on a computer. TVML is a special language that can describe every aspect of TV programs. When a script written in TVML is read by software called TVML Player, the TV program described by the language is generated in real time. In this paper we will introduce attempt of presenting raw information in a TV program-like manner. As an example, we will demonstrate the system to automatically generate a talk show style program from raw dialog text. The paper also discusses an external control mode that allows TVML to be used from external applications, and cites a number of prospective interactive applications that could exploit this capability.

2. Internet and TV
    Considerable progress has been made in the development of media for conveying information from one person to another. The last few years, indeed, has witnessed an explosive diversification of such means, as represented by the ubiquitous term multimedia. Most of the massive volumes of new information that are currently being generated and made available worldwide over the Internet are coded in hypertext. The key to hypertext's success in being universally accepted is that it provides a means of creating links between information. This permits recipients of information to actively follow their own path or create their own order by jumping to further information about a specific topic of interest.
    Dramatic advances over the last few years are most evident in the way recipients of information, spurred by some specific interest, can move or surf through information space following their own path of associations. Commonly referred to as machine Interactivity, it is apparent how appealing this approach is as entertainment by the dissemination of TV games around the world, and how useful it is for work as a powerful business tool.
    In contrast to these active media, television remains a quintessentially passive medium: you turn the set on, select a channel, then sit and watch. The Internet has certainly created a major stir, but the fact remains that TV is still the primary means by which most people around the world get their information.

3. Television as a means of presenting information
    The history of broadcasting as a medium and TV programs as a means of conveying information go back a long way. Through decades of experience, the basic TV program paradigm has been honed and refined as a vehicle for conveying information. Yet, today it is only the broadcast companies that are really satisfied with this arrangement, for up to now, the production ofTV programs has been the exclusive preserve of industry experts.
    This presents a stark contrast with other media, where all sorts of user-friendly consumer-oriented tools have appeared in recent years enabling ordinary users to produce books, images, magazines, and create hypertext documents on the Internet all from the comfort of their own living rooms. This is why the Internet appeals to such a wide spectrum of people, ranging from professionals to ordinary users at home.
    It is generally assumed that individuals cannot make TV programs because the production of programming requires a studio, an announcer, actors, actresses, and so on. Yet, if we confine ourselves strictly to informational type programming-programs developed to present specific information-, then it should be possible to create programs on a computer using computer graphics (CG) for the studio, speech synthesis for the announcer, and so on. Since proposing the TV program making language (TVML) in 1996, we have made considerable progress in researching and developing a system to automatically produce TV programs on a computer [1][2].

4. TV Program Making Language - TVML
    We first present a brief overview of TVML. TVML is a fully self-contained language that can represent every aspect of a TV program. Modeled after the descriptive approach used in program scripts used at TV program production sites, TVML was designed to be a language that is readily readable and that even a total novice can master. The program scripts written in TVML can then be played back by a software component on a PC called the TVML Player. The TVML Player features all the capabilities needed to generate informational type programs from a TVML script. Such informational programs could be used to present news, a guide to some event, or any other body of information that one might want to convey to others. The TVML Player generates studio shots in real-time CG, produces CG characters on a CG set that speak with synthesized voices, and shoots the movements of the actors by cameras within the CG setting. In addition, various video-audio effects are also created in real time including video effects by replaying movie data files, displaying of on-screen fonts, titles by displaying image data, superimposing, background music by playing audio data files, narration by synthesized voices, and more.
    TVML scripts are compiled of multiple lines, and the description of each line corresponds to an event. For example, a typical event might be represented:

character:talk(name=Bob, text="How are you?")

    If you enter this line into the TVML Player, a scene is immediately played in which a CG character named Bob appears in the CG studio, and says"How are you?"Stories are played as the TVML Player reads and executes one line at a time from a TVML script. Figure 1 shows samples of TVML commands, and Fig. 2 presents an example of a TVML script and playback.

Function Event name Command examples (partial set)
CG character character talk(speak lines), walk(walk), look(look at something),...
CG camera camera closeup(closeup), twoshot(two-shot),...
CG studio set set change(change sets),...
CG prop prop position(position prop),...
CG lighting light model(shift lighting),...
Play video movie play(play movie file),...
Still text, still image title display(display still information),...
Superimpose super on(super display),...
Play sound sound play(play audio file), mixer(adjust mixer),...
Narration narration talk(speak lines),...

Figure 1. Events and commands used by TVML
(currently there are approximately 70 commands)

title: display(type=imagefile, filename=openingimage.tiff, wait=no).... (A)
sound: playfile(filename=bgm.wav)
wait(time=6.0)
sound: mixer(source=bgm, action=fadebgm)
character: talk(name=BOB, text="Hello everybody.")
character: talk(name=MARY, text="Thank you for tuning in.")
character: talk(name=BOB, text="Do you know who's the father of the Blues?").... (B)
character: look(name=MARY, what=BOB)
character: talk(name=MARY, text="Father?")
character: look(name=BOB, what=MARY)
character: talk(name=BOB, text="Yes.")
character: look(name=BOB, what=camera)
character: look(name=MINA, what=camera)
character: closeup(what=MASA)
character: talk(name=MASA, text="It's Muddy waters.")
camera: twoshot(namel=MASA, name2=MINA)
character: talk(name=MASA, text="Let's watch him anyway.")
sound: mixer(source=bgm, action=fadeout)
movie: playfile(filename=hoochie. mov, from=0, to=270, wait=no)
super: on(type=text, text="Muddy Waters sings Hoochie Coochie man".).... (C)
movie: wait_playfile(frame=270)
... some lines ommited...
title: display(type=html, filename=MUDDYCAREER.html, wait=no)
narration: talk(who=MASA, text="He was born in 1915. He went to Chicago, and got famous.").... (D)
narration: talk(who=MASA, text="He's the first man who playd electric blues.")


A

B

C

D

Figure 2. TVML script and playback

5. Using TVML to convert information to a program
    Targeting a wide range of potential users, TVML not only provides a user-friendly TV program production tool for pros, it also provides a means of creating informational presentations based on the metaphor of a TV program. By conducting a keyword-based search on an image database, for example, the results could be presented to the viewer. The simplest approach would just be to string the images together sequentially, but if the images could be presented in a TV program-like format with added commentary and explanations, this would open up a whole new dimension and appeal for this database resource.
    TVML is perfectly suited to such a use, since it can be used to write out semantic information such as announcers' lines, superimposed text, text information contained in titles, directly in a TVML script. The image database example could be readily implemented by automatically generating a TVML script using the image index data as commentary for the images turned up in the search. The search results could then be presented to users immediately as a seamless TV program-like presentation by simply loading the generated TVML script in the TVML Player.

6. TVML external control mode
    In situations where someone wants to interconnect with and use TVML from a different application such as the database system in our previous example, there must be a means to control the TVML Player externally or remotely. The TVML Player has thus been equipped with an external control mode that provides this functionality.
    One can see from Fig. 3 that when the TVML Player is started by the external control mode, the Player is controlled by overwriting the control file in the current directory by commands from the external program. Since the TVML always checks the control file while it runs, an external program can overwrite the commands to execute any TVML script at any time and to stop the execution at any time. At the same time, the TVML Player outputs the script playback status to a status monitoring file located in the current directory.
    When this function is being used, the TVML Player continues to be controlled by the external control mode. When implementing an application, the external program generates an appropriate TVML script which is sent via the control file to run the Player and monitor the status file.


Figure 3. TVML Player external control mode


7. An application example: automatic generation of a talk show
    To test the system's ability to automatically generate a TV program-like presentation from a set of data, we used it to create a talk show style program [3].
    Assuming a dialog between two people-an interviewer and a guest-, we assembled a system that automatically generates a typical talk show-like program based on nothing but the dialog text between the interviewer and the guest. Since TVML can be used to make CG characters speak in synthesized voices, it is a simple matter to create a scene showing the two characters speaking the input dialog text. Then, to produce a complete talk show with visual appeal, camerawork and camera switching are required. In this report, we will highlight the camera switching capability. A camera switching engine was developed that uses a statistical approach to generate camera switching decisions automatically, and this was used to create the complete talk show.
    Figure 4 shows the steps that are involved. First, a text file that contains just the dialog between the interviewer and the guest is input, and this is automatically converted into a TVML script. The TVML script is then sent to the TVML Player, where an interview scene is generated. Meanwhile, the camera switching engine produces camera switching decisions, which are asynchronously sent to the TVML Player to control the switching of the TVML internal camera. Transmission of the playback script and camera switching decisions to the TVML Player uses an advanced function (described in the next section) of the external control mode described earlier.
    Note that the camera switching engine was designed based on the results of statistical processing using an approach developed by Ishii et al. based on actual camera switching employed in real on-air talk shows. Essentially, the camera switching decisions are generated using two probabilities: the camera switching occurrence probability when the speaker changes from one to the other, and the camera switching occurrence probability while the same camerawork continuing.


Figure 4. Process flow for automatically generating talk show program


    For the purpose of this trial implementation, we used a suitable interview that we found on the web. Figure 5 shows the dialog text file that was entered, and a sample of the talk show that was automatically generated from it. While a person reading the interview on the web has to continually scroll down and follow the text with his eyes, the obvious advantage of our approach is that the viewer can just sit back and watch the interview unfold like a TV program. Next we plan to develop a system that can be watched with greater enjoyment and assurance by adding more sophisticated camerawork (bust shots, zoom-ins, facial close-ups, etc.) and a fuller range of gestures by the characters (nodding and so on). We are also seeking to develop a range of other types of programs besides the talk show format for automatic generation by the system [4].

LANGUAGE: English
TOPIC: Life in Japan
CAST: Don Foley, BOB

A: Where are you from?
B: Canada.
A: What brought you to Japan?
B: When I was first here,
B: it was so difficult to find a half-decent restaurant in Tokyo.
B: that served real Western food.
B: I really wanted a cheese fondue,
B: when I first lived here and couldn't find one anywhere,
B: so I decided to open up a restaurant myself.
A: How did your restaurant get started?
B: lot of hard work. Lots of closed doors because I'm a gaijin,
B: but I met a lot of very nice Japanese people who were willing to help.
B: If you can make it in Japan, you can make it anywhere.
A: What do you do here?
B: Most of the time I cook. I also spend a lot of time reading cookbooks,
......

Input dialog text data
Automatic generation

Talk show program output
Figure 5. Automatic generation of a talk show program

8. New external control mode and application examples
    Currently, a TVML Player featuring an external control mode with more advanced capabilities than described earlier in Section 5 is under development. These new capabilities include
- a library of Player control commands for developers of external applications,
- capability to control the Player over networks,
- ability to control a single Player from multiple applications,
- capability to send and play more than one line of TVML script at a time,
- ability to stop, play and pause the given script at any time,
- ability to control all Player graphical user interface functions (play, stop, read script, etc.) remotely,
- capability to execute and quit a Player and to place the Player window at any position, etc. remotely,
- ability to acquire the operating status (event processing status, etc.) of the Player in real time, and
- ability to acquire mouse x-y coordinates in the Player window and button information in real time.
Having highlighted the new capabilities of the upgraded external control mode, let us next consider a few of the many potential applications that could exploit these new features.

  1. TV programs that support interrupts and queries
        This would permit the viewer to interrupt a program that was in progress at any time, and address or speak with a person appearing in the program. After the interruption, the program would then resume as before. The script sent to the Player would be controlled by the viewer's interrupts, and multiple stories would be produced for each different version.

  2. Interactive quiz program
        With this capability, the TVML program poses questions to the user, and the user responds. The user responses could be implemented in a number of different ways: (1) by clicking among multiple choices presented on the screen with a mouse, (2) through an external program user interface, or (3) using voice recognition. In all cases, multiple stories would again be generated as described above for each different version.

  3. Hyper control TV (TV permitting parameters to be changed on the fly)
        This would enable a wide range of programming parameters-camerawork, lighting, speaking voice, speed, and so on-to be altered at will and on the fly even as the program is being aired. This could be implemented with interrupts that send controls corresponding to the different parameters to the Player.

  4. Interactive production with multiple participants
        This would allow several people to connect to the same Player, with each person able to asynchronously manipulate a different character or different camera.


9. TVML near-term schedule
    The TVML Player is being distributed as freeware. There are currently two versions of the TVML Player software available: Version 1.1.2 for SGI that runs on SGI graphic workstations, and Version 1.1.3 for Windows that runs on Windows 95, 98, and 2000. The SGI version can be obtained by regular surface mail or by downloading it from the Internet, while the Windows version can only be obtained by downloading it off the Internet. The download site is accessed through the TVML homepage at http://www.strl.nhk.or.jp/TVML/index.html
    The external control mode described in Section 5 is currently only supported on the SGI version of the Player. The TVML Player with the more sophisticated set of external control mode features described in Section 7 is scheduled for release in September 2000.
    In addition, there is TVML Player PRO, a broadcast quality Player tuned specifically for Onyx2 that is not available as freeware. This professional version Player has additional enhancements to obtain programming output that is equivalent to on-air broadcasts. However, the most of the new functions in the TVML Player PRO are now supported in the Version 1.1.3 for Windows. The main difference is picture quality such as high frame rate, realtime unti-aliasing and movie playing quality etc.
    A TVML Editor is now also under development that features a man-machine interface enabling people who are totally unfamiliar with TVML language specifications to generate TVML-based programs. We are also developing a commercial product both for Windows and SGI. Further information is in the TVML homepage.

10. Conclusions
    In this paper we presented an overview of the TV program making language TVML, a powerful tool for presenting diverse types of information in a visually appealing format based on the metaphor of a TV program. As an example, an application was introduced that automatically generates a TV style talk show from a dialog text. The text in this particular case was an interview taken from a web page. Essentially, this experience can be considered as an attempt in a field of media conversion technology that allows the basic content to be converted to a different medium-i.e., from a web page to a TV program. Talk shows could similarly be created from an exchange of mail, chat dialog between two people, or other materials. Building on this promising approach, our next challenge will be to develop the capability automatically convert between various different kinds of media.
    The paper also described how various types of interactive applications can be implemented using the TVML's external control mode.
    In the process of implementing the infrastructure upon which this work was based, the TVML system-from the language to the display system- is now all but complete. By all means, take full advantage of TVML as a viable tool in conducting a wide range of research activities. Development and deployment of TVML is being conducted as a cooperative research venture by the TVML Research and Development Team, consisting of the NHK Science and Technical Research Laboratories and a number of other corporate research institutions (refer to the TVML homepage for details).


REFERENCE

  1. M. Hayashi, H. Ueda and T. Kurihara"TVML (TV program Making Language) - Automatic TV Program Generation from Text-based Script -", Proceedings of Imagina'99 (1999).
  2. H. Ueda, M. Hayashi and T. Kurihara"DeskTop TV Program Creation - TVML (TV program Making Language) Editor -"ACM Multimedia'98 State of the Art Demos (1998).
  3. M.Hayashi,Y.Gakumazawa,Y.Yamanouchi: Automatic Generation of Talk Show >from Dialog using TVML", ICCC'99 Digital Convergence for Creative Divergence Vol.2 pp.325-332,(1999).
  4. M.Douke, M.Hayashi, E.Makino,"A Study of Automatic Program Production Using TVML", Short Papers and Demos, Eurographics '99, pp.42-45, (1999).


Copyright 2000 NHK (Japan Broadcasting Corporation) All rights reserved. Unauthorized copy of the pages is prohibited.

BackHome