¹⁾ Masaki Hayashi, ²⁾ Hirotada Ueda, ³⁾ Tsuneya Kurihara,
⁴⁾ Michiaki Yasumura, ¹⁾ Mamoru Douke, ¹⁾ Kyoko Ariyasu

¹⁾ NHK (Japan Broadcasting Corporation) Science and Technical Research Laboratories
²⁾ Hitachi Kokusai Electric Inc.,
³⁾ Hitachi, Ltd., Central Research Laboratory, ⁴⁾ Keio University

Abstract

This paper describes TVML (TV program Making Language) for automatically generating television programs from text-based script. This language describes the contents of a television program using expressions with a high level of abstraction like "display title" and "zoom-in." The software used to read a script written in TVML and to automatically generate the program video and audio is called the TVML Player. The paper begins by describing TVML language specifications and the TVML Player. It then describes the "external control mode" of the TVML Player that can be used for applying TVML to interactive applications. Finally, it describes the TVML Editor, a user interface that we developed which enables users having no specialized knowledge of computer languages to make TVML scripts. In addition to its role as a television-program production tool, TVML is expected to have a wide range of applications in the network and multimedia fields.

1. Introduction

Broadcasting has a long history as a medium for conveying information. The techniques used in this field, and especially in the production of television programs, have progressed over the years to become highly refined means of presenting information. At present, however, only broadcasting stations are able to use these techniques satisfactorily. In other words, television program production is still the domain of specialists.
In contrast, tools are now being provided that allow individuals to prepare various types of media like documents, pictures, and magazines, as well as Internet-based hypertext, entirely in the privacy of one's own room. One reason for this is that the once limited use of the Internet by scientific and technical professionals has expanded to widespread use by individuals.
Individuals cannot easily produce television programs with conventional techniques because program production requires various video and audio equipment as well as studios, announcers, and so forth. However, if we limit the target programs to the one for information presentation, it becomes quite possible to create studios, announcers, and the like with current technology by computer through the use of computer graphics (CG), voice synthesizers and multimedia computing. This paper describes a mechanism for generating television programs in real time on only a desktop computer through the use of these technologies.
We have developed a new language that can represent an entire television program by text, called TVML^[1][2] (TV program Making Language). TVML is not limited to describing the parts played by actors. Instead, it can be used to describe all program-production work, including pre- and post-production work such as that involving sets, props, cameras, VCR in playback, superimposing, titles, background music, and narration. This paper describes TVML as well as a TVML Player, which converts the television-program script written in TVML to program video and audio in real time. It also describes a TVML Editor, which is a user interface that we have developed for the creation of TVML scripts by users having no specialized knowledge of computer languages.

2. Mechanism of Real-time Generation of Television Programs

The generation of television programs by computer requires some kind of intermediate expressions that can be given to the computer to indicate tasks like directing the performance of an actor and configuring a program. TVML is a text-based language that uses such expressions. The expressions can be easily understood when perused by people and instruct the computer what to do. TVML can describe an entire television program through highly abstract text-based expressions like "display title" and "zoom-in". The software that we developed for interpreting scripts written in TVML and automatically generating program video and audio in real time is called the TVML Player. This is illustrated in Figure 1.

Figure 1: General configuration of system for generating TV programs by TVML

The television programs targeted here are of the types of programs that present information such as news, weather forecasts, event guides, and documentaries. For the most parts, television programs like dramas and live variety shows that are essentially one-time creative works are excluded. The production elements of such information-oriented television programs are relatively simple. These elements are shown in Table 1. together with the techniques used in the TVML Player to generate each element by computer. For example, a studio shot is generated entirely by CG in which computer-generated actors in a computer-generated set give speech by synthesized voice, and the entire scene is "shot" by a camera within the CG world. A motion picture is achieved by playing back a movie file, titles and captions are generated by using the layout-description part of HTML to display text information, audio is produced by playing back audio files, and narration is generated by using a voice synthesizer. The generation of these elements requires two types of data: script data written in TVML and various forms of reference data.
The next section discusses how a television program should be described on a text basis to establish TVML language specifications.

Table 1: Elements in actual program production and methods used in TVML Player
Program production	Method used inTVML Player
Studio shot
Studio set	Real-time CG set
Actor	Real-time CG character with speech by synthesized voice
Lighting	Lighting setup in real-time CG
Camerawork	Camera setup in real-time CG
Motion picture
VCR	Movie file playing(AVI,Quick Time)
Title
Text information	Text lay out section of HTML
Static image	Image data file(TIFF,JPEG)
Super imposing
Text information	Text lay out section of HTML
Sound
Music	Audio file playing(WAV,AIFF)
Naration	Synthesized voice
Video effect
Cut	Cut change only
Sound effect
Cut	Audio mixer

3. TVML Language Specifications

The specifications for the TVML language have been determined with reference to the structure of program scripts used in actual television program production. The following describes basic specifications of TVML with an emphasis on event description and time description.

3.1 Event description
The event format consists of an event type from the list of classes in Table 1, a command provided for the given event type, and a list of parameters required by the command for execution. For example, to make a character named BOB say "Hello, I'm BOB," one would write:

character: talk (name=BOB, text="Hello, I'm BOB.")

Here, "character" is the event type, "talk" is the command, and the items inside the parentheses are parameters. The general format of an event is as follows:

Event type: Command name (arg1=data1, arg2=data2, arg3=data3, .....)

Table 2 lists types of events and associated commands provided in TVML. Here, parameters represented by "arg" in parentheses need not follow a particular order and default values are adopted for any parameters that are omitted. In other words, parameters that are not explicitly specified by the user will be set by the computer. Conversely, the user can specify as many parameters as needed for indicating detailed instructions. For example, if the user desires to indicate a certain speaking speed and an exaggerated character gesture, the event description could be written as follows.

character: talk (name=BOB, text="Hello, I'm Bob, rate=5.0, emotion=excite)

Table 2: Events and commands used by TVML (currently there are approximately 90 commands)
Function	Event type	Examples of command (a part)
CG character	character	talk (make a speech), walk, look (look at something), sit, stand, bow,...
CG camera	camera	closeup (focus on someting), twoshot (twoshot),...
CG studio set	set	openmodel (open modeling data), change (change set),...
CG prop	prop	position (position prop), openimageplate (make a plate with texture image),...
CG lighting	light	model (setup lighting), switch (turn on and off the light),...
Motion picture	movie	play (movie file playing),...
Title	title	display (display title),...
Superimposing	super	on (tune on caption),...
Sound	sound	play (audio file playing), mixer (control mixer),...
Narration	narration	talk (make a speech),...

3.2 Time description
One event is described on one line in TVML. Upon completion of each event, the system advances to the next line to process the next event. Consider, for example, the following two lines.

character: talk (name=MARY, text="Look at this video.")
movie: playfile (filename=test.mov, from=30, to=450)

Here, the computer-generated character named MARY is made to say "Look at this video," and after this, the motion-picture file named "test.mov" is played back from frame 30 to frame 450. There are actually two types of events: an action event in which time is taken to perform some action (like "sit") lasting from the beginning to the end of the event, and a state event that simple specifies a state change (like superimposing ON) with no elapsed time. Consequently, if a user wants to execute two action events simultaneously, "wait=no" must be added as a parameter to the first action event. This expression will be treated as a state command. For example, to make the characters BOB and MARY bow at the same time and to superimpose text after the bowing is completed, the following event descriptions would be used.

character: bow (name=BOB, wait=no)
character: bow (name=MARY)
super: on (type=text, text="Mary & Bob")

Here, if "wait=no" is not included in the first line, the actions would be such that MARY starts bowing only after BOB completes his bow.
When "wait=no" is added to an action event and treated as a state event, it becomes necessary to check whether a certain action event has been completed. This is why a wait command is provided for all action events. The format of the wait command is as follows.

Event type: wait_Command Name (arg1=data1, arg2=data2, arg3=data3, .....)

This wait command blocks further execution of the script until the action specified by the command is completed. In short, the following line:

character: bow (name=MARY),

is equivalent to the following two lines:

character: bow (name=MARY, wait=no)
character: wait_bow (name=MARY).

The above structure can therefore be used to describe simultaneously occurring events. For example, if the user wants to superimpose text 1.5 seconds after starting playback of a motion-picture file from frame 100 to frame 200, the following description can be used.

movie: playfile (filename=test.mov, from=100, to=200, wait=no)
wait (time=1.5)
super: on (type=text, text="This is a test movie.")
movie: wait_playfile()

In addition to events and wait commands, TVML also provides a several commands that are called directly without being treated as events. These are called "direct commands."

4. TVML Player

4.1 TVML Player operation
The TVML Player is software that reads a program script written in TVML and converts the script to program video and audio in real time. The hardware platform of the TVML Player is a Windows PC or any graphic workstation of SGI. In a Windows PC, a Microsoft speech API is used for synthesized voice. The mouth of the computer-generated character is opened in direct proportion to the magnitude of the voice-level to achieve lip-synching. SGI workstation requires an external voice synthesizer(hardware) that is attached to the serial port of the machine. The TVML Player supports AVI movie files and QuickTime and SGI movie files, WAV and AIFF files as audio files, and TIFF and JPEG files as still pictures. It also supports OpenInventor and VRML 1.0 for the modeling data format in computer-generated characters, sets, and props. The TVML Player features a straightforward user interface with buttons for playback, stop, and other functions, enabling selection of TVML-script files and immediate playback. Figure 2 shows an example of a script written in TVML and a sample of a program video generated from this script by the TVML Player.

Figure 2: TVML script and playback

4.2 TVML Player external control mode

Figure 3: TVML Player in external control mode

As described above, the basic operation of the TVML Player is to play back a TVML program script from start to finish. However, let us consider for a moment an interactive application that gives a user the choice of changing the story of a program that he or she is currently watching. In this case, there must be some kind of interface to accept a user's selection and some method for dynamically changing the story development based on the selection made. This can be achieved through TVML because a mechanism has been provided to control the TVML Player from an external interactive application program. We point out again that the TVML Player functions completely as an interpreter in which one line of TVML, corresponding to one TVML event, is read in, syntactically parsed, and executed before moving on to the next line. A real-time interactive application can therefore be constructed by having an external application send a script to the TVML Player in an asynchronous manner. The mode which enables a script to be sent to the TVML Player from an external application is referred to as the "external control mode" of the TVML Player.
As shown in Figure 3, booting up the TVML Player in external control mode allows an external application to communicate with the TVML Player. Consequently, if the TVML Player is always left on, an external program can execute any TVML script (of at least one line) at any time and suspend external operation at any time as well as fully control the GUI functions of the TVML Player. Moreover, TVML Player sends the script playback status to the external application. All the communication between the TVML Player and external applications is done very fast via a shared memory. In external control mode, therefore, with the TVML Player up and running, an interactive application can be constructed by having an external application send commands and scripts to the TVML Player while checking the status comes from the TVML Player.

5. TVML Editor

For non-technical people to use TVML as a program-production tool, a user interface is necessary that enables a user with no specialized knowledge of computer languages to create a television program by intuitive operations. We have also been developing a user interface called TVML Editor^[3] that incorporates a GUI for such users. The TVML Editor allows users to create a television program by computer mouse and window-based operations while interactively checking intermediate results. In the TVML Editor, a television program is prepared by lining up a studio-shot, a motion picture, and title blocks in a time sequence and making clips where necessary. Each of these blocks is divided into cells, and each cell corresponds to a TVML command. Double-clicking on a cell opens an operation window having buttons, scroll bars, and so forth, that enables parameters such as dialogs to be set up easily. A program created on the TVML Editor can be exported as a TVML script for playback on the TVML Player.

6. Conclusions

This paper described the TVML language, designed for describing television programs by text-based scripts, and the TVML Player that interprets the scripts and outputs program video and audio. We also described the TVML Editor, which is a user interface that enables individuals without specialized knowledge of computer programs to prepare television programs. These developments provide an environment in which anybody can prepare a personal television program in a relatively easy manner on a desktop computer.
One objective of TVML is to popularize program production by making television-program production techniques available to the general user as described above. This is not the only purpose of TVML, however, as many possibilities can be envisioned for it. The following gives some examples.

- The TVML system can be used as a training tool and simulator for professional program directors before they begin producing actual programs.

- As high-performance computers are expected to be incorporated in television sets in the future, a TVML Player installed in the built-in computer can provide a powerful infrastructure for interactive television.

- TVML can be used to achieve automatic program-generation[4][5] for programs consisting of regular formats and performances like news programs, because core information given by users can be converted to a TVML script automatically using the regular format of the program.

- If the TVML Player is plugged into a WEB browser, program scripts can be distributed and reused over the Internet.

- TVML-based applications can be created that use television programs as a metaphor to present various forms of data like text, pictures, video, and audio stored in databases.

A variety of applications like those described above are currently being studied in a comprehensive manner. We would like to point out here that the research and development of TVML is being conducted on the basis of open research system, and that joint-research partners are always being looked for.
The TVML Player is distributed without charge as freeware. The procedure for obtaining a copy is described at the WEB site given below. This site includes detailed information not given in this paper and we suggest that you pay the site a visit.

http://www.strl.nhk.or.jp/TVML/index.html

References

[1]	M. Hayashi, "Automatic Production of TV Program from Script - A proposal of TVML," Proceedings of the 1996 ITE Annual Convention, S4-3, pp.589 - 592, (1996)
[2]	M.Hayashi:"TVML(TV program Making Language) Make Your Own TV Programs on a PC!",International Conferences, Virtual Studios And Virtual Production,(Aug.2000)
[3]	Ueda, H., Hayashi, M. and Kurihara, T., "DeskTop TV Program Creation - TVML (TV program Making Language) Editor", ACM Multimedia'98 State of the Art Demos (1998).
[4]	M.Douke, M.Hayashi, E.Makino, "A Study of Automatic Program Production Using TVML", Short Papers and Demos, Eurographics '99, pp.42-45, (1999)
[5]	M.Hayashi,Y.Gakumazawa,Y.Yamanouchi:"Automatic Generation of Talk Show from Dialog using TVML", ICCC'99 Digital Convergence for Creative Divergence Vol.2 pp.325-332,(1999)