Closed captions and open captions together

One technique that has not been attempted, to our knowledge, is a combination of closed and open captioning. A video segment is provided with burned-in open captions and optional streams of player captions.

In this way, dialogue can be rendered via open captions (and probably with near-verbatim completeness in most cases), while closed captions give additional information. Some examples:

Non-speech information

Productions with a lot of non-speech information (NSI) and a lot of dialogue can render much more of the NSI in a separate stream than would be possible if dialogue and NSI were mixed together.

Music

Lyrics can be handled by open captions, while closed captions can annotate and explain the features of the music itself, as already seen in captioned opera and ballet.

Emotion

Manner of delivery can be explained in the closed-caption stream. Because these captions are not really supplementary but an adjunct to the dialogue, it will likely be necessary to tightly synchronize them to the dialogue and position them near the dialogue captions. Since the open- and closed-caption streams will probably be at different vertical positions on the canvas, they may be difficult to follow.

Extended captions

The closed-caption stream could appear only on request. The viewer could select a Tell me more button, which would pause the video and play additional captions. Or the author could set up the video so it always pauses at a given point, whereupon closed captions are sent.

Audio description

An audio-description narrator can be captioned separately from the main audio.