Ted Goranson - Personal Blog

You’ve all seen the scrubber used by YouTube, Netflix and others. You can move back and forth and see a small preview of that location. Its pretty good for what it does. Now suppose you wanted to have a thing that allowed you to navigate a film maybe better, and allow you to simultaneously browse what could be huge numbers of onscreen annotations as well.

Also, you might not really know what you are going to find.

It would do double duty, being tall enough for you to see a frame preview as you move back and forth, but also have strict time calibration, so that you could make and browse annotation anchors. You can see the space where we would put such a thing at the bottom of the early Framethrower design.

The Framethrower Demo

(Image) The last few minutes of our demo.

720:540

The design here is evolved from an idea by Harold Cooper for Framethrower. It leverages what he called space-time frames. The original patent application is here.

The method is simple:

• Take keyframes from a film. We do that every second. Adjust them for size.
• For each keyframe, take vertical slices from left to right. We use 2 pixels wide. For Memento, our example, that will be 232 slices per keyframe, each slice at 2p by 200. The keyframe bounding frame will be 464p wide by 200p high.
• Construct space-time frames by sequentially taking one slice from keyframes and placing them in a synthesized frame. The first slice of the first keyframe is the first slice of the first space-time frame. The second slice of the second keyframe becomes the second in the first space-time frame. When the first space-time frame is filled (with 232 slices), start the second space-time frame with the first slice of the 233rd keyframe and go until the end of the film.

What you end up with has every 2 pixels in horizontal order and can be used to show the associated keyframe. This basically gives you the Netflix scrubber, the time bit. Also, each space-time slice has every pixel in the same location as the corresponding pixel in the keyframe, the space bit. This will be a very big deal.

Space-Time Strips and Magical Dynamism

A script to construct space-time frames from videos was tediously worked out by friend Robert Black. Some examples of what these look like can be seen here. These are already pretty good. If you know the film well, you can pick out locations.

Now let's add some magic. Suppose we have a preview pane that shows the keyframe selected when the cursor is over the related slice. Now move the cursor left:

• The displayed keyframe is the one before, of course.
• Every slice that is displayed changes to the one to the left in the keyframe.
• Every space-time frame bounding box moves left one click, maintaining the rule that every pixel of every displayed slice is in the same location as that pixel in the keyframe.

I hope a little imagination will show you that you only have to move left or right one frame distance to see the entire film, using a little eidetic memory. Every space-time frame becomes a dynamic preview pane. You see simultaneous previews, about 4 minutes (232 seconds for the Memento example) apart.

In effect, we've compressed the film so that you can browse the entire thing in a short horizontal distance and see what is happening everywhere. Part of the compression is the reduction from 30 (or 24) frames a second to one. But the big compression is stacking 232 frames behind each other for fractional display.

Now we have something that allows you to quickly understand what is going on visually in the film. But the goal is to allow you to create, edit, browse and filter annotations that are linked directly to the film. A large class of those link to bits of the film that are clips or sequences. We address that now.

There are 4 types of these, I think:

• Freeform selections of frames and sequences that the user will make for special purposes. He/she may just want to say that the camera has an unusual stance at a place.
• Selections that are associated with a discrete plot point that is captured in a synopsis fragment. For instance, if there is a detailed synopsis that somewhere says Frank turns on the TeeVee then you may want to select that fact rather than the sequence where it happens if that very fact is what is of interest.
• Similar selections of speech and sound that are typically captured in the closed captions. If what you want to link to is; leave the gun; take the cannoli, then you likely want the greater precision than linking to the scene and then noting that is is the spoken phrase that is of interest.
• Scenes and cuts. These are natural divisions in the film.

These aren't the most challenging anchors that we address, but they are likely to be the anchors that most visitors use, at least as beginners. They should find it elegant and easy. The solution we use has the anchors mostly on the strip itself, because the annotations and essays are expected to want a lot of screen space. But there is otherwise no reason that they cannot be drawn above or below the space-time strip. Our solutions for these are in the condensed graphic below, or can be downloaded full size as a PDF.

Temporal Anchors

(Image) A set of solutions to the anchor challenges.