What Is Rotoscoping?

What is rotoscoping?

Rotoscoping means tracing over live-action footage, frame by frame. Wikipedia describes it as an animation technique in which animators trace over motion picture footage to produce realistic action. The original rotoscope projected film frames onto a glass panel so the frames could be traced onto paper. Max Fleischer developed it and received a US patent in 1917. Computers replaced the projector, and the process kept its name.

The word now has two meanings. In animation, the traced drawings are the character. Disney animators used the technique extensively on Snow White and the Seven Dwarfs to make human movement more realistic. In the visual effects industry, rotoscoping means manually creating a matte for an element in a live-action shot, so it can be composited over another background. The hand movements are the same. The result is different: drawings in one case, a cutout in the other.

This page mostly covers the second meaning, because that is the one that costs production time.

What is rotoscoping used for in VFX?

A matte is a black and white image that says which pixels belong to the subject. Once you have one, a lot becomes possible. These are the uses I see most in commercial work.

Use	What the matte does	Example
Background replacement without a screen	Separates the subject from a real location	A presenter placed in front of a new set
Layering CG with real people	Lets a 3D object pass behind or in front of a person	A product animation that wraps around a hand
Isolating one subject	Limits a grade, blur or relight to a single element	Brightening a face without lifting the room
Garbage mattes	Cuts away stands, rigs and screen edges before a key	Removing a light stand from a green screen shot
Removal	Protects the subject while a wire or logo is painted out	Erasing a rig that crosses an actor's arm
Guided effects	Gives a glow or distortion a shape to follow	A light line that traces a moving silhouette

Wikipedia notes that roto is often used to prepare garbage mattes for other matte-pulling processes. That is the quiet job it does in most shots: cleaning up the edges of work done by faster tools. The wider picture of what a visual effects shot contains is in what is VFX.

How does rotoscoping work, step by step?

The work is manual, but the order is always the same.

1. Break the subject into parts. A person is not one outline. Head, torso, upper arm, forearm, hand and each leg move differently, so each gets its own shape.
2. Draw the shapes on a key frame. In Nuke, the Roto node lets you create and edit Bezier and B-spline shapes. Other packages work the same way.
3. Move to a later frame and correct. The software interpolates between key frames. You step forward, fix where the shape drifted, and add key frames only where the motion needs them.
4. Treat the edge. A hard line looks cut out, so the edge gets a feather, and fast movement gets motion blur that matches the footage.
5. Output the matte. The result is an alpha channel the compositor uses to combine layers.
6. Check it on a contrasting colour. Play the shot at speed over a flat colour. Edges that jitter or boil between frames show immediately.

Most of the time goes into step 3, because it repeats for every part, in every shot.

What is the difference between rotoscoping, keying and tracking?

They solve different problems and are often mixed up in briefs.

Method	What it does	Needs	Fails when
Rotoscoping	Cuts a subject out by hand with keyframed shapes	Any footage	Edges are soft, fast or numerous
Keying	Removes a colour range so software builds the matte	A green or blue screen	Light is uneven, the screen wrinkles or colour spills onto the subject
Tracking	Reconstructs how the camera or a surface moves	Texture and parallax in the shot	The surface is blank or the footage is heavily blurred
Clean plate and paint	Covers a spot with a patch from another frame	A locked camera, or a track if it moves	Something moves across the patch

Chroma keying makes a colour range transparent so separately filmed footage can be inserted behind the subject. Wikipedia also notes that a green screen casts green light onto the subject, called spill, which can look unnatural or make parts of the subject disappear. That is exactly where a key stops being enough and roto takes over.

Tracking sits underneath the others whenever the camera moves, because shapes and patches have to stay attached to the scene. How it fits into a tracked and composited job is explained in mixed reality video production.

Is rotoscoping cheating?

No. The belief comes from the animation meaning, where tracing sounds like a shortcut. In practice tracing only supplies the motion. Someone still has to design the character, decide which frames to simplify, exaggerate or drop, and give the result a style that fits the film. The early users were not hiding the method. Disney's artists used it on Snow White to make human movement more realistic, and rotoscoped lines became part of the look of well known music videos.

In visual effects the question hardly comes up. A matte is a technical object, and nobody watching the finished shot should notice that it exists. The only fair measure is whether the edge holds on every frame. A rotoscoped cutout that nobody notices is the standard, and it takes more skill than it looks.

Can AI do rotoscoping now?

AI-assisted tools can produce a first matte quickly. Tools such as the Roto Brush in After Effects follow a subject through a shot after you mark it on one frame, which saves the time spent on step 3 above.

They work best on a clear subject, with simple motion, against a background that differs from it. They struggle in a recognisable list of cases:

• Hair and fur against a busy background
• Motion blur and fast limbs
• Transparent or reflective objects such as glass
• Thin parts like straps, cables and fingers
• Edge flicker, where the matte changes slightly from frame to frame

The usual pattern is a fast machine matte followed by an artist who checks it on a contrasting colour and repairs the failures. For a hero shot in a commercial that review is the work. Assume AI shortens the first pass, and not that it removes the check.

What makes rotoscoping slow and expensive?

Five things move the effort, in rough order of impact.

1. Edge type. Hair, smoke, glass and motion blur take far longer than a product with a hard outline.
2. Length. For most work, duration is render time. For roto, duration is human time, because every extra second means more frames to check.
3. Number of subjects. Two overlapping people can be more than twice the work of one, because each needs a matte and the overlap has to be solved.
4. Footage quality. Compression, noise and rolling shutter blur the edge you are trying to follow.
5. How hard the shot is looked at. A background person needs a looser matte than a hero product in the final frame.

The same logic runs through every cost in CGI work, and it is laid out for a whole commercial in what a CGI commercial costs. In my own mixed reality jobs, the amount of roto is one of the four things that move the price, next to the difficulty of the camera solve, the complexity of the digital elements and the number of formats.

What software is used for rotoscoping?

Nuke has a dedicated Roto node, and since I composite in Nuke, that is where I would do it. After Effects is the common choice for motion graphics work and has its own Roto Brush. Silhouette is a specialist roto and paint package, and Blackmagic Fusion, part of DaVinci Resolve, handles it in a node based setup.

The software matters less than the habits. Separate shapes for separate parts, key frames only where the motion changes, and a check against a flat colour will improve a matte in any of them. If the footage is meant for a larger CGI film, the roto often works best in the hands of the person who composites it, because they know which edges the 3D element will actually touch.

How do you get footage rotoscoped?

Start by deciding whether you need roto at all. The check above gives you the answer for your shot, and the answer is sometimes keying or a clean plate. If roto is the right route, send the original clips, not a social media export, with the frame rate and the delivery format.

Say which shots have hair, smoke, glass or fast motion, because those decide the schedule. If you are still planning the shoot, the cheapest decision is made on set: an evenly lit screen, a locked camera where the story allows it and a clean plate of any area where something will be removed. The trade-off between shooting a plate and building the scene in 3D is covered in CGI vs live action product commercials.

For footage that needs a 3D element placed into it, the tracked and composited route is on the mixed reality production page. Send the clip in a few lines and you will get a straight answer on what it needs, including when the right answer is not roto.

Written by Marco Mori, 3D artist and creative director in Lindau at Lake Constance, Germany, working with brands and agencies worldwide.