TM

Course 36
SIGGRAPH 2001

From Ivory Tower to Silver Screen

Six Visual Effects Companies reveal how academic,
SIGGRAPH and studio-based Research and Development finds its way into production.

Course Organizers
Jill Smolin, Cinesite Visual Effects, co-chair V.E.S. Education Committee
Pam Hogarth, Gnomon, Inc. School of Visual Effects,V.E.S. Education Committee

Speakers from member companies of the Visual Effects Society

www. visual-effects-society.org

Digital Domain Industrial Light

Bob Cook and Magic a Division of
John Gibson Lucas Digital, Ltd. LLC
Doug Roble Tony Hudson
Steve Sullivan
PDI/Dreamworks
Jonathan Gibbs Pixar
Bert Poole Michael Fong
Mark Henne
Sony Pictures Imageworks
Armin Bruderlin The Secret Lab
Jay K. Redd John P. Lewis
Stuart Sumida David Oliver
 From Ivory Tower to Silver Screen
Six Visual Effects Companies reveal how academic,
SIGGRAPH and studio-based Research and Development finds its way into production.

Table of Contents

8:30 INTRODUCTION AND BRIEF OVERVIEW

Jill Smolin

8:45 PIXAR ANIMATION STUDIOS

Animating Monster Fur
Michael Fong,Technical Director
Creating Clothing for Monsters, Inc.
Mark Henne,Technical Director

9:45 SONY PICTURES IMAGEWORKS

Feather Software Design Issue
Armin Bruderlin, Software Engineer
Jay K. Redd, Digital Effects Supervisor
Stuart Sumida, Professor, U.C. Riverside

10:00 BREAK
11:00 THE SECRET LAB
Research and Production at The Secret Lab
John P. Lewis, Software Director,The Secret Lab
David Oliver,Technical Director,TheSecret Lab

NOON LUNCH
1:30 PDI/DREAMWORKS
Getting Under Shrek’s Skin
Jonathan Gibbs, Senior Effects Animator
Bert Poole, Senior Lighting Animator

2:30 DIGITAL DOMAIN

From Research and Development to Production:Three Perspectives
Bob Cook, Software Production Manager
Doug Roble, Software Creative Director
Johnny Gibson, Digital Shader Lead
3:00 Break

3:45 INDUSTIAL LIGHT AND MAGIC

Photomodeling and Hair Combing for AI
Steve Sullivan, Principal Engineer
From Digital Reference to Screen
Tony Hudson, CG Model Supervisor

4:40 CLOSING COMMENTS

Jill Smolin
 From Ivory Tower to Silver Screen
Six Visual Effects Companies reveal how academic,
SIGGRAPH and studio-based Research and Development finds its way into production.

Introduction

In our seat at the local multiplex, we experience incredible images that make
us cheer for the computer-generated hero, laugh when a pig converses with a
dog, or cower at monsters that exist only as pixels.The work that goes into
creating these effects starts in the minds of artists, scholars, programmers and
scientists who turn their work into images that manipulate space, animate
extinct species or make time stand still. Over the course of the day, presenters
from six of the industry's top visual effects companies will discuss the myriad of
ways in which they use research to augment the production of this year's major
feature films, including How the Grinch Stole Christmas, Monsters, Inc., Shrek,
Stuart Little 2 and AI.Topics covered in this course include using research and
development to create dynamics and fur, the collaboration between an ornithol-
ogist, engineers and artists to create computer generated feathers and flight;
research and development created in concert with production; a perspective on
different research and software development timelines; developing skin for an
animated character; and the partnership of software engineers and artists in
creating custom toolsets that are then integrated into the production pipeline.
Because film is a visual medium, our presenters will rely on images to present
this in-depth, technical look at the collaboration of r&d and art.
 From Ivory Tower to Silver Screen
Six Visual Effects Companies reveal how academic,
SIGGRAPH and studio-based Research and Development finds its way into production.

Speaker Biographies

Armin Bruderlin
Software Department
Sony Pictures Imageworks

Dr. Armin Bruderlin is currently a senior software engineer at Sony Pictures

Imageworks in Culver City, California, where he has been working to support
production on special effects for such movies as "Stuart Little" and "The
Hollow Man". Prior to Imageworks, Armin spent nearly two years as a visiting
researcher at ATR (Advanced Telecommunications Research) in Japan near Kyoto,
where he worked on producing believable human animation and intelligent agents.

Armin received a B.C. in Informatik from Furtwagen Technical College in Germany

in 1984, an M.Sc. and a Ph.D. in Computer Science from Simon Fraser University
in Vancouver, Canada, in 1988 and 1995, respectively. Armin's research
interests
include all aspects of making believable computer animated characters, as well
as procedural and hierarchical motion control schemes for computer animation.

Robert Cook
Software Production Manager
Digital Domain

Bob Cook is the Production Manager for the Software Department at Digital Domain. He is a film maker
and artist who has been making images professionally for 23 years, and involved with digital imaging since the
early 1980s. His work has won awards at film festivals and has been collected by major museums.

He has also worked in the administration of systems and software development for 10 years, primarily
within academia. Additionally, while in academia he taught photography, digital imaging, computer animation,
WWW development and design, and critical theory. He received a B.S. in Photography from Texas A&M,
Commerce in 1984, and a M.A. in Humanities from the University of Texas at Arlington in 1994.

Michael Fong,
Technical Director
Pixar Animation Studios

Michael Fong’s life at Pixar began in 1995, during his last semester at UC Berkeley. Beginning as a render-
wrangler, he climbed his way to technical director as a modeller/articulator for A Bug’s Life.Working first with
props and then with characters, Michael found himself researching simulation and rendering with the Bug’s
Life Research and Development effects group. Moving into the role of technical lead/supervisor f the crowds
department, he worked with Bill Reeves and a talented group of artists researching hair and fur. He is cur-
rently sequence supervisor on Monsters, Inc.

Michael’s credits include:Toy Story, A Bugs Life (Crowds Technical Supervisor),Toy Story 2 (for which he
created a preliminary version of monster fur), Monsters, Inc., for which he is fur technical lead and Sequence
Supervisor.

Jonathan Gibbs
Senior Effects Animator
PDI/DreamWorks

Jonathan Gibbs is currently serving as a Lead Effects Animator on the forthcoming PDI/DreamWorks fea-
ture, SHREK. For SHREK he has been leading shader development, working on fur and hair rendering and
continues to develop the crowd system which he co-developed for PDI/DreamWorks' first feature, ANTZ. A
native of St. Louis, Jonathan has a B.S. in Computer Science from Principia College. He holds a Masters degree
in Computer Science from UC-Santa Cruz.

Johnny Gibson
Digital Shader Lead
Digital Domain

John M. Gibson is currently working at Digital Domain where, since 1997, he has worked as a Digital
Artist and Technical Director. His film credits include How the Grinch Stole Christmas, O Brother Where
Art Thou, Supernova and Titanic. Prior to working at Digital Domain Johnny has worked as a Software
Engineer at Disney, a Character Animator at Marvel Films and as a System Designer and Software Engineer at
TRW’s Ballistic Missles Division. Johnny’s various research areas of interest include applied global illumination
techniques, shading systems and renderers. Johnny holds a M.S. in computer science from the University of
California, Riverside.

Mark Henne
Technical Director
Pixar Animation Studios

Mark Henne has spent ten years in production, the last six with Pixar and earlier at Rhythm & Hues.
Lately he has been bringing dynamic clothing into practical use as a Technical Director on Monsters, Inc., a
Pixar/Disney film due out at the end of 2001. His previous projects have included A Bug’s Life and Toy Story,
and the highlights while at R&H were three of the “Polar Bears” spots for Coca-Cola and a car cloth effect
for Lexus.

Mr. Henne has a MS in Computer Science from the University of California Santa Cruz, and a BS from the
University of New Mexico. His expertise is in using simulations for special effects, and the design and imple-
mentation of facial articulation software and controls.
Tony Hudson
Digital Model Supervisor
Industrial Light & Magic a division of Lucas Digital Ltd. LLC

Tony Hudson has been with Industrial Light & Magic in various capacities since 1985. In the 1980’s he
worked in ILM’s model shop as a creature maker, model maker and puppeteer, as well as supervising various
projects. Hudson left ILM in 1990 for Walt Disney Imagineering to work on theme park design, and then he
took a year off to train himself in computer graphics, returning to ILM in 1995 to work on Dragonheart. He
has been interested in animation and visual effects since he was a small child, and is thrilled to be practicing
his passion at ILM. Prior to ILM, Hudson was lead puppeteer and show supervisor for Vagabond Marionettes
and John Hardman Productions.

John P. Lewis
Software Director
The Secret Lab

John (J.P.) Lewis is a software director at the Walt Disney Company’s digital production studio,The Secret
Lab (TSL). Previously John worked in software R&D for Dream Quest Images, Disney?s feature film visual
effects division, now merged with The Secret Lab. He has also worked for visual effects companies
Centropolis and Industrial Light and Magic, as well as think tanks such as Paul Allen’s Interval Research.

John holds several patents and has published research in the areas of computer graphics, neural networks,
computer-human interfaces and computer vision. His research is discussed in books including “The Science of
Fractal Images”, “Computer Facial Animation” and “Music and Connectionism.”

Bert Poole
Senior Lighting Animator
PDI/DreamWorks

Bert Poole is currently a Senior Lighting Animator in the Feature Division of PDI/Dreamworks. Having
joined PDI 5 years ago during the visual development phase of ANTZ, Poole was responsible for surfacing
lead characters such as Bala and Mandible. While spending most of his time on ANTZ and SHREK lighting
production shots, he has been called into visual development on many occasions to handle technically difficult
lighting and surfacing issues. A senior member of the Lighting Department, Poole graduated with a B.F.A.
from Columbus College of Art and Design, Columbus Ohio in 1996.
Jay K. Redd
Digital Effects Supervisor
Sony Pictures Imageworks

Jay Redd is currently the Digital Effects Supervisor on Columbia Picture’s “Stuart Little 2”, the sequel to
the successful file “Stuart Little”. Functioning as CG Supervisor on “Stuart Little”, Jay started his involvement
with the feature in the early days of pre-production and character design, and supervised the technical and
aesthetic research and development for the films’ extensive hair, fur, and lighting requirements. As an amateur
astronomer, he animated and supervised “Contact’s” Opening Shot, a 4,710 frame journey from earth to the
end of the universe, which has received many international awards.

Before joining Imageworks, Jay spent four years at Rhythm & Hues Studios working as a Technical
Director and then Computer Graphics Supervisor on numerous features, commercials, and theme-park rides,
including the Academy Award Winning “Babe”, and the award-winning Seafari. Jay studied at the University of
Utah with an emphasis in film, music composition, and Japanese. His projects thus far have had a running
theme of space and animals.

Doug Roble
Software Creative Director
Digital Domain

Dr. Doug Roble is the Creative Director of the Software Department at Digital Domain, the multiple
Academy Award-winning visual effects studio located in Venice, CA. He has been writing software and doing
research since he joined the company in 1993. In 1999, Doug won a Scientific and Technical Achievement
Academy Award for his 3D tracking/scene reconstruction program, “track”.

Doug’s interests cover widely different areas of computer graphics. He has written many different soft-
ware tools including a fluid dynamics package, a motion capture editing and manipulation tool, a complete
computer vision toolkit and package and many others. He is currently interested in computer vision, physical
simulation, level set theory and motion capture manipulation. Before Digital Domain, Doug received his Ph.D.
in Computer Science from The Ohio State University in 1993. His dissertation was on extracting three
dimensional information from a photograph. He received his Master’s from Ohio State and his Bachelor’s
degree in Electrical Engineering from the University of Colorado.

Jill Smolin
Cinesite Visual Effects
Visual Effects Society

Jill Smolin is the Manager of Artist Development and Education at Cinesite Visual Effects. Prior to working
at Cinesite, Jill worked at Digital Domain. She has been working in and around computer graphics for the past
15 years or so. Having co-chaired the Electronic Schoolhouse for SIGGRAPH 99, Jill also organized and pre-
sented a SIGGRAPH Course in 1999 entitled “A Visual Effects Galaxy.” In addition, she presented a facial ani-
mation panel for SIGGRAPH 2000. Jill is the chair of the Visual Effects Society’s Education Committee, and
currently serves on that organization’s board.
Steve Sullivan
Computer Graphics Software Group
Principal Software Engineer
Industrial Light & Magic, a division of Lucas Digital Ltd. LLC

Steve Sullivan joined Industrial Light & Magic in 1998 as a Principal Engineer. His role specifically focuses
on vision algorithms and computer vision applications for film and video production. Sullivan also heads up
the computer vision team for ILM’s Research and Development group which is currently developing tools for
matchmoving and photogrammetry, as well as facial and full-body motion capture.

In 1996, Sullivan received his PhD in Electrical Engineering from the University of Illinois at Urbana-
Champaign, with an emphasis on automatic object modeling, recognition, and surface representations. After
graduation, Sullivan joined Rhythm & Hues Studios in Los Angeles to develop animation and 3D tracking soft-
ware.

Sullivan was born and raised in Kansas City, Missouri.

Dr. Stuart S. Sumida

Associate Professor
California State University, San Bernardino

Stuart Sumdia is an Associate Professor of Biology at California State University San Bernardino. He is a
vertebrate paleontologist and provides anatomical and locomotary expertise to the film industry.

His academic research focuses on the structure, function, and evolution of fossil animals near the amphib-
ian to amniote transition; animals that are found in sediments of late Pennsylvanian and early Permian age
(approximately 270 to 300 million years old) -- animals nearly 100 million years older than the earliest known
dinosaurs. Other activities in the CSUSB Laboratory for Vetebrate Paleontology include study of and struc-
ture and biology of theropod and ceretopsian dinosaurs.When he finds time, Dr. Sumida consults extensively
for Film and Animation studios on animal anatomy and function and has worked on several films with the
animators at Sony Pictures Imageworks, including "Stuart Little" and "Hollow Man". At Imageworks, Sumida is
currently providing expertise for the feather development of several leading characters on "Stuart Little 2".
His many publications include phylogenetic context for the origin of feathers for American Zoologist. Sumida
also particpated on the 1999 SIGGRAPH Panel "Visual Effects: Incredible Effects vs. Credible Science".

B.A., University of California, Los Angeles

M.A., University of California, Los Angeles
Ph.D., University of California, Los Angeles
 Animating Monster Fur
by Michael Fong
Pixar Animation Studios

Introduction

One of the more challenging issues tackled by the crew working on the movie Monster’s, Inc. was the
creation of a believable main character, James P. Sullivan. Sullivan, an endearing monster who learns that scar-
ing children is not the only way of life, was dreamed up by Pete Docter, the director of Monster’s, Inc.
Fortunately for his
production crew,
Pete had clear
ideas in his mind of
what he wanted
the fully CG
Sullivan to look
like. Unfortunately
for the production
crew, Pete’s ideas
generally revolved
around one of the
more complex
problems of com-
puter graphics, fur.

©Disney/Pixar 2001 As hints of

Sully Sullivan began to
emerge from the
art department, a
technical team was put together to attack the complexities posed by a furry main character. This team, which
at it’s smallest consisted of only one researcher and at it’s largest drew on the knowledge of almost every
technical mind in the studio, was loosely referred to as the fur team.The word team, however, is misleading
because it implies that only a subset of the production crew worked on this problem. In reality, the entire
studio had a hand in developing Sullivan’s fur. For what sometimes seemed like an eternity, this fur “team”
would attempt to solve the problems posed by an enormous monster covered from head to toe in blue and
purple fur.

Computer generated fur is generally considered a difficult problem because there is no official solution
or accepted methodology. It is essentially a research issue that spans many different areas in computer graph-
ics, from modeling, to rendering, to animation.These notes will attempt to explore some of the research
which went into the animation of Sullivan’s fur.
 RESEARCHING HAIR AND FUR

Drawing from Reality

Research began by examining real life examples of fur.Videos of wolves, dogs, bears, llamas, etc. were
obtained and religiously poured over. Art directors would watch the videos tagging the different clips as hav-
ing desirable characteristics or not. Scenes where the fur gave a lot of secondary motion to the animal’s walk
cycle were often pointed to and described as must-have. Short, clean, well-groomed fur, was to be avoided
and instead direction led towards a long, thick, clumpy coat which seemed to fall somewhere between the fur
of a bear and a llama.Technical members of the fur team would then pour over the same set of videos
attempting to isolate the key features the art directors were responding to. After the general characteristics
of the fur were identified, samples of bear, llama, and faux fur were brought in.These physical samples were
invaluable because they gave the team a concrete base to work from.The samples could be examined when-
ever a question about physical characteristics came up and they also provided common ground for the art
department and the TDs. Real fur samples could be pointed to with no confusion from either department
about what was being described. Surprisingly, the faux fur samples turned out to be just as informing as the
real fur.The fur team was able to develop ideas about what made the faux fur look and behave in a “fake”
way. By contrasting the real fur against the synthetic, certain real fur characteristics were isolated and identi-
fied as being mandatory for believability.

Drawing from Beauty Salons and Wool Sweaters

Armed with lofty goals, the next step was to discover what research and work had been done previously
in the field of fur. Previous work is always priceless in that even if it doesn’t seem to apply directly to the
problem at hand there is always something to learn and perhaps augment. Fortunately for the fur team,
there’s a wealth of knowledge and studies out there regarding hair and fur.The majority of wealth doesn’t
come from the computer graphics field, however, but rather from cosmetics and textile research. Quite a bit
of work has gone into researching what gives hair body, what causes baldness, what induces style retention,
and what promotes a thicker, healthier coat. All this work, though not directly related to Monster’s, Inc., help
shed some light on the underlying hair structure that the fur team was hoping to duplicate.

A Physical Description

A hair is a long, thin appendage that grows through the epidermis from follicles set deep in the skin layer
known as the dermis. Examination of a cross section of a hair often starts with the thick, transparent, over-
lapping scale structure on the outside of the shaft known as the cuticle. One layer below the cuticle is the
cortex where pigmented, fibrous protein cells are found.Within the cortex is the medulla, a center shaft of
loosely packed cells. Intercellular cement binds everything together and cystine linkages give the hair struc-
ture stability. As the hair extends from the follicle, the cuticle, cortex and medulla become dehydrated and
cornified giving the hair the majority of its motion and behavior related properties.The more important of
these properties being:
* elastic deformation (stretching, bending and torsion)
* follicle angle
* cross-section
* friction
* static charge
* cohesive forces (oils, dirt, hairspray, etc.)
 An animal’s coat of fur is just a heterogeneous collection of densely packed hairs.The fur obtains its
properties from the single-fiber characteristics of the different types of strands.This point about different
types of hairs making up the coat is a simple but important one. Cheaper versions of faux fur have a very
regular quality to them that gives away their synthetic roots. Real animal fur is made of many different types
of hairs that have different functions and thus different physical characteristics.

DEVELOPING COMPUTER GENERATED FUR

Development of the Sullivan character design was happening concurrently with the above hair and fur
research.The design which Pete Docter and the art department were narrowing in on involved a character,
10 feet tall, covered from head to toe in long wavy fur. Only Sullivan’s palms and the bottoms of his feet
would be bare. Long wavy fur meant
there would be a lot of secondary
motion as well as a lot of complex
interaction with the environment.
Compounding this, millions of hairs
would be needed to cover a 10 foot
monster which meant that the typical
key-frame animation done at the studio
was impractical. Instead, the fur team
focused on a more automatic solution
in the form of a hair simulator.

©Disney/Pixar 2001

Full body shot of Sullivan

The Simulator

Like almost all simulators, the heart of the Monster’s, Inc. fur simulator is its physics model. Early on, the
model chosen for Sullivan’s fur was a simple mass/spring system.This model, despite its lack of complexity,
has the advantages of being fast and efficient. It was also easily incorporated into an existing physics simulator
previously used by the studio.The first version of Sullivan’s fur was developed with a modified version of the
clothing simulator used in the short movie, Geri’s Game.This simulator was later replaced with a completely
new version, but despite the massive reworking, both simulators operated around the same five main con-
cepts:
• Points - are infinitely small masses
• Springs - connect two points and exert a force on them when the spring is stretched or compressed
• Hinges - prevent the angle between two springs from varying from some rest value
• Nailed points - are points that derive their motion from sources external to the simulation
• Forces - like gravity and collisions exert an influence on points

Reducing Reality to the Essentials (or Less)

Unfortunately, the five main simulator concepts were not enough to model all the hair properties identi-
fied in previous paragraphs as contributing to the motion and behavior of fur. Stretching and bending could be
modeled but some properties like torsion and static charge couldn’t be incorporated easily so were left out.
This reduction of reality simplified the hair model but would lead to problems during production.
 A single hair from Sullivan’s coat ended up being represent-
points and ed by a small number of points linked in a chain by a set of stiff
spring model springs. Spring constants and rest lengths controlled how the
hair stretched. Hinges were established along the chain at each
interior point to give rigidity to the hair.The hinge constants
and rest angles, along with the point masses, dictated how
much the hair would bend.The root of point of each hair was
then “nailed” to Sullivan’s skin and thus derived its position and
motion directly from Sullivan’s movement. In this way, as
Sullivan’s skin moves, the nailed points are
©Disney/Pixar 2001
dragged with it. As the nailed points
move, forces are exerted through the
springs and hinges to cause the rest of
the hair to follow along.

Various causes
of hair
movement ©Disney/Pixar 2001

CG FUR IN PRACTICE

From a production pipeline perspective, character animators are asked to animate a bald Sullivan through
a shot.This animation is then fed into the fur simulator which nails a huge number of point/spring/hinge
chains to Sullivan’s skin. As the simulator walks through the shot in time, with timesteps much smaller than a
single frame, the skin moves and the simulator computes the corresponding movement of the hairs. In
essence, the fur simulator plays the role of a secondary animator taking cues from the work done by a lead
character animator.

Interpolation Produces Bland Fur

The ideal solution to Sullivan’s fur would be to model and simulate each and every single strand of the
millions of hairs on Sullivan’s body. Unfortunately, this solution suffers from obvious time and space concerns
and was, rather early on, ruled out as impractical. Instead the decision was made to approximate Sullivan’s fur
solution by simulating only a sparse set of hairs, keyhairs.These keyhairs, which are found at the control ver-
tices of Sullivan’s skin mesh, describe the macro level features of Sullivan’s fur. Instead of representing a single
strand, a keyhair represents the nearby region of fur. Keyhairs are converted into a vector description and fed
into a generalized Catmull-Clark subdivision algorithm to produce interpolated hairs known as inbetween
hairs. In this way, the same subdivision algorithm used to generate Sullivan’s skin is used to interpolate the
hairs and attach them to the surface.
 ©Disney/Pixar 2001

©Disney/Pixar 2001

Inter-hair friction and collision can-

not be modeled directly when hairs are
interpolated. Instead, simulation was
used to describe more macro level fea-
tures of the fur like overall fur volume.
Straight interpolation also implies that
neighboring hairs are very similar to one
another. Unfortunately, research strongly
suggested that this would look fake and
synthetic. Real fur, it was established, was
made of many different types of hairs
©Disney/Pixar 2001 each with their own physical characteris-
tics.To reproduce the complexity of dif-
ferent hair types the builder concept was introduced. A builder is a small snippet of code whose main respon-
sibility is to generate the geometry of the inbetween hairs. A builder is run for each inbetween hair and
essentially modifies the interpolation scheme on a strand by strand basis. A lot of the control, which had been
taken away because only a subset of the hairs could be simulated, was returned through procedural manipula-
tion in the builder.
 void SampleBuilder (...)
{
hairLength = hairWidth = hairWave = 1.0;

t = random float between 0 and 1; /* seeded by hair strand Id */

if (t < 0.10)
{ /* guard hairs */
hairLength = hairLength * 2.0; /* generated 10% of the time */
haiwWidth = hairWidth * 2.0;
}
else if (t < 0.30)
{ /* underhairs */
hairLength = hairLength * 1.5; /* generated 20% of the time */
hairWidth = hairWidth * 0.5;
hairWave = hairWave * 2.0;
}

if (this region of skin has been identified as having extra wavy hair)
{
hairWave = hairWave * 2.0;
}

generateASingleHair (hairLength, hairWidth, hairWave, keyhairs);

}

The above builder psuedocode produces guard hairs 10% of the time, underhairs 20% of the time and
normal hairs all other times. Guard hairs, often referred to as shield hairs, are longer and wider than normal
hairs and generally act as a form of sensory input for an animal.They are particularly important to consider
when developing any type of faux fur because they break up the silhouette edge in an organic way. Underhairs
are very fine hairs found “under” the majority of fur and serve to keep the animal warm and insulated. It was
originally believe that these underhairs hairs would help increase the apparent density of Sullivan’s fur but
were dropped because they didn’t add much to the overall picture and turned out to be expensive to render.
Continuing with the above builder example, depending on where on the skin the hair is being grown, a hair
may also end up being wavier than normal.This type of individual strand control was achieved with a combi-
nation of scalar fields and texture maps which dictate hair characteristics in particular regions of the body.
Texture maps put the control into the art departments hands which allowed Sullivan’s look to progress much
more quickly than if it had been transcribed by technical hands.

Simulation Error

Builders gave the fur team the ability to produce the necessary visual complexity required for CG fur but
they also introduced simulation error because the inbetween hairs were no longer behaving exactly like the
keyhairs. This disparity leads to situations where the keyhairs respond correctly to collisions but the inbe-
tween hairs behave poorly because they are being procedurally modifed without taking collisions into
account. Interpolation, by its very nature, also introduces error because it suffers from sampling problems.
Since collisions can only be detected by keyhairs, very thin collision objects can come into contact with inbe-
tween hairs and no collision is registered.
 These were fundamental problems with Sullivan’s fur design
that were never actually solved. Instead, workarounds were devel-
oped to prevent visual artifacts.When needed, collision objects
were scaled up in size.This scaling happened mathematically, not
visually, so thin objects would appear larger than they were to
the keyhairs.The keyhair resolution was also increased to both
better match what the builder was producing as well as increase
the number samples taken for collisions. Lastly, a fix tool was
developed to allow the user to select and modify portions of
Sullivan’s fur pre and post simulation to correct problems.This
©Disney/Pixar 2001 final aid, despite having only a small feature set, turned out to be
thin collision object falling between keyhairs invaluable in getting shots approved. It allowed simulation param-
eters to be tweaked on a keyhair basis as well as allowed the
user to directly manipulate the output of the simulator.

Incomplete Physics Model

The inability to fully model all the different characteristics governing hair motion often led to undesirable
behavior even for the simulated hairs. Real hairs can only be stretched a certain amount (depending on the
humidity) until either the root is torn from the follicle or the hair breaks. Nailed, point and spring chains, on
the other hand, can be stretched to any length if the necessary force is applied to them.This unreleastic
behavior caused problems when Sullivan’s fur was snagged by collision objects. Keyhairs would get sand-
wiched by objects and then yanked with great force. Hairs would stretch impossible amounts and the illusion
of believable fur would break down.

Sandwiching keyhairs was a very common problem and it would appear all over Sullivan’s body even when
other collision objects weren’t present. Fur could get sandwiched inside the armpits, the elbows, the neck
folds, etc. Sandwiched hairs, because they are in an illegal state and are feeling strong collision forces from
many different directions, would often oscillate in distracting ways that made Sullivan’s fur almost seem as if it
was infested with rats.

The solution to many of the inadequacies of the physics model was to identify and attack each problem
separately.The simulation coders of the fur team spent many hours tracking down the cause for each stretch-
ing and oscillation case that came their way. In the end a variety of different heuristics were developed to
handle the problem scenarios.

Over-stretching was solved by endowing the simulator with the concept of a hair. Previously, everything
was just points, springs, hinges, nails and collision objects. Now, a chain of points, springs and hinges could be
grouped into a hair object which could then have a maximum stretch amount attached to it.When a keyhair
is stretched beyond its maximum, the points realize that they are in an invalid state and essentially nail them-
selves into place until the offending collision objects have moved away. In practice, this appears as if the hairs
have simply slipped out of the grasp of the collision object.

Oscillations were caused by many different things and some of the cases were solved by endowing the
simulator with the concept of a scalp.The Sullivan skin collision object was identified as the fur’s scalp and
was treated differently than other collision objects. Hairs could detect when they were buried inside the
scalp and instead of oscillating, would simply ignore the extreme forces on them and behave in a more sub-
dued way.
 CONCLUSION

The research into Sullivan’s fur began well before the production of Monster’s, Inc. started and didn’t stop
even when production was in high gear. Research like this never really stops because the problem is too open
ended.The software, ideas, and solutions go through evolutionary stages during production as the different
components are put to the test. Often times the original ideas failed because of the demanding scenarios a
monster movie can require. When these ideas failed, the code was augmented or reworked to hopefully pro-
duce a better piece of software.

REFERENCES

D. Baraff. Linear-time dynamics using Lagrange multipliers. Computer Graphics Proceedings, Annual
Conference Series: 137-146, 1996

Tony Derose, Michael Kass,Tien Truong. Subdivision Surfaces in Character Animation. Computer Graphics
Proceedings, Annual Conference Series: 85-94, 1998
 Creating Clothing for Monsters, Inc.
Mark Henne
Pixar Animation Studios

Monsters, Inc., Pixar’s new film for November 2001,

features a little girl named Boo who travels through her
closet at bedtime into the world of monsters. Boo
spends much of the film wearing the oversized t-shirt
she had on when her parents tucked her into bed.

My responsibility was in constructing the shirt, and

finding the right behavior characteristics for it. Just as
importantly, I needed to work with the software devel-
opment team to make it robust enough so that it would
breeze through the simulation software for the vast
majority of the more than one hundred shots where it
appears in the film. Dan Herman and Brad Winemiller
also spent significant time working on the cloth model
and running tests.

©Disney/Pixar 2001 Complete Overhaul of “Geri’s Game” Cloth

We began by looking at the software that had been
Boo developed within Pixar to create the clothing for Geri,
the star of our short film Geri’s Game. Michael Kass had
written that software and made it function for the short film, but we felt we could take advantage of some
lessons learned by treating that program as a prototype and starting over from scratch. The undertaking got
the necessary staffing when David Baraff and Andy Witkin
joined Pixar, and they teamed up with Michael to start again.

Witkin and Baraff had written the engine behind Maya

Cloth, and also felt they could do it one better the second time
around, especially when combined with Kass’ numerical solver
techniques.

Our preproduction team therefore worked while the simu-

©Disney/Pixar 2001
lation software was in development, creating models, running
tests, and prioritizing the work of the software development Geri’s Game
team.The result is a piece of software known as fizt, used for
simulating fizix.

Jumping From the Ivory Tower

Making practical, robust clothing involves solving a lot of problems that aren’t covered in the Siggraph
research papers. Dealing with cloth sandwiched between body parts is one such difficulty, which is similar to
contact with the character’s external environment. But to begin with, a cloth behavior must be found that
satisfies the director’s version of reality, which is not necessarily in line with the normal laws of physics, and
can often be self contradictory. This we call cartoon physics.

This talk covers topics on jumping from the ivory tower down to the pavement.
 Real Physics vs. Cartoon Physics
When making a movie, you’ve got it right when the director says it’s right. In making an animated film,
stylization is the rule; just because something works a particular
way in real life doesn’t mean it’s right for the film.

Our shirt does not have folds as small as one normally gets
with t-shirt fabric. This is intentional, because Pete Docter, the
director of Monsters, Inc., wanted broader folds. Pete felt that
the tight and busy folds characteristic of a real t-shirt would
have too much detail and be distracting. He preferred a smaller
number of clean lines, helping to accentuate the body arcs. The
fabric weight we chose was closer to that of a sweatshirt.

The choice of broader fold patterns had implications for

the tesselation of the cloth surface. Finer folds would have
required more triangles and correspondingly longer compute
times. Larger folds caused problems in tight areas such as
under the arms; it took careful tailoring and control over bend
resistance to avoid rainbow shapes as Boo’s arms are brought
down by her sides. For a while we tried increasing the resolu-
tion and reducing bend resistance specifically in this area, but
eventually ended up with uniform resolution within each panel.
©Disney/Pixar 2001
Nonetheless, Broad Folding Patterns
when Boo’s pose
is closer to a
fetal position, or she has her arms raised high, the extra fabric
in the compressed areas and preference for larger folds can
make the shirt look less than its best. The same is true in real
life; a man’s suitcoat will pull and bunch in unattractive ways if
he is seated without
unbuttoning the jacket.

In animation, char-
acters can move much
faster than people do
in real life, yet we still
©Disney/Pixar 2001 want a reasonable
result in the simula-
Boo’s Shirt Resolution tion. Reasonable may
not be the same as
realistic. In our prototype testing, we had Moo (the test version of
Boo) jumping down from on top of Sullivan’s head. Trouble was, her
shirt would ride up too far because of the fast acceleration. The
solution? An Inertial Field Generator(patent pending), which caps
excessive acceleration from the base coordinate frame. ©Disney/Pixar 2001

Boo’s Shirt Resolution

 Something we wish we could have done, would have been
to provide a way to direct the cloth folds in 2D screen space,
to accentuate the character’s body arc and line of action.
Trouble is, a specified folding line may fight the tailoring or
other natural forces imparted on the cloth. In addition, contin-
uous edge runs within the mesh would be needed to trace
along the fold. Furthermore, it’s not clear what specifically
should be done to the internal forces to enforce the desired
folding. This is an area of current study.

Internal Cloth Forces

There are just a few basic forces established on the triangu-
lar cloth mesh. Keep in mind the basic terms for fabric prop-
erties: the warp runs the length of the bolt, and the weft trav-
els the width, weaving through the warp. The bias is the diago-
nal direction. Knit fabrics also have variable (anisotropic)
behavior in different directions.

First, there is stretch resistance, enforced along the edges

©Disney/Pixar 2001
of the triangles.The resistance can vary, based on the angle of
Moo Jumping Down Off Of Sullivan the edge with respect to the warp and weft thread directions.

Next, adjacent triangles fold along their shared edge. Bending along the fold line is given a resistance
value.

The last basic force type is shear resistance.

Each triangle tries to maintain the angles within
its form. This is used to fight deformations along
the bias.

In our fizt software, many of our forces can

be non-linear. Stretch and shear can have low
resistance for low amounts of deformation, rep-
resenting the taking up of slack in the weave.
However, at a certain point the slack is gone, and ©Disney/Pixar 2001
resistance climbs dramatically. We found the Basic Cloth Force Types
same concept to be useful for bending resistance
too. At a large fold radius, a small resistance
value is appropriate. As the fold grows tighter, bend resistance can grow significantly to enforce a minimum
fold size.

The next most important force is shape retention. Because of internal friction between the threads of
real fabric, once a fold pattern is achieved, the garment will like to hold onto it. Some fabrics, like silk, have
less internal friction and come out of their folds more easily. Finding a good shape retention force will keep
the cloth from oozing once a character’s pose is relatively constant.

The cloth also needs to have knowlege of its own form, and not pass through itself. That means noticing
during a calculation timestep that a point has passed through a triangle cloth face. Since a cloth surface does
not have an inside and an outside this test is challenging to get right, but increasing the number of timesteps
within a frame makes it easier for the solver. There will be times that a knot forms, especially when the cloth
is sandwiched between skin layers, and the best thing the solver can do is notice the knot and temporarily
deactivate internal forces to allow the knot to simply fall away.

For some garments, it may be of value to simulate a semi-rigid curve along a seam, like a wire stiffener.
We didn’t find this of value in our t-shirt, but can see the
potential for other clothing articles.

Collision Objects
Collision objects are things in the environment colliding
against the shirt. The most important collision object in our
case is Boo’s body. Our character collsion objects share a
subset of the data points from the original model, but with
unneccessary detail culled.

Rebuilding the collision mesh is done following the follow-

ing basic principles:
• There should be no holes in the surface.
• The shape should be soft and round, with no sharp
angles between faces.
• The object size should be large relative to cloth trian-
gle mesh.
• Nooks and crannies should be smoothed over.
• Insides of knees & elbows are smoothed to remove
the crease.
• Fingers are replace with mittens, and thumbs removed. ©Disney/Pixar 2001
• A shrink-wrapped envelope shape represents the
hand’s overall silhouette. Boo’s Collision Body
• A version with fingers is available if absolutely
necessary.

The biggest problem with collision objects is sandwiching and interpenetration. Sandwiching occurs when
the cloth is trapped between two body parts or the body and
another collision object. A typical example of this is what can
happen if your garment is a pair of pants, trapped between the
calf and thigh of a squatting character. The simulation software
needs to detect that the cloth points are inside two surfaces,
and try to find a
place of compro-
mise between
the two. Most
importantly, it
needs to be sta-
©Disney/Pixar 2001 ble and not wig-
gle under these
Pants Fabric Sandwiched Between Calf & Thigh
conditions.

©Disney/Pixar 2001

Cutaway Showing Intersecting Calf & Thigh

 ©Disney/Pixar 2001 ©Disney/Pixar 2001 ©Disney/Pixar 2001

Moo Trying a Somersault Cutaway - Moo’s Chin Still Cutaway - Moo’s Chin Tucks Inside Her Chest
Outside Her Chest

This case of Moo trying a somercault illustrates a particularly evil example. Early in the motion her chin
is clear of her chest, but as she tucks into the roll, it suddenly passes inside. The shirt had a good surface to
collide against at one frame, and that surface is suddenly gone the next frame. The end result is that the col-
lar passes through her neck. In this test case, we cheated to get a good test by adjusting the animation data
seen by the simulator so that her head doesn’t tuck in as much.
It still worked even though the rendered animation was kept in
its original state.

Results
In the end, our software and model were very robust. Nine
out of ten shots went through the first time with the default
setup. For the remaining ten percent, we used tricks like turn-
ing off collisions in a particular region for a few frames, or tem-
porarily glued down a few points in the mesh. There were
some shots where we asked for a change in the animation, but
those were few in number. We only had to ask for a little
cleanup, not anything that would change the acting.

Papers & Other Reference

Baraff & Witkin, Large Steps in Cloth Simulation, Siggraph
‘98 Proceedings DeRose, Kass, & Truong, Subdivision Surfaces in
©Disney/Pixar 2001
Character Animation, Siggraph ‘98 Proceedings Cloth and
Clothing in Computer Graphics - Siggraph ‘98 Course Notes
The Collar Passes Through Moo’s Neck
 Feather Software Design Issues
Armin Bruderlin
Sony Pictures Imageworks

One of the main software developments necessary for the upcom-

ing motion picture “Stuart Little II” has been the implementation of a
flexible, robust and efficient feather pipeline to produce realistic feath-
ered creatures.

Stuart’s new companion is Margalo, a female bird. Like Stuart, she is

completely computer-generated. In terms of her looks, Margalo is a
blend of an American goldfinch and a (domestic) canary (which are both related and belong to the same tax-
onomy family, Fringillidae). Her anatomy is more like a canary, her feather patterns more like goldfinch, but
with lighter wing colors and more yellow tones, especially around the head.

Thus, Margalo is a somewhat imaginary bird, both in terms of her looks and, like Stuart, in terms of her
rather human-like behaviors. With respect to our efforts to producing cg-feathers, this meant that we didn’t
have to match an existing bird exactly, but rather satisfy the creative vision of the director. So instead of a
“real” feather coat, Margalo needed “realistic”, convincing and believable feathers. Currently, we are develop-
ing the look of another bird character in the movie, a Peregrine falcon, which is required to look very much
like a falcon in the real world.

From these requirements, we knew that we would have to be able to produce realistic cg-feathers as an
end-result. But we also knew that our feather pipeline needed to be practical, flexible (easy to modify and add
new effects), robust (reliable to push lots of frames through), efficient (as inexpensive as possible, in time and
storage), and easy to use by animators. During an initial research phase — which included field trips to
museums, live-bird demos, in-house lectures on bird anatomy, behaviors and looks, studying books on birds
and feathers — we realized that there was little published material in the computer graphics literature on
this topic. Based on our experience and the information gathered, we decided to divide our research and
development efforts into two camps: designing a realistic feather primitive, and designing a pipeline for gener-
ating a feather coat (consisting of these realistic feather primitives). Both are addressed briefly in the follow-
ing sections.
Feather Primitive

The tools to generate a realistic, individual feather primitive fall into two major categories: design or mod-
eling tools, and rendering or shading tools.

Real feathers come in all kinds of different shapes, sizes, colors and configurations.This is certainly true
for feathers of different birds, but also for the feathers which cover just a single bird.There are contour feath-
ers, flight feathers (remiges), tail feathers (rectrices), down feathers, etc.We wanted to be able to create all
these feather types within a single model. For this purpose, we have identified over 200 descriptive feather
attributes, which our modeling software then maps to specific feather geometries. Examples of these attrib-
utes are: shaftLength, shaftRadius, vaneWidthLeft, vaneWidthRight and downHairDensityVaneRight. Each of
the attributes has a default value, and certain combinations of attribute values define different feather types,
such as down or contour feathers.The final feather geometry is composed of a NURBS surface for the shaft
(quill) of the feather, and one or more NURBS surfaces for each left and right vane, plus any number of
NURBS curves for the down hairs and barbs, depending on the attribute values.With this model, we have
been successful in accurately defining the shapes of a wide variety of feathers.

Real feathers also interacts with light in many intricate ways, and due to simplified NURBS models of our
feathers, special shading methods are necessary to account for effects like reflection, opacity, texturing, color
and self-shadowing of feathers.We also need to simulate the anisotropic (*) nature of the vanes of a feather.
We have developed special feather shaders to account for these effects.

(*) a surface is considered anisotropic if the light intensity reflected to the viewer varies when it is rotat-
ed around its normal while light and viewer direction remain unchanged.

Feather Coat

Generating a convincing feather coat in computer graphics which covers the skin of a whole bird requires
dedicated solutions to a number of problems. First, it is infeasible to individually model and perhaps animate
all of the large number of feathers individually. Other problems which can arise are collision of feathers, both
between neighboring feathers and between feathers and the underlying surface. Finally, feathers are not static,
but move and break up as a result of the motion of the underlying skin and muscles, as well as due to exter-
nal influences, such as wind.

We designed a feather generation system to address these issues. Based on the experience we gained
from the development of a related system to generate hair/fur for “Stuart Little,” we have implemented a
new, improved and more powerful pipeline to put feathers on a bird. Some of the design features have been:
- define and animate a few simple key curves, ribbons or feathers primitives (hundreds), then auto-
matically produce the full feather coat (tens of thousands) from these primitives plus their attrib-
utes. Each final feather produced this way is called a procedural feather.
- provide grooming (combing) and animation tools for these primitives.
- separate static calculations (once-only, e.g. feather position on skin) and frame-dependent (e.g.
feather orientation) calculations for efficiency. The static calculations are done before render-
time, frame dependent information is computed during rendering.
- include a mathematical expression language to modify feather attributes, both during static calcula-
tions and at render-time. For example, based on their feather-id, certain feathers could be length-
ened or shortened at render-time from their predefined length according to an arbitrary
expression; or, we might want to break up the groomed feather look a bit by small random off-
 sets (expressions) in the rotations around the underlying surface normal axis, twist axis or lay-
down axis of each final feather.
- provide feather collision detection and response, i.e. prevent feathers from interpenetrating, on
demand as a pre-rendering step.
- supply a variety of automated and semi-automatic feather placement tools. These range from ran-
dom placement, grid positioning, placement based on density maps or particle repulsion, place-
ment along isoparms, direct placement or combinations of these.These tools are necessary to
achieve various desired looks.Whereas in real birds, feather follicles are often placed along
well-defined dense tracks on skin called pterylae, Margalo does not necessarily follow this rule as
a imaginary bird.
- provide mechanism to turn a procedural feather into a hand-animated “proxy” feather on demand,
for instance if the script requires a feather or feathers in a certain area to move a certain way.
- accomplish seamless integration of all feathers: the procedural feathers, the proxy feathers and the
“hand-animated” secondary and primary feathers along the wings.The latter are specially
physiqued and part of the primary animation, as they express gesturing and are used to interact
with objects.
- achieve smooth migration of all files through the pipeline by a reliable versioning and publishing sys-
tem.

We are still adding new features to the feather pipeline and finetuning the performance as we apply the
tools to the second computer-generated bird character, the falcon. As the look of Margalo is in its final
approval stages, we have been able to produce convincing feathers with our approach.
 Research and Production at the Secret Lab
John P. Lewis, David Oliver
The Secret Lab

Contents:

• Research-Production Lag

• Kolmogorov Complexity: Objective Estimation of Software Schedules is

Impossible.

• Case studies

– Case study: Pose Space Deformation

– Case study: Disney Skin & Muscle System (David Oliver)
– Case study: Dalmatian Spot Removal

• Software-Meets-Production Anecdotes
Research-Production Lag

• Template Matching
(feature tracking, image registration)
Pratt, “Correlation Techniques of Image Registration,” 1974
Early production use: Forest Gump 1994, Speed 1994

• Optic Flow
Horn and Schunck, “Determining Optical Flow,” 1980.
Commercial product: Cinespeed, 1995
Notable production: What Dreams May Come, 1998; Matrix, 1999.

• Structure from Motion

(determine camera and scene points without calibration object)
Early papers: 1989-early 90s
Commercial product: 3D-Equalizer, SFM feature in 1997.

• Early (60-80s) research was not adopted quickly simply because

digital effects were rarely used.

• Pace of research adoption is probably increasing.

The Software Estimation Dilemma

• Production requires useful time estimates.

• Objective estimation of software development schedules is not just difficult

but mathematically impossible.
(see “Some Limits to Software Estimation”, Software Engineering Notes
Vol. 26, July 2001, reprinted here.)

The statement above is contrary to the claims of an industry of software consul-

tants:

“In the Certainty state [of quality management], the objective of

software development and software quality management, producing
quality software on time with a set cost everytime, is possible.”

G. G. Schulmeyer, “Software Quality Lessons from the Quality Experts,” in G. G. Schul-

meyer and J. I. McManus, Eds., Handbook of Software Quality Assurance (2nd
ed.), Nelson Canada, 1995. p. 76.

“[In a mature organization] There is an objective quantitative basis

for judging product quality and analyzing problems with the product
and process. Schedules and budgets are based on historical perfor-
mance and are realistic”

M. C. Paulk, B. Curtis, M. B. Chrissis, C. V. Weber, The Capability Maturity

Model for Software Version 1.1, CMU/SEI-93-TR-24 Feb. 93, p. 19.

“...software estimating can be a science, not just an art. It really is

possible to accurately and consistently estimate costs and schedules
for a wide range of projects,” etc.

W. H. Roetzheim and R. A. Beasley, Software Project Cost and Schedule Estimat-

ing: Best Practices, Prentice Hall, Upper Saddle River, New Jersey, 1995, p. xvii.
Definitions:

The estimation of software schedules, programmer productivity, and program com-

plexity is termed software estimation.
An estimate is objective if there is a well defined set of steps that can be followed
by persons of differing opinion to arrive at a common conclusion. By the Church-
Turing thesis, this set of steps is essentially an algorithm.
An estimate is feasible if it can be obtained in our lifetime. Objective upper bounds
on algorithmic complexity are possible but most are not feasible, see below.
Kolmogorov complexity (also called Algorithm Complexity (AC) and KCS com-
plexity) defines the complexity of a digital object to be the length of the shortest
computer program that produces that object. We will consider the complexity of
a program to be the length of the shortest program with the identical behavior, or
Algorithmic Complexity of the program specification or input/output table.
Kolmogorov complexity is a relatively recent idea (developed since the mid 60s).
It is an interesting concept with connections to mathematical incompleteness and
logical paradoxes.
Kolmogorov Complexity Explained

We know complexity when we see it, but how can we define it formally?
Kolmogorov complexity formalizes an intuitive notion of complexity. Consider
the three patterns:

11111111111111...
12312312312312...
30547430729732...

These strings may be of the same length but the first two strings appear to be
simpler than the third. This subjective ranking is reflected in the length of the
programs needed to produce these strings. For the first string the program is a few
bytes in length, e.g.,

for i:=1 to n print(’1’);

The program for the second string is slightly longer since it will contain either
nested loops or the literal ’123’. If there is no obvious pattern to the third string,
the shortest program to produce it is the program that includes the whole string as
literal data and prints it — the string is incompressible or algorithmically random.
Ideally the complexity of an object should be a property only of the object itself,
but the choice of computer and programming language affects program lengths.
Kolmogorov complexity handles this issue by considering the complexity to be
well defined only for large objects. A translator or emulator from any language or
machine to any other is a fixed-size program, possibly of about 100K bytes or less,
so the choice of an inelegant language or machine adds only a constant amount to
the algorithmic complexity; this amount becomes insignificant in the limit of large
objects.
Objective Estimation is not Possible: Argument

Thesis: Algorithmic Complexity (AC) is an appropriate measure of the complex-

ity of programs. AC addresses the well known problems of code metrics, i.e., that
they may not reflect the actual complexity of the program!
Crux of the argument: AC is not computable. There is no algorithm for comput-
ing algorithmic complexity. AC can be bounded, but not feasibly.
To get an intuitive feel for the ‘complexity of defining complexity’, consider the
number of possible programs involved – there are too many! One recent version
of Microsoft Excel is more than 7 megabytes. Consider instead a far simpler set of
programs, those that are 8 bytes or less in size. Such programs are too small to do
anything useful (or anything at all), but how many of them are there: 26 4. That is 4
billion 4 billions, since 23 2 is roughly 4 billion. Get a very fast computer with a 4
gigahertz clock (so it can compute roughly 4 billion instructions/second). For this
computer to do a simple check on each of the tiny 8-byte programs would require 4
billion seconds for each instruction in the check – the simple check would require
centuries of computer time.
Claim 1: Program complexity cannot be objectively estimated. This is just a re-
statement of the non-computability of AC.
Claim 2: Development time cannot be objectively estimated. This claim is a direct
consequence of claim 1. There is some limit on the speed at which a programmer
can write code, so any development time estimate that is supposed to be indepen-
dent of program size will be wrong if the program turns out to be larger than can
be constructed during the estimated time period.
Claim 3: Absolute productivity cannot be objectively estimated. Consider a state-
ment that N lines of source code were developed in T time, with N/T higher
than measured on other projects. This suggests that higher productivity has been
achieved. But productivity is relevantly defined as the speed of solving the prob-
lem, not the speed of developing lines of code. If the N is significantly higher
than it needs to be for the particular problem then a high N/T ratio may actually
represent low productivity.
Discussion

Consider the following scenario: A company wishes to improve the predictabil-

ity of its software process, so it decides to gather statistical data on development
times. As part of this effort each programmer is assigned a timed series of exer-
cises. It is found that the average programmer at the company can complete each
exercise in an average of 3.7 hours. Now the company is asked to bid on the devel-
opment of an operating system for a major company that is years behind schedule
on their own operating system development project. Based on the newly gathered
statistics the company estimates that it can deliver the new operating system in
about 3.7 hours using one average programmer.
This absurd example is intended to clearly illustrate that estimates of development
time depend on estimates of the size or complexity of a new program, and histor-
ical statistics cannot provide the latter. In the preceding example the complexity
of the proposed operating system is presumably much greater than the complexity
of the programming exercises, but the data do not say anything about the relative
difference in complexity.
What about the case where a new program is very similar to a program that was
written previously? In this case one’s subjective estimate of the complexity and
development time are surely going to be more realistic, on average. But this is
not enough to make statements of certainty. Every experienced programmer has
come across seemingly trivial tasks that turned out to be much more difficult than
initially thought – often so much more difficult that the whole task is abandoned.

Programming and Math are Related

There is another, simpler, suggestion of the impossibility of software estimation:

computer programs and mathematics are deeply equivalent. A proof can be trans-
lated into a string-processing computation and vice-versa:

axioms ⇐⇒ program input or initial state

rules of inference ⇐⇒ program interpreter
theorem(s) ⇐⇒ program output
derivation ⇐⇒ computation
As such, if it were possible to objectively estimate software development times,
then it should be possible to objectively estimate the time needed to write a pro-
gram that proves a theorem.
For example, the Goldbach conjecture (that every even integer exceeding two is
a sum of two primes) dates from 1742 but remains unresolved in 2001. Can you
provide an objective estimate of how long it will take to write a program that
proves/disproves this conjecture?

Conclusions
• Objective estimation is not possible, but subjective estimation is. Find ex-
perienced people with good opinions. They won’t always be right.

• Rather than relying on unfinished software, plan ahead if possible. Finish

the software first, as a background task. This will save money too.
Brooks law: adding more programmers to a late project makes it later (the
same work can be done with less labor if a few programmers are allowed
to work for a longer period).

• Use rapid prototyping.

• Shift in perception. Some “artist” tasks such as lighting can probably be

automated, and probably will be in a decade. Programming cannot be au-
tomated (incompleteness). Which is more of an art, something that can be
automated or something that cannot?
 Some Limits to Software Estimation
J. P. Lewis
Disney TSL
3100 Thornton Ave.,
Burbank CA 91506 USA
[email protected]

Abstract Similarly, a book promoting a software estimation package [15]

Algorithmic (KCS) complexity results can be interpreted as indi- states that “...software estimating can be a science, not just an art.
cating some limits to software estimation. While these limits are It really is possible to accurately and consistently estimate costs
abstract they nevertheless contradict enthusiastic claims occasion- and schedules for a wide range of projects,” etc.
ally made by commercial software estimation advocates. Specifi-
The answer to our question (‘can software be objectively esti-
cally, if it is accepted that algorithmic complexity is an appropriate
mated?’) has both practical and ethical implications. Newspaper
definition of the complexity of a programming project, then claims
headlines frequently describe the cancellation of costly software
of purely objective estimation of project complexity, development
projects that are behind schedule and over budget. With com-
time, and programmer productivity are necessarily incorrect.
puter programs now widely deployed in socially critical roles it
Keywords is recognized that software professionals have a responsibility to
Estimation and metrics, project management, risks, ethical issues. make accurate and truthful characterizations of prospective soft-
ware sytems.
Introduction
Among the important practical problems in software engineering Given the existence of the various software methodologies and
is software estimation — the estimation of development sched- processes alluded to above, it would be easy to conclude that the
ules and the assessment of productivity and quality. Is it possible problem is merely that these methods are not being practiced.
to apply mathematical and scientific principles to software esti- On the other hand, considering the wide variety of competing
mation, so that development schedules, productivity, and quality methodologies and the well considered critiques of some of these
might be objectively ascertained or estimated rather than being a methodologies [1, 2, 3, 4, 8], one may be tempted to adopt an out-
matter of opinion? side perspective and ask whether all of the stated goals of these
methodologies are possible even in principle.
The debate over this question is familiar. In the case of develop-
ment schedules, for example, many programmers find it self ev- In this paper we will look at software estimation from the point of
ident that accurate and objective estimates are not possible. One view of algorithmic or KCS (Kolmogorov-Chaitin-Solomonoff)
reader of an early version of this paper commented, “Software complexity. Section two introduces the notion of algorithmic
practitioners know about poor predictability from empirical evi- complexity. In sections three and four we will find that algorith-
dence. I don’t need to prove it...” mic complexity results can be directly interpreted as indicating
that software complexity, development schedules, and productiv-
On the other hand, there are a large number of design methods, de- ity cannot be objectively and feasibly estimated and so will re-
velopment processes, and programming methodologies that claim main a matter of opinion. Section five considers approximate and
or hint at objective estimation of development schedules, project statistical estimators. The situation here is more optimistic, but
complexity, and programmer productivity. For example, a hand- we give reasons for maintaining a scepticism towards overly opti-
book of software quality assurance describes the benefits of a mistic claims of estimation accuracy. Although these are negative
quality management process [16]: results, we will argue in concluding that the software industry can
“In the Certainty state [of quality management], the ob- benefit from acknowledging and publicizing the inherent limita-
jective of software development and software quality tions and risks of software development.
management, producing quality software on time with ALGORITHMIC COMPLEXITY
a set cost everytime, is possible.” Algorithmic complexity (AC) defines the complexity of a digital
A software process manifesto states [14] object to be the length of the shortest computer program that pro-
duces that object. This definition formalizes an intuitive notion of
“[In a mature organization] There is an objective quan- complexity. Consider the three patterns:
titative basis for judging product quality and analyzing
problems with the product and process. Schedules and 11111111111111...
budgets are based on historical performance and are re- 12312312312312...
alistic” 30547430729732...
These strings may be of the same length but the first two strings axioms ⇐⇒ program input or initial state
appear to be simpler than the third. This subjective ranking is rules of inference ⇐⇒ program interpreter
reflected in the length of the programs needed to produce these
strings. For the first string the program is a few bytes in length, theorem(s) ⇐⇒ program output
for i:=1 to n print(’1’); derivation ⇐⇒ computation
The program for the second string is slightly longer since it will
contain either nested loops or the literal ’123’. If there is no Consider a program that will successively enumerate all possible
obvious pattern to the third string, the shortest program to produce proofs in length order until it finds a proof of the form AC(x) > c,
it is the program that includes the whole string as literal data and which it will then output. The program and computer will need to
prints it — the string is incompressible or algorithmically random. encode the axioms and rules of inference, which are N bits by
assumption. The program will also need to encode the constant
Ideally the complexity of an object should be a property only of
c. Say that the size of the search program including the encoded c
the object itself, but the choice of computer and programming lan-
is AC(c) + N . Append a second algorithmically simple program
guage affects program lengths. Algorithmic complexity handles
that extracts and prints the x from the proof found in the search.
this issue by considering the complexity to be well defined only
Let k be the AC of the second program, plus, if necessary, the AC
for large objects. The choice of an inelegant language or machine
of some scheme for delimiting and sequentially running the two
adds only a constant amount to the algorithmic complexity, since a
programs. We now have a compound program of AC AC(c) +
translator or simulator from any language or machine to any other
N + k.
is a fixed-size program. In the limit of large objects this fixed size
2···
becomes insignificant. It is possible to choose a c, such as 2 2 , that is much larger than
N but whose AC is small. If c is picked as c > AC(c)+N +k then
There are several major variants of AC (endmarker complexity,
either we have a program of AC < c that generates a string that is
prefix complexity, Chaitin’s prefix variant in which conditional
proved to have AC > c (such a proof is then wrong, so the formal
complexity is conditioned on a shortest program rather than on an
system is unsound) or the program cannot prove statements of the
uncompressed string), but they are asymptotically similar (differ-
form AC(x) > c. Since the details of the formal system were
ing by O(log n)). The AC of a string will be denoted AC(x) when
not specified, it is concluded that no formal system can prove that
our purpose does not distinguish these variants; we will switch to
strings are much more complicated than the axioms of the system
the notation K(x) (prefix complexity) for the discussion of ap-
itself.
proximate bounds.
Definitions
The flavor of algorithmic complexity reasoning will be shown
with the following theorem (it will also be used later in the pa- A formal process is a specified and repeatable process that can be
per): followed by independent agents (computer or human) to arrive at
a common conclusion. We assume the Church-Turing thesis, that
Chaitin Incompleteness theorem. A formal theory with N bits any such process is essentially an algorithm even if the process is
of axioms cannot prove statements of the form ‘AC(x) > c’ if c is in fact followed by humans rather than a computer.
much greater than N .
An objective estimate is an estimate obtained via a formal process.
The proof is by contradiction. One makes the reasonable as-
sumption that if a statement AC(x) > c can be proved then it A feasible process is one that can be completed with conceivably
should be possible to extract from the proof the particular x that is realizable time and other resources. Exponential time algorithms,
used. Then by appending this extraction to the proof sequence (of such as searching over the space of programs for one satisfying
AC ≈ N ) one can generate the string x using approximately N some objective, are clearly infeasible. For example, searching all
bits. But the proof has shown that the AC of x is AC(x) > c > N possible program texts up to 100 bytes in length would require
resulting in contradiction. considering some fraction of O(2 800 ) possible texts – assuming
a processor capable of examining a billion (10 9 ) texts per second
The proof is illuminated by recasting the formal system in com- this task would still require a number of centuries (far) larger than
putational form. A formal system consists of a set of symbols; can easily be described by common words denoting large numbers
a grammar for combining the symbols into statements; a set of (‘billion’, etc.).
axioms, or statements that are accepted without proof; and rules
of inference for deriving new statements (theorems). A proof is Additional background on computability and KCS complexity is
a listing of the sequence of inferences that derive a theorem. It found in textbooks [17, 10].
is required that a proof be formally (i.e. mechanically) verifiable. COMPLEXITY AND SCHEDULE ESTIMATES
Thus, there is a correspondence [18] between a formal system and Various software design methods and processes address the is-
a computational system whereby a proof is essentially a string sue of predicting development times. Software cost models es-
processing computation: 1 timate development time as a function of a size measure such as
1 This is an arguably idealized and restrictive notion of proof; see Naur [12] for source line counts or function points. Software process literature
a discussion. Also note that there are alternate ways of defining the correspondence and commercial software management tools have suggested that
between formal and computational systems. cost models can be combined with historical data on development
times to predict the development times of future projects. for the moment, the AC of the program is the AC of the com-
bined subprograms, plus the AC of a small event loop that
In an extensive empirical study Kemerer benchmarked four soft- calls the subprograms based on user input. 2
ware cost estimation algorithms on data gathered from 15 large
projects for which accurate records were available. It was found • arguments: The AC of a function that depends on arguments
that these models had only limited predictive ability in ex post can be defined as the AC of a large table containing the
facto estimation of the development times for completed projects argument-value pairs interleaved in some fashion, plus the
— the selected error measure (magnitude of error normalized by AC of some scheme for delimiting the argument-value pairs,
the actual development time) ranged from 85 percent to more than plus the AC of a small program that retrieves an output given
700 percent [9]. Kemerer indicates that the limited accuracy of the corresponding input. The size of this uncompressed tab-
these models may be accounted for by variations in problem do- ular representation will be called tabular size.
main and other factors, and suggests that the models may be tuned
to be more accurate. • state: State can be considered as an implicit argument to any
routines whose behavior is affected.
The limited accuracy of these cost models is not the fundamental
obstacle to software estimation however. Rather, since cost mod-
els are a function of a size or complexity measure, the issue is how These comments are only a rough sketch at formulating the AC
to estimate the size or complexity of a new project. of real-world programs, but the fidelity of this formulation is not
crucial to our argument: if the complexity of an output-only pro-
Consider the following scenario: A software company wishes to gram cannot be objectively determined, the addition of arguments,
improve the predictability of its software process, so it decides to state, and interactivity will not simplify things.
gather statistical data on development times. As part of this ef-
fort each programmer is assigned a timed series of exercises. It is The following central results of algorithmic complexity show that
found that the average programmer at the company can complete complexity is not feasibly computable.
each exercise in an average of 3.7 hours. Now the company is
asked to bid on the development of an operating system for a ma- • KCS noncomputability theorem: there is no algorithm for
jor company that is years behind schedule on their own operating computing the AC of an arbitrary string. Denote a shortest
system development project. Based on the newly gathered statis- program for producing a particular object x as x ∗ : AC(x∗ |x)
tics, the company estimates that it can deliver the new operating is not recursive (computable). Rephrasing this for our pur-
system in about 3.7 hours using one average programmer. poses, there is no algorithm for finding the shortest program
with a desired behavior.
This absurd example is intended to clearly illustrate that estimates
of development time depend on estimates of the size or complex- • A trivial upper bound on program size (tabular size) can be
ity of a new program, and historical statistics cannot provide the defined but is not feasible. A trivial upper bound on pro-
latter. In the preceding example the complexity of the proposed gram size is easy to define — it is simply size of the output
operating system is presumably much greater than the complexity string or argument table describing the program’s behavior,
of the programming exercises, but the data do not say anything as sketched above. This ‘tabular size’ bound is not feasible
about the relative difference in complexity. Can complexity itself however. Consider a simple function that accepts two 32-bit
be formally and feasibly determined or estimated a priori? integer arguments and produces an integer result. This func-
tion can be represented as an integer-valued table with 2 64
Claim 1: Program size and complexity cannot be feasibly esti-
entries. While small data types such as characters are some-
mated a priori.
times processed in a tabular fashion, this example makes it
Algorithmic complexity shows that the minimal program size for clear that tabular specification becomes infeasible even for
a desired task cannot be feasibly computed, and a trivial upper small functions involving several integers.
bound on program size exists but is not useful.
• An asymptotic upper bound to AC can be computed, but not
Before discussing these results further we need to relate algorith- feasibly. One can write a search program that enumerates
mic complexity to real-world programs. Recall that algorithmic all programs smaller than the tabular size in lexicographic
complexity is defined as the minimum program size needed to order, looking for the shortest one with the desired behav-
produce a desired output string. The complexity of a program that ior. Since many programs will loop the search program must
produces a fixed output will be defined as the AC of that output. interleave the enumeration and execution of the programs.
Since this definition deals with output only we will briefly indicate That is, it runs each program constructed so far for a certain
how arguments, state, and interactivity might be accommodated: 2 It has recently been argued that interactive programs cannot be considered as

conventional Turing machine-equivalent algorithms, basically because (according

to this argument) human interaction should be considered as an intrinsic part of
• interactivity: An interactive program can be considered as an interactive program. In any particular run of a program, however, the human
input can be replaced with a recording of that input with no changes to either the
a collection of subprograms that are called in sequence ac- behavior of the program or to the program text. The AC of a program is therefore
cording to user commands. These subprograms will share independent of this debate over whether interactive programs should be considered
subroutines and a global state. Ignoring arguments and state as Turing machine-equivalent.
 number of time steps, collects results on programs that finish ming, CASE, 4GLs, object-oriented design, design patterns, or
during this time, and then it constructs the next program and other trends resulted in greater productivity, and if so, is there any
begins running it along with previously constructed programs way of objectively determining which of these techniques result
that are still active. The search program will asymptotically in the greatest productivity gains?
identify smaller programs with the desired behavior but there
is no way to know if a long-running program is looping or if Claim 3: Absolute productivity cannot be objectively determined.
it will finish and prove to be a still smaller representation of Consider a statement that N lines of source code were developed
the desired function. This approach is infeasible, in part be- in T time, with N/T higher than measured on other projects. This
cause the number of programs to be checked is exponential suggests that higher productivity has been achieved. But produc-
in the tabular size. tivity is relevantly defined as the speed of solving the problem, not
the speed of developing lines of code. If the N is significantly
The preceding comments indicate that there is no way to objec- higher than it needs to be for the particular problem then a high
tively define the algorithmic complexity of a program. The min- N/T ratio may actually represent low productivity. This is not
imal program size cannot be feasibly computed, and the trivial merely a theoretical possibility: DeMarco and Lister’s program-
upper bound vastly overestimates the size of a realistic program. ming benchmarks empirically showed a 10-to-1 size range among
In fact, the ratio between the tabular size and the (uncomputable) programs written in the same language to the same specification
AC, considered as a function of the tabular size, is known to grow [7].
as fast as any computable function. We conclude that since there is no feasible way to determine
Claim 2: Development time cannot be objectively predicted. program complexity, productivity cannot be compared across
projects. This position has been arrived at previously using less
Since it is clear that development time estimates must consider formal arguments — it is commonly noted that measures such as
program size or complexity among other factors, this claim is a lines of code and function points may not reflect domain and prob-
direct consequence of the fact that program size cannot be objec- lem specific variations in complexity.
tively predicted. There is some limit on the speed at which a pro-
grammer can write code, so any development time estimate that is Proviso: The relative effectiveness of various software engineer-
supposed to be independent of program size will be wrong if the ing methods can be determined by way of a comparative experi-
program turns out to be larger than can be constructed during the ment in which a fixed problem is solved by programming teams
estimated time period. using different methods. Since it is believed that that the effects
of programmer variability are much stronger than those due to
These comments apply to programming problems in general. Is development methods (10-to-1 differences in productivity across
it possible to do better in particular cases? For example, suppose programmers working on the same problem have been found [11])
a company has developed a particular program. If it is asked to a large experiment might be necessary to achieve valid results.
write the same program again, it now has an objective estimate
of the program size and development time. In practice this is WHAT ABOUT APPROXIMATE ESTIMATORS?
a common scenario, since it is often necessary to recode legacy Though we have argued that absolute algorithmic complexity can-
software for a different language or platform. But is there a mid- not be estimated, there remains the possibility of a approximate or
dle ground between having no objective estimate and an estimate statistical estimator E, say of the form
based on an identical completed project? Clearly the parts of a
project that are similar to previous projects will be estimated more AC(x) ≤ E(x) ≤ AC(x) + b
accurately. The remaining parts, even if they are small, can be
problematic. We can guess how long the coding will take, but for some bound b; for practical purposes this would be quite use-
since the necessary coding time for a even small function may ful. The situation in regards to such an estimator is not so clear
range from a few hours to perhaps years (if the routine is equiv- cut, but the following discussion may suggest maintaining a scep-
alent to an as yet unsolved mathematical problem, c.f. the formal ticism towards claims of strong estimation accuracy, even of an
system↔computation equivalence described in section two), there approximate sort.
is no way to objectively know in advance how long the develop- The Chaitin incompleteness theorem (section two) is relevant.
ment will take. Most experienced programmers have encountered Rephrased, it says that an approximate estimator program can-
projects where an apparently trivial subproblem turns out to be not produce a lower bound on the complexity of programs much
more difficult than the major anticipated problems. larger than its own size.
ESTIMATION OF PRODUCTIVITY We will now show that an approximate estimator E of the form
A wide variety of programming disciplines and processes have indicated above also cannot bound complexity to within a fixed
been proposed in the past several decades. Many of these pro- range. (We switch to the notation K(x) indicating the prefix com-
posals are justified by way of claims that programming productiv- plexity.) Consider the supposed estimator as identifying a set B of
ity is increased. On the other hand, recent studies have shown programs of complexity K(p) . . . K(p) + b that contains the given
that programmers average only a few delivered lines of code program p. The complexity of the program can now be expressed
per day. Have structured programming, object-oriented program- using a two-part description, the first part that of identifying the
set B; the second part that of identifying the particular program ity as an appropriate definition of the complexity of programs. AC
given B. Apply the triangle inequality is a precise and developed notion of complexity and it has been
applied in a variety of fields. The application of AC to the com-
K(a|b) ≤ K(a|x) + K(x|b) + O(1) plexity of programs seems evident. For example, AC addresses
to the two-part description: the well known problems of code metrics – that they may not re-
flect the actual complexity of the code, for reasons such as coding
K(K(p)|p) ≤ K(K(p)|B) + K(B|p) + O(1) style, choice of language, or other reasons. The selection of an ap-
propriate definition of complexity is ultimately a point of philos-
K(K(p)|B) ≤ log2 |B| + O(1): given the set, the size of a pro- ophy, however, since it is establishing a correspondence between
gram to identify the given member is at most the size of a pro- an intuitive notion and a formal one. We do not claim that AC
gram that indexes into the set. Because K(K(p)|p) = O(1) is the only complexity suitable for classifying programs, but it is
(complexity of complexity is not computable) this means that intuitively appealing and supports precise reasoning about related
K(B|p) = O(1), i.e., issues.
Claim 4: There is no estimator which produces a correct fixed
bound on the complexity of all inputs. Some of the implications of our perspective on the software engi-
neering debate will now be discussed.
There are weaker alternatives which are still open to considera-
tion, e.g. a suite of programs each of which is only required to Should software estimation be called “engineering”?
bound the complexity of some subset of inputs, etc. Also the The answer to this question has important implications. If soft-
O(1) constants are unknown and so it is possible that an estimator ware estimation is believed to be a codifiable engineering process
could accurately bound the complexity of inputs up to some use- analogous to house building then litigation is a reasonable and
ful threshold complexity. The preceding discussion does suggest, expected consequence of inaccurate estimations. This and sim-
however, that any claim of producing an accurate approximation ilar issues currently divide the software engineering community
to AC should be examined carefully. into two camps — a “process” camp, who believe that quality
software can be developed on time if a particular software pro-
More generally, statistical estimation of complexity has an intrin- cess or programming technology is used, and a “problem solving”
sic problem that does not arise with common applications of statis- camp, who believe that programming is fundamentally a process
tics such as estimating a population mean. The problem is that the of solving problems and as such intrinsically resists codification.
ground truth is unknown and unknowable, so it is not possible to The problem solving viewpoint is represented in the software en-
determine the bias and variance of different estimators. As such, gineering literature by Bollinger [5], who writes
independent and disagreeing observers are not immediately lead
to a common conclusion because they can adopt differing estima- “The creation of genuinely new software has far more in
tors to support their opinions. common with developing a new theory of physics than
it does with producing cars or watches on an assembly
This issue resembles the problem of defining and measuring psy-
line.”
chological characteristics such as intelligence. While there is no
absolute and objective measure of intelligence, ‘intelligence’ can Bollinger further argues that the process viewpoint is not just in-
be somewhat circularly defined as the ability measured by an in- correct but possibly dangerous, since it focuses attention on codi-
telligence test. The merits of particular intelligence tests are then fied procedures and away from the unknown and potentially risky
debated by considering their correlation with measurable things issues in development. Our conclusion supports the problem solv-
(such as school grades) that are considered to be related to in- ing viewpoint at least in so far as the opposing (process) viewpoint
telligence. Similarly, there is no feasible objective measure of rests on hopes of objective estimation of software complexity.
complexity, but ‘complexity’ can be defined as the property mea-
sured by a proposed code metric. The particular estimate must
Ethics
then be justified; this can be done by demonstrating a correlation
between the estimate and an expected correlate of complexity such Though our conclusions may be considered a ‘negative result’,
as measured development time. We conclude that approximate es- we agree with authors [6] who warn that exaggerated claims
timators should be selected based on empirical demonstrations of and overly optimistic estimates are harming the credibility of the
utility. software industry and inviting litigation and possible regulation.
Credibility will not be achieved by continuing to promise that
CONCLUSIONS software predictability is just around the corner. Instead, the soft-
We have argued that program complexity (and hence productivity) ware industry should attend to the intrinsic uncertainties and risks
cannot be objectively identified. Our conclusions will seem ob- of software development and where necessary promote the public
vious to many, and have been arrived at previously using informal discussion and honest assessment of these risks.
arguments. The algorithmic complexity perspective formalizes,
strengthens and simplifies these arguments. REFERENCES
Revisiting our assumptions. [1] D. E. Avison, H. U. Shah, and D. N. Wilson, “Software Quality Stan-
Our arguments rest on the consideration of algorithmic complex- dards in Practice: The Limitations of Using ISO-9001 to Support
 Software Development,” Software Quality Journal 3, p. 105-111,
1994.
[2] J. Bach, “The Immaturity of the CMM,” American Programmer,
Sept. 1994.
[3] J. Bach, “Enough about Process: What We Need are Heros,” IEEE
Software Vol. 12, No. 2, Feb. 1995, pp. 96-98.
[4] T. Bollinger and C. McGowan, “A Critical Look at Software Capa-
bility Evaluation,” IEEE Software Vol. 8, No. 4, pp. 25-41, 1991.
[5] T. Bollinger, “The Interplay of Art and Science in Software,” IEEE
Computer, Oct. 1997, pp. 128, 125-126.
[6] R. Charette, “Are We Developers Liars or just Fools,” IEEE Com-
puter, July 1995 pp. 90-92.
[7] T. DeMarco, Why Does Software Cost So Much? and other Puzzles
of the Information Age, Dorset, New York, 1995.
[8] M. Fayad and M. Laitinen, “Process Assessment Considered Waste-
ful,” Communications ACM, Vol. 40, No. 11, November 1997,
pp. 125-128.
[9] C. F. Kemerer, “An Empirical Validation of Software Cost Estima-
tion Models,” Communications ACM, Vol. 30, No. 5, May 1987,
pp. 416-429.
[10] M. Li and P. Vitányi, An Introduction to Kolmogorov Complexity
and its Applications, 2nd Ed., Springer, New York, 1997.
[11] S. McConnell, Rapid Development: Taming Wild Software Sched-
ules, Microsoft, Redmond, WA, 1996, p. 167.
[12] P. Naur, Knowing and the Mystique of Logic and Rules, Kluwer
Academic, Dordrecht, 1995.
[13] P. Neumann and contributors, “Risks to the Public” column, Soft-
ware Engineering Notes; also P. G. Neumann, B. Simons, and others,
“Inside Risks” column, Communications ACM.
[14] M. C. Paulk, B. Curtis, M. B. Chrissis, C. V. Weber, The Capability
Maturity Model for Software Version 1.1, CMU/SEI-93-TR-24 Feb.
93, p. 19.
[15] W. H. Roetzheim and R. A. Beasley, Software Project Cost and
Schedule Estimating: Best Practices, Prentice Hall, Upper Saddle
River, New Jersey, 1995, p. xvii.
[16] G. G. Schulmeyer, “Software Quality Lessons from the Quality Ex-
perts,” in G. G. Schulmeyer and J. I. McManus, Eds., Handbook of
Software Quality Assurance (2nd ed.), Nelson Canada, 1995. p. 76.
[17] C. H. Smith, A Recursive Introduction to the Theory of Computa-
tion, Springer Verlag, New York, 1994.
[18] K. Svozil, Randomness and Undecidability in Physics, World Sci-
entific, Singapore, 1993, pp. 30-35.
 Disney Skin and Muscle
David Oliver
The Secret Lab

In order to provide more realistic and efficient character deformation tools for feature film, we developed
a technique that produces complex skin reaction to underlying bone animation.The system was developed in
response to a production need for a easy to use sys-
tem that would allow thousands of pieces of charac-
ter animation to be sent through a production
department, with the least amount of per scene
adjustment. Critical to the success of the system was
the extremely rapid turnaround of new software ver-
sions due to the involvement of the software team on
the production process.

Building characters
We use a multilayered approach, in which sim-
ple bone motion, created using a typical inverse kine-
matics system, drives the motion and shape of nurbs
surfaces representing muscles.These muscles in turn,
drive the motion of the nurbs skin layer. The muscle
layer is a collection of surfaces that move in response
to bone motion. By changing the behavior of individ-
ual muscles involved in the motion of a joint, a complex skin behavior can be produced based on less com-
plex bone motion.

The process of building a muscle and skinned character begins with the creation of an IK animation
rig, which determines the position and length of each bone, and the creation of a modeled character skin.
The muscles are then applied, based on the physiology of
real or similar characters, and on the visible muscles in the
modeled skin. As the muscles are being created around a
joint, the joint is moved through its range of motion to
ensure the correct muscle response when the joint is moved
during an production animation scene.

The muscles are added by selecting the bones that

the muscle will traverse, and choosing points on the bone,
along the path of the muscle, where the muscle is attached
to the bone. Once the points are chosen, a
nurbs surface is created by lofting through a
number of cross sections that are posi-
tioned along a curve.The curve shape is
determined by the position of the bone
attachment points.The cross sectional
shape of the muscle changes in response to
changes in the length of the curve along
which it is placed, which in turn changes as
the bones move. As the lofting path curve
gets shorter, the cross sectional shape of
the muscle increases towards a bulged shape, and as the length increases, the cross sectional shape moves
towards a relaxed, smaller shape.

Once the muscles have been created, the modeled character skin is attached to
the muscles, and to any other surfaces that have been created to model fat, or simulate
other non muscle based skin motion. Each vertex in the skin is uniquely associated
with a point on a muscle, so that as the muscles are moved by the bone animation, the
skin is the moved by the muscles.

This initial positioning of the skin vertices is based only on the motion of the
muscles and other driving surfaces, and does not take into account the position of
other skin vertices.This creates a rough
surface, with sharp muscle boundary arti-
facts, and excessive local stretching and
folding.To reduce these artifacts, the skin is
then relaxed, by iteratively applying forces
to each vertex based on distance to neigh-
boring skin vertices. These distances are
compared to the neutral position, and
forces applied based on the change in
length.The neutral length may be altered by
using an attribute map to provide areas of
increased skin tension.The movement created by these changes in length is con-
strained by the skin surface normal, so that movement against the normal is penalized, preventing excessive
loss of volume during extremes of joint movement.

Use in Production.
This system has been used to create over 2000 produc-
tion shots using over 40 characters over several productions. One
of the primary goals in developing the system was to provide ani-
mators and technical directors with the most effective way to pro-
duce realistic and subtle skin motion in the shortest time, with the
least amount of adjustment on a per shot basis. In order to
achieve this, we developed a
series of tools and techniques:
1) Scripts to automatically do a
first pass application of animation
onto a muscle and skinned char-
acter.
2) Groupings of muscles which
allow for limb by limb control of
muscle response.
3) Geometry blending tools to
correct skinning artifacts.
4) Overall skin tension controls
allowed folds to be pulled when
needed.
 5) Editing methods for changing skin to muscle association.

An animation typically is applied to a muscled character after the animation has been approved as
seen on a low resolution, marionette type rig.The first results of the skinning process are generated automat-
ically from scripts, and, as the basic skin system does not use previous frame information, the task of generat-
ing per frame geometry files does not have to performed sequentially, and can be distributed across available
CPUs.This means that initial turnaround from marionette to skinned geometry is quite fast. After evaluating
the results of the first pass, changes can be made either to the animation performance, to the muscle behav-
ior, or to the skin parameters, and the animation re-run. In production, the vast majority of shots are
approved, in terms of skin geometry, in two or three passes.The remaining shots that require more attention
are achieved by adjusting local skin tension through attribute map values, changing muscle attachments and
response, specific temporal geometry targets, successive skin relaxation passes, the addition of shot specific
muscles or other skin driving geometry, and a variety of other muscle and skin system tools.

Advantages, limitations, and continuing work

The muscle and skin system has allowed us to develop a number of realistic and non-realistic charac-
ters. By using a multilayer approach, we allow a collection of surfaces, moving based on bone animation, to
drive a modeled skin. Surfaces may be easily added or removed from the list of skin drivers.This allows pro-
ductions a great deal of freedom to build character rigs which have rough collection of discrete surfaces driv-
ing the motion of a smooth, continuous skin surface. By separating the initial positioning of the skin vertices
from the iterative smoothing process, the system may be placed at any frame without a run up. Local changes
to the original modeled skin are immediately reflected in the animated skin, allowing wrinkles and other spe-
cific deformations to be easily incorporated.

Because of the intermediate muscle layer between bone and skin, the motion of the skin above a joint
can be much more complex than can be achieved using a simple weighted binding technique. Although the
motion of the skin is, ultimately, based on the motion of the bones, the muscle layer provides a way to build
surfaces that interpret bone motion into a more complex association of muscle surface changes.

The more convincing our characters become, the more more aware we become of the deficiencies of
this technique of character deformation. Although the system successfully models a number of skin parame-
ters, there is no method for allowing the skin to move transversely over its underling structure.This creates
more pinching and localized stretching than should occur. Similarly, by using a fixed attachment between mus-
cle and skin to initially position the skin vertices, we often create folds that are difficult to resolve in the
relaxer phase.

We continue to develop the system toward both realistic and non realistic applications. Each produc-
tion develops its own approach to building the intermediate muscle layer, depending on their own needs.We
have incorporated a layer upon layer approach that uses an inner skin layer to allow for more control and
effects in the final, outer skin. As well, we are developing dynamic response, and alternate surface description
methods.

Acknowledgements
The muscle and skin sytem at Walt Disney Feature Animation / The Secret Lab was created through the
unwaivering efforts of Ross Kameny, Dr. Don Alvarez, and myself. Also critical to the success of the system
are the contributions of Kevin Rogers, Philip Schneider, Sean Phillips, Mark Empey and the rest of the charac-
ter finalling team for the “Dinosaur” project.
 Pose Space Deformation: A Unified Approach to Shape Interpolation and
Skeleton-Driven Deformation
J. P. Lewis∗, Matt Cordner, Nickson Fong

Centropolis

Abstract while a purely image-based approach can achieve very realistic im-
Pose space deformation generalizes and improves upon both shape ages, this advantage may be lost if one needs to introduce geome-
interpolation and common skeleton-driven deformation techniques. try and surface reflectance in order to re-light characters to match
This deformation approach proceeds from the observation that sev- preexisting or dynamically computed environments. Film and en-
eral types of deformation can be uniformly represented as mappings tertainment applications require fanciful creatures that fall outside
from a pose space, defined by either an underlying skeleton or a the scope of image-based approaches.
more abstract system of parameters, to displacements in the ob- Some of the most impressive examples of geometry-based (as
ject local coordinate frames. Once this uniform representation is opposed to image-based) human and creature animation have been
identified, previously disparate deformation types can be accom- obtained in the entertainment industry. These efforts traditionally
plished within a single unified approach. The advantages of this use shape interpolation for facial animation and a standard but
algorithm include improved expressive power and direct manipula- variously-named algorithm that we will term skeleton subspace de-
tion of the desired shapes yet the performance associated with tradi- formation (SSD) for basic body deformation [25, 9]. While shape
tional shape interpolation is achievable. Appropriate applications interpolation is well-liked by production animators, it is not suitable
include animation of facial and body deformation for entertainment, for skeleton-driven deformation. On the other hand SSD produces
telepresence, computer gaming, and other applications where direct characteristic defects and is notoriously difficult to control.
sculpting of deformations is desired or where real-time synthesis of These issues, which will be detailed in the next section, lead us
a deforming model is required. to look for a more general approach to surface deformation. We
consider the following to be desirable characteristics of a skeleton-
CR Categories: I.3.5 [Computer Graphics]: Computational based surface deformation algorithm:
Geometry and Object Modeling—Curve, surface, solid and ob-
ject modeling I.3.6 [Computer Graphics]: Methodology and
• The algorithm should handle the general problem of skeleton-
Techniques—Interaction techniques I.3.7 [Computer Graphics]:
influenced deformation rather than treating each area of
Three-Dimensional Graphics and Realism—Animation
anatomy as a special case. New creature topologies should be
Keywords: Animation, Deformation, Facial Animation, Morph- accommodated without programming or considerable setup
ing, Applications. efforts.

1 Introduction • It should be possible to specify arbitrary desired deformations

at arbitrary points in the parameter space, with smooth inter-
Free form deformation has been approached from several distinct polation of the deformation between these points.
perspectives. As an abstract and general problem, good methods
have been obtained both using the well known technique that bears • The system should allow direct manipulation of the desired
this name [32, 12, 17] and other kinematic surface deformation deformations [33].
techniques, and with physical models that simulate the time evo-
lution of a membrane or solid. • The locality of deformation should be controllable, both spa-
The animation of human and creature skin deformation is ar- tially and in the skeleton’s configuration space (pose space).
guably the most common and important application of free form de-
formation in computer graphics. While such creature animation can • In addition, we target a conventional animator-controlled
be considered a special case of general free form deformation, its work process rather than an approach based on automatic sim-
importance and difficulty have lead researchers to propose a number ulation. As such we require that animators be able to visual-
of domain-specific algorithms that will be reviewed in Section 2. ize the interaction of a reasonably high-resolution model with
The problem of realistic facial animation is being actively and an environment in real time (with ‘high resolution’ defined in
successfully addressed by image-based and hybrid techniques. accord with current expectations). Real time synthesis is also
These techniques are not yet suitable for all applications, however: required for applications such as avatars and computer games.
∗ [email protected]
Our solution, termed pose space deformation, provides a uni-
form and expressive approach to both facial skin deformation and
skeleton-driven deformation. It addresses the previously mentioned
drawbacks of shape interpolation and SSD while retaining the sim-
plicity and performance associated with these techniques.
The next section reviews various approaches to free form de-
formation and describes shape interpolation and skeleton subspace
deformation algorithms. The pose space deformation algorithm re-
quires well behaved and efficient scattered data interpolation in high
dimensional spaces; Section 3 considers this issue. The pose-space
deformation algorithm itself is described in Section 4; examples
and applications are shown in the last section.
2 Background
Recent research has delivered significant improvements in many ar-
eas of character animation, including surface representation, model
capture, performance capture, and hybrid (partially image-based)
rendering approaches. In this literature review we focus specifically
on milestones in the surface deformation models and necessarily
omit other important contributions.
p’ p p"
2.1 Surface Deformation Models
Continuous deformation of a character skin was first addressed in
Parke’s pioneering facial animation work [26]. In this work, control
q’ q q"
vertices were deformed by custom algorithmic implementation of
carefully selected high-level parameters (‘raise-upper-lip’, etc.). Figure 1: The skeleton subspace deformation algorithm. The deformed position of a
Komatsu [13] and Magnenat-Thalmann et. al. [23] demonstrated point p lies on the line p
p

defined by the images of that point rigidly transformed by
human body deformation driven by an underlying skeleton. The the neighboring skeletal coordinate frames, resulting in the characteristic ‘collapsing
region and shape of deformation is algorithmically defined in each elbow’ problem (solid line).
of these approaches. Magnenat-Thalmann et. al. developed algo-
rithms for each of the various joints in the hand. The discussion in
Komatsu focuses on the elbow and shows how the skin crease on including manually animated FFDs and custom procedural ap-
the acute side can be obtained by a suitable algorithmic manipula- proaches in the spirit of [26, 23, 13]. Arguably the most com-
tion of the surface control vertices. The algorithms in this early mon practice in character animation (as reflected in commercial
work do not suffer the ‘collapsing elbow’ characteristic of the SSD software, animation books and courses, and some custom software)
algorithm (below). On the other hand, the algorithms are specific to is founded on the twin techniques of shape interpolation and SSD
particular types of joints and are perhaps too simple to portray the [18, 9].
complexity and individual variability of real anatomy.
The short film Tony de Peltrie [3] popularized the use of shape in- 2.3.1 Shape Interpolation
terpolation for facial animation. Forsey [11] describes a character- Shape interpolation (also called shape blending and multi-target
oriented deformation scheme in which the bending of a smooth morphing) is probably the most widely used approach to skin de-
surface can be controlled by anchoring levels of a multi-resolution formation for facial animation [3, 18, 9]. Surface control vertices
spline surface to the underlying skeleton. These efforts are distin- are simply an animated linear combination (not necessarily con-
guished from the previous purely algorithmic approaches in giving vex, i.e., individual weights can be greater than one or less than
the modeler control of and responsibility for the deformation.
The specification and animation of surface deformation remains

zero) of the corresponding vertices on a number of key shapes Sk :
w S . A variation of this technique
k=0 k k
uses a single base shape
an active area of investigation [17, 10]. The Wires technique [22]
S0 and a number of delta shapes, S0
+ k=1 wk (S
k − S0 ). By
is one interesting recent contribution; this approach is notable in
providing a direct manipulation interface in a form immediately fa- writing the delta shape form as (1 − 1 wk )S0 + 1 wk Sk it is
miliar to sculptors (armatures). clear that the space of achievable shapes is identical in both varia-
tions.1 An attractive feature of shape interpolation is that the desired
2.2 Multi-Layered and Physically Inspired Models expressions can be directly specified by sculpting.
The limitations of shape interpolation. Given the popularity and
Chadwick, Haumann, and Parent [7] introduced a multi-layered and effectiveness of this simple approach, it would be desirable to em-
physically inspired approach to skin deformation. In their model ploy it on regions of the body other than the face. The blending of
a free-form deformation abstractly represents underlying body tis- rigid shapes is inconsistent with regions of the body that are bend-
sues and mediates skin movement. Chadwick et. al. demonstrated ing under the action of an underlying skeleton, however. Of course
expressive three-dimensional cartoon characters but deformation of the key shapes could be deformed to the moving articulated figure
a realistic character was not shown. using some other algorithm, but this defeats the purpose of propos-
Other researchers have investigated modeling the underlying ing shape interpolation as the means of obtaining the deformation
body tissues in greater depth [27, 24, 8, 35]. Most recently, sev- in question.
eral groups have undertaken ambitious efforts to produce anatom- Shape interpolation also has some drawbacks for its intended
ically inspired multi-layered models of animals and humans with role of facial animation. For one, the interpolation is not always
considerable verisimilitude. Nedel and Thalmann [19] simulate smooth. Consider interpolating from a smile (shape A) to a neutral
the surface deformation of muscles using spring mesh dynamics; a pose (B) and then to a frown (C). An individual vertex travels in a
modeled skin cross section is reshaped by a ray-casting procedure straight line between A and B and again in a line between B and
that finds the maximum displacement of the underlying tissue. Sev- C. Selecting smoothly changing weights with dw/dt = 0 at the
eral papers by Wilhelms and coworkers have shown anatomically key shapes merely causes the deformation to “ease in” and stop at
representative human and animal models. In Wilhelms and Van each key pose before continuing on – the time derivative of control
Gelder [36] several classes of muscles are algorithmically modeled point motion is smooth, but the motion path itself is only piecewise
with attention to volume conservation; skin is a spring mesh an- linear (parametric versus geometric continuity). In practice ani-
chored to underlying tissue or bone in appropriate areas. Scheepers mators object to the linear nature of the interpolation [34] and have
et. al. [31] produced convincing representations of muscles as well sometimes compensated by sculpting new key shapes as often as
as preliminary but promising skin deformation. every three to five frames [38]. These comments will be revisited
in the discussion of the pose space approach later in the paper.
2.3 Common Practice 1 Provided that the weights sum to one. This is enforced in the delta
In recent years character animation has moved beyond being a re- shape formulation. It is not enforced in the (non-delta) shape interpolation
search topic and sophisticated deforming characters routinely ap- formulation as written, but weights that do not sum to one are a separate
pear in films and on television. Various techniques are employed, effect – they cause the face to change overall scale.
 Figure 2: The ‘collapsing elbow’ in action, c.f. Figure 1. Figure 3: The forearm in the ‘twist’ pose, as in turning a door handle, computed by
SSD. As the twist approaches 180◦ the arm collapses.

2.3.2 Skeleton-Subspace Deformation

This simple algorithm has been repeatedly conceived and appears in A second difficulty with SSD is that, unlike shape interpolation,
commercial software packages under several rather uninformative it does not permit direct manipulation; artists instead directly or
names such as skinning, enveloping, etc. The algorithm is unpub- indirectly edit the meshes of weights wk (for each control vertex
lished but is subsumed by more general published schemes such on a surface there is one weight per skeletal frame that affects the
as [23]. The position of a control vertex p on the deforming sur- vertex). SSD algorithms consequently have the reputation for being
face of an articulated object lies in the subspace defined by the rigid tedious and difficult to control. Artists with a poor understanding
transformations of that point by some number of relevant skeletal of the underlying algorithm have difficulty distinguishing between
coordinate frames (Figure 1). This may be notated results that can be further improved by adjusting weights and results
that cannot be improved since the desired result lies outside the
 achievable subspace, resulting in the impression of unpredictability
p̄ = wk Lk (p) p (“sometimes adjusting the weights helps, sometimes it doesn’t”).
In some cases the SSD defects can be manually corrected us-
(in more detail) ing FFDs and other techniques, and one could consider a scheme
 −1
p̄ = wk Lδk L0k L0p p whereby these fixes are procedurally invoked as the skeleton articu-
lates. But although FFDs work well (and have a direct manipulation
algorithm [12]) the layered FFDs do not reduce the difficulty in ad-
where L0p is the transform from the surface containing p to the justing the underlying SSD. The algorithm introduced in the subse-
world coordinate system, L0k is the transform from the stationary quent sections removes the need for such layered fix-it approaches
−1
skeletal frame k to the world system (L0k L0p together represent p and permits direct specification of the desired deformations.
in the coordinate system of skeletal frame k), and Lδk expresses the
moving skeletal frame k in the world system. The deformation is 2.3.3 Unified Approaches
controlled by the user through the weights wk . Several published algorithms and commercial packages combine
SSD is fairly versatile. For example, secondary animation effects aspects of skeleton-driven deformation and shape interpolation in
such as muscle bulging and swelling of the chest can be achieved by ways that anticipate our approach. In the pioneering work of Burt-
variably weighting the surface to an abstract “bone” whose transla- nyk and Wein, two dimensional characters were animated using a
tion or scale is manually animated. polygonal rubber sheet that afforded both skeletal and local defor-
The limitations of SSD. The first major shortcoming of SSD re- mation control [6]. Van Overveld described a two-dimensional an-
sults directly from the fact that the deformation is restricted to the imation system in which animation is controlled by a skeleton and
indicated subspace. In common situations such as shoulders and character deformation is driven from this skeleton through a scat-
elbows the desired deformation does not lie in this subspace, hence tered interpolation [20]. This work is similar in spirit to ours but dif-
no amount of adjusting the algorithm weights will produce good re- fers in that it used the image plane as a global interpolation domain
sults. This fact leads to considerable frustration by users of the algo- rather than introducing a pose space. Litwinowicz and Williams’s
rithm – the character of the deformation changes as the weights are system [16] is also a precedent and introduced sophisticated scat-
changed, sometimes sustaining the incorrect assumption that some tered interpolation (again in the image domain). Several papers
combination of weights will produce good results. In fact, the SSD consider animation (and indeed image synthesis in general) as a
algorithm can be easily identified in animations by its characteristic special case of neural net learning and interpolation/extrapolation
‘collapsing joint’ defect (Figures 1, 2). [14, 15, 21]. While this viewpoint is valid, in practice it is per-
This problem is extreme in the case of simulating the twist of haps excessively general, for example, a skeleton is merely learned
a human forearm (the pose taken in turning a door handle, Fig- rather than being an intrinsic part of the model. While employed
ure 3). In this case the subspace basis consists of surface points at Industrial Light and Magic the first author of the present paper
rigidly transformed by the forearm frame (no axis rotation) and the developed a system that attempted to blend shape interpolation and
wrist frame (axis rotation). With a rotation of 180 degrees this line SSD algorithms; a small portion of it remains in use in their well
crosses the axis of the arm, i.e., the forearm collapses entirely as known Caricature animation system. Drawbacks of this work in-
the SSD weights transition at some point from the forearm to wrist cluded both a complicated dependence on the details of SSD and
frames. its overall conception as a “correction” to SSD. Some commercial
Figure 4: Shepard’s interpolant operating on a set of colinear points. The derivative Figure 5: Radial basis functions φ(x) = exp(−x2 /2σ2 ), σ = 10 interpolating
is zero at the data points, and the curve extrapolates to the average of the data values. the same set of colinear points as in Figure 4. A different y scale is used to fit the curve.
The curve extrapolates to zero.

packages allow blending between two sculpted deformations as a

function of a single-joint rotation, thereby combining shape inter- normalized by the sum of the weights,
polation and skeleton-driven deformation in a limited but useful set-

ting. ˆ wk (x)dk
d(x) =

wk (x)
2.4 Kinematic or Physical Simulation?
The depth of simulation is a prevalent issue in computer graph- with weights set to an inverse power of the distance: wk (x) =
x−
ics, albeit one that is not always consciously considered. Early xk
−p . (This is singular at the data points xk and should computed
approaches to animation were purely kinematic; an emphasis on as (||x − xk
+
)−p ). With p > 1 the interpolation surface is
physically based modeling appeared in the literature later. Recent once differentiable. Unfortunately this simple scheme has some
sophisticated approaches allow a hybrid of animator-controlled and potentially undesirable properties. Far from the
data the weights


physically governed animation as needed. In rendering we perhaps will be approximately the same, ˆ
d(∞) = w dk /w∞ 1=

∞
see the opposite trend – much of the literature a decade ago focused dk /N , i.e. the interpolated surface converges to the average of
on ever deeper simulations of reality, whereas ‘shallower’ image- the data values. A serious drawback for some applications is that
based approaches are attracting attention at present. the derivative of the surface is zero at the data points (Figure 4).
Similarly, in character deformation both deep and shallow ap-
proaches have their place. Deep models promise universally accu- 3.2 Radial Basis Functions
rate simulation, and the importance of representing humans justifies
Radial basis functions [28, 29] have become a popular choice for
the needed effort. The authors of these approaches acknowledge
scattered interpolation. The interpolant is a linear combination of
that producing anatomically plausible models is a daunting task,
nonlinear functions of distance from the data points:
however.
Pose space deformation is a shallow, purely kinematic approach

N
to deformation (i.e. without reference to underlying forces, mass, ˆ
d(x) = wk φ(
x − xk
) (1)
volume), and it has consequent disadvantages. In particular, accu-
k
racy is reliant on the modeler/animator rather than being guaranteed
by the simulation. On the other hand, our algorithm has clear ad- If N values of d are available then the weights can be easily solved
vantages with respect to simplicity and generality, direct manipula- by a linear system; this can be derived either by least squares fit or
tion, real-time synthesis, and other criteria listed in the introduction. by subspace projection. Taking the latter approach, we reconsider
the available data points as a single point d in an N dimensional
3 Deformation as Scattered Interpolation space, and consider φk () = φ(
xj − xk
) as the kth basis vec-
In abstract, we wish to express the deformation of a surface as a tor. The best approximation to d in the space spanned by φk () oc-
function of either the pose of an underlying skeleton, or equiva- curs (in direct analogy with the three-dimensional case) when the
lently as a function of some other set of parameters such as the weights are such that the error d − Φw (with φk () comprising the
{smile, raise-eyebrow,...} controls desirable in facial animation. We columns of Φ) is orthogonal to each of the φk ():
also wish to directly sculpt the desired deformation at various points
in the parameter space, rather than working in a more abstract space ΦT (Φw − d) = 0
such as the coefficients on various coordinate frames as required by
the SSD algorithm. so (the so-called “normal equation”)
A scattered data interpolation method is required because defor-
mations will be sculpted at arbitrary (rather than regularly spaced) ΦT Φw = ΦT d
poses. Since this interpolation is central to our application (the re-
can be solved for the familiar
sults of the interpolation will be directly visible in the animating
deformation), we will consider the available scattered interpolation w = (ΦT Φ)−1 ΦT d
approaches before settling on a candidate.
A least squares approach leads to the identical result.
3.1 Shepard’s Method Any nonlinear function φ() will interpolate the data, includ-
Shepard’s method [1, 2] is a frequently employed scattered data ing odd choices such as φ(x) = x (which is nonlinear since
interpolation scheme in computer graphics. In this method the in- x =
x − xk
is the argument), provided that the columns of Φ
terpolated value is a weighted sum of the surrounding data points are independent. On the other hand a smooth φ() will result in a
smooth interpolant (a weighted sum of continuous functions is con- pose space, creature deformations that are discontinuous with re-
tinuous). In fact radial basis functions have a universal convergence spect to pose seem unlikely. As such the expensive energy func-
property similar to Fourier series, though the convergence definition tional and non-convex schemes are not necessary. In addition we
is different. want δ to approach zero away from the data, and the width of this
The preceding description maps a k-dimensional input space (ar- falloff should be selectable.
2
bitrary k) to a one dimensional range, i.e., it is the k-dimensional Together these comments support φk (x) = exp( −(x−x k )
)
version of a height field. Surfaces can of course be interpolated 2σ 2
as one possible choice of radial basis (Figure 5). Gaussian radial
by allowing different combinations of the same basis functions in basis functions are reputed to be well behaved and our experience
different dimensions, i.e., vector valued wk . The distance

can supports this judgement. Gaussian radial basis functions with ad-
be generalized to Mahalanobis distance (effectively rotating and justable placement and σ are discussed in the neural net literature
stretching the basis function) [4]. and optimizing over these parameters is possible. This issue does
not arise in our application, however, since the animator decides
3.3 Energy Functionals and Non-Convex Methods where in the parameter space to sculpt a pose (effectively decid-
Various visual reconstruction schemes can be adapted for scattered ing the basis function placement). The falloff σ is also specified
data interpolation. In these schemes the interpolated or approxi- explicitly by the animator, as described below.
mated surface is found as the minimum of a functional such as
 4.1 Algorithm Summary
ˆ − d(x)|2 dx + λP (d)
|d(x) ˆ
The steps in a pose space deformation (PSD) algorithm will now be
described consecutively.
where the first term penalizes deviation of the surface dˆ from the Definitions. A pose is defined as the configuration of any pose
available data d and the second regularizing term votes for surface controls (joints or abstract manipulators) that have changed from
smoothness e.g. by integrating the squared second derivative of the their default values. An abstract manipulator is a UI control or ar-
surface. With small λ many of these schemes can serve as scat- bitrary piece of geometry whose movement will control the inter-
tered data interpolants; reference [5] is a good introduction to these polation of some deformation, such as a muscle bulge or a desired
approaches. facial attribute such as “happiness.” A self-relative configuration of
In some of the most powerful formulations of scattered interpo- the controls is actually considered, for example, an elbow involves
lation the regularizer is considered to hold everywhere except at an two skeletal frames but only one joint angle.
unknown set of edges – this is the piecewise-smooth prior desirable The pose space is the space spanned by the variations of these
in image reconstruction. Since the unknown edges may exist (or controls. If n = 2 pose controls are active and each has three
not exist) at any location in the domain, all combinations of possi- degrees of freedom then a 3(n − 1) pose space is defined, and the
ble edge locations must be considered and the interpolation cost is particular position of the controls defines a point in that space.
prima facie exponential in the surface resolution. Sculpt. The artist first positions some set of pose controls and
then sculpts a deformation for that pose. The artist also assigns a
4 Pose Space Deformation falloff (Gaussian σ), either as a symmetric radius across all controls
or to each control individually (axis stretched falloff).
The crux of our approach is the identification of an appropriate
space for defining deformations. As discussed above, the inter- Define δ(pose). Any control vertices that have moved from their
polation domain is (a subset of) the pose space of an articulated rest position are found. This is done in the local coordinate frame,
character, or equivalently the space defined by some set of parame- i.e., rigid body articulated motion results in zero δ. The δ values
ters such as facial controls. for the deformed vertices are computed (again in the local coordi-
In concept the range of the interpolation function could simply nate system) and they are saved in a database together with their
be the desired movement of the surface control vertices. To make corresponding location in a pose space. (At the boundary of sev-
the job easier for the interpolation we instead interpolate the desired eral surface patches there may be shared vertices that need to be
deviation of a surface vertex (expressed in the local frame) from its coincident to maintain surface continuity. Unlike some SSD imple-
initially computed position (the rigidly transformed position in the mentations interpolation in pose space by definition cannot separate
case of an articulated model). Several reasons for this choice will such vertices).
be mentioned shortly. Solve. When several such deformations have been saved (or
Thus the deforming surface is defined by p + δ with p moved when the artist is ready to try animating) it is necessary to solve
rigidly by the skeleton or other underlying system, and the interpolation problem. For each control vertex that was moved
during sculpting there are now one or more δ values at points in
δ = finterp (configuration) the pose space. Note that the dimension of the pose space can vary
across vertices, for example, a particular vertex might be modified
where configuration is the configuration of the set of joints or pa- in three sculpted deformations but a neighboring vertex might have
rameters controlled by the animator. been modified in only two deformations. The interpolation is done
Our scheme can be bootstrapped on top of an existing software independently for each control vertex (but see additional details be-
system: the model is posed as desired and the desired surface at that low); in our experience using patch surfaces this has not been prob-
pose is sculpted. Our algorithm computes the difference between lematic. Singular ΦT Φ is interpreted as a user error; in practice this
the initial and resculpted model at that pose. This ‘deformation’ has turned out to be the result of saving new deformations without
is associated with the joints or other parameters that have moved moving any pose controls rather than a result of actual numerical
from their default positions to create the particular pose. One or problems.
more deformations will then be interpolated in this subspace using Synthesis. The model is now moved to an arbitrary pose. The
a scattered data approach. location in pose space is determined from the concatenated relative
We now have enough criteria to select a particular interpolation degrees of freedom of the pose controls (simply interpreted as in-
scheme. Although it would be desirable to allow deformations to dependent dimensions). For each deforming control vertex a δ is
change both continuously and discontinuously with respect to the interpolated from the delta values at the stored poses using Eq. (1).
 Evaluate and Repeat. At this point the model interpolates • In both approaches a set of key shapes (or delta shapes) are
through the previously defined deformation(s). The most recently sculpted. The same set of shapes can be used in both ap-
defined deformation may extend too far (or not far enough) in pose proaches.
space, however. There is a rich literature of schemes for optimizing • Whereas shape interpolation is (despite the name) a super-
radial basis parameters including σ [4]. On the other hand, anima- position of a set of shapes, PSD interpolates among these
tors consider detailed control of the animation to be part of their shapes.
craft and are quite happy to have interpolation parameters exposed
to them. We have found that this potentially abstract parameter • The animator’s task in PSD is to choose the interpolation
is comprehensible so long as it is possible to explore the effect of path (and adjust interpolation parameters such as falloff if de-
different values. At a minimum axis-aligned scaling of the falloff sired). With SI the animator’s task is to choose the interpola-
should be available; we have not experimented with Mahalanobis tion path but also solve the problem of representing this path
rotation of the basis. Based on the evaluation the artist may decide by using a set of (non-orthogonal!) basis shapes. In practice
to sculpt additional poses as needed to achieve the desired motion. this has been considered the major difficulty in applying SI
A detail that was omitted previously will now be mentioned: when high quality animation demands large numbers of basis
when a deformed vertex is found the associated pose space is de- shapes [38].
termined as described above. If there are previous deformations of • In shape interpolation the key shapes and the animation pa-
this vertex in the same pose space then the new deformation is sim- rameter space are one and the same – the keys define the axes
ply another point to interpolate. The new deformation’s pose space of the animation parameter space. In PSD the key shapes are
may, however, be different from the previous spaces associated with positioned as desired in a space of desired dimensionality.
the vertex! In such a case a new pose space is started, and the δ is
One can assign each shape in PSD to a separate dimension,
computed as a delta from the previous layered PSD synthesis rather
exactly as with SI. On the other hand, PSD allows one to
than from the base model. This ensures that the previous deforma-
sculpt intermediate expressions (half-smile) and situate them
tions are interpolated while allowing the artist complete freedom in
half-way along the relevant (full-smile) axis. Similarly a
determining the extent of the deformation and the associated pose
sculpted pose that represents the simultaneous activation of
controls. While there is an issue of commutativity, in our experi-
several parameters (e.g. happy but surprised, or smiling with
ence artists consider this iterative layered refinement to be a natural
a wink) can simply be saved at the appropriate location in the
process.
pose space. Psychological research has shown that human
In the preceeding discussion we have not described the represen-
facial expressions are largely described by two “emotional”
tation of rotations and other transformations. This is a well known
axes [30] (Figure 6); this two-dimensional space would be a
issue; well behaved transformations are fundamental and are hope-
convenient high-level pose space for controlling facial anima-
fully addressed early in the development of any character animation
tion.
system.
• The PSD interpolation is smooth if so desired.
4.2 Cost
With n poses three matrices of size n must be inverted for each To illustrate these comments consider Figure 7, which abstractly
surface control vertex. Typically n will be between 1 and 10, say, represents both SI and PSD with an identical set of expressions
so this cost is small. Also it is incurred at a convenient time – during (neutral, half-smile, full-smile, frown). In the SI side of the dia-
setup (as a pose is saved) rather than during synthesis. gram expressions are arranged as independent (but not orthogonal)
For synthesis, the basis function φ(x) can be implemented by dimensions as required by SI. In the PSD diagram the expressions
interpolated table lookup and the sqrt required in the Euclidean are situated in an expression space having a happy-unhappy axis; a
distance can be composed with φ(x) in the table. The cost of Eq. (1) second axis (arousal) and an expression (delighted) on that axis are
is then not much greater than the cost of traditional shape interpola- added to show a multidimensional space.
tion, i.e., real time synthesis is possible with significant models on As illustrated, a PSD path from neutral to half-smile to full-smile
current machines. is monotonic, as might be expected; the motion of a surface point
over this interpolation is also smooth. To interpolate these emotions
using SI requires a zig-zag pattern of weights: the half-smile weight
5 Applications and Discussion goes from zero to one, and then back to zero as the full-smile weight
goes from zero to one.The motion of a surface point would also be
5.1 PSD for Skeleton-Driven Deformation
piecewise linear using SI in this scenario.
An articulated model such as a human will typically have a num-
ber of different deformation subspaces, each with one or several 5.3 PSD for Secondary Animation
deformations; the deformations in different subspaces may overlap
Switches and dials. Additional “dimensions” of deformation can
spatially e.g. to simulate the influence of different muscles. The
be added at any time by adding a new parameter and associating
deformations needed for an elbow, for example, will be interpo-
additional poses with the movement of this parameter. For example,
lated in the one-dimensional subspace defined by the elbow joint
a limb can be modeled in a particular pose both in an unloaded state
angle. Deformations in a shoulder area will need to consider two or
and with muscles sculpted to express carrying a heavy load. The
more degrees of freedom. The neck/chest/leg blend area of many
‘heavy’ pose can be associated with the ‘on’ state of an abstract
quadrupeds is a more complex case – the motion of the skin surface
parameter (e.g. an isolated bone moved into the vertical position);
in these regions may depend on the relative configuration of several
light and heavy loads can then be controlled by flipping this switch.
leg bones as well as the rib cage and possibly the neck region of the
Similarly one can imagine setting up a dial that causes the character
spine. PSD handles all these cases simply and uniformly.
to morph; this would of course require a significant set of additional
Figures 8 and 9 are a simple comparison of PSD and SSD algo-
deformation poses.
rithms in action on human elbow and shoulder regions.
6 Conclusions
5.2 PSD for Facial Animation Pose space deformation is not the last word in surface deformation
The application of PSD to facial animation is best described by for character animation; high quality anatomically based models
comparison with shape interpolation (SI). are certainly preferable. Nevertheless both anatomically based and
 alarmed excited
afraid astonished [6] N. Burtnyk and M. Wein, Interactive Skeleton Techniques for Enhancing Motion
Dynamics in Key Frame Animation, Comm. ACM, vol. 19, no. 10 (October 1976),
angry delighted
annoyed
pp. 564-569.
frustrated happy [7] J. Chadwick, D. Haumann, and R. Parent, Layered Construction for Deformable
pleased Animated Characters. Computer Graphics vol. 23 no. 3 (Proc. SIGGRAPH 89),
content
pp. 243-252.
serene, [8] D. T. Chen and D. Zeltzer, Pump It Up: Computer Animation Based Model of
miserable calm
depressed relaxed
Muscle Using the Finite Element Method. Computer Graphics vol. 26, (Proc.
bored SIGGRAPH 92), pp. 89-98.
tired sleepy
[9] Cinefex, Riverside, CA, various issues, e.g. vol. 66 (June 1996), p. 52 (Dragon-
heart); vol. 64 (Dec 1995), p. 62 (Jumanji).
Figure 6: Schematic diagram of emotion space obtained by multi-dimensional scal-
ing from pairwise similarity ratings, simplified from [30]. [10] T. DeRose, M. Kass, and T. Truong, Subdivision Surfaces in Character Anima-
tion. Proc. SIGGRAPH 98, ACM, pp. 85-94.
[11] D. Forsey, A Surface Model for Skeleton-Based Character Animation. In Second
N=neutral Eurographics Workshop on Animation and Simulation, Vienna, Austria, 1991,
S=smile pp. 55-73.
F H=half−smile
H F=frown [12] W. M. Hsu, J. F. Hughes, and H. Kaufman, Direct Manipulation of Free-Form
arousal delighted
Deformations, Computer Graphics vol. 26 (Proc. SIGGRAPH 92), pp. 177-184.
F N H S [13] K. Komatsu, “Human Skin Model Capable of Natural Shape Variation,” The
N S Visual Computer, vol. 4, no. 3, 1988, pp. 265-271.
happiness
[14] J. P. Lewis, Creation by Refinement: A Creativity Paradigm for Gradient Descent
Shape Interpolation Pose Space Deformation Learning Networks. International Conf. on Neural Networks. San Diego, 1988,
II: 229-233.
Figure 7: Abstract comparison of shape interpolation versus pose space deformation
using the same set of facial expressions. [15] J. P. Lewis, Probing the Critic: Approaches to Connectionist Pattern Synthesis.
IEEE International Joint Conference on Neural Networks Seattle, July 1991.
[16] P. Litwinowicz and L. Williams, Animating Images with Drawings, Proc. SIG-
purely kinematic models have their place. In the current computer GRAPH 94, pp. 409-412.
animation culture animators generally practice their craft by direct
[17] R. MacCracken and K. Joy, Free-Form Deformations with Lattices of Arbitrary
and exhaustive specification of the desired motion combined with
Topology. Proc. SIGGRAPH 96, pp. 181-180.
quick evaluation using real-time playback. Deeper simulation ap-
proaches intrinsically take away some of this control, and animators [18] G. Maestri, Digital Character Animation 2, Vol 1. New Rider, Indianapolis,
1999. ISBN 1056205-930-0.
often argue (rightly or not) that automated processes are inferior or
will not produce a human feel. The performance of current anatom- [19] L. Nedel and D. Thalmann, Modeling and Deformation of the Human Body Us-
ically based models prohibits animation preview and other real-time ing an Anatomically-Based Approach, www preprint.
applications such as telepresence and gaming (one published result [20] C. W. A. M. van Overveld, A Technique for Motion Specification in Computer
is several orders of magnitude slower than real time), and the effort Animation, Visual Computer, vol. 6 (1990), p. 106-116.
needed to produce an anatomically accurate model is not always [21] T. Poggio and R. Brunelli. A novel approach to graphics. AI Memo 1354, C.B.I.P
justified, nor even appropriate if the model is of a fanciful creature Paper 71, MIT, 1992.
whose surface appearance may be inconsistent with any plausible [22] K. Singh and E. Fiume, Wires: A Geometric Deformation Technique. Proc. SIG-
internal anatomy in any case. GRAPH 98, ACM, pp. 405-414.
PSD unifies and improves upon two techniques that have been [23] N. Magnenat-Thalmann, R. Laperriere, and D. Thalmann. Joint-Dependent Lo-
common graphics practice for more than a decade. This relatively cal Deformations for Hand Animation and Object Grasping. Proc. Graphics In-
simple algorithm uniformly handles a variety of deformation situ- terface, 1988, pp. 26-33.
ations ranging from a simple elbow to secondary animation. The [24] N. Magnenat-Thalmann and D. Thalmann. Human Body Deformations using
setup cost of the algorithm is insignificant, and the synthesis cost is Joint-Dependent Local Operators and Finite Element Theory. In N. Badler,
only slightly more than that of shape interpolation, so real-time syn- B. Barsky, and D. Zeltzer, eds., Making Them Move: Mechanics, Control, and
thesis is possible at effective resolutions on current hardware. We Animation of Articulated Figures San Mateo, CA: Morgan Kaufmann, 1991,
expect that this algorithm will be a useful complement to current pp. 243-262.
techniques. [25] Maya Alias/Wavefront, Santa Barbara, CA, 1998.
[26] F. I. Parke, Parameterized Models for Facial Animation. IEEE Computer Graph-
Acknowledgements ics and Applications, vol. 2, no. 9, November 1982, pp. 61-68.
The authors wish to thank Sean Jenkins, David Miya, Amir [27] S. Pieper, Physically-Based Animation of Facial Tissue for Surgical Simulation,
Nasrabadi, Steven Puri, Kosta Saric, Steffen Wild, Lance Williams, SIGGRAPH 89 Tutorial Notes: State of the Art in Facial Animation, ACM, 1989.
Raymond Yeung, and the anonymous reviewers. [28] J. D. Powell, The Theory of Radial Basis Function Approximation. Cambridge
University Numerical Analysis Report, 1990.
References [29] M. J. D. Powell, Radial Basis Functions for Multivariable Interpolation: A Re-
[1] R. Barnhill, R. Dube, and F. Little, Properties of Shepard’s Surfaces. Rocky view. In J. Mason and M. Cox, Eds., Algorithms for Approximation, Oxford:
Mountain J. Math., vol.13, 1983, pp. 365-382. Clarendon, pp. 143-167.
[30] J. A. Russel, A Circomplex Model of Affect. J. Personality and Social Psychol-
[2] T. Beier and S. Neely, Feature-Based Image Metamorphosis. Computer Graphics
ogy, vol. 39, p. 1161-1178, 1980.
vol. 26, no. 2 (Proc. SIGGRAPH 92), pp 35-42.
[31] F. Scheepers, R. Parent, W. Carlson, and S. May, Anatomy-Based Modeling of
[3] P. Bergeron and P. Lachapelle, Controlling Facial Expression and Body Move- the Human Musculature. Proc. SIGGRAPH 97, ACM, pp. 163-172.
ments in the Computer Generated Short ‘Tony de Peltrie’. SIGGRAPH 85 Tuto-
rial Notes, ACM, 1985. [32] T. Sederberg and S. Parry, Free Form Deformations of Solid Geometric Models.
Computer Graphics, vol. 20 no. 4, (Proc. SIGGRAPH 86), pp. 150-161.
[4] C. Bishop, Neural Networks for Pattern Recognition, Oxford: Clarendon, 1995.
[33] B. Shneiderman, The Future of Interactive Systems and the Emergence of Direct
[5] A. Blake and A. Zisserman, Visual Reconstruction, MIT Press, Cambridge, 1988. Manipulation. Behaviour and Information Technology, 1, pp. 237-356.
[34] Pat Taylor, Disney/Dream Quest Mighty Joe Young facial animation, personal
communication.
[35] R. Turner and D. Thalmann, The Elastic Surface Layer Model for Animated
Character Construction. in N. M. Thalmann and D. Thalmann, eds., Proc. Com-
puter Graphis International, New York: Springer Verlag, 1993, pp. 399-412.
[36] J. Wilhelms and A. Van Gelder, Anatomically Based Modeling. Proc. SIG-
GRAPH 97, pp. 173-180.
[37] G. Wyvill, C. McPheeters, and B. Wyvill, Animating Soft Objects. Visual Com-
puter, 2, 235-242, 1986.
[38] Xinmin Zhao, Disney Dinosaur project, personal communication.

Figure 8a. Comparison of PSD and SSD on an animating shoulder – PSD using only two sculpted poses.

Figure 8b. SSD on an animating shoulder. The shoulder area is especially problematic for SSD due to the large range of rotational movement.

Figure 9. Comparison of PSD (at left) and SSD on the extreme pose of an elbow.

Figure 10. Smooth interpolation of four expressions (frown, neutral, smirk, smile) arranged along a single axis in a pose space, c.f. the discussion of Figure 7.
 Pose Space Deformation Notes

Pose Space Deformation (PSD) is a simple algorithm (it can be implemented in a few
dozen lines of code) that combines skinning and blend-shape approaches to deforma-
tion, and offers improvements and additional control.

PSD for Skinning

The problems with skinning algorithms include the famous collapsing elbow problem
and lack of realism. These issues are illustrated in the Siggraph 2000 paper on PSD
(Pose Space Deformation: A Unified Approach to Shape Interpolation and Skeleton-
Driven Deformation).

PSD versus Shape Interpolation

Blend shapes or shape interpolation (SI) has several drawbacks:

• Shapes are not independent. A major consideration in designing face models for
shape interpolation is finding sets of shapes that do not “fight” with each other.
Animators describe this common problem with shape interpolation: the model
is adjusted to look as desired with two targets. Now a third target is added; it
interferes with the other two, so the animator must go back and adjust the previ-
ous two sliders. And so on for the fourth and subsequent sliders. Sophisticated
models (e.g. those on Disney’s Dinosaur) can have as many as 100 blend shapes,
so this is a lot of adjustment due to shape “fighting”.
Likewise, the authors of shape interpolation programs have described artists’
complaints relating to lack of shape independence – with highly correlated shapes
it is not clear which slider should be moved. Some shapes reinforce, others can-
cel, sometimes a small slider movement results in a large change, sometimes
not.
• Animation control is dictated by sculpting Each slider controls one key shape,
each key shape is controlled by one slider, as it has been for 15 years of facial
animation.
• Linear interpolation

Pose space deformation (PSD) algorithms address these problems.

Interpolation versus superposition

The problem of shapes “fighting” is because the shapes are simply added. PSD inter-
polates, so keyshapes do not interfere.

Modeling is decoupled from animation control

It is necessary to be able to control the influence of each keyshape, but the one-for-one
mapping is not the only way to do this.

• Non-control shapes. Suppose “excited” and “happy” are two distinct target
shapes, but in a direct crossfade the intermediate shape is not adequate and a
new model is required. With SI one would need to introduce a new slider for the
intermediate “half-excited-half-happy” model, and this simple crossfade then re-
quires manipulating three sliders. Arguably this is complexity caused by the
system rather than desired by the animator. With PSD, place the halfway shape
halfway between the key shapes and it will automatically be interpolated during
the crossfade.

• Higher order parameters. The decoupling of sculpting from animation control

makes it possible to consider other sorts of control. Example: A character is
sometimes possessed by the devil, sometimes not. Place the keyshapes in a
PSD space with one extra dimension (‘possessed’), then a performance can be
changed from possessed to not by just flipping a switch.

• Psychologically relevant poses. The decoupling of sculpting from animation

control makes it possible to design facial poses by criteria other than motion
extremes. Example: Psychologist J.A.Russel has studied human perception of
emotion and found that emotional similarity is mostly explained by a two-dimensional
space . This space may be the most appropriate one in which to interpolate emo-
tional expressions. See Fig. 1.

• Regularization of parameters. If two sculpted shapes are similar, having one

slider per shape does not reflect this, and thus the ‘slider movement may make
small or large changes’ problem. PSD allows similar shapes to be placed as
neighbors in a chosen control space.

Smooth rather than linear interpolation

PSD allows smooth interpolation if desired, whereas with shape interpolation, in going
from shape A to B and then to C, an individual cv moves in a piecewise linear way –
there is a kink at B. Easing in/out of the transition does not change this fact.
Linear Algebra View of Shape Interpolation

Linear algebra gives another viewpoint on the character of motion resulting from shape
interpolation: Shape interpolation of n shapes each having m control vertices
n

S= wk Sk
k

can be written as a vector-matrix multiply with the keyshape vertices arranged in the
columns of a m × n matrix.
   
c1x | | ··· |
 c1y   | | · · · | 
   
 c1z   | | · · · |  
    w1
 c2x   | | · · · | 
    w2 
 c2y = S1 S2 · · · Sn  
    .. 
 ..   | | · · · |  . 
 .   
    wn
 .   | | ··· | 
 ..   | | · · · | 
cnz | | ··· |

The range of this matrix is at most of dimension n, so the animation is restricted to this
subspace of dimension n << 3m reflecting the fact that individual cvs cannot move
independently. The ‘Bruton’ Dino model appears to have 60*52 + 4*21 + 4*18 + 4*21
+ 15*35 + 16*35 + 11*35 + 18*35 + 17*21 + 17*16 + 17*21 + 11*21 = 6677 cvs and
so can be represented in a 3 (x,y,z) * 2 (symmetry) * 6677 length vector. On the other
hand it appears that there are under a hundred key shapes used to animate this head.
The preceding vector interpretation is valid; the next analogy is only that (an analogy).
Consider the cv’s as “samples” representing the resolution of the model – so the Bruton
model has 18k samples. Also consider the number of samples needed to represent an
object in the subspace of possible movement: 100 or less. This ratio of 100/18k reflects
a movement deficiency - it indicates how much modeling resolution is not used in the
animated movement.
A similar vector space interpretation of PSD is more complex but indicates that the
PSD motion is richer than that produced by shape interpolation. A single coordinate of
a particular cv is deformed as

c= wk R(|
θ − θ
k |)

where θ
is the vector of PSD parameters. The matrix R changes depending on θ,
and
wk are different from one coordinate to the next, so the range is not a simple subspace
– each cv has some amount of independent movement.
PSD versus Shape-by-Example

Shape-by-Example is a technique presented by Sloan, Rose, and Cohen at the 2001

I3D conference; their technique has some broad similarities to PSD. A quick analysis
suggests that their technique requires less computation at the authoring stage but is
slower at runtime synthesis: Shape-by-example deforms a point as

x[j](p) = xk [j]rm,k R(p)
k m

whereas PSD is 
x[j](p) = wk R(|p − pk |)
k
Since the sqrt involved in the distance can be folded into R, PSD deformation appears
to be more efficient.
The shape-by-example paper has some useful tips on improving RBF interpolation.

Implementation notes

This section needs to be filled out...

PSD is at its core a scattered interpolation of deformation based on the creature pose.
We used radial basis functions (RBFs) for the interpolation, since these are controllable,
smooth (if so desired), and have a large literature of techniques and extensions.
Sample Java language libraries for two-dimensional scattered interpolation are avail-
able at

www.idiom.com/˜zilla/Work/PSD/

(translation to C++ should be straightforward).

There are two routines. A Gaussian RBF routine has the constructor

public RbfScatterInterp(double[][] pts, double[] values, dou-

ble width)

where pts are the 2-D locations with data, values are the values at those locations,
and width is the width of the Gaussian kernel. A Thin-Plate RBF routine has the
constructor

public RbfThinPlate(double[][] pts, double[] values)

This routine does a thin plate + affine fit to the data. The thin plate minimizes the
integrated second derivative of the fitted surface (approximate curvature).
In fact any radial kernel different than a linear function will work, so one can choose a
smooth piecewise polynomial for efficiency, or store the kernel in a precomputed table.
It is often desirable that the interpolated function decay to zero far from the data, which
is not true of the thin-plate interpolation. The affine component of the thin-plate code
is useful; this should be incorporated in the Gaussian RbfScatterInterp routine.
Both routines implement the interface

public interface ScatterInterpSparse2

{
float interp(float x, float y);
}

as the call to actually perform the interpolation.

For the character deformation application one needs M-dimensional interpolation of
3-dimensional CVS, for arbitrary M depending on the number of joints and other qual-
ities. This can be accomplished using 3 separate M-to-1 dimensional interpolations.
The extension of the interpolation code from 2-D to M-D requires changing the dis-
tance function from x 2 +y 2 to an appropriate distance between points in M-dimensional
space.
Figure 1: PSD Facial poses selected according to psychological research rather than
traditional motion extremes. Model and figure by Nickson Fong.
Case Study: 102 Dalmatians Spot Removal

Research and development does not always transition smoothly into production.
Can we learn anything from cases where the R&D was not used?
On Disney’s 102 Dalmatians, the story requires that the lead dog character have no
spots through most of the movie; the spots reappear at the end. Animal handlers
determined that there was no effective and safe makeup that could be applied to
the dogs to remove the spots. This left Disney TSL with the task of painting out
all the spots on the dog.

”Ninety-five percent of the spot removal shots required a second,

hand-painting phase”...

...”any spot that procedural could get rid of would help”...

...”It turned out to be a huge job, much bigger than any of us had
imagined.”

— Jody Duncan, “Out, Out, Damned Spot,” Cinefex 84 (January 2001), p. 56.
The Disney TSL software group produced two rapid prototypes of automated spot
removal algorithms (one is described in the Siggraph 2001 Tech Sketch Lifting
Detail from Darkness). The latter algorithm had advantages over the painting
procedure both in time savings and in that it recovered the fur detail from the
spots when possible, resulting in accurate detail without the chattering or sliding
that results from imperfect painting.

Lessons
1. Early prototyping is beneficial, but is not enough in itself
2. Work directly with the artists who will use the software.
(Comment from a 102 artist: “why aren’t we using this?”)
3. Need a project champion in management

Without these, don’t waste your time!

Lifting Detail from Darkness J. P. Lewis
Disney TSL

This application sketch describes a high quality method for separat-

ing detail from overall image region intensity. This intensity-detail
decomposition can be used to automate some specialized image al-
teration tasks. Our work was motivated by the movie 102 Dal-
matians, which featured a dalmatian puppy without spots. Animal
handlers determined that the obvious idea of applying makeup to Figure 1: Hypothetical spot luminance profile, blurred luminance (heavy), and un-
sharp mask (bottom). When the blur size is too large the texture overshoots; when it is
the dog was not possible – evidently there was no suitable makeup too small the blurred curve follows the texture and texture amplitude is reduced.
that was both safe to the dogs and that would stay in place during
normal dog activity (including licking). This left Disney TSL with
the task of painting out all the spots on the dog every few frames
(the paintings were carried for a few frames with a simple scheme).
The spot removal task was larger than anyone guessed, and ulti-
mately required a large number of artists (up to 40) working for
eight months. The problem also proved to be more difficult than
expected from an algorithmic point of view. As the spots often had
visible fur texture, initially it was believed that there must be some
simple compositing technique that could lighten these spots.
To get a feel for the problem, consider one representative approach
inspired by unsharp masking: blur the dog and then subtract the
blurred version from the original, giving a high-pass texture con-
taining the fur. Then correct the intensity of the blurred dog to Figure 2: Demi-dalmatian.
remove the spots (without saying how this is done). Lastly, add the
high-pass to the lightened blurred dog, resulting (we hoped) in a
dalmatian without spots but nevertheless having fur derived from Applications
the original texture. The problem with such an approach is sug- In addition to the spot removal application, the intensity-detail de-
gested in Fig. 1: the scale of the blur must be exactly matched to composition has other specialized applications such as altering or
the spot profile or there will be an overshoot/undershoot around the removing shadows and reducing wrinkles (Fig. 3).
spot edge. This is not achievable since the spot transition regions
have markedly different widths even on opposite sides of a single It should be emphasized that although the effects shown here are
spot – some more adaptive technique is needed. routine work for a photoshop artist,
• The image alterations shown were produced with no artistic
Adaptive Filtering skill in a few seconds each by providing only a crude out-
line for the desired region. Consider Fig. 3: the altered re-
The intensity-detail decomposition problem is reminiscent of the gions have luminance gradients as well as recovered original
problem addressed by Wiener filtering: to separate signal from texture, so the effects could not be produced with a simple
noise. Making use of this observation, by casting detail in the role cloning operation but would require careful airbrushing fol-
of ‘noise’ and intensity in the role of ‘signal’ we were able to ap- lowed by detail painting.
ply a Wiener separation approach; a simple spatial-domain adaptive
Wiener filtering algorithm described in [1] works quite well. • Unlike manual retouching, the detail decomposition can be
keyframed and produces consistent effects across frames.

Intensity Modification References

Once the detail is successfully separated we need a means of alter-
[1] J. S. Lim, Two-Dimensional Signal and Image Processing, Prentice-Hall, 1990.
ing the image region intensity in a simple and controllable fashion.
The membrane (Laplace) equation ∇2 u = 0 produces
an interpo- [2] W. H. Press, S. A. Teukolsky, W. T. Vetterling, B. P. Flannery, Numerical Recip-
lation of specified boundary conditions that minimizes (∇u)2 dA ies, Cambridge, 1993.
(the integrated gradient); as such it provides an adequate way of
interpolating intensity specified on a boundary curve (e.g. a rough
spot boundary). The Laplace equation is a linear system Au = b
with A being a sparse square matrix whose dimension is the num-
ber of pixels in the region. The intensity-detail decomposition was
initially prototyped in MATLAB (which took less than a day) and
used sparse matrix routines in that package. Our algorithm was
later reimplemented in Java, which took about three months, ap-
proximately half of that being devoted to the membrane solution.
The first implementation was a direct (non-sparse) matrix solution,
programmed by Yi-Xiong Zhou. This was adequate for very small
regions but was too inefficient for larger regions – areas of 1502
pixels were requiring several minutes and 0.5G of memory! Fortu-
nately the Laplace equation can be solved with the multigrid tech-
nique [2] (and in fact is the model problem for this approach). A
multigrid implementation of the membrane reduced the solve time
to several seconds even for large regions. Figure 3: Altered regions include shadows under nose, eyes, and other changes.
Lifting Detail from Darkness J. P. Lewis
Disney TSL

Figure 4: Original dalmatian picture

Figure 5: Missing spots detail.

Software-Production Interaction: A slice of life
(This is a revealing, humorous look at life inside the production fishbowl. The names
have been changed to protect the ultrastressed.)

• A facility had a software help phone number, to which each programmer was
assigned on a rotating basis. Programmers shared offices. When one particular
programmer answered the phone, we would usually hear "Hello" (silence for a
while as the artist explains the issue) "Why would you want to do that?"

• A software TD was working with a program that had a large number of

parameters controlling appearance, and there were a number of objects; in total
perhaps a hundred parameters were involved. The program was introduced to a
supervisor, who responded by calling an urgent meeting. In the meeting he said
there should only be THREE PARAMETERS: color, size, and one other. The
software TD removed all the remaining parameters. A few weeks later the
supervisor called the programmer into another urgent meeting, because there was
TOO LITTLE CONTROL over the look.

• I never had any problem finding the soft dev person on our show. At the end of
the day I'd always run into him when I was walking out the door.

• In the early stages of the particle software for our show, we'd get together with
our soft dev person to see the progress. He'd show us a bunch of vectors moving
around and ask "What do you think?" We would ask in reply "Are you making
fun of us?"

• All I wanted was a plug-in for our tracking software. What I got was a two year
plan for writing our own tracking software.

• A particular show was using custom software to produce its signature effect. The
show and its effect were recognized at an industry conference, and an effects
supervisor gave presentations showcasing the work. At an internal meeting he
said that he never wanted to use custom software again.

• Someone was offended when we couldn't say how long it would take to make one
of the features of the program work better. Work better how?

• A programmer was working literally every possible waking hour to produce the
signature software for a show. On his birthday his colleagues arranged a surprise
party for him because he had been working so hard, but he was told that he could
not leave to go to the party. %The software was finished, and the %effect was
well recognized in the industry. After finishing the %software he quit the
company; we know not to mention his %name inside the company.

• "You guys get away with murder." -- A visual effects supervisor, speaking of the
software team.
• A software company sent around an announcement saying that their new feature
would be shown at Siggraph, but it wasn't. Two years later, they told us that their
new feature would be ready in time for Siggraph...

• A particular program was requiring massive resources, both time and memory.
Speeding the program up had been on the production's to-do list for months. The
particular program ran on shared memory multiprocessor machines. A consultant
with experience in the esoteric cache/processor interactions on these machines
was called in, and he quickly wrote a simulation program that showed some
simple reorganization could speed the program up by a dramatic factor (like
10,000), since it was spending most of its time waiting to lock down memory on
other processors in order to write. The consultant believed that the required
reorganization would take a few days. Production was asked whether they wanted
to pay for the consultant to return. They didn't.

• A programmer wrote an interface with a help button next to each command. A

CG supervisor said to the program author that he didn't think the artists would
read the help text. A second programmer interjected that "at a minimum we
should expect literacy from our artists".

• Compositors were complaining that the 3D elements they received were not lit
properly, so a software TD was assigned to investigate. The software TD asked
one of the responsible artists, "what color space are you working in? Are the
background plates linear?" The artists responded, "what space are you talking
about? This is MAYA SPACE!"

• More color consistency troubles,

• and once again someone was assigned to investigate. The investigator found that
one artist had his monitor gamma correction arbitrarily set to 0.8 and the color
balance set to a strong green. The investigator told the artist that he needed to use
the latest standard lookup table. The artist responded "no, it hurts my eyes when
I surf the web."

• A CG effect had some difficult lighting issues, so a TD addressed the problem by

introducing reference spheres into the scene -- if the spheres were lit correctly, the
correct lighting could then be applied to the complex effect. A visual supervisor
saw the scene and instructed the TD to remove all the spheres.

• A programmer was given about four months to develop a complicated effect; he

said that he could complete a version in this time. After working on it for three
weeks, the production asked to see it work on a "test" that was comparable to the
most difficult shot in the film. The programmer protested that it was absurd to
demonstrate something that wasn't scheduled to be finished for another 4 months,
 but they responded that after the test was done the programmer would be given
time to write the code.

• A new production coordinator was trying to reign in the schedules on a particular

production. He asked each of the programmers how long they would take to
finish the task they were working on, without troubling to understand the tasks,
and knowing nothing about software development. One programmer said "I don't
know yet" several times, which rubbed the production coordinator the wrong way
of course. Thereafter the coordinator was riding the programmer, asking every
day for a rigid estimate, still without knowing what it was that was being
developed. One day the programmer was finished, and proudly called the
coordinator over to show that his code was now working. He showed the
coordinator a green screen, all green, nothing else. To the programmer this was
sufficient proof that the software was working -- the program being some kind of
format decoder, and passing green though the system was an adequate test. The
coordinator's reaction cannot be expressed here.
From Ivory Tower to Silver Screen:
Getting Under Shrek’s Skin
By Jonathan Gibbs and Bert Poole
PDI/DreamWorks

Finished Fiona in Production

 These notes chart the course of developing the skin shader for PDI/DreamWorks’ second feature film
SHREK. This is truly a case of “from ivory tower to silver screen”, as research from both the computer
graphics community as well as dermatology and biomedical engineering came together during the visual
development of the film. In the first section we focus on building the technical underpinnings of the skin
shader. The second section focuses on applying this shader to our most challenging character, Princess Fiona.

Simulating the visual quality of skin is critical to the success of having reasonably realistic computer-
generated characters occupy lead roles in feature films. During close-ups, skin is the single surface that
occupies most of our field of view -- the canvas on which the essence of the story unfolds as the emotions of
the characters are revealed.

Skin has complex structures and its reaction to light is not simple. Most of us don't really understand why
skin looks and behaves the way it does, but we are all intimately familiar with its appearance. We know when
it looks right and are distracted when it looks wrong.

So the minimum challenge is to create skin that is not distracting, which is acceptable enough in its
appearance that it doesn't detract from the emotional message that the characters are trying to convey. Skin
that lets the acting of the characters come through and helps us accept them as real.

Beyond that, if we have an understanding of what makes skin appear the way it does, and with control
over those qualities, it should be possible to go beyond mere acceptance. This makes it possible to create skin
that actually enhances the character’s performance; skin that in subtle ways might be beautiful, luminous, or
creepy; perhaps more so than real skin could ever be.

With these goals in mind, we developed a process for simulating the look and reflective quality of real skin
under a wide range of lighting conditions and character motion. This process was applied to SHREK’s lead and
secondary characters. The fundamental component of the process is a shader that defines a lighting model
based on a simulation of skin's physical structure. Hand-painted maps that add blemishes, control the thickness
of different skin layers, and define the distribution of moisture augment the results of the shader. In addition,
portions of the shader can be controlled by the animation system. For example, wrinkles can be applied
around areas of compression or stretching like around the corners of the eyes.

THE DEVELOPMENT TASK

In May 1998, with ANTZ still 5 months away from release, shader development began on SHREK. Perhaps
the most challenging shader to be written was the skin shader. Skin is a complex arrangement of layers of
tissue having varying degrees of translucency and pigment. It contains microstructures of pores and non-
uniform layers of moisture which change how light reflects off of it. Light also enters the skin and bounces
around inside and between layers to created a warm diffuse appearance. These bounces are called sub-surface
scattering. Because skin is layered, various parameters must be specified not only on the surface, but also for
each layer. The layered nature of skin makes it a non-homogeneous medium. All this makes skin very hard to
render. Our intimate knowledge of human skin makes it a very hard surface to fake. We needed a shader that
would simulate these visual properties of skin.

In addition, we had certain practical requirements that the shader must meet. The shader had to be cheap
enough to use on multiple characters in over 1500 shots. This means it can’t take much longer to shade than
our typical, standard, shaders do. It also had to fit into our standard rendering pipeline. Our standard renderer
is a modified a-buffer renderer and although we can do ray tracing we could not afford to use it on a regular
basis for the main characters. A more subtle requirement is that control is more important the strict physical
realism. Since we were not creating a photo-realistic film, the shader has to be adaptable to the wishes of the
directors and art directors. At the time this was being developed, SHREK was still in visual development and
designs were apt to change. Also, we wanted to use this shader on a wide variety of human and human-like
characters.
 There are three visual properties of skin that immediately stood out. First, the color of the skin is view
dependent. This is because of the sub-surface scattering, but the effect is that skin is a different color
depending on whether you view it head on or at a grazing angle. Second, the color is the skin is dependent on
light hitting the surface “nearby”. This is also because of the sub-surface scattering. Unlike hard objects, the
light hitting the skin at a certain location may affect the color at a nearby location because light can be
transported though the skin and emerge at a different location than it entered. Finally, the specular behavior of
skin is also more complex than the standard specular formula currently in use. There are at least two different
sorts of specular behavior: one term is from the skin itself, and one is from the oils and moisture resting on
top of the skin. The specular term from the skin itself is particularly anisotropic.

With this information, we set out to find a solution in the computer graphics literature. Unfortunately, we
quickly found out that there has been very little published on rendering skin. The only appropriate reference
we found was the Hanrahan/Krueger paper from SIGGRAPH 1993 titled “Reflection from Layered Surfaces
due to Subsurface Scattering” [1]. This covers skin as well as other similar layered materials. However, we
quickly decided that this approach would be too expensive for Shrek to use and possibly too difficult to
control. So, back at the drawing board we came up with two separate plans. Plan A was to explore the use of
the Kubelka-Munk model for layered pigments. Plan B was to follow a standard CG tradition and look for
ideas in other fields.

Plan A: Kubelka-Munk
Kubelka and Munk devised the Kubelka-Munk model in 1931 to describe the optical property of a turbid
medium that absorbs and scatters light. A medium that transmits light largely by a diffuse transmission such
that objects are not directly seen through it is considered a turbid medium. It was originally developed for
paint films, but seemed to be a good possibility for skin due to its elegant treatment of layered models. The
Kubelka-Munk model was recently applied to metallic patinas [2] and watercolor rendering [3] where we first
noticed it. Haase and Meyer first introduced the model to the CG community in 1982 in a paper called
“Modeling Pigmented Materials for Realistic Image Synthesis” [4].

In this model, each layer is described by three values: the thickness, the diffuse back-scattering coefficients
and the absorption coefficients. Immediately this seemed like a great idea for skin as it seemed to fit the model
perfectly. It is computationally cheap as it is an implicit model and doesn’t require taking multiple samples
across the depth of the layers or firing many rays to investigate the region below the surface. We began
playing with the model and found it powerful, but very unintuitive. It is difficult to specify a desired color using
these terms, and the same color can be made in various ways especially with a multi-layered system. Excited
about the idea, but unsure if it would lead to a robust model of skin or be usable in production we also began
on Plan B.

Plan B: Steal from Some Real Science

Plan B was a fact-finding trip to the library to learn more about the actual properties of human skin. Below is a
rough diagram of the physical properties of skin. You can see it is a layered structure. The top layer is called
the epidermis. It is very thin, and is where the key skin pigments live. Pigments such as Melanin are primary
responsible for the skin tone. The difference between races is primarily a difference of pigments in the
epidermis. Below the epidermis is the dermis. This is a much thicker layer and is where blood-flow occurs.
The difference between a flush person and a pale person is generally due to the amount of blood in the dermis
at that time. Underneath the dermis is the inside of the body including organs and bones. We simplistically
lump all this stuff together as the “bones”. This simple model is fine since very little light penetrates the
dermis.
 The Layered Structure of Skin

Due to this layered nature, the optical pathways in skin are very complex, as shown in the next diagram.
About 5% of the incident light will reflect off the skin in a specular manner. The rest will penetrate the
epidermis and bounce around inside it. Some of it will bounce back out of the epidermis, some will bounce off
the boundary between epidermis and dermis, some will be absorbed, and some will proceed down into the
dermis. Once inside the dermis the light will bounce around more, possibly getting absorbed and possible
making it back up to the epidermis and possibly making it back out and becoming visible again.

Optical Pathways in Skin

 In currently literature, skin is described using a model similar to the Hanrahan/Krueger SIGGRAPH
paper [1]. However, we found older works from the early 1980s that actually used the Kubelka-Munk model
we were already experimenting with! This was a very exciting find and confirmed we were on the right track.
Particularly useful to us were a series of articles by R. R. Anderson and J. A. Parrish from the Department of
Dermatology at Harvard Medical School, including “Optical Properties of Human Skin” from 1982 [5]. This
was the boost of confidence we needed to continue on. The papers also gave us more information on the
physical nature of skin and good starting points for the coefficients required by Kubelka-Munk.

The Skin Shader

With Plan A and Plan B now pointing to the same solution, we went ahead with writing the prototype of
the skin shader. There was some nervousness about using such a physically based model as the basis for the
shader and whether it would be too hard to control, so we also wrote a “simple skin” shader which just
focused on the visual properties of skin and ignored the Kubelka-Munk model entirely. Quickly, we found that
the added complexity of the more complex shader was worth the struggle. As the next sections will illustrate,
it took us several tries before we understood how to control the system.

The final skin shader simulates several key aspects of real skin: translucency, oily highlights, softness, and
selective backlighting. There are four important layers: the oily layer on top and the two layers of the skin
itself (epidermis and the dermis) and the bone layer underneath. Controlling the thickness of the epidermis
layer via maps allows more or less of the dermis to be visible in different regions (such as the lips). This
layered model was shaded according to an implicit model of sub-surface scattering, giving the warm "flush",
softness, and tonal variation so critical to representing realistic skin. Another key feature of the skin shader is
the ability to map bone regions, which allowed for the painting of areas that do not contain actual bone and
allow backlighting to glow through, such as the ears. The specular aspect of the lighting model has one
component that accounts for the moisture of the skin and another that accounts for the anisotropic effects
caused by the pores.

Texture maps for freckles, blemishes, and other detail are applied to either the dermis or epidermis in order
to get effects that work well in conjunction with the lighting model. If all of the detail sits in one layer, you do
not get the dynamic accumulation and loss of detail as the character moves through light in a scene.

The implicit model of sub-surface scattering has two primary parts. The first is that the skin layers are
combined using the Kubelka-Munk model. The thickness and back-scattering parameters are parameters to
the shader. Instead of providing the absorption as a parameter, we instead gave the “forward scattering” and
computed the absorption as one minus forward scattering. This makes the shader much easier to use since
the animator or TD setting up the shader needs only think about scattering and not about absorption. In the
simple case, the back and forward scattering terms can be identical. The apparent thickness of a layer is
depends on the view angle, so this model gave us the view-dependent colors in a very natural way.

Underneath the skin is the “bone” layer. It is a white diffuse layer with an adjustable transparency. Under most
of the skin it would be opaque, but under certain parts like the nose and ears it would be partially transparent
to simulate cartilage instead of bone. When the bone was not opaque we consider lights from behind the
surface as well as those in front to get a natural backlit effect.

The second part of the implicit sub-surface scattering model is less elegant, but effective nonetheless. As is
typical in most renderers, the lights only compute information about how much light falls on the current
shading sample. This makes it hard to get lighting information for “nearby” points that play an important role
in skin illumination. We solved this problem in a purely visual manner. The effect of nearby light was only
important were the amount of light changed rapidly due to shading or shadow boundaries or high-frequency
bumps. Identifying those locations allows us to mimic the effect by shading certain samples twice and tinting
and mixing the results. For example, we may query the lights with the shadows both on and off and mix in a
slightly redder version the shadows-off color into the shadows-on color based on how much the current point
is in shader. Similar techniques can be applied to other aspects of the shader.
The following images were the early tests of the skin shader before it was given to production. The first image
is a very early proto-Fiona model. The second image is the same skin parameters, but with only one back light
turned on. The next two are very early proto-Shrek models. As you can see, the basic effects are in place, but
we are far from finished. The result of the process is tied up in the visual development of Shrek’s main
characters, particularly Princess Fiona. How we got from these prototype images to the final look you see in
the film is the subject of the rest of these notes.

Very Early Proto-Fiona

Very Early Proto-Fiona Back-lit

 Final Fiona

Very Early Proto-Shrek

Very Early Proto-Shrek

Final Shrek
 REALISTIC SKIN IN A FANTASY WORLD
In completing the research and design of any computer animation technology, the true marks of
refinement can only be found through the fires of production. The skin technology at PDI/DreamWorks is the
classic example, where thorough investigation and design formed only the husk of what was to come. Real
world experience was needed.

In September 1998, SHREK first entered the world of visual development at PDI/DreamWorks. With a
host of humanoid characters filling most of the screen, suspending disbelief was going to be a huge challenge.
Having proved it’s facial animation techniques on ANTZ were more than adequate to push the boundaries, the
lighting sophistication would need an advance of its own to pull off the illusion of humanoid characters in a real
world. In ANTZ the surfaces were visually dynamic and rich, but lacked depth and volume. Much like the hard
shells of their real world counterparts, the closest that the surfaces in ANTZ got to fleshy was a slightly satiny
feel. Much like Shrek’s favorite food, onions, the skin surfaces in Shrek would need layers. The skin system was
developed to do just that, and so in that September, Fiona entered visual development.

One of the main challenges from the onset of SHREK was to discover just how much reality there was
versus how much fantasy there was to be. It was definitely a fine line walked on all sides of production, and in
Fiona, we found the pinnacle of challenges. How fantastic was she? Should she have the visual interest of real
skin, with all its blemishes and black heads? Should her skin be without blemish, in a state of perfection like
that of a Barbie doll? Or should she be given the spa treatment so as to have her face plastered with a
foundation and airbrushed with blush-like make-up? Having modeled a less that real looking Fiona with
exaggerated features, the initial decision was to explore the latter, where there were no visible blemishes, and
her face mainly took on a made-up quality. The fine line had been drawn, moving away from the real world,
and favoring the fantasy world of SHREK.

When development hits production, as in the case of the skin shader, the needs of the moment dictate
the path of evolution, and testing becomes something that happens on the fly. In the world of proprietary
software, we are all high paid beta testers. Dailies sessions featuring Fiona became a period of personal pain as
those of us who slaved the most stumbled about and tried to establish what humans in SHREK’s world would
look like. Some of this was certainly due to the fact that Fiona was a unique experiment that would need much
iteration no matter how good the effort. Creating a world not yet seen is not an easy task. But part of the
pain was certainly due to the fact that we were putting a new technology created with the intention of
mimicking real world phenomena into a fantasy situation and exploring that complex issue in the context of
that artificial world. It was not going to work. SHREK would need a helping hand from the world of reality.
Almost one year later, that help would come from PDI/DreamWorks other division for commercial and film
effects in the form of a test for a potential film job.

Skin Faces a Reality Check

The test involved creating photo-realistic aliens, and integrating them into a live action plate for a
potentially enormous live action job. This was the furnace the skin system had been waiting for. In initial
development conversations, other techniques for solving the problem of skin were discussed and debated.
Eventually the conclusion was made, after some initial testing, that our current skin system was still unproven
and from all appearances, had not yet been exploited to it’s full potential. Skin seemed like the right choice for
a number of reasons. First, skin seemed like it had the right ingredients without having had the benefit of the
right types of testing. In short, it had potential. Second, the other methods discussed avoided the implicit
subsurface scattering approach of skin, and favored complex volume calculations that were both impractical
and too costly for the volume of shots being discussed.

We moved ahead, and decided we would make changes where it was necessary. The decision was made
to develop a slightly custom version of the skin, so as not to disturb SHREK’s production, and have the
freedom to explore alterations. This gave us some freedom due to our small staff of three or four to explore
the issue even more in depth and find those cracks that still had yet to be filled.
The Dermis and Epidermis in Practice
At this point, we did some quick testing of real world conditions versus what the shader could do. We
soon came to the conclusion that the epidermis and dermis relationships established by the initial research
would work well. Experimenting with various combinations, we were able to get believable fleshy feeling
spheres and teapots that seemed to react well to the lighting situations in which we put them. How to then
combine these delicate settings with useful data to generate believable human skin was another thing
altogether. We soon discovered that with the skin system, it was very easy to get the same visual result for a
single frame 6 or 7 different ways. The crucial and extremely difficult task was deciding which of those ways
was the right path. The physically based dermis and epidermis relationship was a hard thing to wrangle and
required a change in both thinking and practice when surfacing. Typically in surfacing jobs at
PDI/DreamWorks, materials involve a diffuse, specular and bump component with the occasional need for a
few other attributes. This is a very straightforward manner of working. What you see is what you get. If you
need a rock painted, you go and dig up your favorite rock textures and go to work. Unfortunately, in the case
of skin, this doesn’t work (as the many plastic-skinned zombies out there in the film world will attest).
Texturally, skin is easy, as we have billions of photo references from around the world to pull from. But as we
have seen from the skin research, this is but the cumulative effect of layers and layers of translucent pores and
cells blanketed over sheets of veins and bone. A great looking still image of skin only takes a decent scan of
someone’s face. A system that works and behaves properly when put in motion though a number of lighting
scenarios is something else altogether.

The dermis-epidermis relationship is a complex one involving abstract methods to achieve a complete
fleshy appearance. Unfortunately, up to that point, we too tried to cheat the function and feed it insufficient
and misplaced data. As a result, the skin we got was a thick deadened plastic look. If we were going to do it
right, we would have to apply our skin in layers. Veins, muscles, blood and subsurface discolorations are in the
dermis, and freckles, pimples, flaked skin, and superficial discolorations are in the epidermis. To break this
honor system would result in failure. As we worked the skin, we had to be honest and clean about it, keeping
these various attributes separate. It really forced us to search and examine each portion and it‘s real world
counterpart. For instance, let us say we wanted to create something as subtle as sun-freckles. There are four
conceivable ways of achieving essentially the same look. The first would be to darken the underlying dermis
portion significantly enough as to dampen down the epidermis layer, and create a darkened blemish.
Unfortunately, this might work for something straight on, but as the model rotated away from camera and
into some light, over those pixels we would have an accumulation of epidermis along the profile edge
obscuring the freckle. The blemish would appear to come and go as the models moved through the scene.

This is an effect that is desirable for both veins and underlying discolorations, but not appropriate for
something as superficial as a freckle. We could indicate the blemish through a thinning of the epidermal
thickness, and use some of the darkened tissue from underneath to reveal it, but this color would shift
depending on the accumulation of epidermis as the normal change with respect to view. We could use the
diffuse multiplier to darken the effect of the skin altogether, but this would result in a deadened inconsistent
appearance. In general, this type of simplistic linear multiplier on top of the skin effect is a bad idea for
something as layered and varying as skin. Unfortunately, it is also probably the most tempting resource
because of its instant gratification. Finally, the proper solution is to let the sun-freckle lie right out there on the
epidermis, and thicken that skin up a bit so as to receive little of the color accumulation from below. Not by
accident, this is in fact the situation with a thickened freckled chunk of cells on any one of us. The freckle is an
easy example, but the entire complexion works this way, and requires that type of investigation and becomes
extremely difficult when the cumulative effect consists of several layers. This is exactly what we found as we
worked through the problem on this real world test. The alien would require an aged look, with freckles and
pores to match a real world human counterpart we had been given that was shot against a blue screen. The
task was daunting and took some time to get the hang of.
 The top image shows Fiona’s cheek dead on with lots of accumulated detail coming from thickness variations in the epidermis with a single
freckle actually residing on the surface. The bottom image shows how the information is lost when the thickness of the epidermis is increased
due to the viewing angle, while the freckle riding the surface remains visible.
If You Can Cheat, Cheat
As we worked through the problem concerning the dermis and epidermis relationship, we found that
something was lacking. We soon found that matching reality isn’t the necessarily the best goal when making
the unbelievable believable. In regards to the translucency, we found we had to set our aims a little higher, and
go a bit over what we would consider a match for the human actor. When looking at skin, we want to
perceive that fleshy feel. It’s easy to have a cadaver on your screen. So we in fact pushed the epidermis to a
point where it was both brighter and thinner than one would expect to find in reality and in fact much
different than the approximations made based on the papers. What this did was provide a hypersensitivity to
lighting conditions and the incident viewing direction. Where the skin rolled across the surface toward the
profile edge, if the surface at that point was perpendicular to a key light, you would have an accumulation of
bright epidermis—the result being a pleasing thick fleshy contrast to the skin found in the regions where you
were looking through less epidermis and into more dermis, or flat on from the viewing direction. This was, in
fact, discovered after watching a one-year-old baby walking around in the early weekend sun. The appearance
of baby skin was the proper goal for an egg-shaped alien head, and because we lacked the sophistication of a
volume-based accumulation of light through the pigment of fatty baby tissue, we cheated where we could.

Another discovery we made was that skin rarely features hard shadows. This provided some interesting
discussion and insights regarding the nature of skin and shadow. We determined that the results of sub-surface
scattering and the possibility of retro-reflective properties on a pore-scale may diffuse both penumbra regions
of shadows and terminator boundaries in shading. Three hacks came out of this to simulate these effects. First,
stemming from discussions regarding the retro-reflective qualities of the moon’s surface and the resulting hard
terminator that gives it the crescent nature in waxing and waning periods, we used a diffuse exponent term to
broaden the diffuse region and pinch the shading region. Second, we applied a shading color, so as the surface
rolled away from a light and into shadow, it would receive an additional color term in that falloff region. This
was especially useful for tinting high-frequency bump mapped regions of the skin where, in reality, light should
be entering the lit side of the pore, moving through the skin, and influencing the back side of the pore with a
color accumulation because of the translucent scattering properties. In a bump mapped situation, with no
displaced geometry on this high-frequency scale, there is no way to take advantage of the physically accurate
way of handling this. Thirdly, we added a shadow color term, so that shadows could role through a warm
contribution in the penumbra region. All three of these heightened the fleshy feel and helped sell the illusion of
volume and depth to our flat surfaces.

Having worked through these issues to a point where we had a fairly believable fleshy soft-skinned alien, we
now turned to the specular nature of skin. Our reference actor was extremely greasy at the photo shoot, and
fortunately for us, this provided a thorough and excellent test for our skin system from which to learn.
Certainly this aspect would be one we would not have tackled on our SHREK heroine.

Adding a Bit of Shine and Elbow Grease

Our skin initially had a single specular term. This would prove to be not enough. As we took some time to
observe and study the appearance of skin under various lighting conditions, we came to a number of
conclusions.

First, skin generally has a broad sheen that occurs in grazing lighting conditions. We observed that as we
held an arm out on a bright sunny day, a hard broad bright area would rim the arm. At first we suspected that
hair and follicles were responsible, as in the case with peach fuzzed cheeks. Soon, though, we realized this
specularity was a result of a dry dead skin layer that occupies the top layer of the epidermis. We believe this
accounts for the broad specular nature of the skin, not including the oils and moisture that is found. So we
allowed the specular term that existed in the shader to speak for these conditions and be represented by a
large dull sheen on the skins surface.
 Second, we determined that the other forms of specularity generally associated with skin are that of the
moisture and oils that reside on the surface and in the pores. These are micro-scale events, occurring in
surfaces that actually ride on top of the skin and are apart from the skin itself. The topology of these globules
of moisture are much more varied and non-uniform than that of the larger skin surface they ride, and as a
result, over a patch of moisture, there will be several incidences of the surface being in the specular direction.
This along with the smooth hard surface of the moisture accounts for the many sharp specular highlights that
are kicked back. To compensate for our inability to reproduce this effect with our large approximations, we
gave the moisture its own bump term that would be applied on top of the skins term. In order to be able to
push this bump to the severity needed to catch highlights at times when the skin surface was not in the
specular direction, we allowed the underlying shading of the skin to be unaffected. In short, this was only used
to calculate the specular contribution of the moisture. No other aspects were considered or needed. This
component took the role of our moisture specular term.

On the alien in particular though, we ran into a problem and an unexpected solution. The large folding
eyelids of the alien screamed for having some moisture accumulation, as most eyelids do. As the skin folded
into the contracted position, we wanted small glints of specularity to be reflected back indicating the fleshiness
of the skin. This looked great once we got it in place, but we saw that as the lid relaxed back down during
blinks, the highlights would be strewn across eyelids like zebra stripes. The problem was that our illusion of
moisture was falling apart. In reality, as the skin bunches up and folds, the moisture should be collecting in the
pores and kicking back in lines across the folds. As the eyelid then unfolds back over the eye, the skin is pulled
tight and the moisture is no longer concentrated in the specular direction. This is what would happen in
reality, but there is no such complexity to our cheat. Fortunately, a technology that was providing for the
wrinkles in our alien would come to the droplet’s rescue. We had been using a wrinkle shader to handle both
bump and displacement wrinkles in the eyelids as the contracted. This was done through applying a function
based on the compression and U and V. As long as the eyelid remained relaxed, so did the UVs. As the eyelid
folded, the compression would occur, and the desired effect would be multiplied in. We then took this same
evaluation, and applied it to our moisture. As the eyelid folded up, the particular highlights would take their
marks. As the UVs decompressed, the highlights would disappear. It worked extremely well.

Having added all of these features, we found that our specular model was still wanting. While our highlights
were of the right size and intensity, we found that they lacked visual interest. As we went back to our human
drawing board and looked at more images, we found that our highlights were lacking the interesting shape and
contour found in real skin. As light moves across a face, it bends and contorts to fit the skin. We had known
that skin pulled tightly over bone witnessed a tight and directed highlight while thicker muscular or fatty skin
loosely hanging tended to have a more uniform diffused specularity. After looking into it further, we
determined that the skin has an anisotropic quality due to the stretching of pores in a particular direction.
Depending on the flex and bow of the skin, the highlight will flex tighter and looser depending on the
stretching of the pore structure. The best way to describe this is with the classic example of brushed
aluminum. Minute directional grooves or events, when bunched will affect the broader perspective. As a result
we added an additional anisotropic direction for the specular and moisture specular to compensate. This
perhaps made the greatest difference in regards to specularity. It made everything look that much better and
helped fight the uniform specular treatment found in most computer-generated surfaces.

At this point, our alien was complete, and our skin system had undergone a large transformation, both in
terms of user knowledge and actual features. As the alien test shipped, and Shrek’s skin continued to struggle,
the decision was made to take the newfound knowledge and apply it to SHREK’s main human character, Fiona.
Back to the Drawing Board
At that point, Fiona’s model had undergone a number of changes. Her appearance was much more human
and much less like one of SHREK's fantasy freaks. While this was a great improvement, the skin still took a
deadened flat appearance. As the good and bad was sifted from the previously separated skin systems, work
began on updating Fiona with the new knowledge and features. An additional directional diffuse term was later
added to the specular components that allowed for a ramp from the diffuse region into the specular region.
This came out of observations where it seemed that areas of the skin approaching the specular direction
warmed up due to the light penetrating through more dermis and less epidermis. After we added this feature,
her ramp into specular regions appeared much more natural, and her skin had a bit more added depth and
volume to it.

As the Fiona grew closer to completion, a make-up artist was brought in to do a one-day session on a
certain production coordinator who had a fair appearance and reddish auburn hair. Many photos were taken
both indoor s and outdoors for skin and hair reference. These became invaluable. Not only were they used
for makeup and hair reference, but we used them to study how much loss of detail occurred at certain ranges,
and tried to match this effect through the use of different setups. While this was rarely utilized in the end,
some very thorough observations helped us fine-tune what was to be our final setup.

Soon, Fiona’s skin had undergone the royal treatment and giant leaps had been made over previous
incarnations. One day, as she came up during dailies, she looked vibrant and alive. Fiona took on some
additional tweaks here and there, but generally her new self was found. Other SHREK characters in
development, such as Farquaad, were retrofitted using Fiona as a template.

The Old Flashlight in the Nostril Trick

In this tale of skin development, we have focused almost entirely on Fiona, but there is one feature that
she does not take advantage of that is worth mentioning. The skin system includes one other cool feature: the
ability to control the thickness of bone. While this might sound unusual, what it allows us to do is specify
areas of the body that contain cartilage rather than bone, and have them treated appropriately in backlight
situations. Much like holding a flashlight up to your hands (or nostril), if in the skin system you have specified
that the area contains little or no bone, you will achieve a warm glow with no additional setup. This was most
noticeably exploited on Shrek’s ears. In several shots in the film, you can see veins appear backlit through
Shrek’s ears. There are several little enhancements like this that are not mentioned here.
Top image shows ears in frontal lighting situation; bottom image shows backlighting situation with translucence
 …TO SILVER SCREEN
SHREK’s skin system saw a long and twisted evolution. First were the indispensable dermatological papers
and studies into pigments and the shading properties of skin. Those discoveries were then coupled with
masterful programming skills to give a foundation on which to build our humans. Most unexpectedly, the
nuances and additional techniques needed would have been hard to discover had Shrek’s skin not been
investigated against the real world with the opportunity for testing. Through observation and investigation,
great strides were made thanks to a live-action test that enabled the testing and discovery to continue against
a real world example, only to have it return and manifest itself in a completely made-up fantasy world called
Shrek. From ivory tower to silver screen, SHREK’s skin system has evolved and developed thanks to many
different experiences and discoveries into a solid piece of filmmaking technology. With many new
advancements in development, the best is yet to come.

Human Character Tested Shortly After Completion of Fiona

BIBLIOGRAPHY
1. Pat Hanrahan and Wolfgang Krueger. Reflection from Layered Surfaces due to Subsurface Scattering.
Proceedings of SIGGRAPH 93, Computer Graphics Proceedings, Annual Conference Series, pages 165-
174.

2. Julie Dorsey and Pat Hanrahan. Modeling and Rendering of Metallic Patinas. Proceedings of SIGGRAPH 96,
Computer Graphics Proceedings, Annual Conference Series, pages 387-396.

3. Cassidy J. Curtis and Sean E. Anderson and Joshua E. Seims and Kurt W. Fleischer and David H. Salesin.
Computer-Generated Watercolor. Proceedings of SIGGRAPH 97, Computer Graphics Proceedings, Annual
Conference Series, pages 421-430.

4. Chet S. Haase and Gary W. Meyer. Modeling Pigmented Materials for Realistic Image Synthesis. ACM
Transactions of Graphics, 11(4), pages 305-335 (October 1992).

5. R. R. Anderson and J. A. Parrish. Optical Properties of Human Skin. The Science of Photomedicine, 1982,
pages 147-194.
 From Research and Development to Production:
A Studio’s Perspective
Robert Cook
Digital Domain

At Digital Domain research and development is a production driven activity.The research may be in
response to immediate production needs, or in anticipation of what might be desired in the future.We organ-
ize these two types of activities — immediate and anticipated needs — into a continuum of projects distin-
guished by their respective development cycle timeframes: immediate production needs tend toward short-
term development, while more speculative needs demand mid to long-term development.Thus we have
short-term, mid-term, and long-term projects, each requiring specific research and development methods par-
ticular to their development cycle.

Research and Development Workflow in Each Cycle

The research and development cycle in long, mid, and short-term projects differs slightly — yet those
slight differences dictate where the work gets done organizationally. Digital Domain divides the projects
between two departments: Software and Technical Directors. Generally, the Software Department handles the
long-term projects, while the Technical Directors handle the short-term projects. Mid-term projects are gen-
erally split between the two, based on the specific demands of the project.The primary distinguishing feature
between the two departments is that technical directors are attached to specific productions, while software
developers are show independent.This division of labor creates a more efficient and cost-effective research
and development pipeline for each of the three different types of projects.

Long-term Development Cycle

This development cycle tends to be much more recursive than the other two in that by the time the
project is complete, it is time to reconsider its features and functionality. It begins with a recognized need.
This may be in the form of a production or infrastructural demand, or a suggestion of what “might be possi-
ble” that would give Digital Domain a competitive edge in future production seasons. Extensive research is
completed by reading and considering academic research, journal publications, books, trade shows, SIG-
GRAPH publications and presentations, and one-on-one conversations with people in the field. A project pro-
posal is created, with an assessment of the current issues, and the proposed ideal solution. Of course, we all
know that in this business ideal is a luxury we rarely get to realize, so a scaled-back, real-world implementa-
tion plan is established.The proposal is considered in light of other projects, budget constraints, and viability.
We then establish a development timeline and budget, then pass that forward to management for approval.
Now don’t get me wrong here: I’m not saying our long-term project programmers spend a lot of time putting
together slick, well written documents! Sometimes they do, its true, but sometimes these proposals are sim-
ply discussed conversationally, then Doug (Software Creative Director) and myself (Software Production
Manager) put together the timeline and budget, and we go from there. I don’t expect thesis proposals, just
great ideas!

Next we go into development which consists of a loop of coding, implementation, user feedback, more
coding, further implementation, etc . . . Eventually, with many of these projects, such as Nuke, our in-house
compositing software, by the time we have reached our goals for the proposed project, technology and mar-
ket forces cause us to re-examine the project, and begin the process all over again from the top.
Mid-term Development Cycle

This cycle generally begins by someone hearing a talk at SIGGRAPH, or reading a journal article or book
and seeing how that research might lead to something useful within our production pipeline.These projects
generally become part of our internal software tools that augment either third party packages, or our own
proprietary packages. Such projects are forwarded, again, in either written or verbal form.We establish a
timeline and budget then discuss the merits of such a project with artists and CG supervisors. If the idea
seems to have relevance and merit (within the context of who we are and what we do), a test, proof-of-con-
cept is developed.That proof-of-concept is implemented sparingly, say as a small effect in a feature or com-
mercial. If the outcome is positive, further time is spent developing it into a full-blown application. If it does
not work, it goes back into the proof-of-concept stage, then is released again and tested. At some point we
either move forward with the idea, or the idea is shelved or scrapped altogether.

This cycle is the most difficult to manage and budget for because they often become victims of “feature
creep” or some other product is released which does the same thing. It is extremely important to define the
feature set before the project commences so that expectations can be managed and the project actually
ends! It is not often that we go down the development path on these sorts of projects and abandon the proj-
ect because it just didn’t work. Most of the time we stop such projects because a commercial product
becomes available which is better or more cost effective that the one we created.There is no discredit to the
developer here. For example, we might be creating a new plugin for Maya.We have good relationships with
Alias so we often know that they are working on a similar addition to Maya. However, we may be able to cre-
ate something more specific to our particular needs in a more timely manner, but without the full functionali-
ty that the Maya developers are creating. If the Maya release actually does what we need it to, then we gener-
ally quit development and/or support for our own internal plugin and use the Maya product. Other times the
release does not have a good match for our production pipeline and artists end up using both.

Short-term Development Cycle

This cycle almost always begins with a specific demand from a show in production. Generally it is the
artists which create the initial specification.This ranges from, “It would be really cool if we could . . .” to very
detailed parameters outlined by the artists and visual effects team.The problem is assigned to a Technical
Director, or sometimes Software production support person, and some preliminary research is completed.
There is a short development cycle here largely due to the production time constraints.The developer cre-
ates a quick, short-term implementation, usually testing it themselves. It will turn in this phase, between
design, implementation and testing, all by a single person, for as long as possible, each iteration improving the
desired effect. Once the developer feels they have something interesting to show, it will be shown to the vfx
team for discussion and input. At that point a determination is made as to whether more work should be
done, or if the problem is solved for the moment.

Many times these projects are then revisited when the developer is between productions. Full documen-
tation of the problem presented and the solution created is completed. Since these projects tend to be very
narrow in focus, solving a very specific problem within a particular production pipeline, many times the proj-
ect is then considered for continued development in order to broaden and generalize the scope of the proj-
ect.The project development might then continue with that one person, or be handed to the Software
department for further research, development and implementation.
Managing Research and Development in a Production Environment

The production environment, as I’m sure you all know, is quite different than most other working envi-
ronments, especially when it comes to research and development of software. Managing projects in this envi-
ronment demands a great deal of flexibility and agility on the part of the developer.There is usually not the
luxury of project managers, design architects, coders, and testers. Most of the time the programmer is all
those things at one time.They must be able to identify projects, create proposals, make scheduling estimates,
design the feature sets, do the coding, complete testing, compile, release, and bug fix.This puts an enormous
amount of responsibility on the programmer.

All of this responsibility is the result of a small staff with a big list.The development times are as short as
possible, even for the long-term projects. Being late on a delivery may mean not just pushing a release date,
but holding up a production involving millions of dollars. Research must be completed as quickly as possible. If
a problem comes up in production, then part of the budget for that project will be earmarked for research.
However, we also wanted a way to track how much time we spent developing new ideas.To that end, we cre-
ated a line item specifically in the Software annual budget for general research. Developers bill to this item
when researching new ideas that are net yet attached to a production. Since part of what is expected of the
developers is the infusion of new ideas and methods, we needed a way to account for that time, and establish
its importance.

Another difficulty for the developers is the quick implementation times, which usually means a very tight
development, testing and debugging cycle. On all projects, short, mid and long-term, the software is released
as soon as it is compiled. For very large projects we might have a small group of artists who do the initial
testing, but most of the time the software is released to the entire company. Now if it works great – no
problem! However, if it doesn’t work perfectly, or perhaps breaks something else people are using the
response will be immediate and usually made direct to the developer. No tech support or marketing buffers
here! So not only do developers have to be great coders, they have to be resilient to direct criticism from
the users.We have established a release policy in order to minimize these problems.The policy includes rules
for not releasing on Fridays or before holidays, and insists on the creation of user documentation.

Furthermore, all the developers are expected to multi-task projects. Some may be working on as many as
four major projects, as well as dealing with day-to-day bug fixing and routine updating — not to mention
maintaining code written by people who don’t even work there anymore.Thus they must be specialist and
generalists. All the developers we hire have an area of expertise, and we expect them to move us forward in
those areas. But we also throw a lot of other issues at them, with varying degrees of relevance to their area
of expertise.

In managing this it is important to be as encouraging and supportive as possible.The object is to create a

buffer zone for the developers so that they can progress with their projects with as few interruptions as pos-
sible.This demands that communication be as open and direct as possible between the managers and devel-
opers.The more questions from artists and management the managers can answer, the more time developers
can spend researching and developing. At Digital Domain we have two department heads: a Creative Director,
and a Production Manager.The Creative Director oversees the research and development of all projects, and
of course, develops his own projects as well.The Production Manager establishes timelines, tracks progress
and expenses, as well as attending all related management meetings company-wide.This two-person approach
has been quite successful for us.

For myself, as Production Manager, identifying the staff’s strengths is crucial to completing the projects
slated. Some are better at managing projects, others at coding; others are more interested in research. No
one has the luxury of doing the thing they do best, or enjoy the most, to the exclusion of everything else —
but finding those points and organizing projects around them creates a much more positive working environ-
ment. Some of the long-term projects may have several people on them, and by assigning tasks based on
known strengths and interests improves the success rate of the projects. Sometimes developers express an
interest in some area they had not been involved with previously.We work to try and give that person an
opportunity to pursue those interests. In asking the staff to be a flexible as possible, the management must be
flexible as well, and open to new ideas and interests.

In Conclusion

As a production company, we unfortunately do not have the luxury of having our own research group in
software.We must rely on the work of others within academia and elsewhere. Communication is the key.
Everything from formal publications to informal phone calls and e-mail keeps those lines open. Once in devel-
opment, that communication must continue, in order to push the limits of the ideas — without it we run the
risk of stagnating the imagination.
 Software Development and
Research at Digital Domain
Doug Roble
Digital Domain

Introduction
After I recieved my Ph.D. and had accepted a job at Digital Domain, I asked Wayne Carlson (Faculty at
Ohio State and an executive at Cranston-Csuri, a Visual Effects company that was big in the 1980s) what I
could expect. He said that I was going to spend most of my time programming and I’d get much faster at it.
He was completely right and he had a very good grasp of the type of work I’d be doing. In this talk I will
examine what it is like to work at a visual effects production company as a software engineer/researcher. It is
very different from grad school!

Description of Software Department

At Digital Domain, the software department is made up of 2 different kinds of developers:
Short Term - “Production Support” software developers work on projects that are required by cur-
rent productions.These tasks take somewhere between a day and 3 months. (That’s typically the
maximum lead time we have for a production.) The tasks are usually written as plugins to existing
packages, either purchased packages (Maya, Houdini) or in-house large packages (Nuke,Track).
Long/Mid Term - These developers work on projects that have a bit wider scope.We try to look at
trends in filmmaking, what the latest research is, or what we think we can do now and write tools
to accomplish these concepts.These tools are either written as major plug-ins to existing packages
or stand-alone tools with their own user interface and core libraries.

The kinds of projects come in different flavors too:

Graphics tools: Anything that deals with making the pictures on the screen.These range from 2D com-
positing tools, 3D animation tools, physical simulation, rendering tools.We usually pull a lot of
ideas from SIGGRAPH and what other companies are doing.
Infrastructure:Tools that make the artists lives easier.These include the distributed process manage-
ment tool, disk space management tools, and database/version control tools.
Systems: Low level systems tools that make the facility run faster.Tape tools, networking tools.This
overlaps with the Infrastructure category.
The software department is comprised of software developers with at least a Bachelor’s degree in
Computer Science. A strong background in mathematics is required as well. Most of the easy problems in
computer graphics are solved (to some extent). Mathematics is essential for pushing the boundaries of what
can be done.

The Difference between a Visual Effects Production Company and Grad School
By and large, production companies don’t have the time or money to fund pure research in computer
graphics. In general, the software developers are working problems that have a known (at least pretty well)
solution.We look at papers in journals and try to figure out a way to approach a problem that will, in all like-
lihood, work.
Then, we start designing a tool that is solid - it’s going to be released into our artist community and it
just doesn’t do to have the software crashing all the time.
We are lucky in that our users are in the same company (we rarely sell the code we write).This user
base knows the developer’s phone extension and will not hesitate to call if something goes wrong. It’s also,
for the most part, a very understanding user base. If something doesn’t quite work, yet there is a slightly
inconvenient work-around, the users will accept it to get the current job out the door.

More Differences
The software we develop has to work on all kinds of data, not just idealized data. It never fails that an
artist, after being given a new tool that the software engineer is particularly proud of, will call the engineer
up, complaining about how it doesn’t work on a particular instance of data.
This data, is usually something the engineer didn’t even consider and it’s the first thing that the artist
tried. If you develop a tool, it really has to be able to handle anything.
Research projects from academic institutes normally don’t require this. Most projects are proof of con-
cept and leave the details up to the software engineers who make the concept into a real tool.This is how it
should be, but it is something that we have to deal with.

Collaboration with Academics

Production companies do collaborate with Academic Institutes. At Digital Domain we’ve worked with Cal
Tech and UCLA.These have all been successful to one extent or another, but ended up being slightly different
that what was expected at the beginning.
Production companies and grad students have two diametrically opposed goals.Visual effects houses want
tools that their artists can use. Grad students want a proof of concept that they can write up into a thesis.
Production companies want tools that will help with the project at hand, grad students want to work on
their idea. If these don’t match exactly, the collaboration will drift apart.The projects start out with all parties
being excited and willing to work, then as school work intercedes with the grad students and productions get
in the way of the software engineers, communication gets harder to maintain.The good intentions are still
there, but the time to act on them isn’t.
At Digital Domain, we realize this.We try to help out researchers as we can in the hopes that the work
they are doing will have an impact on the technology we are developing, but if it doesn’t that’s OK.We have
provided data (when all the legal issues are straightened out) without much expectation of anything in return.
This builds relationships and is easy for both parties. Collaboration can work, but there has to be a concerted
effort throughout the project from both parties.

Industry Secrets
Companies keep things secret so that they have an advantage over other companies.The bidding process
for films is very competitive and if one company has the ability to do an effect that another company can’t
(or can do an effect cheaper), then that company has a leg up on getting a film.
This flies in the face of traditional research behavior.We understand this, but really have to shrug our
shoulders and say, “Hey, that’s the way it is.”
As mentioned above, Digital Domain tries to help by collaborating with researchers in ways that it can.

Buy vs. Make and Getting Scooped

One of the issues that we are constantly dealing with is whether to spend the time and effort and money
to develop a new tool or to purchase a solution from a external vendor. Some visual effects houses use only
their in-house developed software while others have only a small software development team and use pur-
chased software.
Digital Domain is somewhere in between the two extremes.We have decided that it is more cost effec-
tive to buy our Animation and Rendering solutions, but have written quite a few large scale packages.We have
also purchased a few copies of smaller programs that solve only one kind of problem.
The decision to buy the solution or make the solution is a complicated one. Making the solution gives you
the ultimate control, but it also the ultimate risk. Making software results in a site license for the tool. Buying
the software means that you have something that already works. Making large-scale projects means that a
developer will probably always have to devote some time on maintenance and addition features each year.
Buying software means that we’ll probably have to wait for the software vendors release schedule if we need
bug fixes or new features. Making the software always takes a long time (sure, the core engine might be ready
in a couple of weeks, but adding all the user interface, undo, and everything else that goes into a released
package takes time!). Buying the software can be very expensive.
Even if there isn’t a something available to purchase that solves the problem, there is always the risk of
getting scooped.We don’t have the resources to devote a large team to each software project. One person
working on a project might get beaten to a solution by a large team of developers at Alias/Wavefront (or
whatever company) who are working on the same thing. If this happens, we look at what the external devel-
oper has to offer and compare it to the current state of the internal project. It may make more sense to buy
the external solution and let our solution go...This isn’t too big a deal, there are always interesting projects
for our internal developers to work on.

Transfer of Knowledge
SIGGRAPH and other conferences and journals are very important to Visual effects companies. It is
through these things that we can see what other parts of the industry are doing and get ideas for new proj-
ects.
As an example of this, we will discuss a project that has been developed in the long term group of the
software department at Digital Domain.

3D Tracking
When I got to Digital Domain, one of the first things I did was take my Ph.D. research and make a robust,
working tool out of it. It analyzed images and figured out where the camera was when the image was pho-
tographed.This, in itself, was a project that had a long history in computer vision.
My tool added sub-pixel precision techniques, and a lot of control to help the tracking artist figure out
the information for the hardest of shots. In computer vision, the test images that are used are usually very
controlled.The lighting doesn’t change and the thing that is photographed certainly doesn’t catch fire!
Unfortunately, in filmmaking, that kind of thing happens all the time. Modifications to typical computer vision
techniques needed to be made.
The tool has been one of my successes and I have worked on it over the years, adding new features,
improving the work-flow, removing some really bone-headed design decisions...

Optical Flow
Track has a very nice pattern tracker built into the program. If you indicate a section of an image, it will
follow that section as it moves through the sequence.This makes a lot of the tracking fairly automatic.
Optical flow is a computer vision technique that’s been around for a while. Basically, in optical flow, one
tries to figure out where every pixel in one image moves to in the next image. If it works, it’s awesome, you
can figure out all sorts of information about the contents of the image.
I had been aware of this technique in computer vision and noodled around with it, but had little success.
Optical flow is not really suited to real world images, most of the papers on it will assume some constraint
on the camera to make the problem more tractable.
Within the last 4 or 5 years, these limitations have started to be addressed by the computer vision
research community. In fact, Richard Szeliski and James Coughlan have written some lovely papers on a tech-
nique that increased the robustness of optical flow and sped it up as well!
In fact, I had one after-a-SIGGRAPH-panel conversation with Richard Szeliski (that he probably doesn’t
remember) where we discussed some of his techniques.This convinced me that further effort would be
worth it. I already knew how I could use the technique in my tracking program and now that it was better
suited for real world images, I pushed for the project at Digital Domain.
Another point in my favor was that an external company was already working on an Optical Flow prod-
uct.This company was already showing off their version 2.0 of their product and it proved that really interest-
ing things could be done with the concepts.
We decided to develop the solution in-house in that it wouldn’t be too big of a project, having a site-
license was a real plus, there were limitations with the external solution (one couldn’t access some of the
results of the program for other uses and it was expensive) and finally, I had some ideas to make it even bet-
ter.
In the development of the project, I collected many papers on the topic and talked with some of my
friends at Universities. After developing a framework for experimentation, I implemented a couple of different
optical flow techniques and finally merged a couple for my final solution. I will present some results at the
course presentation.

Conclusion
Working at a Visual Effects Production Facility is quite different than working at a University.The financial
pressures are different and the type of work that is expected is different.
We appreciate the work that “The Ivory Tower” does to no end and use the concepts developed there all
the time. Our main challenge is modifying the algorithms so that they perform robustly in a real-world envi-
ronment.
 Photomodeling and Hair Combing for AI
Steve Sullivan and Tony Hudson
Industrial Light and Magic

ILM R&D worked closely with several production groups to develop photomodeling and hair combing
tools for work on AI (to be released Summer 2001).

From Digital Reference to Screen

Steve Sullivan
Industrial Light and Magic

The first part of the presentation describes the use of digital reference and plate photography to reconstruct geo-
metrically accurate models of set pieces, which provide the basis for 3d camera and object matchmoves.

R&D at ILM
-28 people, 7 teams
-focused on applications
-long-term solutions
-also: Systems R&D and R&D TDs

From storyboard to software

-Discussion between Production and R&D about key technology
-Determine if R&D resources are required:
--does development fit production schedule
--is a large-scale solution needed
--is the outcome reasonably certain
--is it a high priority, or needed across shows

AI FC37 - head augmentation

Head augmentation/replacement
-matchmove camera
-photomodel head
-matchmove head
-photomodel body
-manually model neck
-matchmove neck/body

Uses: 2d tracking, structure solvers, nonlinear optimization

AI Matchmove/Photomodeling
2d tracking
Follow a 2d pattern
 --relatively mature
--many commercial solutions
--still plenty of room for improvement

Tracking Reference:
Three-Dimensional Computer Vision, O. Faugeras, MIT Press, 1993

Matchmove/Photomodel
compute camera/object position
--Used to be manual (restricted camera moves)
--Anything goes now - required for most shots
--Relatively mature (commercial solutions)
--Still time-consuming, still an art
--Structure solvers very handy

Pose estimation reference:

"View Correlation," Graphics Gems II, J. Arvo (ed.), Academic Press, 1991

Structure solver reference:

Three-Dimensional Computer Vision, O. Faugeras, MIT Press, 1993

Nonlinear optimization
--Fundamentally hard
--Generally useful
--Initial conditions important (user input)

Optimization reference:
Linear and Nonlinear Programming, D. Luenberger, Addison Wesley, 1984

AI FC37 - plate photography

-free-form camera move
-dark plate (less detail)
-need camera + face/body
-tight head match necessary
-need relationship between face/body

AI FC37 - model from photos

-Start with reference Photos
-Make image correspondences
-Use structure solver to create 3d model
-Edit and build upon model

AI FC37 - camera matchmove

-Track 2D points
-Solve for 3d points + camera move
AI FC37 - head matchmove
-Track 2D points
-Use model mesh to track face/head
-Combine with camera matchmove

AI FC37 - body model

-Track 2d points
-Use camera move + tracks to compute body shape

AI FC37 - neck model

-neck machinery modeled manually
-needs to fit between head and body

AI FC37 - body/neck matchmove

-Use existing head + camera motion
-Solver finds neck parameters matching body to head

AI FC37 - conclusion
-Combination of techniques
-Mix 'n match manual with automatic
-Always driven by a user
 Developing tools to model and animate hair
Tony Hudson
Industrial Light and Magic

The second part of this talk discusses a suite of tools developed for hair modeling, animation, and editing.
They permit accurate, interactive placement of guide hairs, with visualization tools for previewing the behav-
ior of interpolated hairs.

Even though digital hair has been around for years the ways we use the technology has evolved and as a
result tools are now being develop to where “styling” digital hair has become possible and just in the nick of
time.

Past uses of hair have mostly been limited to overall "furry" characters, where styling was accomplished
by a limited set of paint tools for controlling length and distribution over the topology. However, on Artificial
Intelligence or "A.I.", it quickly became obvious that a technique for "styling" digital hair would be necessary
to achieve the look for characters, which have specific haircuts, and that these hairstyles would need to be
able to receive dynamic forces.This eventually led to the development of hair creation and styling tools. This
give the modelers a very flexible interface in which hair could be modeled with lo-resoulution splines and a
quick "preview" function, which allows the artist to interact in a meaningful way with the hair and see exactly
what he’s doing.

ILM's software R & D department ultimately provided us with tools to place and plant hairs along a sur-
face, to "style" the hair while keeping the root planted, and to clone sculpted hairs to other locations along
the topology.We can also import hair distribution maps, so that we can see how the hair will ultimately be
rendered.

In addition to styling, the R&D group has also provided us with new tools that allow the animation of hair.
Our "Caricature" animation tool now gives us the ability to add dynamically animated hair and to tweak the
parameters to create specific use. Also possible is a form of "digital hairspray", to keep those nasty locks of
hair in place.