The Beginning of the end for the Uncanny Valley.
Digital Domain reveals its part in Holy Grail of VFX.
CGSociety :: Production Focus
6 January 2009, by Renee Dunlop
The Curious Case of Benjamin Button has one of the strangest character arcs, and involved a most unusual production pipeline and timeline. The Benjamin Button project first came to Digital Domain back in the mid 1990’s when it belonged to Ron Howard, but the technology to bring the story to life was simply not available. Then in 2002, Director David Fincher got involved. Digital Domain (DD) had worked with Fincher for fifteen years, and considers him, according to Button’s VFX Executive Producer Ed Ulbrich, “the patron saint of Digital Domain, one of our most loyal clients. He is a rare director in that he is absolutely fearless, perfectly comfortable pioneering into areas where no one has gone before.”
THE HISTORY OF THE AGE
With a new screenplay by Eric Roth in hand and new technology begging for its moment of truth, the task of how to proceed was back on the table. Fincher didn’t want to use different sized actors wearing makeup, concerned that each time the role was handed off to a new actor the audience would lose the emotional thread and sympathy for the character. The primary goal to Fincher was that the character Benjamin carry the likeness and performance of actor Brad Pitt throughout the film. Various approaches were considered and eliminated. Though Benjamin begins his life as a diminutive old man, he wouldn’t be convincing by simply shrinking Pitt to the appropriate size. Fincher had the idea of casting petite actors to portray the childhood bodies at various stages of Benjamin’s life, but there was no way to photograph Pitt in makeup and stitch his head onto an actor’s body while avoiding lighting, continuity, and tracking issues. Gesticulating while talking, a smile pulling tendons in the neck and chest, the timing of the breathing, were all interconnected. It became clear that DD had to do a photorealistic and perfectly animated CG head, the holy grail of CG, and attach it seamlessly to various sized actors.
Ed Ulbrich. VFX Executive Producer.
By 2004 the film was getting traction from the studios and DD was chewing on the details. “We studied the latest advancements in facial animation,” explained Ulbrich. “One large budget CG movie in production at the time was using state of the art marker based capture. It was stylized, but edging towards photoreal. We looked at some of the best work in that and other films and were very clear that, if our worst shot looked as good as the best shot in that film, we would fail. It’s not a critique of that film, but a comment of the state of the art of using marker based capture. You can put 500 markers on the face, but you still don’t get what happens between those markers, and that was what was critically important to us.” Keyframing was not an option either, since Brad Pitt’s behavioral and gestural performance would be filtered through an animator’s interpretation, not to mention continuity issues working across a team.
Warner Brothers and Paramount commissioned DD to do a one shot test. In five weeks DD created Benjamin, worked out the tracking issues and put a CG head on a body. Everyone was happy, the film was greenlit. But now, DD had to make a character speak, hold up in a close-up, and handle several hundred shots of a character that would make the audience laugh and cry and carry the first 52 minutes of the film. Though the test had worked, it was not sufficient for the entire project. That method was limited to how Pitt looks today, but DD had to model and animate Pitt over a series of decades.
It must be said that several other VFX houses did a lot of other work for the Benjamin Button feature. Extensive Matte Painting work was tackled by the team at Matte World Digital; Greg Strause and his crew at Hydraulx did the Russian snow and the matte painting of scenes in Paris, New York and Russia, as well as head replacements for Cate Blanchett's dance performances and the CG elements for the baby Button. Asylum VFX handled the many scenes in the tugboat, and Lola VFX also worked on the 'youthening' effects, with supervision by Edson Williams.
PIECING IT TOGETHER
Enter Rick Baker. Starting with life casts of Pitt, Baker sculpted three different lifelike maquettes, Benjamin at 60, 70, and 80 years of age. Baker also did life casts from the shoulders up of the various actors playing Benjamin’s body, then grafted on the various heads, resulting in three different busts of Pitt at different ages. Those sculptures were then scanned into a mesh to be retargeted with Pitts acting.
DD’s Character Supervisor Steve Preeg was aware of the work of psychologist Dr. Paul Ekman who researched human emotion and reaction to stimuli. Ekman believed that the face had a series of basic poses he could catalogue, thus creating FACS (Facial Action Coding System), a library of everything the human face can do. Ekman contended our facial muscles are more or less hard wired with root expressions that are universal. It’s a theory used by animators for years, but is usually used by referencing the shapes for keyframing emotion.
By utilizing this theory along with Mova Contour, DD volumetrically captured Pitts’ face doing roughly 120 expressions and applied the information to various 3D models, resulting in literally thousands of models of Pitt. At the same time, they scanned the three maquettes of Benjamin’s head. This type of volumetric capture, where a head is scanned in real time in three dimensions at 24 FPS, results in millions of polygons of surface capture without the dead zones that result between markers. The result was the old Benjamin lined up perfectly to the Mova Contour scan of Pitt’s facial poses. By retargeting the expressions onto the scanned Benjamin heads, DD effectively had the three old Benjamin CG characters performing the FACS with Brad Pitt’s full range of emotions and expressions.
DD then cut up the mesh heads into sub shapes of different poses, creating a relational database of thousands of micro expressions or components of expressions entirely based on the volumetric capture of Pitt. By using this process, DD avoided the issue of human interpretation of how the face would change according to an animator, instead staying completely true to Pitt’s acting. This became the basis of what DD referred to as “emotion capture”.
The next task was bringing this model to life, creating software tools that allowed DD, on a frame by frame basis, to compare the live action performance of Pitt playing Benjamin to the database of micro expressions, then reconstruct a CG model that matched, and seamlessly attach it to the various live action actors in the film. A piece of cake, right? Maybe not. And there is more to go.
ACTING THE PERFECT PART
VFX Supervisor Eric Barba put Tracking Supervisor Marco Maldonado and Jessie James Chisholm, the Integration Lead, on set for all the VFX work over a seven month period. “DD would parse out whatever details the other venders might need. We put together a plan for the elaborate system of tracking the heads in. Traditional tracking is pretty great for most things in a scene because, if the track is not quite right, normally you do 2D reconciliation to make that little bit seamless. That is what this industry is used to. But trying to track the spine and neck together with the CG head is incredibly challenging. When the body and camera is moving around, the lens distortion, that traditional tracking doesn’t give the view depth information to get the accuracy necessary.”
