Several people asked, in the Davy Jones appreciation forum, about how ILM rendered Davy and his digital crew of half-human, crustacean-covered Pirates. We have some answers, direct from Christophe Hery, Lead Research & Development Engineer at Industrial Light & Magic.
Hery joined ILM as a senior technical director (TD) in 1993 after working for the previous four years in Paris at BUF and Thompson Digital Images and as director of R&D for Label 35, a Parisian cartoon studio.
At ILM, Hery was an Associate Visual Effects Supervisor on Spawn, Senior Technical Director for ‘Sleepy Hollow,’ and CG Supervisor for ‘Star Wars: Episode II – Attack of the Clones,’ ‘Jurassic Park III’ and other films. More recently, as an R&D lead, he has been working on techniques for lighting and rendering skin.
In 2003, Hery received a Technical Achievement Award for the development of practical methods for rendering skin and other translucent materials using subsurface scattering techniques. He’s currently developing tools and processes for a project having to do with creating talking digital clones.
Meanwhile, the 2006 Oscar race is gathering steam. Candidates for the visual effects bakeoff were announced on December 15 and 'Pirates of the Caribbean: Dead Man’s Chest' featured prominently. Disney has released the DVD so you can stop frame through the effects now. And to kick off its Pirates campaign, ILM has created an all-about Pirates web site at www.ilm.com/theshow/. Check it out.
I hear that you developed new rendering techniques for Pirates. Why?
We used ZBrush. It’s amazing. You can decide as you model what to put in the vertices, displacement or geometry, and decide whether to use geometry or texture maps for details. But, the resulting geometry count was really heavy, with the displacement, bump maps, and textures; we had a lot of CV’s. Even though these characters were sometimes only three pixels high in screen space, they’re hero characters and we didn’t want to lose detail. But, sometimes we had more vertices, more geometry than we had pixels to render.
That would definitely slow things down.
It meant that occlusion passes, anything that is ray traced, or even anything that used subsurface scattering because it relies on point-based sampling, was much slower. We’d get a summation for all the geometry in one pixel at the end, but it took a lot of time for not a lot of result. It is a waste to have all those rays for only one pixel. There is some automatic LOD (level of detail) for ray tracing built into the RenderMan engine, but not enough and it never removes the geometry. It’s up to the modeler to do that for you.
You were using RenderMan?
Yes, but it would have been as slow with other ray tracers.
What did you do?
After discussing this with Pixar, they implemented a new technique inside RenderMan derived from a technique in a paper in NVIDIA’s GPU Computer Gems 2 and gave us the DSO to try before adding it to a new release. When I integrated it, renders that were taking 12 or 13 hours went down by a factor of three to four. We got the same or better results in a quarter of the time.
How does it work?
The idea is that usually when you do occlusion passes, you do a distribution of rays around a normal. You intersect all the geometry, sum it up, and the average of the ray tracing becomes the occlusion value; the average from all those rays becomes your occlusion for that point.
You end up sending a lot of rays to avoid missing geometry details. But with this method, we pre-export the geometry in terms of the point cloud, just like we do with scattering, with the density we want. Then, we never do ray tracing. The points represent the geometry internally.
Imagine that instead of having a wireframe with B-splines, you now get a little disk at each intersection. It’s a flat disk oriented at a tangent of the surface in the direction of the normal.
Internally, each of those disks represents some sort of direction. You could call that an angle. Now you can do the ray tracing analytically because you can go from this point and see if it’s intersecting another disk. You know how to do a ray intersection for a disk faster than for arbitrary geometry.
So, you organize the disks in memory and do a fast fake ray trace through them. You aren’t really ray tracing; it’s all analytical. It’s an approximation. It has a somewhat different feel than the real solution, but the advantage is that there’s no noise at all.
So instead of doing occlusion with ray tracing by sending random rays around a normal, here you’re sending fake rays deterministically around the point. From frame to frame they will not change, so, it’s faster with no noise.
It led me to rewrite the way I write subsurface scattering.
In what way?
Before, for Lemony Snicket, I was dumping two point clouds – high res and low res – and scattering through the two. In Pirates, the geometry was so dense that the low resolution was as dense as the high resolution. So, I dumped only one and internally derived a hierarchical structure for the other.
Inside the documentation for RenderMan 13, you can read about the extension for scattering. It’s the same approach – using point-based techniques.
What about Global Illumination?
Another advantage with doing ambient occlusion through this point base approach was that we got one bounce global illumination for free. Once you dump the point cloud with all those disks, you store the direct illumination they would receive from outside light on the disks as well. So once you do the fake ray tracing, you don’t have to re-execute the bounce.
Are there any disadvantages?
Because it’s a sparser representation, you might get low-frequency blotchiness and see things that seem to be swimming, but we don’t get the fine grain we usually get from ray tracing.
It was a miracle for Pirates. The geometry was so heavy even inside Zeno.
Last question. Did you write many new RenderMan shaders for Pirates?
We were using subdivision models so it’s more difficult to assign shaders and specific shading properties locally. In the past, eyes, mouth, ears were broken out so we could assign specific shading properties.
But for Pirates, we had one complex model, one mesh, with all those details. Yet, they still had to give clothing, seashells, barnacles, and skin tentacles different materials.
Pat Myers made an ubershader – multiplied my shader by three times, I think – with many properties inside. You could trigger from one to the other based on the maps so we could refine locally that way. One area would react like skin, another like clothing, another like barnacles.
Even with our old pipeline, the way we were modeling before, it would have been impossible because some of the geometry was from displacement, so it didn’t exist prior to rendering and there would have been no way to assign materials to CVs. We had to do it through a map like that.
I hear that you developed new rendering techniques for Pirates. Why?
We used ZBrush. It’s amazing. You can decide as you model what to put in the vertices, displacement or geometry, and decide whether to use geometry or texture maps for details. But, the resulting geometry count was really heavy, with the displacement, bump maps, and textures; we had a lot of CV’s. Even though these characters were sometimes only three pixels high in screen space, they’re hero characters and we didn’t want to lose detail. But, sometimes we had more vertices, more geometry than we had pixels to render.
