Flarg -- Daniel Wexler's Corner

Renderfarm Statistics

Saturday, January 27, 2001

I worked at PDI during Shrek, but left after it was finished.

PDI/DreamWorks uses this hardware 24/7 for our next feature length film, Shrek and all of our commercial and film effects work. These statistics were compiled in January, 2001.

The network consists of two (soon four) Foundry BigIron core routers with Foundry layer-2 edge switches. Gigabit Ethernet is used for all interswitch links and the network interfaces for all fileservers. Desktop and renderfarm machines are switched 100Mb.

Processors

The renderfarm has dual processor machines running two frames in parallel. The renderer is coarsely multithreaded and uses both CPUs if available. Our deformation and effects tools are single threaded so we use a single processor/frame. When necessary, we run a single frame on a dual processor machine to double the memory -- trading a CPU for memory.

Product
CPUs
OS
Description
SGI Origin200 406 IRIX 6.5 Dual R10000 180MHz, 512MB SGI RAM, 3U + 1 rackmount, 9GB HDD
PC Vendor #1 292 Linux Dual PIII 450MHz, 1GB PC-100 SDRAM, 2U rackmount, 39GB HDD
SGI 1200 324 Linux Dual PIII 800MHz, 2GB PC-133 SDRAM, 2U rackmount, 39GB HDD
PC Vendor #2 270 Linux Dual PIII 800MHz, 2GB PC-133 SDRAM, 1U rackmount 39GB HDD
SGI O2 190 IRIX 6.5 Single processor O2s, R10000 , 256-512MB SGI RAM, 9GB HDD
Total
1482 cpus
836 boxes,
443 dual processor Linux boxes, 203 dual O200s, and 190 O2s.

 

File Servers

Almost all the final frames and most shots will be online by the end of a production. We use a proprietary binary revision control system to create shadow trees from a master to synchronize departments and the renderfarm.

Name
TB
Description
Frames and Shots 2.75 2x8-processor cray-linked Origin 2000s, 26x100MB + 2GB ethernet
Master Trees 1 Clustered Network Appliance F840, WAFL/RAID4
Data Servers 2 Clustered Network Appliance F840, WAFL/RAID4
Shadow Trees 0.5 2 x clustered Network Appliance F740, WAFL/RAID4
Home Directories 0.2 Clustered Network Appliance F760, WAFL/RAID4
Binaries 0.05 Clustered Network Appliance F760, WAFL/RAID4
Total
6.5 TB


 

Random Information

Projections
  • When we hit 2000 CPUs we will have 1,000,000x the compute power of the first computer at PDI in 1980.
  • We average about 2 Gigabytes/frame and about one frame/hour, so with 2000 CPUs we will go through about 4 Terrabytes/hour. This requires a server and network capable of 1.2 Gigabytes/second.
  • For twenty years, the overall average frame time has consistently been half an hour.
Statistics
  • 225,000 rendering jobs/week
  • 500,000 pixels/second and 500,000 triangles/second if you include all 1000 CPUs
  • GeForce2 Ultra gets 1B pixels/second and 32M triangles/second on a single CPU
  • 1 hour/frame average rendering time
  • 12 min/layer rasterization time, 10 min/layer shading time
  • 3 layers/frame average
  • 4 min/frame average for motion renders
  • 1000:1 ratio of input to output data
  • 20M polygons/frame, 12M polygons/layer
  • 2 GB/frame, 1 GB/layer textures
  • 15 shadowing lights/layer, 55 MB depth maps/layer
  • 200 MB texture files/layer, 100 MB textures loaded into ram/layer
Linux Tools
  • PDI has been using UNIX for twenty years and Linux since September 1999.
  • We switch to Linux on the desktop in January 2001 (from IRIX).
  • We use proprietary tools for lighting, rendering and motion and a combination of Maya and inhouse tools for modeling and effects.
  • Custom built kernel to modify the amount of space allocated to each process via sbrk() for really large processes.
  • Custom playback tools that use the X11 video extensions and shared memory to get a full 24 fps playback from both uncompressed and motion jpeg compressed files.
  • We have our own color correction tools which work with the NVidia and HP graphics cards to match our film prints.
  • We are switching from SGI R10000 O2s with 256MB to Dual PIII 800MHz with 2GB of PC-133 SDRAM in January. That's roughly a 12x CPU performance increase and a 8x memory increase for our next film, Tusker.
  • We have about 4 million lines of C code in our libraries and commands.
  • We don't use C++. We do use C, a little Python, Perl and CSH.
  • Our applications have clean and efficient user interfaces.
  • We have our own proprietary lighting tool and deferred rendering system.
  • Our user interface software is completely X11/OpenGL based and only relies on features used in Quake.