It’s no secret that visual effects are a crucial part of filmmaking. Yet, managing hundreds of terabytes of files and data is a daunting challenge. For Weta Digital, a five-time Academy Award-winning digital effects company based in Wellington, New Zealand, technology increasingly drives results.
“It’s critical for different departments handling animation, modeling, picturing, shading, lighting and more to be tightly coordinated and have access to files,” states Matt Cunningham, head of systems for Weta Digital.
The 1,200-person company—which has produced stunning visual effects for some of the highest grossing movies of the past two decades, including Avatar, The Lord of the Rings Trilogy, Planet of the Apes, King Kong, The Hunger Games: Mockingjay, Part 2, and The Hobbit—needed to take its IT and storage infrastructure to a new level.
“We are rendering 24/7, and existing systems had reached their capacity,” Cunningham says. “In order to keep teams on schedule, we needed to scale up network and storage performance. We realized that we had to find a more efficient and cost-effective way to achieve top-tier performance.”
In some cases, tens of millions of files exist for a particular scene, and an element might appear in hundreds of shots. The data, which is generated in high-end workstations, must be available for every rendering, and this translates into a need for enormous storage capacity and flexibility.
In some cases, data elements are broken into smaller pieces. “This means that a file that’s a few gigabytes may be broken up into 32K or 64K chunks, which is a really small piece of data,” Cunningham explains. “An 8-megabyte file might require hundreds of operations. It must come together in eight pieces rather than 800 pieces.”
Integrating Existing and Advanced Storage Technology
About two years ago, Weta Digital had begun to encounter performance bottlenecks, particularly in storage systems. The company relies on a highly tiered approach to process and manage about a quarter of a petabyte of data every night.
“We separate data into different classes depending on what use it has,” Cunningham says. “We have high-performance systems, lower-performance systems and older systems.” The company wanted to keep its existing network attached storage (NAS) infrastructure intact—partly for financial reasons, but also to add more advanced technology. “We wanted to be able to stitch together all the systems,” he adds.
Weta Digital went live with a single Oracle ZFS appliance in September 2014. It optimizes performance for Oracle databases and optimizes data lifecycle management through highly efficient internal caching and internal acceleration.
Since then, the company has scaled up to a total of three systems. This has increased total storage capacity by a factor of about five times. The benefits have been substantial, Cunningham reports.
“We can run at maximum capacity and maximum throughout,” he explains. “We have achieved a big bump in capacity while seeing better overall performance.” In fact, he says the storage framework has produced about a 2x gain in speed across the board, including latency, throughput and connectivity.
Says Cunningham: “This storage environment has produced groundbreaking change. It has allowed us to consider new ways to think about storage and our network, and it has produced enormous gains.”