Latest

Technology

The Linux revolution in cinema: from Titanic to the oceans of Avatar

From the sinking of the Titanic to the oceans of Avatar: 30 years of the Linux revolution in cinema's visual effects

The Linux revolution in cinema: from Titanic to the oceans of Avatar
The Linux revolution in cinema: from Titanic to the oceans of Avatar

When we think about the birth of contemporary cinema's great visual effects, the mind usually travels to Silicon Valley, to the labs of hugely expensive proprietary software or to computers bearing fruit logos. Yet the true hero behind the biggest blockbusters in cinema history dresses in black, has a beak and is called Tux.

Today, 100% of Hollywood's major visual effects (VFX) and animation films are rendered and processed on Linux. The open source operating system holds absolute dominance over the servers, render farms and artists' workstations at the world's most prestigious studios, such as Industrial Light & Magic (ILM), Wētā FX and Digital Domain.

This is the story of a mass migration that changed cinema forever.

Goodbye to barriers: install Microsoft Office on Linux using Flatpak

Titanic (1997) and the end of SGI's monopoly

In the mid-1990s, making digital cinema meant going through one company no matter what: Silicon Graphics Inc. (SGI). Its workstations ran IRIX (a variant of UNIX, the "parent" of Linux, MacOS and BSD) and cost tens of thousands of dollars per unit. They were powerful machines, but prohibitively expensive and hard to scale when time was short.

A sample SGI Irix workstation, at the time the best for VFX
A sample SGI Irix workstation, at the time the best for VFX

But when James Cameron set out to film Titanic in 1997, the visual effects studio Digital Domain hit a wall: simulating the digital water, the thousands of virtual passengers floating and the colossal sinking of the ship required computing power that far exceeded its budget for SGI hardware.

The technical revolution

Faced with this, Digital Domain decided to take a historic risk. They bought 160 computers with 64-bit DEC Alpha processors (the fastest on the market for floating-point work at the time) and, instead of installing Digital's costly commercial UNIX, they installed a version of Red Hat Linux 4.1.

To bring to life all that world we see in the film, they used Renderworld (a pioneering ocean simulation software developed by Areté) and the early internal versions of NUKE, their own digital image compositing tool. As an extra curiosity, Renderworld is the same software used for the water effects we see in Contact (the pod scene), Waterworld, 20,000 Leagues Under the Sea, The Fifth Element and Sphere.

This software ran two to three times faster on the Linux Alpha farm than on the expensive SGI workstations. Thus Linux proved it could handle more than 400,000 frames working 24 hours a day, 7 days a week, without complaining about anything, for the peace of mind of Digital Domain and James Cameron.

And with that, the myth that Linux was just a "toy for enthusiasts" of free software began to crumble. Hollywood discovered that open source was faster, infinitely cheaper and modifiable according to the needs of the shot.

The Lord of the Rings (2001-2003) and artificial minds

If Titanic proved that Linux was good for simulating static and fluid environments, Peter Jackson's The Lord of the Rings trilogy, developed by the New Zealand studio Weta Digital (now Wētā FX, and by far one of my favorite FX studios), proved that Linux could simulate life and the chaos of war.

The challenge for The Fellowship of the Ring (2001) and The Two Towers (2002) was colossal: recreating battles with more than 200,000 digital warriors (orcs, elves, humans) where each one was not a simple animated clone, but an "agent" making its own combat decisions.

Not to mention Gollum, a fully digital character whose level of realism is so high that only another piece of work by Weta can rival it, even with today's technology.

A life-size Gollum figure; the figure and the film are almost identical, a demonstration of the power of VFX on Linux
A life-size Gollum figure; the figure and the film are almost identical, a demonstration of the power of VFX on Linux

A challenge without equal

To achieve this, Weta designed an immense server farm based entirely on Linux (using Red Hat distributions and, later in spin-off productions, a mass migration to Ubuntu-based architectures) connected to thousands of processing cores.

But in all of this, the crown jewel was MASSIVE (Multiple Agent Simulation System in Virtual Environment), a software created specifically by Stephen Regelous for these films. MASSIVE gave each virtual character a "brain" (a fuzzy logic tree): they could see the terrain, hear the enemies and decide whether to attack, block or flee. To paint and texture the frames, they relied on Pixar's RenderMan running on command-line Linux.

Amazon and Coinbase let AI agents pay to access content on the Internet

The pipeline (workflow) was automated through Python and Perl scripts native to Linux environments. Programmers modified the kernel and network driver code in real time to avoid bottlenecks when the hundreds of terabytes of textures saturated the storage networks (SAN).

An interesting fact: the legendary battle sequence at Helm's Deep or the Pelennor Fields required the Linux farm to process images for entire weeks. If just one of these computers had had to render the trilogy on its own, it would have taken decades, and if it had failed mid-batch, weeks or months of work would have been lost. Errors were not allowed.

Avatar: The Way of Water (2022) and massive cloud processing

Jumping to the modern era, we return to James Cameron and Wētā FX with Avatar: The Way of Water (now you know why Weta is my favorite FX studio). If the first installment in 2009 already used tens of thousands of Linux cores, for the sequel the computing needs grew exponentially due to the photorealistic rendering of water and underwater motion capture.

For this blockbuster, local storage was no longer enough. Wētā FX combined its local data center (made up of thousands of HP blade servers running Linux systems) with a hybrid cloud infrastructure through Amazon Web Services (AWS), which logically operates under Linux containers and instances.

AI finds flaws in Windows and triggers a clash with Microsoft

Raising the bar

To make all that work a reality, hyper-optimized Linux versions of SideFX Houdini (for destructive and fluid simulation), Autodesk Maya and Weta's proprietary rendering engine, Manuka, were used, the latter designed specifically to calculate in a physically exact way how light interacts with the skin, water and plants of Pandora.

Bazzite, the great Linux distro for those who love PC gaming

This new Avatar installment required the absurd amount of 1.248 billion render hours. That is equivalent to about 142,000 years of processing for a single modern computer. At the peak of production, the Linux-based infrastructure handled between 200 and 250 Terabytes of data A DAY, accumulating a total of 140 Petabytes of storage.

All of that using Linux; on Windows, handling that level of data is impossible.

Academy Software Foundation (ASWF), the legacy for the industry

The romance between Hollywood and Linux is so deep that in 2018 the Academy of Motion Picture Arts and Sciences itself (the body in charge of handing out the Oscars) teamed up with the Linux Foundation to create the Academy Software Foundation (ASWF).

The goal of the ASWF is to standardize and protect cinema's open source ecosystem, ensuring that critical open source libraries, such as OpenVDB (for simulating clouds and smoke) or OpenColorIO (for color management and an essential part of software such as mpv), continue to be developed freely and collaboratively on Linux systems.

Hollywood understood almost thirty years ago that the magic of cinema does not belong to the patents of a single technology corporation, but to the freedom and flexibility of open source. Every time you see dragons fly, planets explode or oceans swallow ships on the big screen, there is a Linux kernel holding up the illusion.

José Maldonado

Author

José Maldonado

Activist and blogger on technology, free software and blockchain.

The latest
From Huang to Botín, from Zara to Byron: corporate responsibility in global capitalism

Keep reading