Everything that was once considered to be integral to film production – sets, make-up, costumes, cameras, etc. – is called “physical production” in the digital effects industry. Many blockbuster films, including Avatar (2009), the highest grossing film in history, are now produced without physical production. These films are the result of what is called “virtual production,” and this development is transforming notions of creativity and creative labour in the film industry.

Virtual production techniques are supposed to reduce costs and make film production more efficient. But at the moment, virtual production means adding more people and technology (and a wider diversity of technologies and specialists who know how to use them). Virtual production expands the scale of film production, and increases the complexity of the industrial apparatus that is commercial filmmaking.

Virtual production is thus an unlikely place for the return of the film auteur — or the notion of the film as a work of individual authorship — but it is here, in the convergence of cinema and 3D imaging, where the auteur is making its 21st century comeback.

Earlier this week, at the Interacting With Immersive Worlds Conference at Brock University, I had the opportunity to hear about state-of-the-art virtual production from digital effects artist, Dejan Momcilovic, who spoke about his work at the New Zealand-based effects firm WETA Digital. Momcilovic spoke about motion capture, a technique used to create the digital effects in Avatar, Lord of the Rings, King Kong, among many other well-known films.

Motion capture is the process of measuring and recording movement in a form that can be processed by computers. Whereas cinematic techniques record the appearance of movement, motion capture extracts information about movement. It’s a method of analyzing movement, which is why, for a long time, motion capture was associated more with rehabilitation medicine than with entertainment.

The video game industry changed all that. After motion capture was used extensively and successfully in action and sports games (Rockstar’s L.A. Noire is probably the most well-known recent example of motion capture in video game production), the film industry began experimenting with it too.

Even if you’ve never heard of motion capture, you’ve probably seen “making of” videos or stills in news articles of actors with dots (or “markers” as they’re called in the industry) placed on their faces. Several cameras (or several dozen, in WETA’s top-of-the-line setup) record the movement of the markers. More detailed and subtle movement require more markers, so typically, the majority of the markers are placed on the actor’s face.

Once motion capture has been used to record a performance in a scene, effects artists use the motion data to make computer-generated bodies move. Bodies that are entirely computer-generated thus have the appearance of live-action filming.

Integrated with other digital effects techniques, motion capture allows directors to film other-worldly characters and environments “as if” they were actually in front of the camera. In virtual production, the characters and the environment of the film can be rendered first and then, using a virtual camera, the director can “film” the scene.

Criticism of motion capture has come mainly from traditional animators who were highly suspicious of this new technique of bridging live-action and animation. As Maureen Furniss writes in her excellent overview these debates, traditional animators saw motion capture as a shortcut around animation production work.

Judging from Momcilovic’s calm and collected manner of speaking about WETA’s digital capture work, today’s motion capture experts no longer see themselves as struggling to meet the artistic standards set by traditional animation.

So it seems like a pretty happy story. Motion capture gives digital effects artists another creative tool. And digital effects artists open up new ways of making film for actors, writers, directors and so on, unconstrained by the technical and aesthetic limitations of traditional cinema. Everyone wins. Or so it seems.

As I watched actor after actor being scanned, transformed into code and then inserted into virtual worlds, I couldn’t help but think of the 1981 film, Looker.

Looker was one of first Hollywood films to seize upon the worries about the replacement of actors by computer simulations. As is common with science-fiction thrillers, Looker revolves around a devious corporation, Digital Matrix, Inc., which, in this case, manipulates television audiences with computer-generated television advertisements. After scanning the actor’s body and producing a 3D model, Digital Matrix programs the virtual actor’s movements so as to hypnotize the viewer (the film doesn’t explain why programmed motion would hypnotize people). Since the real actors might cause trouble for the corporation, Digital Matrix kills the actors/models after digitizing them.

Although Looker is an all-too-familiar tale about the power of media to manipulate the weak and vulnerable audience, it does provide an early commentary on the way 3D capture and digital rendering techniques encourage a view of actors’ bodies and actions as “raw material.”

As with most media impact narratives, which focus on the consequences of technology on society and culture, Looker oversimplifies the relationships between digital media, cinema, and the broader history of techniques for recording movement. Histories of cinema also tend to oversimplify the technological and cultural history of cinema. While the techniques of documenting motion and producing the illusion of motion are at least as old as chronophotography (Leland Stanford and Etienne-Jules Marey’s horse locomotion experiment is probably the most well-known example), cinema was not the inevitable outcome of photography. Animation techniques, for instance, could have led to a very different mode of producing film along with a very different cultural forms of cinema.

But chronophotography comes out of the same obsession with breaking motion down to its basic, mathematic description as motion capture does a century and a half later. Marey’s delight in shifting between media, as demonstrated in his Analysis of the Flight of a Seagull (1887) sculpture, also suggests that the notion of motion as something that can be extracted from a body and shifted into another has been around much longer than contemporary motion capture techniques.

One of the key differences between motion capture and traditional cinematic techniques is that motion capture seeks to render 3D models based on “deep” (rather than “surface-level”) scans of bodies in motion. Digital effects firms like WETA produce 3D models with physiological and anatomical depth, scanning not just the surface of the body but also the body’s “interior,” including muscles, tendons, and bones.

As Momcilovic noted in his talk, some of the actors in WETA’s productions have undergone MRI scanning to make their 3D renderings of movement as realistic as possible. Although MRI may not be a standard component of motion capture yet, it does suggest that the overall trajectory in this area of visual effects is toward “deep” scanning of bodies.

Information from MRI and other kinds of scanning allow the production of virtual models of the actor which perform gestures and movements and respond to similar conditions and events (e.g., impacts) as the actor’s biological body. Motion capture today effectively enables bio-mechanical reconstruction of the actor’s body in digital code.

Looker might have missed the manner in which digital techniques would be integrated into acting and film production more generally, but it does anticipate changes in creative labour facilitated by 3D motion capture and other virtual production techniques.

There are at least two ways of looking at the impact of virtual production on creative labour.

On the one hand, labour that was previously considered “merely” technical is now considered to be creative. Visual effects artists like Momcilovic are increasingly recognized as creative contributors to the film’s aesthetic quality (as well as to its marketability).

On the other hand, motion capture reinforces long-standing hierarchies of labour in the film industry. Momcilovic noted that virtual production gives directors like James Cameron unprecedented control over the entire fictional world of the film. All Cameron needs to do is say, “make that hill bigger,” and the designers make it happen by altering the virtual landscape.

In this way, virtual production, as innovative as it may be, reinforces a very traditional sense of the film as the artistic expression of its director, reminiscent of auteur theory. Even effects artists sometimes regard the film they are working on as the director’s work of art. As Momcilovic put it, “Every decision on Avatar would at least go by [Cameron] and he’d have something to say about it … In a way, it was like watching a genius do what he does.”

Virtual production, as it is currently practiced by filmmakers and interpreted by critics and at least some effects artists, bolsters the sense (however distorted it may be) that films like Avator spring forth from the inspired mind of the director. The actors’ contributions, and those of the effects artists, are thus placed in the background.

Rather than “replacement by computers,” as predicted by Looker, motion capture and the broader array of techniques that enable the virtualization of production are bringing about some very real shifts in creative labour and in the way audiences interpret films produced in this way. Unfortunately, in the short term at least, these shifts appear to be backward, to the 1950s to be precise, when auteur theory, or the notion that the film is the personal expression of its director, became popular among film scholars and critics.

How can the return of the film auteur in the context of an increasingly computerized mode of production be explained?

One possibility is consider the way digital techniques are linked to other artistic practices. It may be, as Lev Manovich suggests in his 2001 book, The Language of New media, that digital cinema is more like painting than photography. As Manovich writes,

“The manual construction of images in digital cinema represents a return to the pro-cinematic practices of the nineteenth century, when images were hand-painted and hand animated. At the turn of the twentieth century, cinema was to delegate these manual techniques to animation and define itself as a recording medium. As cinema enters the digital age, these techniques are again becoming commonplace in the filmmaking process. Consequently, cinema can no longer be clearly distinguished from animation. It is no longer an indexical media technology but, rather, a subgenre of painting.” (p. 295)

What Manovich didn’t anticipate was that techniques like motion capture would be used to bridge animation and live-action. Nevertheless, his suggestion that digital techniques allow filmmakers to manually construct (or program) every detail of the cinematic image, much like a painting, provides a partial explanation for the appeal of the auteur theory in this context. Motion capture is industrially organized and represented as a way for the director to have more control over each “brush stroke” of the cinematic “painting.”

As film production becomes increasingly complex, notions of cinema-as-art may become more simplistic. While the notion of the director as the lone genius toiling away to make the magic happen seems hopelessly out of date, it is precisely that idea which seems to be gaining currency among those who are closest to the action in virtual production.

John Shiga