Rings of Power

In the week following it debut on September 1, The Lord of the Rings: The Rings of Power was seen by an estimated 25 million viewers in the first week, and that number has since exceeded 100 million. While the tale reaches back 1000 years earlier than the events depicted in Tolkien’s iconic trilogy, the production itself pushed filmmaking relentlessly into the future. The production of eight episodes was mounted over the course of two pandemic-interrupted years, and included the design and construction of facilities and procedures that will be used on future seasons.

From the very start of preproduction through delivery, The Rebel Fleet, the New Zealand-based workflow experts, played a crucial role in the execution. Using The Rebel Fleet’s custom software platform, Konsol, the company took the workflow to a new level of integration, streamlining and automating the gathering of precise and universal metadata and meshing those process with collaboration tools like Qtake, Moxion and ShotGrid. Furthermore, they ensured consistency of color and other image information across the entire production and post process to create smooth and well-informed collaboration across all departments, bridging post, editorial and visual effects even as the shoot unfolded.

Given the unprecedented size and complexity of the undertaking, the potential gains in efficiency and the attendant creative flexibility were deemed worthy of the additional effort. It’s a symbiotic relationship: ambitious projects relentlessly drive innovation, and innovation makes such projects possible.

“Each department used specific cutting-edge technologies to achieve their facet of the mosaic,” says The Rebel Fleet CEO Michael Urban. “We see our job as helping to blend those facets – blurring the boundaries between departments so that the final product reflects a unified vision – in other words, compelling entertainment.”

Amazon would take some of their first steps into end-to-end production as a studio on the project. Post supervisor Jake Rice, a veteran of two decades in post, saw the job as an opportunity to design and implement a global, technology-forward philosophy. He worked with visual effects master and overall producer Ron Ames on designing an approach.

“Ron and I wanted to do everything in the cloud, and to completely integrate dailies onset, metadata capture, visual effects and post – and we knew we’d need partners to do that,” says Rice. “Amazon gave us their blessing, along with unlimited cloud storage and a team of people from their tech vision and content security to help us.”

Rice and Ames saw that the camera capture process, still rooted in film procedures, could be antiquated and was ripe for improvement. “We felt that if every shot was going to be VFX, we really needed to harness this,” says Rice. “We need to be aware of everything that’s going on while the cameras are rolling.”

Around that time, they met with Mike Urban and The Rebel Fleet, who were to handle DIT , video assist, onset dailies and cloud archive, integrating with the directors of photography.

“We just dove in together, and soon we were friends,” says Rice. “The Rebel Fleet were people I could throw ideas and challenges at every step of the way, and they were game – more than any other company I’d ever worked with. Not only were they thrilled to work in their own country on a project like Rings of Power – they were also happy to be working with partners who weren’t just going to write the PO and expect them to go and figure it out for them. We wanted to develop solutions that would truly streamline and elevate the process. That’s exciting for anyone working in a tech-intensive field.”

Nine months prior to the shoot, the team began developing Konsol, The Rebel Fleet’s metadata collation and management toolset – one system to rule them all, to coin a phrase. The result is Konsol, the Rebel Fleet’s metadata collation and management toolset.

“Every department needed to be on board with how we were going to capture this info and bring it into the VFX team, and how they were going to realize it,” says Rice. “Then you ask what you’re going to need on the set. We knew that Qtake, a tool that most people use for video playback, has a lot more capability. We needed ways to input the X, Y and Z axes of the camera, and where and how it moves. Lens size, shape, aperture changes, day, time, take, character, number, color value, light color and placement. Can we harness all this data and not just throw it away? The Rebel Fleet developed all these fields of valuable information. For example, green screen overlay sources were added, so the director could comp in and envision the eventual background. We wanted to track that element so we’d know exactly what it was going to be.”

These supplemental files are critical for the visual effects team. Without LIDAR scans, the witness camera, the script, and 3D models, the job is nearly impossible, or at least extremely time-consuming. Nothing kills creativity like having to wait.

Eventually, there were approximately 800 fields of metadata that travelled with every single shot through the entire process. When VFX eventually began pulling plates, they were able to harness all of that data and apply it to the work that they were doing.

“There was never a question of what that lens size was, for example,” says Rice. “It’s better for them in every sense. From every camera angle, from every camera roll, from every day. It's a massive, massive database that was made possible because The Rebel Fleet listened, and along with Konsol, agreed to develop a solution.”

Adam Harriman is in charge of on-set production for Wētā FX, a key member of the extensive visual effects aspect. He describes his job as forensic, but on a film set rather than at a crime scene.

“To help connect the digital world to the real world, we map out sets to scale using engineering tools, LIDAR and other photography to support that,” says Harriman. “We use photography to capture texture information of sets and people and costumes and props. And basically, every time a scene is set and they’re about to shoot, we’re there creating a little map of the position of the cameras, the lights, the actors, the blue screen, and the terrain. Our data wrangler is responsible for noting down camera bodies, camera lenses, frame rates, the height of cameras off the ground, tilt or angles of the cameras. They are our eyes and our ears onset. We get all the pieces to fit together, and then pass them on to visual effects and post , whether it’s at Wētā or other facilities. So it’s really important that our data is accurate, to scale, and organized precisely, so nobody has to waste time trying to figure out the pieces. They can open up the scene and it all comes together.”

The Rebel Fleet’s entirely new system seamlessly integrated the smart lens and other metadata information with Qtake, working to adapt the software code. Every department had an iPad with Qtake, each filled in with data specific to their concerns. The VFX people might make continuity notes, and the camera assistant might record focal distance.

“Before, that was all manual,” says Harriman. “Now, the VFX data wranglers, the camera assistants, the script supervisor, et cetera, don’t have to do that anymore. Instead of us all writing separate notes and then comparing notes to our own databases at the end of the day, it’s done. That extends to every aspect of the metadata.”

 A single point of contact mean that everyone could see what everyone else was entering, and that info all cycled through Qtake and Moxion. The entire database is available, with each take. Repetition and mistakes are minimized.

“The ease of it became so apparent after a month,” says Harriman. “It really gave us visibility into what was being collected for all the departments, since we had been focused on what we needed to collect as VFX onset. I really enjoy that. I'm hoping we can take this workflow to other sets. It’s an amazing way to avoid doubling up on work, and everyone works from the same correct, consistent data.”

When the data is organized and easily shared among departments, efficiencies compound along the way. “We don’t want any facility or any artist to lose time trying to connect the dots or searching for data,” says Harriman. “We work with a naming convention and put that information into that Qtake database, and our photographers and our surveyors and our LIDAR technicians tag all of their data with those names. Because it’s tied back into Konsol, it automatically syncs with the data we have in-house. But because we also service other facilities, they’re able to do that sync as well. They don’t just get the plate data, the witness camera data and the shooting data. They also get the boots-on-the-ground data that we collect for them, including models of locations and sets and meshing data. And as long as we’re working hand-in-hand with The Rebel Fleet or with our own data structure on the ground, when those facilities get it, everything is simply married together. And that’s what we want.”

In providing DIT services and on-set dailies color, The Rebel Fleet made sure directors of photography Oscar Faura, Alex Disenhof and Aaron Morton could concentrate on the creative aspects of their jobs. The cameras were ALEXA Mini LFs, usually paired with ARRI DNA lenses, which use rehoused medium format spherical glass. These were occasionally augmented with Angenieux zooms.

At most locations, a dailies truck was parked nearby where directors of photography could review their images on calibrated monitors, usually at the end of the day.

“To me, the highest compliment is that The Rebel Fleet allowed me to not worry about it,” says Disenhof. “I don’t know all the technical ins and out of their process, and I mean that in the best way. I had all these tools at my disposal. We worked ten-hour days, which I loved, partly because it allowed me to sit down with the dailies colorist afterwards and really look at the work. The images I was seeing were extraordinarily precise. When I got to the truck, I could see that the integrity of our intention was maintained in both SDR and HDR. Sometimes we only had ten minutes in the truck, but that was enough to ease my mind, and remember what I did, and see what worked or what didn’t. I could also give a few notes – maybe we weren’t able to quite get something to where we really wanted it in the time we had, and I could ask for an adjustment. We had the use of Livegrade on set, and we could make quick adjustments, knowing they would be maintained all the way through to the end.”

The Rebel Fleet’s metadata innovations paid off for Disenhof, too. “I’m a real stickler for matching, to be honest,” he says, “whether I’m matching another cinematographer’s work, picking up shots from their scene, or going back to match some work that I did several months before, or whatever it may be. Especially on these long TV shows, you end up shooting similar sets over and over. There may be two or three months’ split because of an actor’s availability, for example. So it was absolutely invaluable to have easy access to the metadata and the imagery in those cases.

“I had a lot of night lighting, firelight and moonlight,” Disenhof continues. “Those are usually the main sources of the show at night. So in general we wanted a consistent night look throughout. Knowing what our camera settings were from scene to scene, and understanding how we had done a scene – sometimes four months previously – was important. To be able to pull those up right away was really great.”

Digital Imaging Technician Hadley Parsons is also part of The Rebel Fleet team. He came onto the project several months in advance of principal photography and was far more deeply involved in the workflow design than a standard DIT. He was also on the team that coaxed the various metadata systems to work together. Once on the set, Parsons could keep a constant eye on the system, QCing the images and making sure the metadata is communicating to the hub.

Disenhof is pleased with how the imagery turned out. “It’s vibrant and beautiful and really kind of whimsical,” he says. “The behind-the-scenes contributions of The Rebel Fleet – that’s the oil in the machine that makes it go. We’re all expected now to make these incredibly ambitious and cinematic pieces of work on schedules that in previous years would have been impossible. We have all the equipment, and a great budget, but we still have to deliver this very high-quality storytelling in a truncated amount of time. In the past, it’s been people writing these things down, flipping through notebooks to figure out how we did something. Everything, including the color process, is so much more streamlined now. As a cinematographer, all I have to do is ask and it’s there. I don’t have to think about it. I can focus on making sure that we’re moving quickly and making our days and just being creative. It has allowed us to trim away a lot of the inefficiencies of a more analog world.” 

Inconsistency in terms of color and image have plagued filmmakers since the dawn of digital cinematography, frustrating creatives, adding time and expense, and diminishing the final results. On The Lord of the Rings: The Rings of Power, Supervision Dailies Colorist Pete Harrow made sure these problems didn’t crop up. The first step was working with Oscar Faura and Wētā to fine-tune the LUT they had established for the show.

Harrow says that one small but important improvement on the shoot was the ability to accurately record and pass along the DIT’s notes and adjustments when the picture data was sent down the line.

“Quite often you’ll get the director’s notes coming through the script supervisor,” he says. “The director will say ‘I love that take.’ But it’s very rare that the director of photography gets to say, ‘Actually I didn’t quite have that light set as I wanted, so that’s an NG take for me.’ You always hear DPs say that the take they were unhappy with is the take that got used. With the system we designed, the DIT can note what the DoP is thinking or saying at the time, and it passes through the next phase to the editorial team, and the DoP’s wishes can be respected. Again, that’s an incremental enhancement to the final result.”

In the dailies van, the system automatically applies CDLs for every shot. There, The Rebel Fleet’s system puts a complete back catalog of every look created at the DoP’s fingertips.

“Let’s say we’re shooting scene 12,” says Harrow. “I can jump to scene 11 and have a look at what we did there. I can make sure there’s no jarring or jumping too much between the looks. And then I sort of tweak it from there. Typically, the DoP would pop in at the end of the day, and quickly go through the shots.”

At that point, an entire library of stills, one for each setup, was exported to Moxion. This separate folder was shared among the two DoPs, their DITs and their dailies colorists as a look bible. “When the DIT comes to the set the next day, they can open up that look and see exactly how the dailies colorist had it,” says Harrow. “Rather than working on what they did the day before, they’re working on the new updated grade, which is what is going into editorial.

“This is where The Rebel Fleet really stands out,” says Harrow. “Rather than just thinking about the DIT, it’s seeing the entire flow and the opportunities for streamlining from the top down. Rather than thinking in terms of departments, we keep the flow of information and color and look heading straight back to the set, and then out to dailies. Information flows in both directions. It’s a different way of thinking.”

When the second unit team needed to see a shot that will intercut with something shot months prior, they could instantly refer to high resolution stills depicting the updated looks, which include any changes made in the trims. Rich metadata could be delivered straight into post and VFX with the on-set team’s intent intact.

“It was great working with Ron Ames and Jake Rice to build this pipeline,” says Harrow. “The idea – having one source of truth in terms of metadata – was something that Mike (The Rebel Fleet CEO Mike Urban) and I had talked about for years. Ron was a real stickler – he wanted every metadata field to be populated. He kept us all on that flow, making sure that every bit of metadata that could be captured was there and in the system. And the whole idea is to make better images at the end of the job.”

Harrow knew he was on the right track when an editorial assistant visited him on a Monday after both units had big shoot days the previous Friday. The assistant passed along a case of beer and offered his thanks, saying, “Typically, that amount of footage would have meant me and another assistant working the whole of Saturday checking the dailies, building bins and applying info from a .pdf of the script supervisor’s notes to have it ready for the editor Monday morning. But you’d done all of that. I just had to check it, and I was finished by mid-day Saturday. So thank you!”

“That was satisfying,” says Harrow, who also serves as Director of Technology at The Rebel Fleet. “It makes for better images and a better show, but it’s also better for filmmakers and their ability to work with a healthy lifestyle – which is a concern for all of us.”

On The Lord of the Rings: The Rings of Power, Moxion, which allows post and production to securely share and review footage within minutes of creation using cloud storage, was harnessed with Qtake as part of The Rebel Fleet’s system. Moxion’s CEO, Hugh Calveley, was also a co-founder of The Rebel Fleet. So perhaps it’s not surprising that the two systems could be integrated. Calveley says that the synergy between tech advancement and creative ambition made this project a highlight of his impressive career.

“Not just because of the amazing ensemble of highly talented artists and technicians,” he says. “But because they really engaged with our technology, using pretty much everything that we had on offer. Prior to Rings of Power, no production has used all facets of what we do. That also shapes our future offerings. Companies like The Rebel Fleet are really a key driver of the future of companies like Moxion. That union drives innovation, which is exciting.”

The ability to very quickly see what’s going on enables editors to look at a scene as it’s being shot on the other side of the world. What happens in the edit suite can then inform further decisions on the set. On a shoot like Rings of Power, where almost every shot has a visual effects element, VFX can suggest an angle or a frame that save two days’ time and money.”

Production and postproduction have been happening more concurrently for years, but the pandemic sped up that change. “At The Rebel Fleet, we’re always trying to act as a live bridge between those processes, and solve the problems that come up,” says Urban. “Looking to the future, we want to make that process solid and standard. The goal is for us to blend into the background, with everything set up and ready to go.”

Since The Lord of the Rings: The Rings of Power, The Rebel Fleet has taken on other projects including Sweet Tooth, a Netflix television series, and The Wilds, an Amazon Original series produced in Australia, where the company has a team of five, including two engineers, ensuring smooth workflow.

 “We see more opportunities to apply what we’ve learned on Rings of Power,” says Urban. “The systems and process and software that we’ve developed, and the expertise and experience in implementing them, are valuable in today’s productions, where organization and efficiency is so important, especially as productions continually get more complex and effects-heavy. You just can’t wait and see when something is going to be shooting partly on the other side of the world. Filmmakers need to be able to communicate, and that’s never been more true than it is today. We love driving innovation, especially when it’s in service of a tremendous project like Rings of Power. That’s why we work so hard at it.

“The mandate from above is to make sure the content is compelling so that people want to watch it,” says Urban. “It’s not to make sure that the metadata is tidy and that it’s a well-packaged product. But our job is to contribute to a better project by connecting those dots and bringing everyone together, thereby helping to achieve the overall goal.”