The visual-effects artists at Monsters Aliens Robots Zombies Inc. were staring at actor Paul Bettany’s head and thinking about how to do their job faster. So they consulted their in-house artificial-intelligence researcher.
The head wasn’t the problem. It was the tracking marks the filmmakers had placed on it, which the artists needed to get rid of as they transposed a red-and-silver robotic head onto him – to transform him into Vision, the Marvel Avengers character and co-star of the Disney+ show WandaVision.
Visual effects, or VFX, work is in extraordinary demand in the age of bingeable, streaming television. VFX shops everywhere are trying to figure out ways to be more efficient. The Toronto studio, abbreviated MARZ, had been seeking technological ways to solve this problem.
Machine learning offered a solution. One-10th of the time spent bringing Vision’s robotic head to life was spent removing those tracking markers, frame by frame. After a month of work, the AI researcher came back with a way to automate that process – keeping down costs for clients and saving MARZ’s artists a great deal of grief.
As the three-year-old studio establishes itself in this crowded industry, it’s betting it can stand out by using artificial-intelligence techniques to save time, effort and cost. It’s now building out an entire AI team, with two dozen hires and dozens more to follow, who MARZ hopes will develop programs to automate other mundane tasks. Much like an AI company, and less like a traditional VFX company, MARZ also just raised a round of venture capital – $6.5-million – to boost its growth.
“We’re trying to create products that can put jetpacks on the artists,” says Jonathan Bronfman, MARZ’s co-president, “so you can take a single artist and substantially increase their bandwidth.”
Mr. Bronfman is also president of JoBro Productions, which has produced films such as The VVitch starring Anya Taylor-Joy, and he is a millennial scion of the ever-branching Bronfman family – the son of Paul Bronfman, himself a long-time film production executive.
Jonathan Bronfman says he first encountered visual-effects executive Lon Molnar “freezing our butts off on set in Northern Ontario,” as the pair worked on films such as The VVitch. Mr. Molnar was chief executive of the award-winning VFX shop Intelligent Creatures, and as the pair chatted, both realized they saw an opportunity in premium television.
Amid those talks, Mr. Bronfman attended a party where he met Matt Panousis, a lawyer who was in the midst of exiting a startup. The two men discussed the new studio, and Mr. Panousis later agreed to come aboard as chief operating officer.
The demand for television series with movie-like quality had been rising for a decade or so, and MARZ made this its focus. But there was a capacity shortage for visual effects, too. “Every studio in the world is backed up,” says Joe Raasch, who co-ordinates Seneca College’s Visual Effects for Film and Television program.
Yet the industry’s profit margins are thin. Many VFX studios have come and gone in recent years. If MARZ wanted to stand out, it needed to do something different than its competitors. The principals wanted to try something technologically different, but weren’t sure what that would look like.
So for the first while after the company launched in 2018, it took on ordinary projects. Its team created a reflective mask for a character in HBO’s Watchmen and inserted a second Paul Rudd in scenes from Netflix’s Living with Yourself. After talking about investing in artificial intelligence for a while, MARZ executives hired their first AI researcher in November, 2019, to figure out how to apply the technology to VFX.
Along came WandaVision. Mr. Bettany plays a robot in the show who occasionally disguises himself as a human. But to piece the show together, the human Mr. Bettany had to be digitally disguised as a robot.
To do that, he wore tracking markers on his head so artists could transform it into Vision’s in postproduction. Normally, those artists would need to remove the markers in each frame manually.
To get around this, MARZ asked Marvel for more data – in this case, as many photos of Mr. Bettany as possible, with and without the tracking markers. Then the company handed the data over to its burgeoning, one-person AI team. Over the course of a month, the researcher worked with MARZ staff to train a machine-learning algorithm to recognize the markers and the textures around them, then to remove the markers and seamlessly fill in the space left behind.
For a 10-day project, this could save an entire day’s worth of menial work. Extended across whole shows, it could also save clients money. This, the executives realized, was the technology they needed. “We started small,” Mr. Panousis says. “It was more just to give ourselves some confidence in what it could do.”
The confidence came fast. After WandaVision, the executives began building a whole team around using AI, hoping to develop products that could be applied to numerous VFX situations – to scale, in tech parlance. Now, 23 of MARZ’s 200-odd employees are AI specialists, and executives hope to hire 35 to 40 more with their new funding.
One of MARZ’s most recent hires was Danny Cohen-Or, a widely cited computer graphics researcher and professor at Tel Aviv University, who is now chief scientist. His lab had already turned its attention to applying AI to editing faces. “This completely new state of mind requires a lot of flexibility, which I found in MARZ and in the mindset of their founders,” Dr. Cohen-Or says by e-mail. “We are just in the infancy of the revolution.”
MARZ has already used its AI for VFX system on 17 productions, including AMC’s The Walking Dead and Peacock’s Dr. Death. Mr. Raasch, the Seneca visual-effects program creator, says the industry is alight with the kind of innovations the company is trying to develop. “The two main things people are talking about for the future are machine learning and real-time VFX,” he says, referring in the latter case to rapid video-game-style graphics processing.
MARZ knows it’s not alone in investing in AI. One company Mr. Molnar has worked with in the past, Vancouver’s Ziva Dynamics, uses AI techniques to help with the animation of muscular movement.
Such digital tools have also become increasingly valuable. Lord of the Rings director Peter Jackson’s company Weta Digital sold its visual-effects tools division to a game-development platform in November for US$1.6-billion. Weeks later, Netflix bought the well-known Vancouver company Scanline VFX for an undisclosed sum. “It really is a crazy time in the industry,” Mr. Raasch says.
The team at MARZ believes its tools could generate strong demand. As the team perfects the company’s software – including by training it with more data – it’s considering licensing it out to others. As companies such as Facebook owner Meta Platforms Inc. explore virtual “metaverse” experiences, Mr. Molnar says, the sector’s capacity to meet demand will be compounded. “That compounding factor will need to rely on technology to solve that additional capacity issue,” he says.
But the humans MARZ hires are still the core of the business. “Hollywood doesn’t want AI,” Mr. Bronfman says. “What Hollywood wants is perfect visuals very quickly at affordable rates. We’re able to process our projects through AI, and then have artists come in and fix those results. … What [AI investment] allows us to do is fulfill our value proposition to Hollywood.”
Your time is valuable. Have the Top Business Headlines newsletter conveniently delivered to your inbox in the morning or evening. Sign up today.
Follow Josh O’Kane on Twitter: @joshokaneOpens in a new window