AI-Restoration in the Film Archive
Uprezzing the classics — and “generating” new ones?
In February 2020, YouTuber Denis Shiryaev uploaded a new version of Lumière Brother’s iconic 1896 film, The Arrival of a Train. Shiryaev modified the film using AI tools: the video was “uprezzed” to 4K resolution, making the images sharper; new frames were generated, making the movements smoother; the images were colorized and the film grain was removed. Described as “AI-restoration” all these new modifications made one of the first films in the history of cinema look more like the contemporary videos we encounter every day. The video was watched by millions and brought new life to the trend of uprezzing and colorizing archival films.
Since then, Shiryaev has uploaded many more videos, from Louis Le Prince’s mysterious film Roundhay Garden Scene (1888) to city travelogs from the early 1900s. These videos in total have garnered almost 65 million views and Shiryaev has gathered close to 600k followers.
In November 2020, The New York Times published an article by Sam Anderson, reflecting on another “AI-restored” Lumière film, The Snowball Fight (1897): “On an intellectual level, we all understand that historical people were basically just like us… and yet it’s hard, across such wide gulfs of time, to really feel this connection. So to watch this snowball fight, to see these people so alive, is a precious gift of perspective.” Anderson’s words voice the novel, facile connection to the past that thousands of viewers probably feel when watching these videos.
However, these two videos are not available online anymore. AI-restored versions of The Arrival of a Train and The Snowball Fight are taken down due to copyright claims. In March 2021, Shiryaev received a dispute letter from Institute Lumière (IL), the representative of the Lumière estate. In the letter Shiryaev shared, the IL informs him that the copyright of the film is still protected. Their claim: since Auguste Lumière died in 1954 and that copyright is protected for 70 years since the death of the last author, this film will not be public domain until 2024. Surprisingly, the first films ever shot are still protected under copyright law. The length of protection afforded by copyright law in the U.S. has been periodically extended, as major intellectual properties (such as Mickey Mouse) have approached entrance to the public domain, and their copyright owners have successfully lobbied for these extensions. In France, the length of protection was similarly extended from 50 to 70 years with changes to the law in the ’90s.
IL also claims this work is an infringement on the authors’ “moral rights,” a copyright law that is active in France and several other countries. The “moral rights” are perpetual and prohibit any alterations that are not intended by the author. Thus, IL argues colorizing and changing the frame rate is at odds with what the creators intended. Shiryaev explains IL’s process as an attempt to “copyright the history,” and as he was in a financially insecure position, he didn’t want to risk fighting back.
But what is the proper way to preserve and present the past? How should institutions and archivists think of AI tools in relation to restoration, copyright and “moral rights?” For this, I talked to Elif Rongen-Kaynakçi, the silent film curator of the Eye Film Museum in the Netherlands, known for its work in film restoration and is one of the biggest archival film collections.
With excitement, Rongen-Kaynakçi commented that many people in the archival film community has been discussing copyright issues lately. (For example, the 2022 Conference hosted by Domitor, an influential society dedicated to the study of early cinema, focused solely on copyright this year.) Rongen-Kaynakçi says that many YouTubers, including Shiryaev, use films from the Eye Collection and uprez and colorize them using AI. For example a video on Eye Film Museum’s YouTube channel might attract 6000 views, while the same video, modified by AI, might draw 6 million views on another channel. Eye’s approach is a little different than Institute Lumière’s. They see these YouTube channels as an entry point for millions of people, many of whom may never have heard of them, into their work. They want the videos to be seen, but they also want the labor of the institution to be acknowledged.
Rongen-Kaynakçi brings up a complicated hypothetical scenario: What if, for example, a TV channel wants to broadcast the AI-restored/colorized version of the silent films? Where would they get the video from and who would they license it from?
Rongen-Kaynakçi pointed to the work of Dr. Claudy Op den Kamp, who is doing invaluable research regarding copyright in moving images. In a recent talk, she mentions that there is a large shift in technology in the post-internet era: it used to be harder to reach the source material and easier to control film distribution. Today, films circulate much faster and without effort. Copying is very easy, thus the legality is muddy and complicated. She mentioned that when using archival materials that are preserved by institutions, “the process is not exclusively legal.” It is dictated by the rules of the archive and is a negotiation between different parties. Copyright is not only a legal issue, it is an ethical and institutional issue as well.
And there is another question: could the work of Shiryaev be considered restoration? Eye Museum’s Chief Curator Giovanna Fosatti defines film restoration as “actions undertaken to bring the film back to a form that is as close as possible to its original form.” But problems arise when we try to define what the original form is. It can be “textual”, such as the plot, intertitles and credits of the film. Or “material”, the original colors of the film, the original medium (35mm, 16mm…). The usual restoration processes attempt to bring the film back to an original form through these two categories. From this perspective, “AI-restoration” is less of a restoration but more of a creative modification.
But then there might be a more speculative category, let’s call it “situational” — contextualization of the film as perceived by the audience of a new era. Can AI-restoration be about this “situational” aspect? As Anderson’s article suggests, these films provide a new bridge to the past. Since they look similar to the images we look at on our phones and computers, we react with surprise: the past also looked just like the present. This type of restoration might be recreating the perception of the time. When audience members watched The Arrival of a Train in 1896, the images appeared to them as cutting-edge and totally new. When we watch an AI-restored version of the same film today, we again share a similar surprise. This is the tension between what is emotionally authentic versus materially authentic.
As the questions around archives and copyright are being renegotiated, a new angle is also appearing in the background. What do we do with archival images in the time of AI image creation? As I have been writing this article, Dall-E and similar projects have become extremely popular. I had a chance to access Midjourney, a platform similar to Dall-E. and asked the platform to create images with the prompt “Lumière Brothers, The Arrival of a Train”. Here are the images the AI produced:
These images don’t look like the original film, however they have the feel of it. These images wouldn’t be the same if the films of Lumière Brothers weren’t online and accessible to AI algorithms. Nor would they exist without the vast archive of images used to train the algorithm. So then, what are these images? Are these original? Are they subject to copyright? These novel issues can provide new angles for bigger questions: How should the past be preserved, owned and presented?
For more news, discourse, and resources on immersive and emerging forms of nonfiction media, sign up for our monthly newsletter.
Immerse is an initiative of the MIT Open DocLab and Dot Connector Studio, and receives funding from Just Films | Ford Foundation, the MacArthur Foundation, and the National Endowment for the Arts. The Gotham Film & Media Institute is our fiscal sponsor. Learn more here. We are committed to exploring and showcasing emerging nonfiction projects that push the boundaries of media and tackle issues of social justice — and rely on friends like you to sustain ourselves and grow. Join us by making a gift today.
