It is uncertain whether individuals beyond the dominant search engine entity Google will have the opportunity to wield the reins concerning the AI-based movie creation tool that has been hinted at by the company. Nevertheless, the prospect is undeniably intriguing.
Google’s Research division unveiled a film showcasing the new text-to-video innovation named Lumiere on Wednesday.
In a LinkedIn post, Inbar Mosseri, a key member of the team, mentioned that the tool can “generate clear, high-quality videos using plain text prompts,” as reported by New Atlas. These videos can last up to five seconds. Sample inputs include scenarios like “A soft baby sloth trying to figure out a computer with an orange knitted hat” and “an escaped tiger munching on snacks in the park.”
While recent attention has been on tools such as ChatGPT, which provides textual responses to prompts, and Dall-E, which generates images, Lumiere represents a significant advancement in text-to-video technology. If Lumiere truly showcases “state-of-the-art text-to-video generation results,” as described by Google, we might be entering a new era where video creation from text prompts becomes commonplace.
As illustrated in the demonstration, Lumiere has the capability to not only convert text to videos and images but also apply stylized techniques that use an image to generate videos in a similar fashion. Another notable feature is the ability to insert missing elements into a video seamlessly.
For instance, the tool can animate famous artworks like Da Vinci’s Mona Lisa and Van Gogh’s Starry Night, depicting scenarios such as a girl yawning and looking fatigued. While the rendition of Starry Night is nearly flawless, the Mona Lisa depiction appears to show the subject laughing rather than yawning.
While many of the depicted animals, such as “a muskox grazing on lovely wildflowers” and “A happy elephant wearing a birthday hat walking under the sea,” appear realistic, some dog animations fall short of complete realism. For example, a golden retriever running in a field and a toy poodle skating are convincingly portrayed, but subtle details like their facial expressions—and possibly their eyes—reveal their CGI nature.
Despite the promising features of the picture editing tools, the operational explanation of Lumiere remains complex and opaque. The tool is described as “a space-time diffusion model,” a concept reminiscent of Doc Brown’s experiments in “Back to the Future.” According to Google Research, this model enables the text-to-video generation by processing information across various space-time scales, resulting in videos that exhibit “practical, diverse, and coherent motion.”
In contrast to existing models that produce frames independently and then stitch them together, Lumiere focuses on holistic video creation, akin to choreographing a dance performance rather than staging a puppet show, as explained by Jason Alan Snyder, the global chief technology officer at Momentum Worldwide.
By comprehensively visualizing the entire film, understanding character movements, object interactions, and temporal changes, Lumiere sets itself apart from conventional filmmaking approaches. This unique method eliminates the need for patching together disjointed frames, resulting in seamless and natural motion throughout the video, with the exception of potential challenges with puppy animations.
