On Tuesday, Artificial Security unveiled Stable Video Diffusion, a novel AI research tool that offers the capability to transform static images into short videos, albeit with varying degrees of success. This tool serves as an initial public glimpse into two AI models that leverage the image-to-video technique and can be executed locally on a machine equipped with an Nvidia GPU.
The previous year witnessed Artificial Security making a significant impact with the introduction of Stable Diffusion, an “open weights” model for image synthesis that catalyzed a surge in open image synthesis endeavors. This initiative inspired a thriving community of enthusiasts who have further refined the technology through their personalized enhancements. Now, the company aims to replicate this success within the realm of AI video synthesis, acknowledging that the technology is still in its nascent stages.
Presently, Stable Video Diffusion encompasses two variants: one capable of generating 25 images (referred to as “SVD-XT”), and another designed for producing image-to-video synthesis with a length of 14 images. These models generate concise MP4 video clips, typically spanning 2 to 4 seconds, at a resolution of 576 x 1024, operating at speeds ranging from 3 to 30 frames per second.
During our local evaluation, the creation of a 14-frame sequence on an Nvidia RTX 3060 graphics card required approximately 30 minutes. However, individuals can expedite model testing significantly by leveraging cloud services offered by platforms such as Hugging Face and Replicate (some of which may involve costs). Our experiments revealed that the generated visuals often incorporate dynamic effects like panning, zooming, or animated elements such as dust or flames, while preserving a portion of the original image. Nevertheless, instances where individuals depicted in images exhibit lifelike movements remain infrequent.
Stability underscores that this concept is still in its developmental phase and primarily intended for research purposes given its current limitations. The company iterates on its website, “While we eagerly share our designs to keep pace with the latest advancements and actively seek your input, this concept is not yet poised for real-world or professional applications.” Your insights and feedback on aspects of security and quality are crucial for refining this model in preparation for its future iterations.
The research paper on Stable Video Diffusion briefly mentions the utilization of “a vast video dataset comprising approximately 600 million samples” to curate the Large Video Dataset (LVD), encompassing 580 million annotated video clips spanning a cumulative content duration of 212 years. While this achievement is noteworthy, it aligns with the trajectory of advancements in this field.
Stable Video Diffusion joins a lineage of AI models offering similar functionalities, with notable predecessors including solutions from Meta, Google, and Adobe. Additionally, the community has witnessed the emergence of open-source tools like ModelScope, alongside the highly regarded AI video model from Runway, namely the Gen-2 model (Pika Labs also stands out as an AI video provider). Artificial Security hints at ongoing work on a text-to-video model, enabling the creation of short video sequences based on textual prompts rather than images.
For local testing, an accessible approach involves utilizing the Pinokio system, which streamlines setup dependencies swiftly and encapsulates the process within its environment. The resources and repositories for Stable Video Diffusion are readily accessible on GitHub for interested users.
