Live Sketch Online
Breathing Life Into Sketches Using Text-to-Video Priors.
A sketch is a simple and effective way for people to communicate their thoughts visually. A sketch that can move adds more possibilities to the expression of ideas and is widely used by designers for different purposes.
Turn your sketch into live video with AI.
Live Sketch
Breathing Life Into Sketches
Animate a single-subject sketch with AI
- Turn your sketch to live
- Live Sketch can animate a single-subject sketch by providing a text prompt indicating the desired motion.
- Automatic sketch animation
- The output is a short vector animation that can be easily edited.
- Character animation
- Live Sketch can be used to create animations of characters for various purposes, such as storytelling, education, entertainment, or design.
- Control over the animation
- Live Sketch offers a degree of control over the generated results by simply modifying the prompts that describe the movement.
- Artistic expression
- Live Sketch can be used to explore different ways of expressing ideas and emotions through sketch animation.
- Sketch entertainment
- Live Sketch can be used to have fun and enjoy sketch animation.
Frequently asked questions
If you can’t find what you’re looking for, email our support team and if you’re lucky someone will get back to you.
What is the main goal of Live Sketch?
The main goal of Live Sketch is to present a method that automatically adds motion to a single-subject sketch, based on a text prompt indicating the desired motion. The output is a short animation in vector representation, which can be easily edited.
How does the method create the appearance of motion?
The method creates the appearance of motion by predicting an offset for each control point of the sketch’s strokes for every frame. These offsets deform the sketch in order to animate it. The method models the motion through two components: a local path that predicts unconstrained offsets for each point, and a global path that predicts the parameters of a global affine transformation matrix for each frame.
What are the limitations of the method?
The limitations of the method include: (1) It assumes that the input sketch depicts a single subject, and may degrade in quality when applied to sketches involving multiple objects or scenes. (2) It may harm the sketch’s identity or introduce temporal jitter when deforming the sketch. (3) It inherits the limitations of the text-to-video priors, such as being unaware of specific motions, having biases due to the data, or producing artifacts.
What is the difference between the local path and the global path in the method?
The difference between the local path and the global path in the method is: (1) The local path predicts an offset for each control point of the sketch’s strokes, which can capture small local deformations, such as bending an arm or opening a mouth. (2) The global path predicts the parameters of a global affine transformation matrix for each frame, which can control global affine transformations, such as translation, rotation, scaling, or shearing.
What is the main source of information for the method?
The main source of information for the method is the motion prior of a large pretrained text-to-video diffusion model. The method uses a score-distillation loss to guide the placement of strokes, based on the text prompt and the initial sketch.
How does the method compare to other baselines?
The method compares favorably to other baselines, such as image-to-video diffusion models and a method tailored for animating human figures. The image-to-video diffusion models fail to maintain the unique characteristics of the sketch and suffer from visual artifacts. The human figure animation method is limited to a specific skeleton and a fixed set of animations. The proposed method can handle various sketches and motions, and preserve the sketch’s appearance better.
How can the user control the generated results?
The user can control the generated results by simply modifying the text prompts that describe the motion. The method can create different motions that align with the semantics of the initial sketch, such as filling a glass with wine or making a boxer jump. The user can also adjust the method’s parameters, such as the learning rate and the translation weight, to balance between the quality of motion and the fidelity of the sketch.
What are the benefits of using vector representation for sketches?
The benefits of using vector representation for sketches are: (1) It allows for easy editing and manipulation of the sketch, such as changing the stroke color, width, or shape. (2) It preserves the sharpness and clarity of the sketch, regardless of the resolution or zoom level. (3) It reduces the storage size and bandwidth requirements compared to raster images.
What are the challenges of using vector representation for sketches?
The challenges of using vector representation for sketches are: (1) It requires a differentiable rasterizer to render the vector sketches into images that can be fed to the text-to-video model. (2) It may not capture the fine details or textures of the sketch, such as shading, hatching, or brush strokes. (3) It may depend on the specific sketch representation, such as the type of curves, the number of control points, or the attributes of the strokes.
What is the role of the text-to-video diffusion model in the method?
The role of the text-to-video diffusion model in the method is to provide a motion prior that guides the placement of strokes, based on the text prompt and the initial sketch. The method uses a score-distillation loss to extract a signal from the text-to-video model, which measures how well the generated video matches the text prompt. The method does not require extensive training, but instead leverages the pre-trained text-to-video model.
What are the evaluation metrics used to measure the performance of the method?
The evaluation metrics used to measure the performance of the method are: (1) A user study, where participants are asked to rate the videos generated by the method and the baselines on a scale of 1 to 5, based on three criteria: motion quality, sketch fidelity, and overall impression. (2) A quantitative analysis, where the method and the baselines are compared using three metrics: the Fréchet Inception Distance (FID), the Structural Similarity Index (SSIM), and the Learned Perceptual Image Patch Similarity (LPIPS).
What are the main contributions of Live Sketch?
The main contributions of Live Sketch are: (1) A novel method for animating sketches based on text prompts, using a neural displacement field and a score-distillation loss. (2) A comprehensive evaluation of the method and the baselines, using both user study and quantitative analysis. (3) A demonstration of the method’s versatility, generalization, and controllability, using various sketches, motions, and parameters.