# Kandinsky S2V 1.0

Kandinsky S2V 1.0 menghasilkan video avatar dari bingkai referensi dan trek ucapan. Orang dalam bingkai tersebut dianimasikan agar sesuai dengan ucapan — seluruh tubuh, bukan hanya kepala. Ucapan berfungsi sebagai kondisi, bukan keluaran: model tidak menghasilkan audio, dan soundtrack dari hasil akhir adalah trek yang Anda berikan.

Kedua kondisi tersebut melewati [Qwen2.5-Omni-7B](https://huggingface.co/Qwen/Qwen2.5-Omni-7B): hidden states-nya mengenkod prompt dan audio tower-nya mengenkod ucapan, yang masuk ke transformer melalui cross-attention adapters yang diselaraskan per frame, sehingga setiap latent frame hanya memperhatikan ucapan yang dicakupnya. Sebuah klip dihasilkan dalam chunk berukuran 5 detik (125 frame pada 25 fps), masing-masing membawa memori FramePack dari chunk sebelumnya, sehingga trek yang lebih panjang tetap konsisten melintasi batas chunk.

Kandinsky S2V 1.0 hanya terintegrasi sebagai [pipeline modular](../../modular_diffusers/overview); tidak ada kelas `DiffusionPipeline` standar untuknya.

## Checkpoint

| Checkpoint | Transformer | Blok | Penjadwal | Langkah-langkah | Panduan |
|---|---|---|---|---|---|
| [Lite][lite] | 2B | [KandinskyS2VAutoBlocks](/docs/diffusers/main/id/api/pipelines/kandinsky_s2v#diffusers.KandinskyS2VAutoBlocks) | [FlowMatchEulerDiscreteScheduler](/docs/diffusers/main/id/api/schedulers/flow_match_euler_discrete#diffusers.FlowMatchEulerDiscreteScheduler) | 50 | 3.0 |
| [Pro][pro] | 21B | [KandinskyS2VAutoBlocks](/docs/diffusers/main/id/api/pipelines/kandinsky_s2v#diffusers.KandinskyS2VAutoBlocks) | [FlowMatchEulerDiscreteScheduler](/docs/diffusers/main/id/api/schedulers/flow_match_euler_discrete#diffusers.FlowMatchEulerDiscreteScheduler) | 50 | 3.0 |
| [Distilasi Lite][lite-distill] | 2B | [KandinskyS2VDistilledAutoBlocks](/docs/diffusers/main/id/api/pipelines/kandinsky_s2v#diffusers.KandinskyS2VDistilledAutoBlocks) | [PiflowScheduler](/docs/diffusers/main/id/api/schedulers/piflow#diffusers.PiflowScheduler) | 16 | off |
| [Pro distilled][pro-distill] | 21B | [KandinskyS2VDistilledAutoBlocks](/docs/diffusers/main/id/api/pipelines/kandinsky_s2v#diffusers.KandinskyS2VDistilledAutoBlocks) | [PiflowScheduler](/docs/diffusers/main/id/api/schedulers/piflow#diffusers.PiflowScheduler) | 16 | off |

Checkpoint yang telah didistil mengonfigurasi `PiflowScheduler` dengan `shift=10.0` dan `n_grid=5`, bukan nilai default-nya.

[lite]: https://huggingface.co/kandinskylab/Kandinsky-S2V-1.0-Lite-Diffusers [pro]: https://huggingface.co/kandinskylab/Kandinsky-S2V-1.0-Pro-Diffusers [lite-distill]: https://huggingface.co/kandinskylab/Kandinsky-S2V-1.0-Lite-distill-Diffusers [pro-distill]: https://huggingface.co/kandinskylab/Kandinsky-S2V-1.0-Pro-distill-Diffusers

`ModularPipeline.from_pretrained` memilih blok dari checkpoint. Langkah-langkah secara default menggunakan nilai di atas. Classifier-free guidance adalah komponen `guider`, bukan argumen pemanggilan: checkpoint dasar menyediakannya pada 3.0. Versi yang telah didistilasi tidak memiliki guider dan tidak memiliki `negative_prompt`, karena guidance sudah menjadi bagian dari student.

## Penggunaan

```py
import librosa
import torch
from diffusers import ModularPipeline
from diffusers.utils import encode_video, load_image

pipe = ModularPipeline.from_pretrained("kandinskylab/Kandinsky-S2V-1.0-Lite-distill-Diffusers")
pipe.load_components(dtype=torch.bfloat16)
pipe.to("cuda")

image = load_image("reference.png")
audio, _ = librosa.load("speech.wav", sr=16000)  # a mono waveform at 16 kHz, for the model
speech, sample_rate = librosa.load("speech.wav", sr=None)  # the original track, added to the output video

video = pipe(image=image, audio=audio, output="videos")[0]
encode_video(
    video,
    fps=25,
    output_path="avatar.mp4",
    audio=torch.from_numpy(speech)[None],
    audio_sample_rate=sample_rate,
)
```

Checkpoint Pro tidak muat di GPU 80 GB berdampingan dengan encoder ucapan dan teks 7B ketika setiap komponen berada di perangkat. Offload-kan melalui `ComponentsManager`:

```py
import librosa
import torch
from diffusers import ComponentsManager, ModularPipeline
from diffusers.utils import encode_video, load_image

manager = ComponentsManager()
pipe = ModularPipeline.from_pretrained("kandinskylab/Kandinsky-S2V-1.0-Pro-Diffusers", components_manager=manager)
pipe.load_components(dtype=torch.bfloat16)
manager.enable_auto_cpu_offload(device="cuda", memory_reserve_margin="40GiB")

image = load_image("reference.png")
audio, _ = librosa.load("speech.wav", sr=16000)
speech, sample_rate = librosa.load("speech.wav", sr=None)

video = pipe(image=image, audio=audio, output="videos")[0]
encode_video(
    video,
    fps=25,
    output_path="avatar.mp4",
    audio=torch.from_numpy(speech)[None],
    audio_sample_rate=sample_rate,
)
```

Margin adalah memori yang tetap bebas di luar bobot saat model dipindahkan ke GPU. Margin ini harus menampung decode VAE dari sebuah chunk, sekitar 33 GiB pada resolusi 512x768: dengan default 3 GB, transformer tetap berada di GPU selama proses decode, yang kemudian kehabisan memori. Dengan 40 GiB, puncak penggunaan sekitar 50 GiB pada GPU 80 GB. Aktifkan offload setelah `load_components`, sehingga berlaku untuk setiap komponen.

## Catatan

- `image` adalah gambar PIL dan `audio` adalah gelombang mono float dalam [-1, 1] pada 16 kHz, sebagai tensor atau array dengan bentuk `(samples,)`; pipeline tidak membaca file.
- Blok tingkat atas (`text_encoder`, `vae_encoder`, `denoise`, `decode`) juga dapat dijalankan secara mandiri; berikan nilai `num_latent_frames` dan `fps` yang sama ke masing-masing blok saat Anda menjalankannya secara terpisah.
- Jumlah chunk mengikuti panjang rekaman suara kecuali jika `num_chunks` diteruskan. Video dipangkas ke rekaman dalam kondisi apa pun, sehingga chunk yang melampaui akhir rekaman tetap dihitung lalu dibuang, disertai peringatan.
- Dengan panduan, jalur tanpa syarat mengganti prompt negatif *dan* membisukan ucapan, sehingga panduan bersifat gabungan atas teks dan ucapan.

## KandinskyS2VModularPipeline[[diffusers.KandinskyS2VModularPipeline]]

#### diffusers.KandinskyS2VModularPipeline[[diffusers.KandinskyS2VModularPipeline]]

```python
diffusers.KandinskyS2VModularPipeline(blocks: diffusers.modular_pipelines.modular_pipeline.ModularPipelineBlocks | None = None, pretrained_model_name_or_path: str | os.PathLike | None = None, components_manager: diffusers.modular_pipelines.components_manager.ComponentsManager | None = None, collection: str | None = None, workflow: str | None = None, modular_config_dict: dict[str, typing.Any] | None = None, config_dict: dict[str, typing.Any] | None = None, **kwargs)
```

[Sumber](https://github.com/huggingface/diffusers/blob/main/src/diffusers/modular_pipelines/kandinsky_s2v/modular_pipeline.py#L91)

ModularPipeline untuk generasi speech-to-video Kandinsky S2V 1.0.

## KandinskyS2VAutoBlocks[[diffusers.KandinskyS2VAutoBlocks]]

#### diffusers.KandinskyS2VAutoBlocks[[diffusers.KandinskyS2VAutoBlocks]]

```python
diffusers.KandinskyS2VAutoBlocks()
```

[Sumber](https://github.com/huggingface/diffusers/blob/main/src/diffusers/modular_pipelines/kandinsky_s2v/modular_blocks_kandinsky_s2v.py#L108)

Pipeline modular untuk generasi speech-to-video dengan Kandinsky S2V 1.0: video avatar dari frame referensi dan track ucapan, dihasilkan per chunk.

Komponen: text_encoder (`Qwen2_5OmniThinkerForConditionalGeneration`) tokenizer (`Qwen2_5OmniProcessor`) text_encoder_2 (`CLIPTextModel`) tokenizer_2 (`CLIPTokenizer`) vae (`AutoencoderKLHunyuanVideo`) video_processor (`VideoProcessor`) transformer (`KandinskyS2VTransformer3DModel`) scheduler (`FlowMatchEulerDiscreteScheduler`) guider (`ClassifierFreeGuidance`)

Inputs: prompt (`str`, *optional*, defaults to The video presents a stable medium close-up of a subject facing the camera directly. The subject exhibits natural, lifelike body dynamics: the head and shoulders move fluidly with the rhythm of speech. The lighting is even, maintaining a balanced natural exposure to clearly define facial features without washout. Throughout the sequence, the subject engages in continuous speech marked by precise, high-fidelity articulation. The lips move in a rhythmic, natural pattern: alternating between brief firm closures and relaxed openings. During these openings, the tips of the upper teeth are momentarily and realistically visible, exhibiting natural enamel texture. The upper lip lifts subtly over the teeth during enunciation, while the jaw performs minimal vertical oscillations synchronized with the speech cadence. Facial micro-expressions are active yet restrained; eyebrows rise and lower slightly with minute asymmetry. The eyes maintain a steady, engaged gaze forward, punctuated by smooth, deliberate blinks. ): The prompt to guide the generation. negative_prompt (`str`, *optional*): The prompt for the unconditional pass of the guidance. Defaults to `DEFAULT_NEGATIVE_PROMPT` when the guidance needs one. audio (`Tensor | ndarray`): The speech track that drives the generation: a mono float waveform in [-1, 1] at 16000 Hz, shaped (samples,). num_latent_frames (`int`, *optional*, defaults to 32): Latent frames per chunk; 32 gives 125 frames (5 s at 25 fps). fps (`int`, *optional*, defaults to 25): Frame rate the chunk length is computed at. num_chunks (`int`, *optional*): Number of chunks to generate. Defaults to however many cover the speech track; the video is trimmed to the track either way. image (`Image`): The reference frame to animate. height (`int`, *optional*): The height in pixels of the generated video. Defaults to the 512 bucket closest to the reference frame. width (`int`, *optional*): The width in pixels of the generated video, chosen together with `height`. generator (`Generator`, *optional*): Torch generator for deterministic generation. num_inference_steps (`int`, *optional*, defaults to 50): The number of denoising steps. scale_factor (`tuple`, *optional*, defaults to (1.0, 2.0, 2.0)): Per-axis RoPE scale for the resolution bucket. output_type (`str`, *optional*, defaults to pil): Output format: 'pil', 'np', 'pt'.

Output: video (`list`): Video yang dihasilkan.

## KandinskyS2VDistilledModularPipeline[[diffusers.KandinskyS2VDistilledModularPipeline]]

#### diffusers.KandinskyS2VDistilledModularPipeline[[diffusers.KandinskyS2VDistilledModularPipeline]]

```python
diffusers.KandinskyS2VDistilledModularPipeline(blocks: diffusers.modular_pipelines.modular_pipeline.ModularPipelineBlocks | None = None, pretrained_model_name_or_path: str | os.PathLike | None = None, components_manager: diffusers.modular_pipelines.components_manager.ComponentsManager | None = None, collection: str | None = None, workflow: str | None = None, modular_config_dict: dict[str, typing.Any] | None = None, config_dict: dict[str, typing.Any] | None = None, **kwargs)
```

[Sumber](https://github.com/huggingface/diffusers/blob/main/src/diffusers/modular_pipelines/kandinsky_s2v/modular_pipeline.py#L109)

ModularPipeline untuk checkpoint pi-Flow distilled Kandinsky S2V 1.0.

Guidance didistilkan ke dalam student, sehingga blok-bloknya tidak memiliki guider dan tidak memiliki negative prompt, dan satu forward pass dijalankan per segmen.

## KandinskyS2VDistilledAutoBlocks[[diffusers.KandinskyS2VDistilledAutoBlocks]]

#### diffusers.KandinskyS2VDistilledAutoBlocks[[diffusers.KandinskyS2VDistilledAutoBlocks]]

```python
diffusers.KandinskyS2VDistilledAutoBlocks()
```

[Sumber](https://github.com/huggingface/diffusers/blob/main/src/diffusers/modular_pipelines/kandinsky_s2v/modular_blocks_kandinsky_s2v_distilled.py#L103)

Pipeline modular untuk generasi speech-to-video dengan checkpoint pi-Flow distilled Kandinsky S2V 1.0: 16 segmen per chunk dan tanpa guidance, yang di-distill ke dalam student.

Komponen: text_encoder (`Qwen2_5OmniThinkerForConditionalGeneration`) tokenizer (`Qwen2_5OmniProcessor`) text_encoder_2 (`CLIPTextModel`) tokenizer_2 (`CLIPTokenizer`) vae (`AutoencoderKLHunyuanVideo`) video_processor (`VideoProcessor`) transformer (`KandinskyS2VTransformer3DModel`) scheduler (`PiflowScheduler`)

Inputs: prompt (`str`, *optional*, defaults to The video presents a stable medium close-up of a subject facing the camera directly. The subject exhibits natural, lifelike body dynamics: the head and shoulders move fluidly with the rhythm of speech. The lighting is even, maintaining a balanced natural exposure to clearly define facial features without washout. Throughout the sequence, the subject engages in continuous speech marked by precise, high-fidelity articulation. The lips move in a rhythmic, natural pattern: alternating between brief firm closures and relaxed openings. During these openings, the tips of the upper teeth are momentarily and realistically visible, exhibiting natural enamel texture. The upper lip lifts subtly over the teeth during enunciation, while the jaw performs minimal vertical oscillations synchronized with the speech cadence. Facial micro-expressions are active yet restrained; eyebrows rise and lower slightly with minute asymmetry. The eyes maintain a steady, engaged gaze forward, punctuated by smooth, deliberate blinks. ): The prompt to guide the generation. audio (`Tensor | ndarray`): The speech track that drives the generation: a mono float waveform in [-1, 1] at 16000 Hz, shaped (samples,). num_latent_frames (`int`, *optional*, defaults to 32): Latent frames per chunk; 32 gives 125 frames (5 s at 25 fps). fps (`int`, *optional*, defaults to 25): Frame rate the chunk length is computed at. num_chunks (`int`, *optional*): Number of chunks to generate. Defaults to however many cover the speech track; the video is trimmed to the track either way. image (`Image`): The reference frame to animate. height (`int`, *optional*): The height in pixels of the generated video. Defaults to the 512 bucket closest to the reference frame. width (`int`, *optional*): The width in pixels of the generated video, chosen together with `height`. generator (`Generator`, *optional*): Torch generator for deterministic generation. num_inference_steps (`int`, *optional*, defaults to 16): The number of denoising steps. scale_factor (`tuple`, *optional*, defaults to (1.0, 2.0, 2.0)): Per-axis RoPE scale for the resolution bucket. output_type (`str`, *optional*, defaults to pil): Output format: 'pil', 'np', 'pt'.

Output: video (`list`): Video yang dihasilkan.