Diffusers documentation

Flux

Diffusers

Join the Hugging Face community

and get access to the augmented documentation experience

Collaborate on models, datasets and Spaces

Faster examples with accelerated inference

Switch between documentation themes

to get started

Flux

Flux 是一系列基于扩散 Transformer 的文本生成图像模型。如需了解更多关于 Flux 的信息，请参阅 Flux 的创建者 Black Forest Labs 撰写的原始博客文章。

Flux 的原始模型检查点可在此处找到。原始推理代码可在此处找到。

Flux 在消费级硬件设备上运行可能成本较高。不过，你可以执行一系列优化，使其运行得更快，并且对内存更加友好。有关更多详细信息，请参阅本节。此外，Flux 可以通过量化来提升内存效率，但代价是推理延迟会增加。要了解更多信息，请参考这篇博客文章。如需获取详尽的资源列表，请查看此 gist。

缓存还可以通过存储和复用中间输出来加速推理。

Flux 提供以下变体：

模型类型	模型 ID
时间步蒸馏	`black-forest-labs/FLUX.1-schnell`
Guidance-distilled	`black-forest-labs/FLUX.1-dev`
填充 Inpainting/Outpainting（引导蒸馏）	`black-forest-labs/FLUX.1-Fill-dev`
Canny Control（引导蒸馏）	`black-forest-labs/FLUX.1-Canny-dev`
深度控制（引导蒸馏）	`black-forest-labs/FLUX.1-Depth-dev`
Canny Control (LoRA)	`black-forest-labs/FLUX.1-Canny-dev-lora`
深度控制 (LoRA)	`black-forest-labs/FLUX.1-Depth-dev-lora`
Redux（适配器）	`black-forest-labs/FLUX.1-Redux-dev`
Kontext	`black-forest-labs/FLUX.1-kontext`

所有检查点都有不同的用途，我们将在下文详细说明。

时间步蒸馏

max_sequence_length 不能超过 256。
guidance_scale 必须为 0。
由于这是一个时间步蒸馏模型，因此它受益于更少的采样步数。

import torch
from diffusers import FluxPipeline

pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-schnell", dtype=torch.bfloat16)
pipe.enable_model_cpu_offload()

prompt = "A cat holding a sign that says hello world"
out = pipe(
    prompt=prompt,
    guidance_scale=0.,
    height=768,
    width=1360,
    num_inference_steps=4,
    max_sequence_length=256,
).images[0]
out.save("image.png")

Guidance-distilled

经过指导蒸馏的变体大约需要 50 个采样步骤即可生成高质量的内容。
它没有max_sequence_length方面的任何限制。

import torch
from diffusers import FluxPipeline

pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", dtype=torch.bfloat16)
pipe.enable_model_cpu_offload()

prompt = "a tiny astronaut hatching from an egg on the moon"
out = pipe(
    prompt=prompt,
    guidance_scale=3.5,
    height=768,
    width=1360,
    num_inference_steps=50,
).images[0]
out.save("image.png")

填充 Inpainting/Outpainting

Flux Fill 流水线与常规修复流水线不同，不需要将 strength 作为输入。
它同时支持图像修复（inpainting）和图像外扩（outpainting）。

import torch
from diffusers import FluxFillPipeline
from diffusers.utils import load_image

image = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/cup.png")
mask = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/cup_mask.png")

repo_id = "black-forest-labs/FLUX.1-Fill-dev"
pipe = FluxFillPipeline.from_pretrained(repo_id, dtype=torch.bfloat16).to("cuda")  # or "mps", "xpu", "cpu"

image = pipe(
    prompt="a white paper cup",
    image=image,
    mask_image=mask,
    height=1632,
    width=1232,
    max_sequence_length=512,
    generator=torch.Generator("cpu").manual_seed(0)
).images[0]
image.save(f"output.png")

Canny Control

注意： black-forest-labs/Flux.1-Canny-dev 不是 ControlNetModel 模型。ControlNet 模型是独立于 UNet/Transformer 的组件，其残差会被加到实际的基础模型上。Canny Control 是一种替代架构，它通过将输入控制条件进行通道级拼接，并促使 Transformer 尽可能紧密地遵循该条件以学习结构控制，从而在效果上达到与 ControlNet 模型相同的结果。

# !pip install -U controlnet-aux
import torch
from controlnet_aux import CannyDetector
from diffusers import FluxControlPipeline
from diffusers.utils import load_image

pipe = FluxControlPipeline.from_pretrained("black-forest-labs/FLUX.1-Canny-dev", dtype=torch.bfloat16).to("cuda")  # or "mps", "xpu", "cpu"

prompt = "A robot made of exotic candies and chocolates of different kinds. The background is filled with confetti and celebratory gifts."
control_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/robot.png")

processor = CannyDetector()
control_image = processor(control_image, low_threshold=50, high_threshold=200, detect_resolution=1024, image_resolution=1024)

image = pipe(
    prompt=prompt,
    control_image=control_image,
    height=1024,
    width=1024,
    num_inference_steps=50,
    guidance_scale=30.0,
).images[0]
image.save("output.png")

Canny Control 也可以通过该条件的 LoRA 变体实现。用法如下：

# !pip install -U controlnet-aux
import torch
from controlnet_aux import CannyDetector
from diffusers import FluxControlPipeline
from diffusers.utils import load_image

pipe = FluxControlPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", dtype=torch.bfloat16).to("cuda")  # or "mps", "xpu", "cpu"
pipe.load_lora_weights("black-forest-labs/FLUX.1-Canny-dev-lora")

prompt = "A robot made of exotic candies and chocolates of different kinds. The background is filled with confetti and celebratory gifts."
control_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/robot.png")

processor = CannyDetector()
control_image = processor(control_image, low_threshold=50, high_threshold=200, detect_resolution=1024, image_resolution=1024)

image = pipe(
    prompt=prompt,
    control_image=control_image,
    height=1024,
    width=1024,
    num_inference_steps=50,
    guidance_scale=30.0,
).images[0]
image.save("output.png")

深度控制

注意： black-forest-labs/Flux.1-Depth-dev 并非 ControlNet 模型。ControlNetModel 模型是独立于 UNet/Transformer 的组件，其残差会被添加到实际的基础模型中。Depth Control 是一种替代架构，它通过将与输入控制条件的通道级拼接，并确保 Transformer 尽可能紧密地遵循该条件以学习结构控制，从而在效果上达到与 ControlNet 模型相同的结果。

# !pip install git+https://github.com/huggingface/image_gen_aux
import torch
from diffusers import FluxControlPipeline, FluxTransformer2DModel
from diffusers.utils import load_image
from image_gen_aux import DepthPreprocessor

pipe = FluxControlPipeline.from_pretrained("black-forest-labs/FLUX.1-Depth-dev", dtype=torch.bfloat16).to("cuda")  # or "mps", "xpu", "cpu"

prompt = "A robot made of exotic candies and chocolates of different kinds. The background is filled with confetti and celebratory gifts."
control_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/robot.png")

processor = DepthPreprocessor.from_pretrained("LiheYoung/depth-anything-large-hf")
control_image = processor(control_image)[0].convert("RGB")

image = pipe(
    prompt=prompt,
    control_image=control_image,
    height=1024,
    width=1024,
    num_inference_steps=30,
    guidance_scale=10.0,
    generator=torch.Generator().manual_seed(42),
).images[0]
image.save("output.png")

Depth Control 也可以通过该条件的 LoRA 变体实现。用法如下：

# !pip install git+https://github.com/huggingface/image_gen_aux
import torch
from diffusers import FluxControlPipeline, FluxTransformer2DModel
from diffusers.utils import load_image
from image_gen_aux import DepthPreprocessor

pipe = FluxControlPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", dtype=torch.bfloat16).to("cuda")  # or "mps", "xpu", "cpu"
pipe.load_lora_weights("black-forest-labs/FLUX.1-Depth-dev-lora")

prompt = "A robot made of exotic candies and chocolates of different kinds. The background is filled with confetti and celebratory gifts."
control_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/robot.png")

processor = DepthPreprocessor.from_pretrained("LiheYoung/depth-anything-large-hf")
control_image = processor(control_image)[0].convert("RGB")

image = pipe(
    prompt=prompt,
    control_image=control_image,
    height=1024,
    width=1024,
    num_inference_steps=30,
    guidance_scale=10.0,
    generator=torch.Generator().manual_seed(42),
).images[0]
image.save("output.png")

Redux

Flux Redux 流水线是 FLUX.1 基础模型的适配器。它可以与 flux-dev 和 flux-schnell 一起使用，用于图像到图像的生成。
您可以先使用 FluxPriorReduxPipeline 获取 prompt_embeds 和 pooled_prompt_embeds，然后将它们输入到 FluxPipeline 中进行图生图生成。
在使用 FluxPriorReduxPipeline 与基础 pipeline 时，您可以在基础 pipeline 中设置 text_encoder=None 和 text_encoder_2=None，以节省 VRAM。

import torch
from diffusers import FluxPriorReduxPipeline, FluxPipeline
from diffusers.utils import load_image
device = "cuda"  # or "mps", "xpu", "cpu"
dtype = torch.bfloat16


repo_redux = "black-forest-labs/FLUX.1-Redux-dev"
repo_base = "black-forest-labs/FLUX.1-dev" 
pipe_prior_redux = FluxPriorReduxPipeline.from_pretrained(repo_redux, dtype=dtype).to(device)
pipe = FluxPipeline.from_pretrained(
    repo_base, 
    text_encoder=None,
    text_encoder_2=None,
    dtype=torch.bfloat16
).to(device)

image = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/style_ziggy/img5.png")
pipe_prior_output = pipe_prior_redux(image)
images = pipe(
    guidance_scale=2.5,
    num_inference_steps=50,
    generator=torch.Generator("cpu").manual_seed(0),
    **pipe_prior_output,
).images
images[0].save("flux-redux.png")

Kontext

Flux Kontext 是一个支持在图像生成过程中进行上下文控制的模型，可实现编辑、优化、重新打光、风格迁移、角色定制等功能。

import torch
from diffusers import FluxKontextPipeline
from diffusers.utils import load_image

pipe = FluxKontextPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-Kontext-dev", dtype=torch.bfloat16
)
pipe.to("cuda")  # or "mps", "xpu", "cpu"

image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/yarn-art-pikachu.png").convert("RGB")
prompt = "Make Pikachu hold a sign that says 'Black Forest Labs is awesome', yarn art style, detailed, vibrant colors"
image = pipe(
    image=image,
    prompt=prompt,
    guidance_scale=2.5,
    generator=torch.Generator().manual_seed(42),
).images[0]
image.save("flux-kontext.png")

Flux Kontext 附带一个完整性安全检查器，应在图像生成步骤之后运行。要运行安全检查器，请从 black-forest-labs/flux 安装官方仓库，并添加以下代码：

from flux.content_filters import PixtralContentFilter

# ... pipeline invocation to generate images

integrity_checker = PixtralContentFilter(torch.device("cuda"))
image_ = np.array(image) / 255.0
image_ = 2 * image_ - 1
image_ = torch.from_numpy(image_).to("cuda", dtype=torch.float32).unsqueeze(0).permute(0, 3, 1, 2)
if integrity_checker.test_image(image_):
    raise ValueError("Your image has been flagged. Choose another prompt/image or try again.")

Kontext 图像修复

FluxKontextInpaintPipeline 支持在固定掩码区域内修改图像。它目前同时支持基于文本的条件控制和基于图像参考的条件控制。

仅文本

图像条件

将 Flux Turbo LoRA 与 Flux Control、Fill 和 Redux 结合使用

我们可以将 Flux Turbo LoRA 与 Flux Control 以及其他流水线（如 Fill 和 Redux）结合使用，以实现少步推理。下面的示例展示了如何针对深度 Flux Control LoRA 以及来自 ByteDance/Hyper-SD 的 turbo LoRA 进行操作。

from diffusers import FluxControlPipeline
from image_gen_aux import DepthPreprocessor
from diffusers.utils import load_image
from huggingface_hub import hf_hub_download
import torch

control_pipe = FluxControlPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", dtype=torch.bfloat16)
control_pipe.load_lora_weights("black-forest-labs/FLUX.1-Depth-dev-lora", adapter_name="depth")
control_pipe.load_lora_weights(
    hf_hub_download("ByteDance/Hyper-SD", "Hyper-FLUX.1-dev-8steps-lora.safetensors"), adapter_name="hyper-sd"
)
control_pipe.set_adapters(["depth", "hyper-sd"], adapter_weights=[0.85, 0.125])
control_pipe.enable_model_cpu_offload()

prompt = "A robot made of exotic candies and chocolates of different kinds. The background is filled with confetti and celebratory gifts."
control_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/robot.png")

processor = DepthPreprocessor.from_pretrained("LiheYoung/depth-anything-large-hf")
control_image = processor(control_image)[0].convert("RGB")

image = control_pipe(
    prompt=prompt,
    control_image=control_image,
    height=1024,
    width=1024,
    num_inference_steps=8,
    guidance_scale=10.0,
    generator=torch.Generator().manual_seed(42),
).images[0]
image.save("output.png")

关于在使用 Flux LoRA 时调用 unload_lora_weights() 的说明

在卸载 Control LoRA 权重时，请调用 pipe.unload_lora_weights(reset_to_overwritten_params=True)，将 pipe.transformer 完全重置为其原始状态。随后，生成的 pipeline 即可与 DiffusionPipeline.from_pipe() 等方法配合使用。有关此参数的更多详细信息，请参阅此 PR。

IP-Adapter

查看 IP-Adapter 以了解 IP-Adapter 的工作原理。

IP-Adapter 允许你在文本提示之外，使用图像来提示 Flux。当需要描述仅靠文字难以准确表达的复杂概念，且手头有参考图像时，这一功能尤为实用。

import torch
from diffusers import FluxPipeline
from diffusers.utils import load_image

pipe = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev", dtype=torch.bfloat16
).to("cuda")  # or "mps", "xpu", "cpu"

image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/flux_ip_adapter_input.jpg").resize((1024, 1024))

pipe.load_ip_adapter(
    "XLabs-AI/flux-ip-adapter",
    weight_name="ip_adapter.safetensors",
    image_encoder_pretrained_model_name_or_path="openai/clip-vit-large-patch14"
)
pipe.set_ip_adapter_scale(1.0)

image = pipe(
    width=1024,
    height=1024,
    prompt="wearing sunglasses",
    negative_prompt="",
    true_cfg_scale=4.0,
    generator=torch.Generator().manual_seed(4444),
    ip_adapter_image=image,
).images[0]

image.save('flux_ip_adapter_output.jpg')

使用提示词“wearing sunglasses”的 IP-Adapter 示例

优化

Flux 是一个非常大的模型，加载所有建模组件大约需要 50GB 的 RAM/VRAM。启用以下部分优化以降低内存需求。

组卸载

Group offloading 通过卸载内部层的分组（而非整个模型或权重）来降低 VRAM 使用量。你需要在 pipeline 的所有模型组件上调用 apply_group_offloading()。offload_type 参数允许你在 block 级别和 leaf 级别的卸载之间切换。将其设置为 leaf_level 会将最低层级的 leaf 级参数卸载到 CPU，而不是在 module 级别进行卸载。

在支持异步数据流传输的 CUDA 设备上，设置 use_stream=True 以重叠数据传输和计算，从而加速推理。

可以在流水线中为不同组件混合使用块级和叶级卸载。

import torch
from diffusers import FluxPipeline
from diffusers.hooks import apply_group_offloading

model_id = "black-forest-labs/FLUX.1-dev"
dtype = torch.bfloat16
pipe = FluxPipeline.from_pretrained(
	model_id,
	dtype=dtype,
)

apply_group_offloading(
    pipe.transformer,
    offload_type="leaf_level",
    offload_device=torch.device("cpu"),
    onload_device=torch.device("cuda"),
    use_stream=True,
)
apply_group_offloading(
    pipe.text_encoder, 
    offload_device=torch.device("cpu"),
    onload_device=torch.device("cuda"),
    offload_type="leaf_level",
    use_stream=True,
)
apply_group_offloading(
    pipe.text_encoder_2, 
    offload_device=torch.device("cpu"),
    onload_device=torch.device("cuda"),
    offload_type="leaf_level",
    use_stream=True,
)
apply_group_offloading(
    pipe.vae, 
    offload_device=torch.device("cpu"),
    onload_device=torch.device("cuda"),
    offload_type="leaf_level",
    use_stream=True,
)

prompt="A cat wearing sunglasses and working as a lifeguard at pool."

generator = torch.Generator().manual_seed(181201)
image = pipe(
    prompt,
    width=576,
    height=1024,
    num_inference_steps=30,
    generator=generator
).images[0]
image

运行 FP16 推理

Flux 可以使用 FP16 生成高质量图像（例如，以加速在 Turing/Volta GPU 上的推理），但与 FP32/BF16 相比，其输出会有所不同。问题在于，在 FP16 模式下运行时，文本编码器中的某些激活值需要进行裁剪，这会影响整体图像。因此，强制文本编码器以 FP32 进行推理可以消除这种输出差异。详情请参见此处。

FP16 推理代码：

import torch
from diffusers import FluxPipeline

pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-schnell", dtype=torch.bfloat16) # can replace schnell with dev
# to run on low vram GPUs (i.e. between 4 and 32 GB VRAM)
pipe.enable_sequential_cpu_offload()
pipe.vae.enable_slicing()
pipe.vae.enable_tiling()

pipe.to(torch.float16) # casting here instead of in the pipeline constructor because doing so in the constructor loads all models into CPU memory at once

prompt = "A cat holding a sign that says hello world"
out = pipe(
    prompt=prompt,
    guidance_scale=0.,
    height=768,
    width=1360,
    num_inference_steps=4,
    max_sequence_length=256,
).images[0]
out.save("image.png")

量化

量化通过以较低精度的数据类型存储模型权重，有助于降低超大模型的内存需求。不过，量化对视频质量的影响会因视频模型的不同而有所差异。

请参阅量化概述，以了解支持的量化后端以及如何选择支持您使用场景的量化后端。下面的示例演示了如何使用 bitsandbytes 加载用于推理的量化 FluxPipeline。

import torch
from diffusers import BitsAndBytesConfig as DiffusersBitsAndBytesConfig, FluxTransformer2DModel, FluxPipeline
from transformers import BitsAndBytesConfig as BitsAndBytesConfig, T5EncoderModel

quant_config = BitsAndBytesConfig(load_in_8bit=True)
text_encoder_8bit = T5EncoderModel.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    subfolder="text_encoder_2",
    quantization_config=quant_config,
    dtype=torch.float16,
)

quant_config = DiffusersBitsAndBytesConfig(load_in_8bit=True)
transformer_8bit = FluxTransformer2DModel.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    subfolder="transformer",
    quantization_config=quant_config,
    dtype=torch.float16,
)

pipeline = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    text_encoder_2=text_encoder_8bit,
    transformer=transformer_8bit,
    dtype=torch.float16,
    device_map="balanced",
)

prompt = "a tiny astronaut hatching from an egg on the moon"
image = pipeline(prompt, guidance_scale=3.5, height=768, width=1360, num_inference_steps=50).images[0]
image.save("flux.png")

FluxTransformer2DModel 的单文件加载

FluxTransformer2DModel 支持加载 Black Forest Labs 提供的原始格式检查点。当尝试加载社区发布的模型微调版本或量化版本时，这一功能同样非常有用。

FP8 推理的稳定性可能因您使用的 GPU 类型、CUDA 版本以及 torch 版本而异。建议您在机器上使用 optimum-quanto 库来运行 FP8 推理。

以下示例演示了如何在显存低于 16GB 的情况下运行 Flux。

首先安装 optimum-quanto

pip install optimum-quanto

然后运行以下示例

import torch
from diffusers import FluxTransformer2DModel, FluxPipeline
from transformers import T5EncoderModel, CLIPTextModel
from optimum.quanto import freeze, qfloat8, quantize

bfl_repo = "black-forest-labs/FLUX.1-dev"
dtype = torch.bfloat16

transformer = FluxTransformer2DModel.from_single_file("https://huggingface.co/Kijai/flux-fp8/blob/main/flux1-dev-fp8.safetensors", dtype=dtype)
quantize(transformer, weights=qfloat8)
freeze(transformer)

text_encoder_2 = T5EncoderModel.from_pretrained(bfl_repo, subfolder="text_encoder_2", dtype=dtype)
quantize(text_encoder_2, weights=qfloat8)
freeze(text_encoder_2)

pipe = FluxPipeline.from_pretrained(bfl_repo, transformer=None, text_encoder_2=None, dtype=dtype)
pipe.transformer = transformer
pipe.text_encoder_2 = text_encoder_2

pipe.enable_model_cpu_offload()

prompt = "A cat holding a sign that says hello world"
image = pipe(
    prompt,
    guidance_scale=3.5,
    output_type="pil",
    num_inference_steps=20,
    generator=torch.Generator("cpu").manual_seed(0)
).images[0]

image.save("flux-fp8-dev.png")

Diffusers

Flux

时间步蒸馏

Guidance-distilled

填充 Inpainting/Outpainting

Canny Control

深度控制

Redux

Kontext

Kontext 图像修复

将 Flux Turbo LoRA 与 Flux Control、Fill 和 Redux 结合使用

关于在使用 Flux LoRA 时调用 unload_lora_weights() 的说明

IP-Adapter

优化

组卸载

运行 FP16 推理

量化

FluxTransformer2DModel 的单文件加载

FluxPipeline

class diffusers.FluxPipeline

__call__

encode_prompt

FluxImg2ImgPipeline

class diffusers.FluxImg2ImgPipeline

__call__

encode_prompt

FluxInpaintPipeline

class diffusers.FluxInpaintPipeline

__call__

encode_prompt

FluxControlNetInpaintPipeline

类 diffusers.FluxControlNetInpaintPipeline

__call__

encode_prompt

FluxControlNetImg2ImgPipeline

class diffusers.FluxControlNetImg2ImgPipeline

__call__

encode_prompt

FluxControlPipeline

class diffusers.FluxControlPipeline

__call__

encode_prompt

FluxControlImg2ImgPipeline

类 diffusers.FluxControlImg2ImgPipeline

__call__

encode_prompt

FluxPriorReduxPipeline

class diffusers.FluxPriorReduxPipeline

__call__

encode_prompt

FluxFillPipeline

class diffusers.FluxFillPipeline

__call__

encode_prompt

FluxKontextPipeline

class diffusers.FluxKontextPipeline

__call__

encode_prompt

FluxKontextInpaintPipeline

class diffusers.FluxKontextInpaintPipeline

__call__

仅使用文本进行图像修复

使用图像条件进行图像修复

encode_prompt

call

call

call

call

call

call

call

call

call

call

call