Qwen-Image-SynthSat-v1.0-warmup-stage2

The idea of the "warmup stages" is to sort of pretrain the LoRA weights so that they can be further trained with more refined data, but maybe have a boost in overall knowledge and stuff.
This stage is 640p, re-using the weights from stage1.
Training Stages

SynthSat-v1.0-warmup-stage1 is 256p only, using freshly initialized weights.
SynthSat-v1.0-warmup-stage2 is 640p only, re-using the weights from SynthSat-v1.0-warmup-stage1.
SynthSat-v1.0-warmup-stage2.5-nsfw is 256p+640p, re-using the weights from SynthSat-v1.0-warmup-stage2-step1400, using only NSFW sources (which is a mix of NSFW+SFW).
Next stage is either warmup-stage3 doing 640p+1328p or v1.0 doing 640p+1328p+1536p on a more refined dataset.
Current Stage Details

INFO:musubi_tuner.hv_train_network:Using timestep bucketing. Number of buckets: 10
INFO:musubi_tuner.hv_train_network:Load dataset config from /media/xzuyn/Toshiba1/musubi-stuff/dataset_configs/SynthSat-v1.0-warmup-stage2.toml
INFO:musubi_tuner.dataset.image_video_dataset:glob images in /media/xzuyn/Toshiba1/000_SynthSatWarmup
INFO:musubi_tuner.dataset.image_video_dataset:found 27666 images
INFO:musubi_tuner.dataset.config_utils:[Dataset 0]
  is_image_dataset: True
  resolution: (640, 640)
  batch_size: 8
  num_repeats: 1
  caption_extension: ".txt"
  enable_bucket: True
  bucket_no_upscale: True
  cache_directory: "/media/xzuyn/NVMe/LClones/musubi-tuner/dataset_cache/SynthSat-v1.0-warmup-stage2-640"
  debug_dataset: False
    image_directory: "/media/xzuyn/Toshiba1/000_SynthSatWarmup"
    image_jsonl_file: "None"
    fp_latent_window_size: 9
    fp_1f_clean_indices: None
    fp_1f_target_index: None
    fp_1f_no_post: False
    flux_kontext_no_resize_control: False
    qwen_image_edit_no_resize_control: False
    qwen_image_edit_control_resolution: None
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (304, 416), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (320, 320), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (320, 384), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (320, 576), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (320, 720), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (336, 1216), count: 2
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (352, 480), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (352, 640), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (352, 1152), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (368, 240), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (368, 944), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (384, 512), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (384, 544), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (384, 1056), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (400, 400), count: 2
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (400, 528), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (400, 592), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (400, 736), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (400, 848), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (400, 1024), count: 6
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (416, 560), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (416, 688), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (416, 976), count: 18
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (432, 512), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (432, 592), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (432, 640), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (432, 944), count: 93
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (448, 272), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (448, 288), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (448, 608), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (448, 672), count: 2
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (448, 912), count: 53
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (464, 336), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (464, 464), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (464, 704), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (464, 800), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (464, 816), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (464, 880), count: 100
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (480, 256), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (480, 304), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (480, 480), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (480, 560), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (480, 624), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (480, 640), count: 3
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (480, 688), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (480, 736), count: 2
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (480, 848), count: 1011
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (496, 320), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (496, 368), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (496, 656), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (496, 672), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (496, 816), count: 185
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (512, 272), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (512, 512), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (512, 592), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (512, 688), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (512, 704), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (512, 720), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (512, 736), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (512, 768), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (512, 800), count: 815
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (528, 384), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (528, 592), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (528, 624), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (528, 672), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (528, 704), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (528, 768), count: 5636
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (544, 400), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (544, 544), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (544, 672), count: 4
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (544, 752), count: 5271
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (560, 416), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (560, 512), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (560, 528), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (560, 560), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (560, 640), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (560, 672), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (560, 704), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (560, 720), count: 1000
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (576, 368), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (576, 384), count: 2
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (576, 432), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (576, 464), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (576, 496), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (576, 576), count: 2
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (576, 704), count: 2362
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (592, 304), count: 25
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (592, 384), count: 4
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (592, 400), count: 3
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (592, 432), count: 2
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (592, 448), count: 2
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (592, 528), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (592, 592), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (592, 688), count: 252
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (608, 336), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (608, 368), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (608, 384), count: 2
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (608, 400), count: 4
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (608, 480), count: 2
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (608, 608), count: 2
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (608, 624), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (608, 672), count: 182
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (624, 384), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (624, 416), count: 6
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (624, 448), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (624, 480), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (624, 624), count: 3
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (624, 640), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (624, 656), count: 156
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (640, 352), count: 3
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (640, 400), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (640, 416), count: 13
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (640, 432), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (640, 448), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (640, 480), count: 2
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (640, 624), count: 2
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (640, 640), count: 1069
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (656, 416), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (656, 432), count: 4
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (656, 624), count: 109
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (672, 448), count: 9
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (672, 464), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (672, 496), count: 2
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (672, 608), count: 96
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (688, 384), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (688, 416), count: 2
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (688, 432), count: 3
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (688, 464), count: 7
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (688, 512), count: 2
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (688, 544), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (688, 560), count: 2
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (688, 592), count: 123
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (704, 384), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (704, 400), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (704, 448), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (704, 464), count: 3
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (704, 496), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (704, 512), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (704, 576), count: 418
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (720, 336), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (720, 400), count: 4
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (720, 432), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (720, 448), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (720, 464), count: 4
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (720, 480), count: 7
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (720, 496), count: 2
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (720, 512), count: 2
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (720, 528), count: 5
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (720, 560), count: 2085
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (736, 336), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (736, 400), count: 3
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (736, 432), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (736, 464), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (736, 480), count: 3
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (736, 496), count: 6
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (736, 512), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (736, 528), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (752, 288), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (752, 416), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (752, 480), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (752, 496), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (752, 544), count: 368
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (768, 416), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (768, 432), count: 2
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (768, 512), count: 4
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (768, 528), count: 4748
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (800, 400), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (800, 448), count: 2
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (800, 512), count: 383
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (816, 448), count: 2
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (816, 480), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (816, 496), count: 123
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (832, 464), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (848, 464), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (848, 480), count: 497
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (880, 464), count: 67
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (912, 448), count: 52
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (944, 432), count: 64
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (976, 416), count: 12
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (1024, 400), count: 8
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (1056, 384), count: 12
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (1104, 368), count: 9
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (1152, 352), count: 1
INFO:musubi_tuner.dataset.image_video_dataset:bucket: (1280, 320), count: 4
INFO:musubi_tuner.dataset.image_video_dataset:total batches: 3593
INFO:musubi_tuner.hv_train_network:preparing accelerator
accelerator device: cuda
INFO:musubi_tuner.hv_train_network:DiT precision: torch.bfloat16, weight precision: None
INFO:musubi_tuner.hv_train_network:Loading DiT model from /media/xzuyn/NVMe/LClones/musubi-tuner/source_models/qwen_image_bf16.safetensors
INFO:musubi_tuner.qwen_image.qwen_image_model:Creating QwenImageTransformer2DModel
INFO:musubi_tuner.qwen_image.qwen_image_model:Loading DiT model from /media/xzuyn/NVMe/LClones/musubi-tuner/source_models/qwen_image_bf16.safetensors, device=cpu
INFO:musubi_tuner.utils.lora_utils:Loading model files: ['/media/xzuyn/NVMe/LClones/musubi-tuner/source_models/qwen_image_bf16.safetensors']
INFO:musubi_tuner.utils.lora_utils:Loading state dict with FP8 optimization. Dtype of weight: None, hook enabled: False
Loading qwen_image_bf16.safetensors: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1933/1933 [02:46<00:00, 11.59key/s]
INFO:musubi_tuner.modules.fp8_optimization_utils:Number of optimized Linear layers: 840
INFO:musubi_tuner.modules.fp8_optimization_utils:Number of monkey-patched Linear layers: 840
INFO:musubi_tuner.qwen_image.qwen_image_model:Loaded DiT model from /media/xzuyn/NVMe/LClones/musubi-tuner/source_models/qwen_image_bf16.safetensors, info=<All keys matched successfully>
INFO:musubi_tuner.hv_train_network:enable swap 12 blocks to CPU from device: cuda
QwenModel: Block swap enabled. Swapping 12 blocks out of 60 blocks. Supports backward: True
import network module: networks.lora_qwen_image
INFO:musubi_tuner.networks.lora:create LoRA network. base dim (rank): 16, alpha: 4.0
INFO:musubi_tuner.networks.lora:neuron dropout: p=0.125, rank dropout: p=0.0, module dropout: p=0.0
INFO:musubi_tuner.networks.lora:create LoRA for U-Net/DiT: 840 modules.
INFO:musubi_tuner.networks.lora:enable LoRA for U-Net: 840 modules
load network weights from /media/xzuyn/NVMe/LClones/musubi-tuner/source_models/Qwen-Image-SynthSat-v1.0-warmup-stage1.safetensors: <All keys matched successfully>
QwenModel: Gradient checkpointing enabled. Activation CPU offloading: True
prepare optimizer, data loader etc.
INFO:musubi_tuner.hv_train_network:use came_pytorch.CAME | {'weight_decay': 0.01, 'enable_8bit': True, 'enable_cautious': True, 'enable_stochastic_rounding': True}
==== CAME Modifications ====
- Stochastic Rounding enabled.
- Cautious Masking enabled.
- 8-bit enabled: block_size=2048, min_8bit_size=16384.
==== CAME Modifications ====
override steps. steps for 1 epochs is / 指定エポックまでのステップ数: 3593
INFO:musubi_tuner.hv_train_network:preparing fused backward pass stuff
running training / 学習開始
  num train items / 学習画像、動画数: 27666
  num batches per epoch / 1epochのバッチ数: 3593
  num epochs / epoch数: 1
  batch size per device / バッチサイズ: 8
  gradient accumulation steps / 勾配を合計するステップ数 = 1
  total optimization steps / 学習ステップ数: 3593
INFO:musubi_tuner.hv_train_network:set DiT model name for metadata: /media/xzuyn/NVMe/LClones/musubi-tuner/source_models/qwen_image_bf16.safetensors
INFO:musubi_tuner.hv_train_network:set VAE model name for metadata: /media/xzuyn/NVMe/LClones/musubi-tuner/source_models/vae_diffusion_pytorch_model.safetensors
Downloads last month: -; Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for PJMixers-Images/Qwen-Image-SynthSat-v1.0-warmup-stage2

Base model
Qwen/Qwen-Image
Adapter
(385)
this model
Collection including PJMixers-Images/Qwen-Image-SynthSat-v1.0-warmup-stage2

Qwen-Image LoRAs

Collection
10 items • Updated 6 days ago