README.md

---
license: other
license_name: mixed-see-model-card
pipeline_tag: image-to-image
library_name: diffusers
base_model: black-forest-labs/FLUX.2-klein-base-4B
tags:
  - sar-to-eo
  - remote-sensing
  - synthetic-aperture-radar
  - flow-matching
  - image-to-image
  - lora
  - arxiv:2609.37496
---

# GeoSET

GeoSET is a generalist foundation model for SAR-to-EO image translation (SET). A pretrained text-to-image flow
transformer is repurposed for SAR conditioning: the text stream is replaced by a spatial SAR stream fed by a
speckle-robust SAR encoder, and the generator is pretrained on 3,204,744 curated SAR–EO pairs spanning diverse
sensors, spatial resolutions and ground sampling distances. The resulting parent is adapted to downstream
datasets by low-rank adaptation (LoRA), updating 0.60% of the generator parameters, or by full fine-tuning.

- **Paper:** https://arxiv.org/abs/2609.37496
- **Code:** https://github.com/KAIST-VICLab/GeoSET
- **Project page:** https://kaist-viclab.github.io/GeoSET_site/
- **Comparison-method weights:** [`baselines/`](https://huggingface.co/JeonghyeokDo/GeoSET/tree/main/baselines) in this repository: the sixteen prior methods of the paper's Tables 2 and 3 on the four public benchmarks

## Files

| Folder | Content | Size | Licence |
|---|---|---:|---|
| [`generalist/transformer`](https://huggingface.co/JeonghyeokDo/GeoSET/tree/main/generalist/transformer) | `GeoSETTransformer2DModel`, Stage 2 generalist (500,000 updates, EMA), 3.852B parameters, float32 | 15.41 GB | CC BY-NC-SA 4.0 |
| [`generalist/sar_encoder`](https://huggingface.co/JeonghyeokDo/GeoSET/tree/main/generalist/sar_encoder) | `GeoSETSAREncoder`, Stage 1 speckle-robust SAR encoder (50,000 updates) | 137.7 MB | CC BY-NC-SA 4.0 |
| [`generalist/vae`](https://huggingface.co/JeonghyeokDo/GeoSET/tree/main/generalist/vae) | `AutoencoderFlux2`, the frozen FLUX.2-klein-base-4B autoencoder converted to float32 | 336.2 MB | Apache-2.0 |
| [`lora/<dataset>`](https://huggingface.co/JeonghyeokDo/GeoSET/tree/main/lora) | one LoRA adapter per benchmark (rank 16, α 16, 23.10M parameters, EMA after 20,000 updates at batch size 16) | 92.4 MB each | CC BY-NC-SA 4.0 |
| [`baselines/<dataset>/<method>`](https://huggingface.co/JeonghyeokDo/GeoSET/tree/main/baselines) | 61 retrained comparison-method checkpoints | – | per method |

`generalist/` is a complete `diffusers` pipeline: `model_index.json`, the three model folders, `scheduler/`
and the pipeline code (`pipeline.py`). The same code files are also at the repository root and under
`src/geoset/` on GitHub. The full fine-tuned models are not released; the code that produces them is on GitHub.

## Usage

```python
import torch
from PIL import Image
from diffusers import DiffusionPipeline
from huggingface_hub import snapshot_download

root = snapshot_download("JeonghyeokDo/GeoSET", allow_patterns=["generalist/*", "lora/sar2opt/*"])
pipe = DiffusionPipeline.from_pretrained(f"{root}/generalist", custom_pipeline=f"{root}/generalist",
                                         torch_dtype=torch.float32).to("cuda")
pipe.load_lora("sar2opt")                                   # optional; resolved from {root}/lora/sar2opt
eo = pipe(Image.open("sar.png"), num_inference_steps=50, guidance_scale=2.0,
          generator=torch.Generator("cuda").manual_seed(20260812)).images[0]
eo.save("eo.png")
```

`custom_pipeline` points at the same folder because the pipeline, model and scheduler classes ship with the
checkpoint. Keep `torch_dtype=torch.float32`; the pipeline casts the autoencoder to bfloat16 itself. The
input is an 8-bit SAR image, read as one channel, or a list of equally sized images; every side must be a
multiple of 16. Images are not resized or cropped inside the pipeline, so SAR2Opt tiles (600 × 600) are
center-cropped to 512 × 512 first. The paper's results use **50 integration steps with two-pass classifier-free guidance of
weight 2.0**.

## LoRA adapters

| Adapter | Benchmark | Test pairs | Input size |
|---|---|---:|---|
| [`lora/qxs-saropt`](https://huggingface.co/JeonghyeokDo/GeoSET/tree/main/lora/qxs-saropt) | QXS-SAROPT | 3,999 | 256 × 256 |
| [`lora/sar2opt`](https://huggingface.co/JeonghyeokDo/GeoSET/tree/main/lora/sar2opt) | SAR2Opt | 627 | center 512 × 512 crop of the 600 × 600 tiles |
| [`lora/sar2eo`](https://huggingface.co/JeonghyeokDo/GeoSET/tree/main/lora/sar2eo) | SAR2EO | first 4,000 of the 21,260 test pairs | 256 × 256 |
| [`lora/spacenet6`](https://huggingface.co/JeonghyeokDo/GeoSET/tree/main/lora/spacenet6) | SpaceNet6 | 495 | 256 × 256 |

`pipe.load_lora(name)` merges one adapter into the transformer weights (`name` is one of the four keys above,
or the path of an adapter directory). Without it, the pipeline runs the pretrained generalist. An adapter is
merged once; load the pipeline again to switch to another one. The frozen train/test lists, the dataset
preparation and `scripts/translate.py`, which translates a whole test split, are in the
[GitHub repository](https://github.com/KAIST-VICLab/GeoSET). Metrics of every checkpoint are in
[MODEL_ZOO.md](https://github.com/KAIST-VICLab/GeoSET/blob/main/MODEL_ZOO.md).

## Training

All stages train on 256 × 256 crops with AdamW and a gradient-norm clip of 1.0.

| | Stage 1 | Stage 2 | Stage 3, LoRA | Stage 3, full FT |
|---|---|---|---|---|
| Trained | SAR encoder | all 3.852B generator parameters | 23.10M adapter parameters (0.60%) | all 3.852B generator parameters |
| Updates × global batch size | 50,000 × 432 | 500,000 × 256 | 20,000 × 16 | 20,000 × 16 |
| Learning rate | 10⁻⁵, 500-update warmup | 10⁻⁴ peak, 1,000-update warmup | 10⁻⁴ | 2 × 10⁻⁵ |
| Hardware, time | 4 × NVIDIA B200, 14.8 h | 4 × NVIDIA B200, 129.3 h | 1 GPU, 0.98 h per dataset | 1 GPU, 1.34 h per dataset |

Stage 1 trains the SAR encoder, initialised from the FLUX.2-klein-base-4B autoencoder encoder, to reconstruct
SAR observations from speckle-perturbed copies through the frozen decoder. Stage 2 initialises the generator
from FLUX.2-klein-base-4B, copies the image-stream parameters into the SAR stream, and learns conditional flow
matching from Gaussian noise to the EO latent; the SAR condition is dropped with probability 0.1 so that
two-pass classifier-free guidance can be used at inference. Stages 2 and 3 keep an EMA with decay 0.9999, and
the released weights are the EMA parameters.

**Training data.** Pretraining: GUSO, TerraMesh, SARLO-80, SAR-1M and 3MOS, selected by the keep tables
published on GitHub (`corpus/keep_v1/`). Adaptation: the training splits of QXS-SAROPT, SAR2Opt, SAR2EO and
SpaceNet6. No dataset is redistributed here.

## Comparison methods, in this same repository

`baselines/` holds the sixteen prior methods of the paper's Tables 2 and 3, retrained on the same training
splits and evaluated on the same test pairs: 61 checkpoints, one folder per dataset and method (Seg-CycleGAN on
SpaceNet6 only), each with its own card and licence. Start at
[`baselines/README.md`](https://huggingface.co/JeonghyeokDo/GeoSET/blob/main/baselines/README.md); the
training and inference code is in
[`baselines/`](https://github.com/KAIST-VICLab/GeoSET/tree/main/baselines) on GitHub.

```bash
hf download JeonghyeokDo/GeoSET --include "baselines/sar2opt/c-diffset/*" --local-dir hf
```

## Licences

**This repository is mixed-licence, so the Hub tag is `other`.**

- **GeoSET weights** (`generalist/transformer`, `generalist/sar_encoder`, `lora/`):
  [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/).
- **Autoencoder** (`generalist/vae`): the FLUX.2-klein-base-4B autoencoder of Black Forest Labs, converted to
  float32, under [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0).
- **Code** (the `.py` files here and the GitHub repository): Apache-2.0. The transformer and autoencoder code is
  adapted from [black-forest-labs/flux2](https://github.com/black-forest-labs/flux2) (Apache-2.0).
- **Comparison methods** (`baselines/`): the terms of the code each was trained with; the texts are in
  [`baselines/licenses/`](https://huggingface.co/JeonghyeokDo/GeoSET/tree/main/baselines/licenses). Check the
  method you intend to use; the repository-level tag is not a substitute.

The generator is initialised from
[FLUX.2-klein-base-4B](https://huggingface.co/black-forest-labs/FLUX.2-klein-base-4B) (Apache-2.0). GeoSET is
not a FLUX product and is not endorsed by Black Forest Labs. See
[LICENSE-WEIGHTS.md](https://github.com/KAIST-VICLab/GeoSET/blob/main/LICENSE-WEIGHTS.md) and
[NOTICE](https://github.com/KAIST-VICLab/GeoSET/blob/main/NOTICE) on GitHub.

## Citation

```bibtex
@article{do2026geoset,
  title={GeoSET: Generalist Foundation Model for SAR-to-EO Image Translation},
  author={Do, Jeonghyeok and Kim, Munchurl},
  journal={arXiv preprint arXiv:2609.37496},
  year={2026}
}
```

Our prior work on SAR-to-EO image translation, [C-DiffSET](https://github.com/KAIST-VICLab/C-DiffSET):

```bibtex
@article{do2026cdiffset,
  title={C-diffset: Leveraging latent diffusion for sar-to-eo image translation with confidence-guided reliable object generation},
  author={Do, Jeonghyeok and Lee, Jaehyup and Lee, Seungchul and Kim, Munchurl},
  journal={IEEE Transactions on Circuits and Systems for Video Technology},
  year={2026},
  publisher={IEEE}
}
```