---
license: apache-2.0
library_name: lclm
base_model:
  - Qwen/Qwen3-Embedding-0.6B
  - Qwen/Qwen3-4B-Instruct-2507
tags:
  - latent-context-language-model
  - context-compression
  - long-context
---

# 0.6b-4b LCLM, 4× compression

Latent Context Language Model: an encoder–decoder compressor described in
*End-to-End Context Compression at Scale*
([arXiv](https://arxiv.org/abs/2606.09659)).

The text to compress should be wrapped between `<|memory_start|>` and
`<|memory_end|>`.

> Running these checkpoints requires the LCLM codebase:
> <https://github.com/LeonLixyz/LCLM>. Standard `transformers.AutoModel` /
> `vllm.LLM` will not load this format on its own.

## Versions

- **v1.1** (current, `main`): improves the agent with latent context and
  online latent compaction. The SFT mixture adds tool-use trajectories and
  synthetic trajectories that call an `expand` tool on compressed documents.
  This is the checkpoint evaluated in the current version of the paper.
- **v1.0**: the first release (May 2026). Load it with
  `LCLM.from_pretrained("latent-context/0.6b-4b-LCLM-4x", revision="v1.0")`.

## Quick load

```python
from latent_context import LCLM

model = LCLM.from_pretrained("latent-context/0.6b-4b-LCLM-4x")

prompt = (
    "<|memory_start|>"
    "<long document, code, or text to compress>"
    "<|memory_end|> "
    "Summarize the document above."
)
# model.generate(...): see inference/hf.py in the codebase
```

For faster encoding, `inference/optimized_encoder.py` adds FlashAttention,
`torch.compile` and CUDA graphs to the encoder (inference only).

## vLLM serving (two-stage CLI)

The vLLM path runs the encoder and the decoder in **separate
processes** that hand off via a `.pt` file on disk. Running both in
one process OOMs: vLLM grabs all GPU memory at init, leaving none
for the HF encoder.

```bash
# Step 1: HF encoder over a jsonl of prompts → embeds.pt
python -m inference.vllm_inference.encode \
    --checkpoint latent-context/0.6b-4b-LCLM-4x \
    --prompts-jsonl prompts.jsonl --out embeds.pt

# Step 2: vLLM decoder reads embeds.pt → completions.jsonl
python -m inference.vllm_inference.decode \
    --checkpoint latent-context/0.6b-4b-LCLM-4x \
    --embeds-pt embeds.pt --out completions.jsonl
```

See `inference/examples/README.md` in the codebase for the
`prompts.jsonl` schema and an end-to-end RULER NIAH eval driver.

## Configuration

| field | value |
|---|---|
| encoder | Qwen/Qwen3-Embedding-0.6B |
| decoder | Qwen/Qwen3-4B-Instruct-2507 |
| compression_ratio | 4 |
| encoder_window_size | 1024 |
| pooling | mean |
| encoder_mask_type | causal |
| boundary_overlap | 0 |
| adapter_type | mlp |

Code: <https://github.com/LeonLixyz/LCLM>