# VLLM Launch Configs

Ready-to-use VLLM launch scripts for tool calling with open source models.

## Quick Reference

| Script | Model | Parser | Memory | Speed | Best For |
|--------|-------|--------|--------|-------|----------|
| `hermes3_llama31_70b_fp8.sh` | Hermes-3 70B | `hermes` | ~40GB | 25-35 tok/s | Best tool calling quality |
| `llama33_70b_fp8.sh` | Llama-3.3 70B | `llama3_json` | ~40GB | 60-90 tok/s | Open WebUI compatibility |
| `qwen2_72b_fp8.sh` | Qwen2 72B | `hermes` | ~45GB | 60-90 tok/s | Multilingual + reasoning |
| `mistral_nemo_12b_fp8.sh` | Mistral-Nemo 12B | `mistral` | ~15GB | 100-150 tok/s | Fast iteration |

## Usage

```bash
# Make executable
chmod +x configs/hermes3_llama31_70b_fp8.sh

# Launch
./configs/hermes3_llama31_70b_fp8.sh
```

## Customization

All scripts are designed for single-GPU setups. To adapt:

- **Multi-GPU:** Change `--tensor-parallel-size` and `CUDA_VISIBLE_DEVICES`
- **Different port:** Change `--port 8000`
- **Less VRAM:** Reduce `--max-model-len` (but keep above 32K for tool calling)
- **Non-Blackwell GPU:** Remove the `VLLM_ATTENTION_BACKEND` and `VLLM_USE_FLASHINFER` exports

## Critical: Context Length

All configs use 128K context. **Do not reduce below 32K** for tool calling workloads. See [../guides/CONTEXT_LENGTH_FIX.md](../guides/CONTEXT_LENGTH_FIX.md) for why.