---
license: mit
tags:
- sensor-language-action
- physiological-signals
- clinical
- multimodal
- time-series
pipeline_tag: question-answering
library_name: pytorch
language:
- en
base_model:
- Qwen/Qwen3.5-2B
---

# Sensor-Language-Action Models

[![Paper](https://img.shields.io/badge/paper-arXiv-red)](https://arxiv.org/abs/2610.08244)
[![Webpage](https://img.shields.io/badge/website-project-blue)](https://yang-ai-lab.github.io/OpenSLA/)
[![Code](https://img.shields.io/badge/code-GitHub-black)](https://github.com/yang-ai-lab/OpenSLA)
[![License](https://img.shields.io/badge/license-MIT-green)](https://github.com/yang-ai-lab/Sensor-Language-Action-Model/blob/main/LICENSE)
[![Python](https://img.shields.io/badge/python-3.10%2B-brightgreen)](#-installation)


## 🔥 News

- **[2026-10-06]** Our paper is available on [arXiv](https://arxiv.org/abs/2610.08244).
- **[2026-10-06]** Code released on [GitHub](https://github.com/yang-ai-lab/OpenSLA), and model released on [HuggingFace](https://huggingface.co/yang-ai-lab/OpenSLA).
- **[2026-10-06]** [Project website](https://yang-ai-lab.github.io/OpenSLA/) is live!

## 📖 Introduction

**OpenSLA** is a new family of models - *Sensor-Langauge-Action* (SLA) models. It takes a multi-channel sensor history together with language context and
produces a structured action prediction and a sensor-state caption. We release
two variants:

- **OpenSLA-B**: sensor tokens are projected and prepended to the language
  prompt as a flat token prefix.
- **OpenSLA-H**: builds on B with a hierarchical sensor encoder that compresses
  the signal into local, per-channel, and global memory tokens.


## 📖 Table of Contents

1. [Installation](#-installation)
2. [Quick Start](#-quick-start)
3. [Pretrained Weights](#-pretrained-weights)
4. [Usage](#-usage)
5. [Datasets](#-datasets)
6. [Citation](#-citation)

## 💿 Installation

```bash
git clone https://github.com/yang-ai-lab/Sensor-Language-Action-Model.git
cd Sensor-Language-Action-Model
pip install -r requirements.txt
```

### Dependencies

- Python >= 3.10
- PyTorch >= 2.5
- Transformers >= 5.8.1 (for the Qwen3.5 backbone)
- PEFT >= 0.12


## 🚀 Quick Start

`demo.ipynb` loads a checkpoint, builds an input batch, and runs prediction.
The same flow in Python:

```python
import torch
from opensla import OpenSLA, SensorBatch

model = OpenSLA.from_checkpoint(
    "pretrained_weights/opensla_h_clinical.pt", domain="clinical",
    dino_checkpoint="pretrained_weights/waveform_encoder.ckpt",
    numeric_config="pretrained_weights/numeric_config.json",
    action_group_types="pretrained_weights/action_group_types.json",
)

batch = torch.load("data/preprocessed_batch.pt", weights_only=True)
predictions = model.predict(batch["input_text"], SensorBatch(**batch["sensors"]))
```


## 📦 Pretrained Weights

| Model | Domain | Files |
|-------|--------|-------|
| OpenSLA-H | clinical | `opensla_h_clinical.pt`, `waveform_encoder.ckpt`, `numeric_config.json`, `action_group_types.json` |

Download the files from this repository and put them under `pretrained_weights/`
in the code repository.

## 👩‍💻 Usage

### Input Format

The model takes preprocessed signals as a `.pt` dictionary with `domain`,
`input_text` (one text context per sample), and `sensors`. The sensors are:

- **Waveform**: `waveform` of shape `[B, M, C, T]`, with `M` one-minute slots,
  `C` physical channels, and `T = 7500` samples per slot (60 s at 125 Hz), plus
  a boolean `waveform_channel_mask` (`[B, M, C]`) marking observed
  channel-minutes and int64 `waveform_modality_ids` (`[B, C]`) indexing
  `opensla.WAVEFORM_MODALITIES[domain]`.
- **Numeric**: observed events as parallel `[B, E]` tensors (`values`,
  `rel_time_min` in minutes before the decision, `measure_ids`, `source_ids`,
  `event_mask`), plus per-measure summary features (`summary_features`,
  `summary_measure_ids`).

### Command line

Point `configs/template.json` at your weights and input batch, then:

```bash
opensla --config configs/template.json --dry-run   # print the resolved config
opensla --config configs/template.json             # write outputs/predictions.jsonl
```

On Slurm:

```bash
sbatch --account=YOUR_ACCOUNT --partition=YOUR_PARTITION scripts/run_template.sbatch --config configs/template.json
```


## 📊 Datasets

The models are trained and evaluated on six datasets from three healthcare
settings, with an additional MIMIC-IV held out as an external clinical cohort. All of them are
publicly available and can be accessed through the following links.

| Dataset | Domain | Sensors | Source |
|---------|--------|---------|--------|
| MC-MED | Clinical | ECG, plethysmography, respiration, arterial pressure; vital signs, labs, ventilator and other charted measurements | [PhysioNet](https://physionet.org/content/mc-med/1.0.0/) |
| MIMIC-III | Clinical | same as MC-MED | [PhysioNet](https://physionet.org/content/mimiciii/1.4/) |
| MIMIC-IV | Clinical, external evaluation only | waveform-linked subset | [PhysioNet](https://physionet.org/content/mimiciv/2.0/), [waveforms](https://physionet.org/content/mimic4wdb/0.1.0/) |
| MOVER | Operating room | ECG, plethysmography, arterial/central venous pressure, capnography, airway pressure, EEG; vital signs, hemodynamics, ventilation gases, labs | [UCI MOVER](https://mover.ics.uci.edu/) |
| VitalDB | Operating room | same as MOVER | [vitaldb.net](https://vitaldb.net/dataset/), [PhysioNet](https://physionet.org/content/vitaldb/1.0.0/) |
| MetaboNet | CGM | glucose trace; basal insulin delivery | [metabo-net.org](https://metabo-net.org/) |
| PEDAP | CGM | glucose trace; basal insulin delivery | [Jaeb Center](https://public.jaeb.org/datasets/diabetes) |


## 📝 Citation

If you use this code or models in your research, please cite our paper:

```bibtex
@misc{xu2026opensla,
  title  = {Sensor-Language-Action Models},
  author = {Xu, Yuekai and Shuai, Zitao and Yang, Yuzhe},
  journal = {arXiv preprint},
  year = {2026}
}
```

## Acknowledgments

The waveform encoder adapts from the sensor encoder from
[OSF](https://github.com/yang-ai-lab/OSF-Open-Sleep-FM) (MIT License).