---
license: apache-2.0
datasets:
- ai2-adapt-dev/rlvr_gsm8k_zs
metrics:
- accuracy
base_model:
- swiss-ai/Apertus-8B-Instruct-2509
pipeline_tag: reinforcement-learning
tags:
- rlvr
- grpo
- gsm8k
- apertus
results:
  - task:
      type: text-generation
    dataset:
      name: gsm8k
      type: mathematical
      split: test
    metrics:
      - name: GSM8K (Validation, 0-shot, T=0) Accuracy
        type: GSM8K (0-shot, T=0)
        value: 66.23

---

# RLVR Training Apertus 8B with GRPO on GSM8K dataset

## Results

<p style="font-size:16px;">
  <strong>Validation accuracy improved</strong> from 
  <span style="color:#c0392b; font-weight:bold;">46.41%</span> 
  to 
  <span style="color:#27ae60; font-weight:bold;">66.23%</span>.
</p>


<figure>
  <img src="https://cdn-uploads.huggingface.co/production/uploads/6738b18cab421271320a596c/2pfAqYDLZGmSlQ1LKwWDV.png" alt="Full validation set accuracy" width="500">
  <figcaption><strong>Figure 1.</strong> Full validation set accuracy.</figcaption>
</figure>


<figure>
  <img src="https://cdn-uploads.huggingface.co/production/uploads/6738b18cab421271320a596c/CK3yFwOCspUF-HyAKQflv.png"
       alt="Validation set, average tokenes used per senquenced (capped at 512 tokens)"
       width="500">
  <figcaption><strong>Figure 2.</strong> Validation set, average tokenes used per senquenced (capped at 512 tokens).</figcaption>
</figure>

<figure>
  <img src="https://cdn-uploads.huggingface.co/production/uploads/6738b18cab421271320a596c/v77-3ixk_SV03YKDAVuAR.png"
       alt="Training reward curve"
       width="500">
  <figcaption><strong>Figure 3.</strong> Training reward.</figcaption>
</figure>

---

<h2>Compute</h2>

<p style="font-size:16px;">
  Training performed on a <strong>GPU node</strong> with 
  <strong>4× NVIDIA H100 (95 GB)</strong>, 
  running for approximately <strong>5 hours</strong>.
</p>

---

<h2>Hyperparameters</h2>

<table>
  <tr><th colspan="2" style="text-align:left;">Rollouts</th></tr>
  <tr><td><code>num_unique_prompts_rollout</code></td><td>32</td></tr>
  <tr><td><code>num_samples_per_prompt_rollout</code></td><td>8</td></tr>
  <tr><td><code>temperature</code></td><td>0.8</td></tr>

  <tr><th colspan="2" style="text-align:left; padding-top:10px;">Optimization</th></tr>
  <tr><td><code>learning_rate</code></td><td>3.0e-7</td></tr>
  <tr><td><code>beta</code></td><td>0.01</td></tr>
</table>


---

## Notes

- Note: format reward was not applied because neither the instruct or the base models were able to get a correct answer. Thus the model is not able to use `<think>  </think>`.
- Funny observation: the model memorized the dataset. in one try, the model answered the question but because the format was not familiar, it started reciting another question from same dataset; another time it outputed the html code, assumingly from where it saw the question.

--- 


<h2>Acknowledgements</h2>

<p style="font-size:16px;">
  This work builds upon and was inspired by the following contributions:
</p>

<ul style="font-size:16px;">
  <li>
    <a href="https://arxiv.org/abs/2411.15124" target="_blank">
      <strong>RLVR: Verifiable Rewards for Reasoning Models</strong>
    </a> — for introducing the verifiable reward framework used in this experiment.
  </li>
  <li>
    <a href="https://github.com/allenai/open-instruct" target="_blank">
      <strong>Allen Institute for AI — Open Instruct</strong>
    </a> — for providing open-source infrastructure for RLHF/RLVR training.
  </li>
  <li>
    <a href="https://arxiv.org/abs/2509.14233" target="_blank">
      <strong>Apertus Project</strong>
    </a> — for releasing the <em>Apertus-8B</em> base and instruct models used in this work.
  </li>
</ul>