---
license: apache-2.0
language:
- en
base_model:
- allenai/MolmoE-1B-0924
pipeline_tag: text-generation
library_name: peft
tags:
- agent
- lora
- finetune
---


Testing a QLoRA adaptor for [allenai/MolmoE-1B-0924](https://huggingface.co/allenai/MolmoE-1B-0924), 

Targets top 10 experts that are activated when pointing is involved and image pooling and projection layers of Vision backbone

Trained on 47 screenshots of a low-poly video game with ragdoll casualties

Evaluated on 44 screenshots of aforementioned video game

Molmo has an edge case where it declares there are no humans in an image:
![img1 (2)](https://cdn-uploads.huggingface.co/production/uploads/6367f8dd46919b9619bc7bf2/8zsuqnz-QCTamBDOgWGM-.png)

This custom QLoRA successfully reduces the occurance of these cases
![img1 (1)](https://cdn-uploads.huggingface.co/production/uploads/6367f8dd46919b9619bc7bf2/-HENqZx5SiLYX35tx3ADs.png)

However, pointing to non-human objects is observed to increase.

Comparison of Model performance with and without QLora on Eval dataset
|Model| MolmoE-1B | MolmoE-1B w/ QLora | 
|----------|------|------|
| Precision | 82.4 | 81.5 |
| Recall | 63.5 | 72.1 |

Dataset: [reubk/RavenfieldDataset](https://huggingface.co/datasets/reubk/RavenfieldDataset)