reubk/MolmoE_1B_LoRA_Pointing

Testing a QLoRA adaptor for allenai/MolmoE-1B-0924,

Targets top 10 experts that are activated when pointing is involved and image pooling and projection layers of Vision backbone

Trained on 47 screenshots of a low-poly video game with ragdoll casualties

Evaluated on 44 screenshots of aforementioned video game

Molmo has an edge case where it declares there are no humans in an image:

This custom QLoRA successfully reduces the occurance of these cases

However, pointing to non-human objects is observed to increase.

Comparison of Model performance with and without QLora on Eval dataset

Model	MolmoE-1B	MolmoE-1B w/ QLora
Precision	82.4	81.5
Recall	63.5	72.1

Dataset: reubk/RavenfieldDataset