library_name: transformers
license: mit
pipeline_tag: question-answering
tags:
- lora
- knowledge-editing
- question-answering
Model Card for Knowledge-Packed LoRA Adapters
This model card describes LoRA adapters fine-tuned to incorporate new knowledge into Large Language Models (LLMs), while preserving previously learned information. The approach and potential pitfalls of LoRA-based LLM updates are discussed in the paper: How Much Knowledge Can You Pack into a LoRA Adapter without Harming LLM?.
Model Details
- Developed by: Sergey Pletenev, Maria Marina, Daniil Moskovskiy, Vasily Konovalov, Pavel Braslavski, Alexander Panchenko, and Mikhail Salnikov
- Model type: LoRA adapter for causal language modeling
- Language(s) (NLP): English
- License: MIT
- Finetuned from model: meta-llama/Meta-Llama-3.1-8B-Instruct
Uses
Direct Use
The model can be used to answer questions based on newly injected knowledge, for example, using facts from a specific domain. However, be mindful of the potential biases and knowledge spillover effects described in the paper.
Out-of-Scope Use
The model's performance may degrade when applied to tasks significantly different from the training data or when the training data is imbalanced. The model may exhibit biases learned from the training data and should not be used in high-stakes applications without careful evaluation and mitigation strategies.
Bias, Risks, and Limitations
The model may regress to overrepresented answers when the training data is biased towards certain entities. Fine-tuning can negatively impact the model's performance on external question-answering benchmarks. The model may also become more confident and refuse to provide an answer in only a few cases.
How to Get Started with the Model
See the Github repository for instructions on generating the dataset and training LoRA adapters: https://github.com/memyprokotow/lora_vs_persisted/tree/master
Training Details
Training Data
The training data consists of a mixture of known and new facts, created using the head-to-tail pipeline with Dbpedia. The authors experimented with varying amounts of new knowledge. More details about the training data generation process can be found in the paper and the Github repo. Datasets used for the paper can be downloaded from:
Training Procedure
The model is fine-tuned using LoRA.
Evaluation
The model's performance was evaluated on external question-answering benchmarks and by analyzing knowledge spillover effects. See the paper and Github repo for more details.
Citation
@misc{pletenev2025knowledgepackloraadapter,
title={How Much Knowledge Can You Pack into a LoRA Adapter without Harming LLM?},
author={Sergey Pletenev and Maria Marina and Daniil Moskovskiy and Vasily Konovalov and Pavel Braslavski and Alexander Panchenko and Mikhail Salnikov},
year={2025},
eprint={2502.14502},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2502.14502},
}