SyReC-Mistral-7B-Reconstructor-v3

Model Description

This model is a specialized, fine-tuned version of mistralai/Mistral-7B-Instruct-v0.3. It has been explicitly trained to perform holistic syntactic and semantic reconstruction, a complex reasoning task that involves reconstructing a coherent, grammatically correct paragraph from a "bag of words" stripped of its original order.

This is the third and most successful version in a series of experiments. Unlike its predecessors which suffered from catastrophic forgetting or topical "blind spots," this v3 model was fine-tuned on SyReC v3.0, a large-scale, diverse, and balanced dataset. This holistic training regimen has endowed the model with a robust ability to infer grammatical structure and logical flow across a wide range of topics and writing styles, from concrete scientific descriptions to abstract philosophical arguments.

The result is a model that not only understands the reconstruction task but excels at it, demonstrating semantic comprehension superior to the base model while also gaining a genuine, measurable skill in structural reconstruction.

Intended Use

This model is a specialist tool for tasks requiring high-fidelity source grounding and strict adherence to constraints.

• Primary Use Case: Solving the syntactic reconstruction task as defined by the SyReC benchmark.
• Downstream Applications:
  • High-Fidelity RAG (Retrieval-Augmented Generation): Answering questions based only on the provided context documents, with a significantly reduced tendency to hallucinate or inject outside knowledge.
  • Fact-Based Summarization: Creating summaries that are more extractive and factually grounded in the source text.
  • Complex Instruction Following: Adhering to strict positive and negative constraints within a prompt (e.g., "use only these words," "do not mention X").

This model is not intended for general-purpose creative writing, as its training has biased it towards more literal and structured outputs.

Performance and Evaluation

The v3 model represents a complete success, curing the critical failures of previous versions.

(Replace with the URL to your "Overall Performance" chart)

(Replace with the URL to your "Blind Spot" chart)

• Cured Catastrophic Forgetting: Unlike v1, this model maintains S-Tier semantic understanding across all test paragraphs, often exceeding the base model's comprehension.
• Eliminated the "Humanities Blind Spot": Unlike v2, this model now performs exceptionally well on abstract, argumentative texts, demonstrating a complete recovery on its previous weak points.
• Demonstrated True Reconstruction Skill: The model consistently achieves significant, non-zero BLEU scores on complex texts, proving it has learned to reconstruct multi-word phrases and sentence structures.

How to Use

You can use this model with the transformers library pipeline. Ensure the input is formatted using the model's chat template.

from transformers import pipeline, AutoTokenizer
import torch

# The model you fine-tuned
model_id = "ambrosfitz/SyReC-Mistral-7B-Reconstructor-v3"

# The system prompt the model was trained with
system_prompt = "You are an expert at syntactic reconstruction. Reconstruct the original, coherent paragraph using only the provided words. You must use every word exactly once."

# A sample scrambled paragraph (from the SyReC benchmark)
scrambled_text = "a, a, a, a, a, allow, an, as, assumed, belonging, brahe’s, but, called, circle, circle, circles, closed, conic, consistent, curve, curves, data, did, discovered, doing, ellipse, ellipse, eventually, family, find, flattened, for, had, he, him, his, initially, is, is, its, kepler, kind, known, mars, next, not, object, observations, of, of, of, of, orbit, orbit, orbits, path, planet, planets, sections, shape, simplest, so, somewhat, space, that, that, the, the, the, the, the, the, through, to, to, were, were, with, with, working"

# Setup the pipeline
pipe = pipeline(
    "text-generation",
    model=model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Format the prompt using the chat template
messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": scrambled_text},
]
prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

# Generate the output
outputs = pipe(prompt, max_new_tokens=256, do_sample=False)
full_output = outputs["generated_text"]

# Correctly parse the output for Mistral-instruct models
try:
    reconstructed_text = full_output.split("[/INST]").strip()
except IndexError:
    reconstructed_text = "Error: Could not parse model output."


print("--- Model Reconstruction ---")
print(reconstructed_text)

Training Details

• Base Model: mistralai/Mistral-7B-Instruct-v0.3
• Dataset: ambrosfitz/SyReC_v3 - A balanced, holistic dataset of ~8,000 examples sourced from Wikipedia, the Stanford Encyclopedia of Philosophy, US Patent abstracts, and Project Gutenberg.
• Fine-tuning Method: Parameter-Efficient Fine-Tuning (PEFT) using LoRA.
• Training Strategy: The model was trained for 2 epochs with a conservative learning rate of 5e-5 to ensure the new skill was learned without overwriting the model's foundational knowledge.
• Framework: Hugging Face transformers, peft, and trl.

Training Loss

The training loss showed a consistent downward trend, with a significant breakthrough late in the training process, indicating the model successfully converged on an optimal solution. The final training loss was approximately 1.52.

Step	Training Loss
25	2.2159
1000	1.7040
2000	1.5966
3000	1.6446
3775	1.4311
3875	1.5206

Limitations and Bias

• This model inherits all the biases of its base model, mistralai/Mistral-7B-Instruct-v0.3.
• The training data, while diverse, is sourced from texts that are primarily Western and written in English, and will reflect the biases of those sources.
• Due to the specialized nature of the fine-tuning, the model may exhibit slightly reduced creativity or performance on general-purpose, open-ended tasks compared to the base model (the "alignment tax").

Citation

If you use this model or the SyReC methodology in your work, please consider citing the project.

@misc{syrec_mistral_reconstructor_v3_2025,
  author    = {Fitzgerald, Ambrose},
  title     = {SyReC-Mistral-7B-Reconstructor-v3: A Holistically-Trained Model for Syntactic Reconstruction},
  year      = {2025},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/ambrosfitz/SyReC-Mistral-7B-Reconstructor-v3}
}

---