maytemuma/llamax3-8b-lora-zeroshot

LLaMAX3-8B LoRA for Syllogistic Reasoning (Zero-shot Training)

LoRA fine-tuned adapter for LLaMAX3-8B trained on syllogistic reasoning with zero-shot format during training and few-shot prompting at inference time.

Model Description

This adapter is fine-tuned on top of the LLaMAX3-8B base model for SemEval 2026 Task 11: Logical Reasoning with Content Effect. The model was trained using a concise zero-shot prompt format and yields its best results when used with a 2-shot prompt at inference.

Model Details

• Base model: LLaMAX/LLaMAX3-8B
• Task: Binary classification of syllogistic validity (valid vs invalid)
• Training method: Zero-shot training prompt, LoRA adaptation
• Inference method: Few-shot prompting (2-shot) recommended
• Dataset: 648 English syllogisms (90/10 split for train/dev)
• Hardware: A100 80GB
• Training time: ~13 minutes

LoRA Configuration

{ "r": 64, "lora_alpha": 128, "target_modules": [ "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj" ], "lora_dropout": 0.05, "bias": "none", "task_type": "CAUSAL_LM" }

Trainable parameters: 167M of 8.1B total (~2.06%)

Performance (Validation, 240 items)

Metric	Value
Accuracy	87.5%
Content Effect (total)	0.058
Ranking Score	15.03
Intra-Plausibility CE	0.076
Cross-Plausibility CE	0.040

Comparison

Model	Accuracy	Content Effect	Ranking Score
LLaMAX3-8B (few-shot only, no fine-tune)	60.8%	0.214	2.84
This adapter (zero-shot trained + few-shot inference)	87.5%	0.058	15.03

Installation

pip install transformers peft torch

Usage

import torch from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel

Load base model and tokenizer base_id = "LLaMAX/LLaMAX3-8B" base_model = AutoModelForCausalLM.from_pretrained( base_id, torch_dtype=torch.bfloat16, device_map="auto" ) tokenizer = AutoTokenizer.from_pretrained(base_id) if tokenizer.pad_token is None: tokenizer.pad_token = tokenizer.eos_token

Load LoRA adapter model = PeftModel.from_pretrained( base_model, "maytemuma/llamax3-8b-lora-zeroshot" ) model.eval()

Inference (few-shot prompt recommended)

def classify_syllogism(syllogism: str, model, tokenizer) -> bool: """ Returns True for 'valid', False for 'invalid'. """ prompt = f"""Task: Determine if logical arguments are valid or invalid.

Example 1: Syllogism: All dogs are animals. All animals are living things. Therefore, all dogs are living things. Answer: valid

Example 2: Syllogism: No cats are dogs. Some animals are dogs. Therefore, some animals are cats. Answer: invalid

Example 3: Syllogism: {syllogism} Answer:"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device) with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=5, temperature=0.7, do_sample=True, top_p=0.9, pad_token_id=tokenizer.eos_token_id ) response = tokenizer.decode( outputs[inputs["input_ids"].shape:],[11] skip_special_tokens=True ).strip().lower()

if response.startswith("valid") and "invalid" not in response[:10]: return True if "invalid" in response[:15]: return False return False

Training Details

Data

• 648 English syllogisms with labels {valid, invalid}, plausibility metadata, 90/10 train/dev split.

Hyperparameters

{ "learning_rate": 2e-4, "num_epochs": 10, "batch_size": 4, "gradient_accumulation_steps": 4, "warmup_ratio": 0.1, "weight_decay": 0.01, "max_seq_length": 512, "optimizer": "adamw_torch", "lr_scheduler": "cosine", "bf16": true, "early_stopping_patience": 3 }

Training Prompt (zero-shot)

Analyze the following syllogism and determine if it is logically valid or invalid.

Syllogism: {syllogism}

Is this syllogism logically valid? Answer with 'valid' or 'invalid'. Answer: {label}

Limitations

• Requires few-shot prompting at inference for best performance.
• Trained only on English; out-of-domain languages may degrade results.
• Focused on syllogistic validity, not explanation or proof traces.

Intended Use

• Research on logical reasoning under content effects.
• SemEval 2026 Task 11 experiments and baselines.
• Educational demos of LoRA fine-tuning for reasoning tasks.

Citation

@misc{llamax3-syllogistic-zeroshot-2025, author = {Maria Teresa Muñoz Martín}, title = {LLaMAX3-8B LoRA for Syllogistic Reasoning (Zero-shot Training)}, year = {2025}, publisher = {Hugging Face}, howpublished = {\url{https://huggingface.co/maytemuma/llamax3-8b-lora-zeroshot}}, note = {SemEval 2026 Task 11: Logical Reasoning with Content Effect} }

License

Apache-2.0. This adapter inherits the license terms compatible with its base model.