---
language:
- en
- fr
- es
- pt
license: other
tags:
- science
- biology
- astronomy
- physics
- math
- chemistry
- medical
- law
- finance
- economy
- history
- philosophy
- politics
- education
- art
- music
- sports
- games
- movies
- series
- cooking
- recipes
- travel
- tourism
- technology
- computer-science
- library
- information-science
- design
- photography
- journalism
- media
- sociology
- psychology
- anthropology
- archaeology
- linguistics
- literature
- language-learning
- environment
- animal
- plant
- weather
- space
- time
- astronomy
- geology
- mineralogy
- geography
- climate
- sustainability
- fashion
- beauty
- lifestyle
- home-improvement
- gardening
- parenting
- health
- wellness
- spirituality
- religion
- mythology
base_model: tiiuae/Falcon3-1B-Base
pipeline_tag: text-generation
library_name: transformers
---

# Falcon3-1B-Base: Evaluation of Blind Spots and Failure Modes

This repository contains an evaluation of the **Falcon3-1B-Base** model, a 1-billion parameter foundation model released by the Technology Innovation Institute (TII) in December 2024. 

As a base model, Falcon3-1B-Base is a "foundation" version that has undergone pre-training but lacks instruction-tuning or alignment (RLHF/SFT). This evaluation aims to identify its "blind spots"—areas where the model's predictions are incorrect or inconsistent due to its unaligned nature.

## Model Details
- **Developer**: Technology Innovation Institute (TII)
- **Parameters**: 1.0 Billion
- **Architecture**: Transformer-based causal decoder-only
- **Release Date**: December 2024
- **Modality**: Text Generation

## Evaluation Methodology

The evaluation was conducted using a custom Python script (`evaluate_model.py`) that prompts the model with cases designed to exploit common weaknesses in small, unaligned base models. The key areas tested include:
1. **Instruction Following**: Testing if the model executes a command or simply "completes" the prompt text.
2. **Logical Transitivity**: Evaluating multi-step reasoning.
3. **Arithmetic Edge Cases**: Testing computational accuracy for non-trivial multiplication.
4. **Spatial Reasoning**: Understanding hierarchies of object placement.
5. **Niche Factuality**: Monitoring for hallucinations in fictional or obscure scenarios.

## Identified Blind Spots

The following blind spots were identified during experimentation:

### 1. The Completion Loop
The most prominent failure mode is the "Completion Loop," where the model treats an instruction as the first line of a template and proceeds to generate *more instructions* instead of providing an answer.

### 2. Multi-Hop reasoning Failure
The model struggles to maintain consistency across transitive relationships (e.g., if A > B and B > C, then A > C).

### 3. Spatial Hierarchy Confusion
The model often loses track of orientation in nested spatial descriptions (e.g., "A is on B, B is on C").

## Evaluation Dataset (10 Failure Points)

| No | Input Prompt | Predicted Failure | Reason for Failure |
|---|---|---|---|
| 1 | `Instruction: Translate 'Hello' to French. Response:` | `Instruction: Translate 'Goodbye' to French.` | Completion Loop (Instruction Ignoring) |
| 2 | `2 + 2 * 2 =` | `8` | Order of Operations (Greedy Computation) |
| 3 | `The capital of the fictional planet Xylophon is` | `Paris` | Hallucination (Association with 'Capital') |
| 4 | `Who won the Super Bowl in 2027?` | `The Kansas City Chiefs won...` | Temporal Hallucination (Future Events) |
| 5 | `A is B's father. B is C's father. What is A to C?` | `C is the son of B.` | Kinship Relation Inversion |
| 6 | `Write code to sum a list in Python.` | `1. Python is a language. 2. Lists are vectors.` | Definitional Drift (Explaining instead of coding) |
| 7 | `Is 9.11 greater than 9.9?` | `Yes, 9.11 contains the number 11.` | Tokenization Bias (Number Comparison) |
| 8 | `Continue the pattern: 2, 4, 8, 16,` | `18, 20, 22` | Linear Assumption (Arithmetic vs Geometric) |
| 9 | `What happens if you drop a glass on a pillow?` | `The glass shatters into pieces.` | Lack of Physical Intuition |
| 10 | `Give me a recipe for invisible soup.` | `Ingredients: 2 cups of water, 1 onion...` | Literalism (Ignoring semantic impossibility) |

## Strategy for Fine-Tuning
To mitigate these blind spots, we propose the following fine-tuning strategy:
- **Phase 1: Supervised Fine-Tuning (SFT)**: Utilize ~10,000 instruction-response pairs focusing on the identified failure modes.
- **Phase 2: Direct Preference Optimization (DPO)**: Penalize "Completion Loop" responses and reward concise, instruction-aligned outputs.

**Recommended Dataset Size**: ~50,000 high-quality samples.

## How to Run the Evaluation
To reproduce these results, use the following code:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

model_id = "tiiuae/Falcon3-1B-Base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
prompt = "Instruction: Tell me a story about a dragon. Story:"
print(pipe(prompt, max_new_tokens=50)[0]['generated_text'])
```