Tilde

README.md

@@ -41,7 +41,6 @@ datasets:
 - uonlp/CulturaX
 - bigcode/the-stack
 - common-pile/arxiv_papers
-library_name: transformers
 ---
 **Developed by:**  [Tilde.ai](https://tilde.ai/tildeopen-llm/)   
 **Funded by:**  European Commission via [EuroHPC JU Large AI Grand Challenge](https://www.eurohpc-ju.europa.eu/winners-announced-large-ai-grand-challenge-2024-06-26_en)   
@@ -105,161 +104,48 @@ outputs = model.generate(
 )
 ```
 # Evaluation
-## Belebele Benchmark: Reading Comprehension
-**What is Belebele Benchmark?** [Belebele](https://aclanthology.org/anthology-files/anthology-files/pdf/acl/2024.acl-long.44.pdf) is a multiple-choice machine reading comprehension (MRC) dataset spanning 122 language variants. This dataset enables the evaluation of mono- and multi-lingual models in high-, medium-, and low-resource languages. Each question has four multiple-choice answers and is linked to a short passage from the FLORES-200 dataset. The human annotation procedure was carefully curated to create questions that discriminate between different levels of generalizable language comprehension and is reinforced by extensive quality checks.
-Results
-
-**Why does this Matter?** Belebele tests LLM's ability to provide answers based on a given text -- a standard use case in retrieval augumented generation workflows. 
-
-**What did we do?** We used the standard implementation of the [belebele](https://github.com/eleutherai/lm-evaluation-harness/tree/main/lm_eval/tasks/belebele) task from the LLM Evaluation Harness. We set tokenisers to ```use_fast=False```. We report **5-shot** accuracy.
-
-| 5-shot | **Gemma 2 27b** | **ALIA 40b** | **EuroLLM Prev. 22b** | **TildeOpen 1.1 30b** |
-|----------|:-------------:|:----------:|:------------:|:-------------------:|
-| Bulgarian | 79.8% | 78.8% | **85.3%** | 84.7% |
-| Czech | 81.4% | 78.3% |  85.3% | **85.8%** |
-| German | 81.2% | 80.6% | **85.0%** | 84.3% |
-| English | **88.9%** | 83.0%  | 87.6% | 88.3% |
-| Estonian | 72.1% | 73.7% | 82.0% | **82.6%** |
-| Finnish | 79.0% | 78.1% | 84.3% | **85.0%** |
-| French | 82.6% | 80.1%  | **85.7%** | 85.0% |
-| Hungarian | 77.9% | 76.2% | 83.3% | **86.2%** |
-| Icelandic | 70.8% | 58.2%  | 54.3% | **85.7%** |
-| Italian | 82.1% | 77.8% | 81.0% | **82.4%** |
-| Lithuanian | 76.1% | 76.1% | **85.2%** | 83.3% |
-| Latvian | 78.4% | 77.7%  | **84.6%** | **84.6%** |
-| Dutch | 80.2% | 78.9% | 83.2% | **85.0%** |
-| Polish | 78.3% | 77.9%  | 82.2% | **83.0%** |
-| Portuguese | 83.8% | 80.1%  | 86.1% | **87.1%** |
-| Romanian | 80.3% | 78.8%  | 85.3% | **85.9%** |
-| Russian | 79.4% | 79.4% | 84.2% | **84.6%** |
-| Slovak | 78.9% | 78.0% | 84.1% | **85.0%** |
-| Slovenian | 78.0% | 80.0%  | 83.7% | **85.1%** |
-| Spanish | 82.1% | 78.4% | **84.1%** | 83.8% |
-| Serbian | 79.8% | 78.4% | 74.1% | **84.2%** |
-| Swedish | 80.6% | 76.3% | **85.3%** | 84.4% |
-| Turkish | 77.4% | 62.3% | 79.9% | **82.7%** |
-| Ukrainian | 78.0% | 77.0% | 83.9% | **85.1%** |
-| **Average** | 79.5% | 76.8% | 82.5% | **84.7%** |
-
-## MultiBLiMP Benchmark: Grammar Test
-**What is MultiBLiMP?** [MultiBLiMP](https://arxiv.org/pdf/2504.02768) is a massively multilingual test of core grammar. It gives models pairs of almost-identical sentences—one grammatical and one ungrammatical—and asks whether the model assigns a higher probability to the correct one. Version 1.0 covers 101 languages
-
-**Why does this Matter?** MultiBLiMP tests models' ability to distinguish correct and erroneous language. Just like humans, producing mostly correct language is not a big achievement. Rather, it is very bad to make any mistakes at all. 
-
-**What did we do?** 
-We used the standard implementation of the [MultiBLiMP](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/multiblimp) task from the LLM Evaluation Harness. We set tokenisers to ```use_fast=False```. We report **0-shot** accuracy.
-
-| Language | **Gemma 2 27b** | **ALIA 40b** | **EuroLLM Prev. 22b** | **TildeOpen 1.1 30b**
-|----------|-------------|----------|---------------------|-------------|
-| Bulgarian | 95.4% | 98.8% | 97.7% | **99.6%** |
-| Czech | 98.6% | **98.9%** | 98.5% | 98.5% |
-| German | 98.8% | 98.7% | 98.0% | **99.4%** |
-| English | 98.4% | 98.7% | 98.7% | **99.4%** |
-| Estonian | 92.0% | 95.6% | 95.8% | **98.3%** |
-| Finnish | 93.0% | 96.3% | 95.2% | **98.5%** |
-| French | 98.2% | 98.8% | 98.7% | **99.3%** |
-| Serbo-Croatian | 94.6% | 98.5% | 96.4% | **99.6%** |
-| Hungarian | 95.9% | 98.8% | 97.8% | **100.0%** |
-| Icelandic | 88.5% | 80.3% | 74.4% | **98.8%** |
-| Italian | 96.0% | 96.7% | 96.6% | **98.2%** |
-| Latvian | 91.6% | 95.2% | 96.9% | **99.1%** |
-| Lithuanian | 95.3% | 99.0% | 99.0% | **99.7%** |
-| Dutch | 94.0% | 96.6% | 96.5% | **99.2%** |
-| Polish | 97.0% | 97.5% | 97.6% | **99.3%** |
-| Portuguese | 96.1% | 97.6% | 97.1% | **98.2%** |
-| Romanian | 97.7% | 98.9% | 98.5% | **98.9%** |
-| Russian | 94.7% | 96.6% | 97.3% | **99.4%** |
-| Slovak | 97.7% | 98.8% | 97.7% | **99.3%** |
-| Slovenian | 99.0% | **100.0%** | **100.0%** | 98.8% |
-| Spanish | 95.6% | 98.0% | 97.3% | **98.7%** |
-| Swedish | 95.8% | 85.1% | 93.8% | **100.0%** |
-| Turkish | 97.6% | **98.7%** | 97.9% | 96.4% |
-| Ukrainian | 95.6% | 98.0% | 97.3% | **99.2%** |
-| **Average** | 95.7% | 96.7% | 96.4% | **99.0%** |
-
-## Knowledge tests
-
-### ARC Benchmark Results
-**What is ARC?** [ARC](https://arxiv.org/pdf/1803.05457) - The AI2 Reasoning Challenge is a multiple-choice science question benchmark **in English**, derived from U.S. grade-school standardized exams. It has two subsets — ARC Easy and ARC Challenge — designed to test factual knowledge and common-sense.
-
-**Why does this Matter?** ARC probes a model’s ability to answer non-trivial questions by applying world knowledge. Although the answer can sometimes be inferred from the question, in the classic lm-evaluation-harness ARC implementation the answer choices for each question are **not** provided during inference, thus placing emphasis on world knowledge, rather than on the model's reasoning capabilities.
-
-**What did we do?** 
-We use multilingual translations of ARC provided by [Eurolingua](https://huggingface.co/datasets/Eurolingua/arcx); please refer to the [publication](https://arxiv.org/pdf/2410.08928). Other than the data source, we replicate the standard [LM Evaluation Harness configuration for ARC](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/arc). Our exact configuration is available at [TBA]. We set tokenisers to ```use_fast=False```. We report **5-shot** accuracy.
-
-| 5-shot |  | **ARC Easy**| |  | **ARC Challenge**| |
-|----------|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|
-| **Language** | **ALIA 40b** | **EuroLLM Prev. 22b** | **TildeOpen 1.1 30b** | **ALIA 40b** | **EuroLLM Prev. 22b** | **TildeOpen 1.1 30b** |
-| Danish | 79.9% | **80.1%** | 79.6% | 53.4% | 52.6% | **53.7%** |
-| German | 79.6% | **79.9%** | 78.0% | 53.4% | **53.6%** | 51.7% |
-| Spanish | **82.9%** | 81.7% | 79.4% | **57.3%** | 56.1% | 52.4% |
-| French | **81.7%** | 81.1% | 78.6% | **56.0%** | 54.5% | 52.8% |
-| Italian | 80.5% | **81.6%** | 78.5% | **56.4%** | 54.8% | 54.1% |
-| Dutch | **80.1%** | 80.0% | 78.8% | **54.0%** | 53.8% | 52.2% |
-| Portuguese | **81.7%** | 81.1% | 79.0% | **56.9%** | 55.5% | 54.1% |
-| Swedish | 80.3% | **80.5%** | 78.7% | 53.8% | 53.1% | **54.1%** |
-| **AVG WEST** | **80.8%** | **80.8%** | 78.8% | **55.2%** | 54.2% | 53.1% |
-| | | | | | | |
-| Bulgarian | **79.8%** | 79.2% | 79.5% | **53.8%** | 51.8% | 52.8% |
-| Czech | **79.5%** | **79.5%** | 78.8% | 51.5% | 52.3% | **53.9%** |
-| Estonian | 72.4% | 73.0% | **73.1%** | 49.6% | 49.8% | **52.0%** |
-| Finnish | 73.8% | **74.2%** | 73.3% | 48.7% | 51.1% | **52.1%** |
-| Hungarian | 74.0% | 73.9% | **74.9%** | 49.3% | 49.0% | **49.6%** |
-| Lithuanian | 76.4% | 76.1% | **77.9%** | 50.3% | 51.6% | **53.0%** |
-| Latvian | 76.2% | **76.4%** | 75.9% | 50.7% | 49.8% | **50.9%** |
-| Polish | **79.2%** | 78.2% | 78.0% | **54.5%** | 53.3% | 52.7% |
-| Romanian | **79.6%** | 78.8% | 78.8% | **55.5%** | 53.7% | 54.5% |
-| Slovak | 78.8% | 79.2% | **79.6%** | 52.5% | 53.0% | **54.7%** |
-| Slovenian | **78.3%** | 78.5% | **78.3%** | **53.4%** | 52.2% | 52.7% |
-| **AVG EAST** | **77.1%** | 77.0% | **77.1%** | 51.8% | 51.6% | **52.6%** |
-
-### MMLU Benchmark Results
-**What is MMLU?** [MMLU](https://arxiv.org/pdf/2009.03300) is a massive multitask test consisting of multiple-choice questions from various branches of knowledge, **in English**. The test spans subjects in the humanities, social sciences, hard sciences, and other areas that are important for some people to learn. This covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. Questions are four option multiple choice and assess factual knowledge, reading comprehension, and reasoning across disciplines. The questions can be grouped under four topics - stem, humanities, social_sciences and other, allowing for individual evaluation of each group.
-
-**Why does this Matter?** Similarly to ARC, MMLU measures broad, general purpose factual knowledge and some reasoning capabilites. The possible answer choices are included during prompting, which can allow the model to employ reasoning to discard false answers, rather than just relying on knowing the correct one. It should be noted that some question groups are exclusive to the anglocentric world, e.g. US history or law.
-
-**What did we do?** We use multilingual translations of MMLU provided by [Eurolingua](https://huggingface.co/datasets/Eurolingua/mmlux), please refer to the [publication](https://arxiv.org/pdf/2410.08928). Other than the data source, we replicate the standard [LM Evaluation Harness configuration for MMLU](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/mmlu/default). Our configuration is available at [TODO]. We set tokenisers to ```use_fast=False```. We report **0-shot** accuracy.
-
-| 0-shot | **ALIA 40b** | **EuroLLM Prev. 22b** | **TildeOpen 1.1 30b** |
-|----------|:-----------------:|:---------------------:|:-------------------:|
-| Bulgarian | 48.3% | 52.0% | **56.3%** |
-| Czech | 49.1% | 51.7% | **56.4%** |
-| Danish | 50.2% | 51.1% | **56.6%** |
-| German | 51.0% | 51.8% | **56.2%** |
-| Greek | 50.7% | 50.6% | **50.9%** |
-| Spanish | 53.3% | 53.4% | **56.3%** |
-| Estonian | 48.7% | 49.2% | **55.3%** |
-| Finnish | 47.4% | 48.9% | **55.4%** |
-| French | 53.1% | 53.8% | **56.4%** |
-| Hungarian | 49.9% | 44.4% | **55.2%** |
-| Italian | 52.3% | 53.7% | **57.2%** |
-| Lithuanian | 47.3% | 49.4% | **54.7%** |
-| Latvian | 46.9% | 48.0% | **54.0%** |
-| Dutch | 50.8% | 53.0% | **56.5%** |
-| Polish | 50.6% | 49.6% | **55.6%** |
-| Portuguese | 52.4% | 53.7% | **56.4%** |
-| Romanian | 51.0% | 52.1% | **56.2%** |
-| Slovak | 49.0% | 52.2% | **56.3%** |
-| Slovenian | 48.2% | 50.7% | **55.3%** |
-| Swedish | 49.6% | 51.2% | **56.1%** |
-| **Average** | 50.0% | 51.0% | **55.7%** |
-
-### National Exams Results
-**What are National Exams?** A curated suite of **multlingual** publicly available past questions from national-level standardized exams across multiple countries (e.g., high-school exit and university-entrance exams), please refer to the [publication](https://aclanthology.org/2020.emnlp-main.438.pdf). The dataset is available on HuggingFace [here](https://huggingface.co/datasets/mhardalov/exams). Items are presented in multiple-choice format. 
-
-**Why does this Matter?** Similarly to MMLU, the model is tested on factual knowledge and reasoning capabilites. However, it should be stressed that for each language the bench is **unique** (the exams are different) and available in the **source language** (i.e. not translated). This places emphasis on the model's regional knowledge and eliminates translation noise that is present in many other multilingual benchmarks. Possible answer choices are once again included during inference, allowing for the model to employ reasoning if factual knowledge is lacking.
-
-**What did we do?** [TODO]
-
-| 5-shot | **ALIA 40b** | **EuroLLM Prev. 22b** | **TildeOpen 1.1 30b** |
-|----------|----------|-------------------|-------------------|
-| Bulgarian | 62.4% | 66.8% | **67.8%** |
-| Croatian | 70.8% | **72.5%** | 71.9% |
-| Hungarian | 48.9% | **51.9%** | 48.9% |
-| Italian | **65.5%** | 64.6% | 65.0% |
-| Macedonian | 74.2% | 72.0% | **80.2%** |
-| Polish | 61.2% | 61.4% | **63.5%** |
-| Portuguese | **61.4%** | 60.9% | 59.2% |
-| Albanian | 55.6% | 55.0% | **75.6%** |
-| Serbian | 64.7% | 57.3% | **66.9%** |
-| **Average** | 62.7% | 62.5% | **66.6%** |
+## Per-Character Perplexity
+**What is Perplexity?** Perplexity measures how well a language model predicts text. A model with low perplexity makes accurate predictions consistently, while a high perplexity means the model is frequently "surprised" by unexpected words or patterns. Lower perplexity indicates the model has learned language patterns more effectively. It's less "surprised" by what it encounters because it better understands how the language works.
+Perplexity fairly evaluates how well each model handles:
+- Spelling accuracy across a diverse vocabulary
+- Grammar rules that span multiple words
+- Sentence structure and flow
+- Language-specific patterns (how different languages form plural forms or compound words)
+
+**Why Character-Level?** Different language models use different internal vocabularies - some break text into whole words, others into word fragments, and some into individual characters. This makes direct comparison difficult.
+Character-level perplexity creates a standardised comparison by calculating how well each model would theoretically perform if we measured their predictions character-by-character. We're not changing how the models work - instead, we use mathematical conversion to approximate their character-level performance based on their predictions.
+
+**Why does this Matter?** Models with lower perplexity generally perform better on real-world tasks like text generation, translation, and understanding context. It's a reliable indicator of overall language competency across different applications.
+
+**What data did we use?**
+We use WMT24++ as it is a multilingual, language-parallel evaluation set that none of the models have seen during training. WMT24++ is a composite of texts from news, literature, speech, and social media; thus, it is suitable for foundational model benchmarking.
+
+| Language | TildeOpen 30b | Gemma 2 27b | EuroLLM 22B Prev. | ALIA 40B |
+|-----------------|---------|------------|----|------|
+| Bulgarian | **2.0539** | 2.2184 | 2.1985 | 2.1336 |
+| Czech | **2.1579** | 2.3522 | 2.3221 | 2.2719 |
+| Danish | **2.003** | 2.1517 | 2.1353 | 2.0805 |
+| German | **1.8769** | 1.9285 | 1.9452 | 1.904 |
+| English | 2.0378 | **1.9525** | 2.0568 | 2.0261 |
+| Spanish | 1.9503 | 1.9752 | 2.0145 | **1.9369** |
+| Estonian | **2.1711** | 2.5747 | 2.3852 | 2.325 |
+| Finnish | **2.0497** | 2.288 | 2.2388 | 2.1831 |
+| French | **1.8978** | 1.9355 | 1.9282 | 1.9084 |
+| Croatian | **2.1147** | 2.544 | 2.4905 | 2.2433 |
+| Hungarian | **2.0539** | 2.2228 | 2.2256 | 2.1635 |
+| Icelandic | **2.0873** | 3.0329 | 4.7908 | 3.957 |
+| Italian | **1.9565** | 2.0137 | 2.0098 | 1.9887 |
+| Lithuanian | **2.1247** | 2.4175 | 2.3137 | 2.3075 |
+| Latvian | **2.1439** | 2.5355 | 2.3141 | 2.3276 |
+| Dutch | **1.9333** | 2.0312 | 2.0079 | 1.9904 |
+| Norwegian | **2.1284** | 2.2862 | 2.3506 | 2.2253 |
+| Polish | **2.0241** | 2.1294 | 2.0803 | 2.0803 |
+| Portuguese | **1.9899** | 2.0597 | 2.0272 | 2.0187 |
+| Romanian | **2.0196** | 2.1606 | 2.1641 | 2.1114 |
+| Russian | **2.0424** | 2.09 | 2.1095 | 2.0871 |
+| Slovak | **2.1192** | 2.338 | 2.3029 | 2.2609 |
+| Slovenian | **2.1556** | 2.4443 | 2.3398 | 2.2589 |
+| Serbian | **2.2469** | 2.6351 | 4.2471 | 2.3743 |
+| Swedish | **2.041** | 2.1809 | 2.1464 | 2.1211 |
+| Turkish | **2.0997** | 2.247 | 2.2202 | 2.232 |
+| Ukrainian | **2.1376** | 2.2665 | 2.2691 | 2.2086 |