docs: update README

README.md

@@ -1,3 +1,79 @@
 ---
 license: apache-2.0
+language:
+- ru
+tags:
+- distill
+- fill-mask
+- embeddings
+- masked-lm
+- tiny
+- feature-extraction
+- sentence-similarity
+datasets:
+- GEM/wiki_lingua
+- xnli
+- RussianNLP/wikiomnia
+- mlsum
+- IlyaGusev/gazeta
+widget:
+- text: Москва - <mask> России.
+- text: Если б море было пивом, я бы <mask>
+- text: Столица России - <mask>.
 ---
+# ruRoberta-distilled
+
+Model was distilled from [ai-forever/ruRoberta-large](https://huggingface.co/ai-forever/ruRoberta-large) with ❤️ by me for 120 hours using 4 Nvidia V100.
+
+## Usage
+
+```python
+from transformers import pipeline
+
+
+pipe = pipeline('feature-extraction', model='d0rj/ruRoberta-distilled')
+tokens_embeddings = pipe('Привет, мир!')
+```
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModel
+
+
+tokenizer = AutoTokenizer.from_pretrained('d0rj/ruRoberta-distilled')
+model = AutoModel.from_pretrained('d0rj/ruRoberta-distilled')
+
+
+def embed_bert_cls(text: str) -> torch.Tensor:
+    t = tokenizer(text, padding=True, truncation=True, return_tensors='pt').to(model.device)
+    with torch.no_grad():
+        model_output = model(**t)
+    embeddings = model_output.last_hidden_state[:, 0, :]
+    embeddings = torch.nn.functional.normalize(embeddings)
+    return embeddings[0].cpu()
+
+
+embedding = embed_bert_cls('Привет, мир!')
+```
+
+## Logs
+
+See all logs at [WandB](https://wandb.ai/d0rj/distill-ruroberta/runs/lehtr3bk/workspace).
+
+## Configuration
+
+- Activation GELU -> GELUFast
+- Attention heads 16 -> 8
+- Hidden layers 24 -> 6
+- Weights size 1.42 GB -> 464 MB
+
+## Data
+
+Overall: 9.4 GB of raw texts, 5.1 GB of binarized texts.
+
+Used data:
+- [GEM/wiki_lingua](https://huggingface.co/datasets/GEM/wiki_lingua)
+- [xnli](https://huggingface.co/datasets/xnli)
+- [RussianNLP/wikiomnia](https://huggingface.co/datasets/RussianNLP/wikiomnia)
+- [mlsum](https://huggingface.co/datasets/mlsum)
+- [IlyaGusev/gazeta](https://huggingface.co/datasets/IlyaGusev/gazeta)