ShallowMind-abeat/blahblahthron-1.1b

NOTE : This version is is not fully trained, I stopped it way earlier because i needed something to experiment with ;)

A custom causal language model implemented as a nn.Module now wrapped into a PreTrainedModel—fully compatible with transformers.from_pretrained(trust_remote_code=True) usage.

Highlights:

Custom GPT-style LM with rotary attention, RMSNorm, SwiGLU, and tied embeddings.

Fully integrated: use from_pretrained(), save_pretrained(), and Transformers pipelines.

Includes QLoRA-compatibility—handles 4-bit finetuning via peft.

Load

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "ShallowMind-abeat/blahblahthron-1.1b",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("ShallowMind-abeat/blahblahthron-1.1b")

Then use immediately in pipelines or with Trainer:

from transformers import pipeline

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)