pnawrot/nanoT5-base

Google's T5-v1.1-base pre-trained for 24 hours (80k steps / 256 batch size) on a single GPU in nanoT5 library for efficient pre-training.

For more details about training check out the paper about this work.

For more details about the model refer to the original paper and original model weights.

It can be further fine-tuned on SuperNatural-Instructions dataset to achieve comparable performance to the same model pre-trained on 150x more data through "a combination of model and data parallelism [...] on slices of Cloud TPU Pods", each with 1024 TPUs.