Blog
Long-form behind the videos.No filler, real projects.
Build notes, coding lessons, and honest reviews from the projects behind each video.
Blog
Build notes, coding lessons, and honest reviews from the projects behind each video.
If you want to understand how large language models actually work, GuppyLM lets you train one yourself in about five minutes on a single Colab notebook.
If you want to understand how large language models actually work, GuppyLM lets you train one yourself in about five minutes on a single Colab notebook.
GuppyLM is an educational repository that walks you through the entire process of training a small but functional language model from scratch. It is not a production model. It is a learning tool designed to demystify what happens inside an LLM, from data preparation through tokenizer training, model architecture, the training loop, and inference.
The model itself is tiny, around 8.7 to 9 million parameters. It trains on a T4 GPU in roughly five minutes. After training, it can hold a conversation, though the conversation is that of a small, naive fish named Guppy who is mostly interested in rocks, bubbles, and food. The point is not that the model is powerful. The point is that you can see, read, and modify every line of code that makes it work.
GuppyLM is built on a vanilla transformer architecture implemented in PyTorch. The code is intentionally short, readable, and easy to modify. The pipeline runs through five stages: preparing the training data, training a tokenizer, constructing the model, running the training loop, and performing inference.
The training data is curated to give the model its fish personality. After training on a T4 GPU for about five minutes, the model produces responses that are grammatically simple and topically narrow, but coherent. Ask Guppy about the meaning of life and it will tell you the answer is food. Ask if it is hungry and it will promise to swim to the top and eat everything. The personality is consistent because it was learned during training, not injected via a prompt.
The repo includes a ready-to-run Colab notebook, so you can train the model yourself without any local setup. The trained weights are also available on Hugging Face, so you can try the model in inference mode without training at all.
This is explicitly an educational project. The model is tiny, its capabilities are extremely limited, and its outputs are intentionally narrow. If you are looking for a model to use in a real application, GuppyLM is not it.
The simplicity that makes it a great learning tool also means it skips many of the optimizations and techniques used in production-scale models. You will understand the core transformer architecture, but you will not learn about distributed training, mixture-of-experts, RLHF, or any of the other techniques that make modern LLMs performant.
The fish personality is charming but also a limitation. The training data is narrow, which means the model cannot generalize beyond its domain. This is instructive, but it also means you should not expect the model to be useful for anything practical.
GuppyLM is for developers who want to understand LLMs from first principles rather than treating them as API endpoints. If you have been calling GPT or Claude through an API and want to know what is actually happening under the hood, this repo is one of the most accessible entry points available. It is also useful for educators who need a concrete, runnable example to teach from.
The takeaway: if you want to understand LLMs from the ground up instead of just calling APIs, GuppyLM is one of the best small-scale educational repos available. It will not make you an expert, but it will make the architecture click in a way that reading papers alone rarely does.