GuppyLM: A Simple Language Model with a Fishy Personality

GuppyLM is a charmingly simple language model designed to mimic the conversational style of a small fish named Guppy. With just 9 million parameters, this model is built to demonstrate that creating your own language model is accessible to anyone, even without advanced technical skills or powerful hardware. In just five minutes using a Colab notebook, you can train this model from scratch, covering all aspects from data generation to inference.

GuppyLM is trained on 60,000 synthetic conversations across 60 topics, focusing on the life and experiences of a fish in a tank. It speaks in short, lowercase sentences and is preoccupied with basic sensory experiences like water, temperature, and food. The model is intentionally limited in scope, avoiding complex human abstractions, and is designed to run efficiently on a single GPU, even in a browser.

The architecture of GuppyLM is straightforward, employing a vanilla transformer with 8.7 million parameters, 6 layers, and a vocabulary of 4,096 tokens. It uses standard components like LayerNorm and ReLU, avoiding more complex mechanisms to keep the model simple and understandable.

GuppyLM's personality is friendly and curious, often thinking about food and reacting to its environment with a childlike simplicity. It can be easily interacted with by downloading the pre-trained model from HuggingFace or training your own version using the provided dataset and scripts.

The project structure is well-organized, with scripts for model configuration, data preparation, training, and inference. The dataset used for training is synthetic, created through template composition to ensure consistency in Guppy's personality. This approach allows for a wide range of conversational topics while maintaining the model's playful and simplistic nature.