Soul Player C64: A Retro AI Experience on Commodore 64

Soul Player C64 is a fascinating fusion of retro computing and modern AI, bringing a transformer model to life on the iconic Commodore 64. This project implements a 2-layer, decoder-only transformer with approximately 25,000 int8 parameters, running entirely on the Commodore 64's 1 MHz processor. The architecture mirrors that of advanced models like ChatGPT but is scaled down to fit the constraints of the vintage hardware. With a unique approach to softmax score normalization, the model achieves meaningful attention weights, allowing it to generate responses, albeit slowly—around 60 seconds per token.

To get started, users can load the pre-built Soul Player on a C64 emulator or real hardware. The model understands lowercase letters and basic punctuation, generating responses that are more of a nostalgic novelty than cutting-edge AI. For those interested in customization, the project allows training of new models using a simple text corpus. The training process involves quantization-aware training, ensuring the model's weights are optimized for the C64's integer arithmetic.

The repository provides all necessary scripts and resources, including training, building, and testing tools. Users can train their own models, build C64 binaries, and even chat with the model locally using Python scripts. Despite its limitations, such as a small vocabulary and slow processing, Soul Player C64 is a testament to the ingenuity of combining old and new technologies, offering a unique glimpse into what AI might have looked like in the early 80s.

The project is open-source under the GNU General Public License v3, inviting enthusiasts to explore and expand upon this retro AI experiment.