MacMind: A Retro Neural Network in HyperTalk

MacMind is a unique implementation of a transformer neural network, crafted entirely in HyperTalk and trained on a vintage Macintosh SE/30. With 1,216 parameters, this single-layer, single-head transformer learns the bit-reversal permutation, a crucial step in the Fast Fourier Transform, purely from random examples. Every aspect of the neural network, from token embeddings to backpropagation, is manually coded in HyperTalk, a language from 1987 not originally designed for complex mathematical operations.

The project serves as a tangible demonstration that the core processes behind modern AI models, like forward pass and backpropagation, are fundamentally simple and understandable. Despite the small scale of MacMind compared to models like GPT-4, the mathematical principles remain unchanged. This transparency allows users to modify and inspect every part of the model, offering a hands-on educational experience.

MacMind's training involves generating random 8-digit sequences and using self-attention and gradient descent to discover the bit-reversal permutation. This permutation is foundational to the Fast Fourier Transform, an essential algorithm in computing. The model's attention map reveals the same butterfly routing pattern discovered by Cooley and Tukey in 1965, showcasing the model's ability to independently learn complex mathematical structures.

The HyperCard stack consists of five cards, each serving a distinct purpose, from training the model to visualizing the attention map. Training can be conducted in real-time, with options to adjust parameters and observe the learning process. The inference card allows users to test the trained model on new inputs, verifying its predictions against the expected permutation.

MacMind was developed and tested on a Macintosh SE/30 but can also run on modern systems via emulators like Basilisk II. The project highlights the power of simple, interpretable AI models and the potential for educational tools that demystify the workings of neural networks.