- Published on
Mini-AGI β Dynamic continual learning model trained on 8GB VRAM
- Authors
- Name
- DeepHorus
A new open-source AI model called Mini-AGI is making waves for its innovative approach to continual learning on consumer hardware. Designed to run on a single GPU with just 8GB of VRAM, this byte-level language model can train from scratch and continuously learn from data streams without catastrophic forgetting. The system stores its weights as ordinary files on disk, paging them onto the GPU as needed, making parameter count limited only by available disk space rather than expensive video memory.
This project addresses a fundamental limitation in current AI models: personalization. Most language models today are frozen after training, allowing only minor fine-tuning. Mini-AGI offers a different paradigm where the model can genuinely belong to its user, trained on their hardware and data, continuously learning from daily interactions. The developers emphasize this is currently a small-scale experiment demonstrating the possibility of continual learning without catastrophic forgetting.
The architecture features a novel approach where characters pass through two dense prelude blocks followed by a recurrent block that can be applied up to 24 times. Unlike traditional transformers, it uses adaptive depth with a halting mechanism that determines how many processing steps each character needs. This allows simple characters to be processed quickly while complex ones receive more computational resources.
Three key constraints guided the design: fitting within 8GB VRAM, preventing catastrophic forgetting during continuous learning, and being able to read any data format. The model uses a byte-level alphabet of 256 values, eliminating the need for tokenizers and allowing it to process any type of text or code data.
The system grows new capacity during training when it runs short and prunes unused components. Weights are stored on disk and loaded as needed, with training and generation using the same code path. This eliminates the distinction between training and serving modes found in conventional models.
Developers note this is an experimental demonstration, not a frontier-level capability. The model's actual performance depends on available hardware, data quality, and training time. The weights have not been published yet, as the initial training run is still processing its first pass through the corpus, expected to take several weeks.
The project represents a significant step toward truly personal AI systems that users can fully own and control. While current capabilities are modest, the framework could enable future developments where personal models continuously adapt to individual users' needs without relying on cloud-based services or frozen base models.
