A significant update to llama.cpp, the open-source C++ implementation of LLaMA, dramatically reduces memory requirements for running large language models on consumer hardware. Pull request #613, merged yesterday by contributor jart, introduces memory-mapped file I/O (mmap) for loading model weights, making inference commands load “100x faster” and allowing models up to twice as large to be loaded safely.

The change modifies the file format so weights can be mapped directly into memory without copying. This lets the operating system’s file cache serve pages directly to inference processes, reducing memory contention. Single-file models like LLaMA 7B and multi-file models like LLaMA 13B are supported. Project maintainer ggerganov called the implementation “very compact” and noted that llama.cpp is “much more simpler now.”

The news spread quickly on Hacker News, where a thread about the 30B model running on just 6GB of RAM drew over 1300 points and 400 comments. Some commenters celebrated the achievement as a vindication of systems-level optimization over Python-heavy ML stacks. Others warned that the memory savings may not hold up under scrutiny, and that Python frameworks already rely on efficient C++ backends.