Part 1: The 24GB Wall — What Actually Broke (this post)
Part 2: The Hardware Shopping List — Where Do You Even Buy That Memory
Part 3: The API Break-Even Point — Was Buying Ever Worth It
I set out to run a local AI agent on a single MacBook Air M5 with 24GB, and over four months I hit seven walls in sequence. This series is the record of those walls, and of where they eventually led: the question of whether to buy more hardware or just switch to an API. This first post is the diagnosis. What broke, why, and which numbers came out of it settled.
The short version: the problem was never model quality. It was memory and context. And of the seven problems, six were solved with configuration. The seventh could only be solved with money — which is why Parts 2 and 3 exist.
The Starting Point: Why Hermes Agent
The setup itself was simple. Ollama running on a MacBook Air M5 (24GB unified memory), with Hermes Agent on top, and a Telegram bot as the client. Send a message from anywhere, and the machine at home would handle it and answer.
I picked Hermes over other agent frameworks for three reasons. First, its four-layer memory architecture keeps context coherent even in long conversations. Second, its self-learning skill files mean a repeated task, once taught, gets written down and reused. Third, its MCP-based tool integration gives a standardized way to attach external tools. I couldn't find a better fit for a personal, always-on automation agent.
Model assignment was straightforward: qwen3:4b-instruct as the main orchestrator — small, but reliable at tool calling. EXAONE 3.5 7.8B handled anything requiring Korean, running as a sub-agent. The idea was to let a small model drive the flow and hand off Korean-language work to a model that was actually good at it.
On paper it was clean. In practice it started breaking within the first week — and the very feature I'd picked Hermes for, that four-layer memory, would later turn out to be the thing eating all my context.
Wall 1: macOS Metal's 75% GPU Memory Cap
Trying to load a bigger model, the load itself failed outright. 24GB should have been enough. It wasn't. Digging into the logs, the culprit wasn't the model or Ollama — it was that macOS caps the GPU memory Metal can allocate at roughly 75% of total unified memory.
So 24GB of unified memory doesn't mean 24GB is available to a model. Realistic usable capacity is closer to 18GB. Subtract what the OS and other apps hold onto, and what's actually left for a model lands around 15–16GB.
That number changed everything. Roughly, by quantization level:
| Model size | Memory needed at Q4 | On M5 24GB (usable ~18GB) |
|---|---|---|
| 7–9B | 5–6GB | Comfortable |
| 12–14B | 8–10GB | Fine |
| ~30B | 18–20GB | Right at the edge / effectively no |
| ~46B | 26–28GB | No |
This table is the first number that runs through the whole series: a 46B-class model needs 26–28GB usable. And the coefficient — listed capacity × 0.75 for usable memory on macOS — carries straight into Part 2's hardware math. You cannot take spec-sheet numbers at face value. That was the first lesson.
Wall 2: Context Length Mismatch in Hermes' Provider Config
The model loaded, but long conversations started cutting off in odd places — the agent would forget something I'd clearly said a few turns earlier. My first instinct was that the four-layer memory system was misbehaving, so I spent a while digging through the memory code.
Wrong place to look. The real cause was context-length auto-detection in Hermes' provider configuration. Hermes was set up to ask Ollama for the model's context length and use whatever came back — but that auto-detected value didn't match what Ollama actually had configured. Hermes believed it had room and kept stuffing the prompt, while Ollama was already truncating the tail end.
The fix was almost embarrassingly simple: stop relying on auto-detection and hardcode context_length in the config file. One line, and days of chasing a phantom bug disappeared. Every added layer of automation adds a new place where that layer might simply be wrong — that was the lesson here.
Wall 3: keep_alive and Prefill Bottlenecks
It worked, but it was slow. Sending a Telegram message meant a long wait for the first response — tens of seconds for even a short question, which made real use impractical.
Two overlapping causes. One was model unload/reload: Ollama drops a model from memory after a period of inactivity, and a Telegram bot's sporadic call pattern meant paying the reload cost almost every time. The other was prompt prefill: with a long system prompt and memory context, there was simply a lot to process before the first token came out.
Three environment variables fixed most of it:
| Variable | Value | Effect |
|---|---|---|
| OLLAMA_KEEP_ALIVE | -1 | Keeps the model resident indefinitely, removing reload time |
| OLLAMA_FLASH_ATTENTION | 1 | Optimizes attention computation, speeding up prefill |
| OLLAMA_KV_CACHE_TYPE | q8_0 | Quantizes the KV cache to reduce memory footprint |
The third one mattered most. KV cache grows with context length, and quantizing it to q8_0 freed up meaningful memory. Within the 18GB ceiling from Wall 1, that headroom was the difference between having room for a longer context or not. Walls 1 and 3, it turned out, were really two faces of the same problem.
Wall 4: EXAONE 3.5's 32K Ceiling
This was the one that hurt most. EXAONE 3.5 7.8B's Korean output was genuinely good, good enough that I tried promoting it from sub-agent to main orchestrator. Running one strong Korean model beat juggling two smaller ones — or so I thought.
Its context ceiling turned out to be hard-capped at 32K. Running Hermes as the main model needs at least 64K. That four-layer memory structure I'd praised earlier consumes a good chunk of context on every single turn, and once you add the system prompt and tool definitions, 32K doesn't survive more than a few real turns of conversation.
This wasn't a wall configuration could clear. It's a hard architectural limit of the model itself — no tuning gets around it. EXAONE came back down off the main-model role and returned to being a callable Korean-language skill tool, invoked only when needed. That turned out to be its right place all along.
Revisiting the Model Lineup
After hitting that fourth wall, I redid model selection from scratch — testing Korean quality, coding ability, and tool-calling accuracy as three separate axes. Candidates were EXAONE 3.5 7.8B, Qwen3.5 9B, and a community-tuned Gemma4-12B.
Currently running Qwen3.5 9B as the main model. Its Korean isn't as polished as EXAONE's, but it wins on context handling and tool-call reliability. The lesson: for an agent framework, "calls functions correctly" matters far more than "writes elegant Korean." That same ranking carries into Part 3, where the same criteria get applied to picking an API model.
Separately, I also tried Ollama on a Windows laptop with an RTX 5070 8GB. 8GB of dedicated VRAM capped things at 7B-class models — fast, but limited by capacity. That experience, fast but small, turns out to be decisive in Part 2's "unified memory vs. dedicated VRAM" argument.
Trouble Shooting
| Symptom | Cause | Fix |
|---|---|---|
| 24GB machine, larger model still fails to load | macOS Metal caps GPU memory around 75% | Recalculate usable capacity as ~18GB; downsize model/quantization accordingly |
| Agent forgets earlier parts of long conversations | Mismatch between Hermes' auto-detected context length and Ollama's actual value | Hardcode context_length in the config file |
| Wasted time chasing the memory system as the culprit | Symptom (forgetting) and actual cause (context config) lived in different layers | Log actual tokens delivered before suspecting higher-level systems |
| Slow first response over Telegram | Model unload/reload + prompt prefill bottleneck | OLLAMA_KEEP_ALIVE=-1, OLLAMA_FLASH_ATTENTION=1, OLLAMA_KV_CACHE_TYPE=q8_0 |
| Extending context blows the memory budget | KV cache grows proportionally with context length | Quantize KV cache to q8_0 to shrink footprint |
| Can't run EXAONE 3.5 as the main model | Hard 32K context ceiling (Hermes needs 64K+) | Demote from main role; reassign as a Korean-language skill tool |
| RTX 5070 laptop caps out at 7B models | 8GB dedicated VRAM is the limiting factor | Route larger models toward unified-memory hardware instead |
Part 1 Ledger: What's Settled, What's Left
This series keeps a running settled-facts ledger at the end of each post. The next post opens with whatever question is left unanswered.
| Item | |
|---|---|
| Settled #1 | macOS usable memory = listed capacity × 0.75 |
| Settled #2 | A 46B-class model at Q4 needs 26–28GB usable |
| Settled #3 | Agent model selection: Korean fluency < tool-calling accuracy |
| Settled #4 | Hermes as main model requires 64K+ context |
| Settled #5 | 8GB dedicated VRAM = fast, but capped at 7B |
| Open question | Where — and for how much — do you actually buy that 26–28GB? |
Takeaways / What's Next
Six of the seven problems were solved by configuration and model choice. The last one was purely a matter of physical memory capacity — and if even one wall can't be cleared by tuning, no amount of tuning elsewhere raises the ceiling. Four months of work landing on "you need to spend money" was a slightly deflating conclusion.
Still, the real payoff of this stretch was turning a vague need into a number. Not "I need more memory" but a specific 26–28GB. That's what makes Part 2's shopping trip possible — the ledger above becomes its input.
Part 2 takes that number to five different paths: Mac mini, Mac Studio, mini PCs, eGPU, and a self-built desktop, and prices every one of them out. Spoiler: 2026's memory price surge makes that shopping trip go somewhere nobody would expect. A coefficient far crueler than "listed capacity × 0.75" was waiting.
Next → Local LLM Upgrade Log, Part 2: The Hardware Shopping List
Comments
Post a Comment