https://d3v0ps.cloud/tags/vram/ Recent content in Vram on d3v0ps.cloud Hugo en <a href="https://creativecommons.org/licenses/by-nc/4.0/" target="_blank" rel="noopener">CC BY-NC 4.0</a> Thu, 28 May 2026 20:22:37 +0000 https://d3v0ps.cloud/posts/2026/05/running-llama.cpp-in-low-vram-a-practical-setup-guide/ Thu, 28 May 2026 20:22:37 +0000 https://d3v0ps.cloud/posts/2026/05/running-llama.cpp-in-low-vram-a-practical-setup-guide/ <h2 id="tldr">TL;DR</h2> <p>I published <a href="https://github.com/aaronbolton/llamacpp-low-vram">aaronbolton/llamacpp-low-vram</a> as a minimal, copy-paste starting point for running <code>llama-server</code> on a GPU with limited VRAM. Adjust <code>--n-cpu-moe</code> and <code>--ctx-size</code> for your hardware, and you get an OpenAI-compatible local endpoint without buying a bigger card.</p> <hr> <p>Running a large model on a modest GPU is mostly a VRAM budgeting problem. The model weights, the KV cache, and the attention computation all compete for the same pool of memory. This post explains the launch script I use, why each flag is there, and how to tune the two settings that have the most impact.</p>