Running a Local LLM on Old Hardware: Quantization Explained

By Webster Noble

Running a Local LLM on Old Hardware: Quantization Explained

You don't need a shiny new GPU rig to run a local LLM on old hardware. Quantization lets you squeeze surprisingly capable models onto machines with 4-8 GB of RAM. including that older laptop you've got sitting in a corner. This guide uses a 2015-era laptop baseline, and honestly, the hardware is more capable than you'd think.

[!TIP]
Quantization shrinks model weight precision to cut RAM and disk usage. A 7B model that needs 14 GB full can run in 4 GB after a good quantization pass.

Why a Local LLM on Old Hardware Still Matters

Most LLM conversations assume you've got 32 GB of RAM and an RTX 3090. That's fine if you're building a dedicated AI box, but what about the rest of us? Old hardware is everywhere. workstations from 2015, laptops that used to be your daily driver, spare machines you can't bring yourself to throw away. They've all got life left in them.

The proprietary cloud model says: buy new hardware or rent ours. Quantization says: let's work with what you've got. That fits the theme here. reclaim your digital life without buying more hardware to run it.

What You Need

  • A Linux machine with at least 4 GB RAM (8 GB is the sweet spot)
  • ~5 GB free disk space for a quantized model
  • Ollama installed (or any tool that runs GGUF models)
  • An internet connection for the initial download

If you haven't installed Ollama yet, check out Self-Host a Local LLM With Ollama — it's a one-command install.

My Laptop Baseline

This is the kind of machine I have in mind for this guide: Linux 7.0, 7.6 GiB of RAM, and Intel Broadwell integrated graphics. The screenshot is the real baseline from this laptop, not a benchmark result. The model still needs to be tested on the machine before I call a particular setup fast or slow.

Terminal showing the author's Linux, memory, Intel graphics, and Mesa baseline" />

How Quantization Works (Without the Math)

Every model weight is stored as a number — originally 16-bit or 32-bit floating point. That's precise but big. Quantization says: what if we use 8-bit, 5-bit, or even 4-bit numbers instead? You lose a tiny bit of precision, but the model shrinks dramatically.

Here's the practical impact:

Precision Size (7B model) RAM Needed Quality Loss
FP16 ~14 GB ~16 GB None
Q8 ~7 GB ~8 GB Negligible
Q5 ~5 GB ~6 GB Very slight
Q4 ~4 GB ~5 GB Slight, acceptable
Q3 ~3 GB ~4 GB Noticeable
Q2 ~2.5 GB ~3 GB Significant

The sweet spot for most people is Q4 or Q5. you get 3x compression with quality that's good enough for daily tasks.

Step-by-Step: Getting a Quantized Model

Step 1 — Understand GGUF Files

GGUF (formerly GGML) is the file format for quantized models. You'll see filenames like mistral-7b-instruct-v0.2.Q4_K_M.gguf. Here's what that means:

  • Mistral-7B — the model name and size (7 billion parameters)
  • Q4 — quantization level (4-bit)
  • K_M — the quantization method (k-quant, medium variant)

The K_M variants are the best quality-to-size ratio. Avoid Q2_K unless you're truly memory-constrained.

Step 2 — Pick a Model That Fits Your RAM

The general rule: you need roughly the model file size in available RAM. Here's what fits common old hardware:

  • 4 GB RAM → Q3/Q4 models under 3 GB, or tiny models (Phi-3 Mini, Gemma 2B)
  • 8 GB RAM → Q4/Q5 7B models (Mistral, Llama 3.1, Qwen 2.5)
  • 16 GB RAM → Q4/Q5 13B-14B models, or even Q8 7B models

Step 3 — Download a Quantized Model

If you're using Ollama, it handles quantization automatically. The ollama pull command grabs a pre-quantized model:

ollama pull mistral

For more control, download a GGUF file directly from Hugging Face:

mkdir -p ~/models
wget -O ~/models/mistral-7b-Q4_K_M.gguf \
  https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGUF/resolve/main/mistral-7b-instruct-v0.2.Q4_K_M.gguf

You can also use hf (the Hugging Face CLI):

pip install huggingface-hub
hf download TheBloke/Mistral-7B-Instruct-v0.2-GGUF \
  mistral-7b-instruct-v0.2.Q4_K_M.gguf --local-dir ~/models

Step 4 — Run It

With Ollama, just run the model you pulled:

ollama run mistral

If you downloaded a GGUF file directly, use llama.cpp:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release
./build/bin/llama-cli -m ~/models/mistral-7b-Q4_K_M.gguf -i

You'll see the model load, then you can type prompts. On an 8 GB machine with a Q4 model, expect a few tokens per second. slower than a GPU, but perfectly usable for drafting, summarizing, and brainstorming.

Step 5 — Optimize for Your Hardware

A few tricks to squeeze more out of old hardware:

  • Use mmap mode — llama.cpp loads models with mmap by default, which means it doesn't load the whole model into RAM at once. This is good for old machines with limited memory.
  • Lower context length — a smaller context window uses less RAM. --ctx-size 2048 is enough for most tasks and saves a chunk of memory.
  • Pin the model to a single CPU — if your machine is sluggish during inference, try taskset to limit the model to one core so the rest of the system stays responsive.
  • Use swap — not ideal, but a 4 GB swap file can let a model that's slightly too big for your RAM actually run. It'll be slower, but it works.

Common Pitfalls

  • Model downloads but won't start? You're probably out of RAM. Check with free -h — if available memory is less than the model file size, try a smaller quantization level or a tinier model.
  • Output is garbage? You've gone too aggressive on quantization. Q2 and Q3 on smaller models can make them nearly useless. Back off to Q4 or Q5.
  • Incredibly slow? CPU-only inference on a 7B model is going to be slow on older hardware. Consider a 3B or even 1.5B model — Phi-3 Mini and Gemma 2B run surprisingly well on limited hardware.
  • Ollama pulls too much? Ollama uses its own quantization levels which don't map exactly to GGUF naming. If you need a specific quant, download the GGUF file directly and run it with llama.cpp.

Alternative Open-Source Options

  • llama.cpp — the engine behind most quantized model inference. Supports GGUF files, runs on everything from a Raspberry Pi to a server, and is the most flexible option for old hardware.
  • GPT4All — a desktop app that downloads and runs quantized models with zero config. Great if you don't want to touch a terminal. For more polished UIs, see Open Source ChatGPT Alternatives You Can Self-Host.
  • Ollama — the easiest path. One command to install, one to pull a model, one to run it. Handles quantization under the hood. See Self-Host a Local LLM With Ollama for the full walkthrough.

References

Conclusion

Quantization is the reason local AI isn't just for people with expensive hardware. You can run a genuinely useful language model on a laptop from 2015 with 8 GB of RAM. and for a lot of daily tasks, it's more than good enough. The cloud wants you to believe you need their servers. You don't.

Why This Matters

Understanding running a local llm on old hardware: quantization explained helps you make better decisions about your infrastructure. This post covers what you need to know and why it matters for day-to-day operations.

Try It

Pull a quantized model, run it on your oldest machine, and ask it something you'd normally send to ChatGPT. You might be surprised how capable a well-quantized 7B model actually is on limited hardware.