Running LLMs On Your Own Machine: The Stack That Actually Works

Running a capable language model on your own machine went from a weekend science project to a fifteen-minute install in about two years. The models got smaller and better at the same time, and the tooling stopped being hostile.

Whether it's worth doing depends on what you want. Local models are still behind the frontier hosted ones on hard reasoning. What they give you instead is privacy, zero marginal cost, and no rate limits — which for a large set of everyday tasks matters more than the last ten percent of quality.

Here's the stack that actually works, and how to decide whether you need it.

Star counts are from GitHub at the time of writing.

Ollama — start here

ollama/ollama · ~180,000 stars · Go · MIT

Ollama is the reason local models became accessible. Install it, run ollama run qwen3, and you have a model in your terminal. It handles downloading, quantisation formats, GPU offloading and memory management without asking you about any of them.

It also exposes an OpenAI-compatible API on localhost:11434. That detail is the important one: any tool written against the OpenAI API can usually be pointed at Ollama by changing a base URL. Your existing scripts keep working, with no per-token bill.

If you're evaluating whether local models are useful to you at all, install this and spend an hour with it before reading further.

llama.cpp — the engine underneath

ggml-org/llama.cpp · ~127,000 stars · C++ · MIT

llama.cpp is the inference engine much of this ecosystem is built on, Ollama included. Written in C++ with no Python dependency, it runs on nearly anything — Apple Silicon, CUDA, ROCm, Vulkan, plain CPU.

Use it directly when you want control Ollama abstracts away: specific quantisation levels, precise layer offloading between GPU and CPU, or embedding inference into a C++ application. Its GGUF quantisation format is what lets a model that nominally needs 140GB run in 20GB with modest quality loss.

The practical knowledge worth having: quantisation is the trade you're making. Q4 quantisation roughly quarters memory use against FP16, with quality loss that's small for most tasks and noticeable for hard reasoning. If a model feels dumber than its reputation, check what quantisation you're running before blaming the model.

vLLM — when you're serving other people

vllm-project/vllm · ~91,000 stars · Python · Apache-2.0

vLLM is built for throughput rather than convenience. Its PagedAttention technique manages GPU memory the way an operating system manages RAM, which lets it batch many concurrent requests efficiently.

The distinction from Ollama is single-user versus multi-user. Ollama is excellent for one person at a terminal. vLLM is what you deploy when an application has real users hitting it concurrently. If you're building a product on open models, this is the serving layer.

Overkill for personal use. Essential if you're serving.

Open WebUI — the interface

open-webui/open-webui · ~151,000 stars · Python

A ChatGPT-style interface that runs locally and talks to Ollama or any OpenAI-compatible endpoint. Conversation history, multiple models, document upload with RAG, multi-user accounts, web search integration.

This is what turns a local model from a terminal curiosity into something you'll actually use daily. Docker install, connects to Ollama automatically.

One caveat worth flagging: its licence is a modified one rather than a standard open-source licence, with restrictions around branding for larger deployments. Fine for personal and small-team use. Read it before building a commercial product on it.

AnythingLLM and LibreChat — the alternatives

Mintplex-Labs/anything-llm · ~66,000 stars · MIT — a desktop app plus server, oriented around documents and agents. The full RAG pipeline is built in: drop in PDFs, it chunks, embeds and retrieves. If your main use is chatting with your own documents, this is the shortest path.

danny-avila/LibreChat · ~43,000 stars · MIT — closest to the ChatGPT experience, and built to mix providers: local models alongside Anthropic, OpenAI and others in one interface, with MCP support. MIT licensed, which makes it the cleanest choice if commercial use is on the horizon.

What you actually need to run this

The honest hardware answer:

8GB RAM — 7-8B models at Q4. Usable for summarising, drafting, simple code. Noticeably weaker at reasoning.

16GB — 14B models comfortably, 32B at aggressive quantisation. This is where local models start feeling genuinely useful.

32GB+ — 32B models comfortably, 70B at low quantisation. Close enough to hosted quality that you'd have to be paying attention to notice on most tasks.

Apple Silicon punches above its weight here because memory is unified — a 32GB M-series Mac runs models that would need a serious discrete GPU on a PC. On NVIDIA, VRAM is the binding constraint, not system RAM.

What local models are actually good for

Where they win outright:

Anything involving private data. Client documents, financial records, trading journals, anything under NDA. The data never leaves your machine. This is the single strongest argument and it has nothing to do with quality.

High-volume, low-difficulty work. Classifying thousands of records, extracting fields, summarising a document backlog. Per-token pricing makes this expensive with hosted APIs and free locally.

Offline and unreliable connections. Works on a plane, works when an API has an outage, works when your provider changes pricing.

Development. Building against a local endpoint means you're not burning credits while debugging a loop that fires a hundred requests.

Where hosted models still win: genuinely hard reasoning, long complex code, anything where being wrong is expensive. Use the right tool.

A setup worth copying

For most people the sensible configuration is not "local instead of hosted" but both:

1. Ollama running as a service, with one small fast model and one larger capable one pulled.
2. Open WebUI or LibreChat as the interface, configured with both your local models and a hosted API key.
3. Route by task — local for private data, bulk work and drafting; hosted for the hard problems.

Cost after setup: electricity. And you retain the option to work entirely offline, which becomes valuable the first time a provider has an outage in the middle of something.

The thing to check before you commit

The models move faster than the tooling. A guide naming specific models is out of date in a quarter — which is why this one mostly doesn't. What's stable is the infrastructure: Ollama, llama.cpp and vLLM have been the answer for a while and will likely stay the answer.

So learn the tools, and stay loose about the models. Pull whatever is current, benchmark it on your own tasks rather than trusting leaderboards, and swap it out when something better lands. With Ollama that's one command, which is rather the point.