Over the last eighteen months, the most frequent question from enterprise clients has flipped. It is no longer “can we use generative AI?”, but a far more specific version: “can we process these documents without them leaving our perimeter?”.
It is a legitimate question, and today it has a technically solid answer for a significant share of use cases. Three factors together have made it possible: compact models that are finally useful, tooling that has matured to the point of being manageable by an ordinary IT team, and growing regulatory pressure (GDPR, the AI Act, NIS2, sector requirements in healthcare and finance) that makes the data boundary a matter of governance, not of technical preference. This article tries to separate the signal from the noise.
Distillation and quantisation, without the mythology
The accessibility of local models rests on two distinct techniques, often confused. Distillation trains a compact “student” model to reproduce the behaviour of a much larger “teacher” model: it is not lossless compression, it is a selective transfer of competence. The result behaves far better than its size would suggest — but it remains a model of 3–8 billion parameters. Quantisation, by contrast, reduces the numerical precision of the weights, typically from 16 to 4–5 bits (formats such as GGUF): the memory footprint drops by roughly a factor of four, with a marginal loss on most applied tasks.
The point that is rarely made is that the degradation is not uniform. Narrow, verifiable tasks — entity extraction, classification, extractive summarisation, data normalisation — hold up very well; long multi-step reasoning and analysis over very large contexts degrade sharply. The design consequence is simple: you design for the task, not for the benchmark. A 7B model that solves three well-defined use cases properly is worth more than a generalist that solves twenty poorly.
The minimal stack, and what is almost always missing
The runtime. Ollama has established itself as the de facto standard for local execution: it handles model downloads, versions and exposure via API, with an experience reminiscent of Docker: excellent for the single user and for pilots; for multi-user server scenarios it is worth evaluating throughput-oriented runtimes such as vLLM from the outset, because in production the bottleneck is not memory but concurrent requests.
The context. A local model, on its own, knows nothing about your company: you need a retrieval layer that indexes documents, wikis and tickets and passes the model the relevant fragments. Lightweight implementations such as LightRAG make this layer accessible without heavy infrastructure. And it is worth being explicit: in most document projects quality depends more on retrieval than on the model — we have written about this in detail, and it is the same lesson taught to us by our own platform, built precisely on these components: Ollama, vLLM, LightRAG.
Adaptation. Tools such as Unsloth have made fine-tuning with LoRA/QLoRA techniques practicable on modest hardware. The operational advice, though, is to treat it as the third option, not the first: first the prompt, then retrieval, and only if both fail in a measurable way do you introduce an artefact that has to be versioned and put back to the test at every change of base model.
What the tutorials leave out. Between a proof of concept and an enterprise system there is a layer that never appears in the guides: identity and permissions (the index must respect the user's permissions), logging, continuous quality evaluation, a model update process. This is where most of the real cost is concentrated.
Three scenarios where local genuinely works
1 · Document pre-triage on sensitive data. Contracts, HR files, clinical documentation, insurance claims. The model does not decide: it extracts, classifies, highlights anomalous clauses and prepares the work for a human reviewer. The value lies in the reading time saved; the risk is contained because the output is always verified.
2 · Semantic search across the internal knowledge base. Minutes, technical documentation, ticket history: natural-language retrieval over archives that nobody can navigate by folders any more. Here too the heavy lifting is done by retrieval; the model synthesises.
3 · Development assistance in isolated environments. Defence, critical infrastructure, certified environments: proprietary code does not leave the repository, and often the network to the outside world simply does not exist. Here local is not an optimisation — it is the only option.
A fourth, growing scenario joins these: operations at the edge — construction sites, plants, remote locations — where connectivity is intermittent and reliability matters more than the marginal quality of the answer.
Where local does not pay off
Honesty on this point is what separates a technical assessment from a marketing campaign. Extended reasoning — complex comparative analyses, synthesis across dozens of heterogeneous documents, editorial-quality text — remains the territory of frontier models, and the gap is not closed by quantisation. Unsized multi-user workloads: a model that answers in two seconds on a laptop can take twenty with ten concurrent requests; you size for throughput, not for memory. And the hidden total cost: a local model that is not kept up to date is technical debt — it needs periodic re-evaluations, regression tests and someone to own it. Below a certain volume threshold, the API remains far cheaper than running it in-house.
Sizing the hardware: practical rules
- Rule of thumb: for a model quantised to 4 bits, the GB of memory required are roughly the billions of parameters × 0.6 — plus space for the context window.
- Apple Silicon: unified memory helps — with 16 GB you can handle 7–8B models, with 32–64 GB you move up to 14–32B.
- Dedicated GPU: 8–12 GB of VRAM comfortably cover the 7–8B range at interactive speed.
- CPU only: suitable for overnight batch processing, much less so for synchronous interaction.
- Shared server: a 24–48 GB GPU serves around ten concurrent users on 7–8B models. Beyond twenty people it is almost always more efficient — operationally too — than installing on every endpoint.
Three questions to decide (and the architecture that follows)
Three questions, in this order: can the data leave the perimeter? (an organisational question before a technical one); is the task narrow and verifiable? (if you cannot say what a correct answer looks like, no infrastructure will save the project); does the volume justify the infrastructure? (below certain thresholds the API is the rational choice).
In most cases the answer is not binary, and the most sensible architecture is hybrid: a routing layer that directs requests involving sensitive data to the local model — within the dedicated perimeter — and everything else to the cloud, with explicit, traceable policies. Data sovereignty becomes a property of the system, not a functional sacrifice.
A 90-day adoption path
Weeks 1–3 · Perimeter. Data classification and the selection of two real use cases, each with a business owner. Weeks 4–8 · Field test. A working prototype and, above all, an evaluation set: 30–50 real cases with the expected answer — the most underrated investment, the one that separates a measure of success from an impression. Weeks 9–12 · Pilot. Release to 10–20 users, metrics, a documented decision: extend, correct or stop.
In summary
Local AI does not replace the cloud and is not a universal answer. It elegantly solves a well-defined class of problems: the data cannot move, the task is circumscribed, verifiability matters more than brilliance. For that class — which in regulated organisations is anything but marginal — the technology is mature, runs on hardware often already in the building and can be evaluated within a quarter at a contained cost. The rest, as ever, is a question of architecture: if you want to set it up with a minimum of rigour, let's talk.
