Run an LLM on Your Own Machine with Ollama

Run an LLM on Your Own Machine with Ollama

Published
AuthorDevansh

Running a large language model on your own machine takes two commands. The first installs Ollama. The second downloads a model and starts a chat. Nothing leaves the machine, there is no API key, no per-token bill, and no rate limit. The catch is everything else: the model has to fit in your memory, it runs at the speed of your hardware, and a model small enough for a laptop is markedly worse than the one behind a chat app. This post covers the install, what actually fits in a given amount of RAM, how to pick a model, the two settings that trip up almost everyone, and how to wire a local model into your own code.

If you have never run a server of your own, start with our Ubuntu server setup guide, which covers the same basics - SSH, a firewall, and a service that stays up - and our Docker Compose guide if you would rather run Ollama in a container. This post assumes you can open a terminal and want the local-AI specifics.

Install Ollama in one command

On Linux, the install from the Ollama project README is a single command: curl -fsSL https://ollama.com/install.sh | sh. On macOS and Windows, download the installer from ollama.com/download and install it like any other app. Ollama is one program that bundles three things: an inference engine, a model registry, and a command line. Once it is installed, running your first model is one more command: ollama run llama3.2 pulls the Llama 3.2 3B model, about 2 GB to download, and drops you into a chat. List what you have with ollama list, stop a model with ollama stop, and pull a model without running it with ollama pull. The workflow is deliberately close to Docker: pull, run, list, stop.

Will it fit? The part that decides everything

A model is a file, and it has to fit in memory before it can do anything. As listed in the Ollama library on 21 September 2026, Gemma 3 at 270 million parameters is 292 MB, while Qwen 3 at 235 billion parameters is 142 GB and GPT-OSS at 120 billion parameters is 65 GB. The download size is not the running size: a loaded model also carries its key-value cache, its working buffers, and the runtime, so budget about 1.2 times the download size in memory you actually have free. Ollama's own quickstart puts the same idea in one sentence about its default model: the download is about 7.2 GB, 8 GB of available VRAM is recommended, and with less VRAM Ollama can use system RAM but responses get slower. As a starting point for coding, Qwen2.5-Coder 7B is the usual recommendation, and Llama 3.2 3B at 2.0 GB needs about 4 GB of RAM. Check the library on the day you install, because model sizes and the default model change, and never buy hardware for a model size you read in a months-old guide.

Your prompts stay on the machine, with one exception

Ollama's FAQ is explicit: Ollama runs locally, and it does not see your prompts or data when you run locally. That is the whole point of self-hosting a model, and it is why local models fit work you would never paste into a hosted chat: internal documents, customer data, unreleased code. But there is one exception to check before you trust it. The same binary can also call Ollama's hosted models, and those requests do go to a server. If you want local-only, set OLLAMA_NO_CLOUD=1 so the hosted path is off entirely. The model file itself sits in a folder on disk, works offline, and does not change under you until you pull a new version, which is a property no hosted API can offer.

Use it from code: the local API on port 11434

Ollama behaves like a server, not an app. When it is running, it exposes a local REST API at http://localhost:11434, and anything that can send an HTTP request can use it. It speaks the OpenAI and Anthropic request formats as well as its own, so code written against a hosted API often works against a local model with only the base URL changed. Send a prompt to /api/generate for a single completion, or to the chat endpoint for a conversation. Two settings trip up almost everyone. First, the context window: every model has a maximum context, and older or smaller models cap at 4K to 8K tokens while Llama 3.2 supports up to 128K, so a long document that works on one model will be cut off on another. Set it with OLLAMA_CONTEXT_LENGTH or the Modelfile setting, and remember that memory use grows with context size. Second, keep-alive: by default a loaded model unloads after a while, so the next request pays the load time again. Set OLLAMA_KEEP_ALIVE=5m when you want it to stay warm. For a server instead of a laptop, run it in Docker with docker run -d -p 11434:11434 ollama/ollama and point your apps at that host.

When a local model is the wrong tool

A local model is the right call when privacy matters, when the work is offline, or when you want a fixed cost instead of a per-token bill. It is the wrong call when you need the best reasoning available, when the model has to stay current, or when the job needs tool calling that only large models do reliably: small models under 7 billion parameters often fail to produce valid tool-call JSON, which breaks agent workflows built on top of them. The honest middle ground is to prototype locally and serve the heavy work from a hosted model or a GPU server, and our Web3 DevOps explainer covers the same build-versus-buy thinking for infrastructure in general. If you want a local model running on a server you own, with monitoring and alerts on it, book a free infra audit with DevOpsBY. We run the servers behind Web3 launches, and a local model is just another service that needs uptime, disk, and someone watching it.

Disclaimer: This post is educational only, not financial advice. Figures are as reported by the sources on 2026-11-12 and may change - always confirm version numbers, requirements, and model sizes in the official docs before acting on them.

Related Articles

DevOps & Infrastructure Services

Need hands-on help? We build and manage the infrastructure behind the concepts covered in this article.

DevOpsBy Assistant

How can we help?

Select a topic to get started: