Serve an LLM Like an API with vLLM

Serve an LLM Like an API with vLLM

Published
AuthorDevansh

Ollama is for one machine and one user. vLLM is for serving a model to many users at once. It is a high-throughput inference engine for large language models that loads a Hugging Face model on a GPU and serves it two ways: as a Python API for batch work, and as an OpenAI-compatible HTTP server your existing apps can call without changes. Its two tricks are PagedAttention, which manages the attention key-value cache far more efficiently than naive allocation, and continuous batching, which keeps the GPU busy by adding and removing requests on the fly instead of waiting for a whole batch to finish. This post covers both modes, the hardware you need, and the mistakes that waste GPU money.

If running a model on one machine is new to you, read our Ollama guide first. It covers model sizing and memory, which decides everything here too. If the GPU server itself is new, our Ubuntu server setup guide covers the host basics. This post assumes you have a GPU machine and want the serving specifics.

Two modes: offline batch, or a live server

vLLM has two modes, and picking the wrong one is the most common waste. Offline batch inference is for a fixed set of prompts: classification, extraction, summarization, or synthetic data over a dataset. You load the model once in Python with the LLM class, pass a list of prompts, and vLLM schedules them together to keep the GPU busy, which is much faster than sending prompts one at a time. The shape of the code, from the vLLM quickstart, is: import LLM and SamplingParams from vLLM, create the engine with llm = LLM(model="facebook/opt-125m"), set sampling with SamplingParams(temperature=0.8, top_p=0.95), and generate with outputs = llm.generate(prompts, sampling_params). The same class also offers llm.chat for chat-formatted prompts. The second mode is the live server: vllm serve starts an HTTP server that implements the OpenAI chat completions API, so the openai Python package, curl, Open WebUI, and any tool that speaks the OpenAI API can call your model with no code changes. Use the batch mode when nobody is waiting for an answer, and the server mode when a person or an app is.

Install, and what the GPU needs

vLLM installs from PyPI into a Python environment, and it compiles some GPU kernels the first time it uses them, which needs the CUDA compiler on the machine. By default it downloads the model from Hugging Face, so a gated model needs a Hugging Face token with read permission, and the model has to be in a format vLLM supports: it loads Hugging Face-compatible models by inspecting the architectures field in the model's config.json. The hardware question is the same as with any LLM: the model weights have to fit in GPU memory, plus the key-value cache for the requests in flight. A 7 billion parameter model in 16-bit weights needs roughly 14 GB before the cache, so a single 24 GB card serves it and a 70 billion parameter model does not fit on one card at all. That is where the next section comes in. Check the installation page on the day you install, because the supported versions move, and never size a GPU purchase from a months-old guide.

Scaling past one GPU

vLLM uses only one GPU unless you tell it otherwise. To split a model across several GPUs, use tensor parallelism: pass tensor_parallel_size=4 to the LLM class in offline mode, or the matching flag to vllm serve. There is also a production path worth knowing about: the project ships an official Docker image, vllm/vllm-openai, for running the server as a container, which is the sane way to deploy it on a host you also use for other services. Whatever the shape, put the same operational basics around it that you would put around any server: the model weights are large files on disk, so alert on disk usage at 80 percent, not 99 percent; the server is a process that can die, so run it under a supervisor and alert on process health; and the API key is a secret, so set it with the VLLM_API_KEY environment variable and never bake it into an image. Our Grafana monitoring guide covers the dashboards and alerts, and the same four alerts apply here: latency, GPU memory, disk, and process health.

Measure it before you trust it

Two numbers tell you whether the server is actually fast: time to first token, which is what a user feels, and tokens per second across all requests, which is what your GPU budget buys. vLLM exposes Prometheus metrics from the server, so point your existing monitoring at it and watch both numbers under real load, not a single test prompt. If time to first token is high while the GPU sits idle, the bottleneck is usually the queue or the prompt processing, not the model. If tokens per second flatlines while GPU memory is full, you are cache-bound and need either a smaller model, fewer concurrent requests, or another card. Record a baseline on the first day, because the first slowdown after launch is otherwise a mystery with no before to compare against.

When vLLM is the wrong tool

vLLM is built for throughput: many requests, one model, GPUs kept busy. It is the wrong call for one-off chats on a laptop, where Ollama is simpler, and for a model you only call a few times a day, where a hosted API is cheaper than a GPU idling at full power. It is also the wrong call if the model you need is not open weights: vLLM serves models from Hugging Face, not closed APIs. The decision rule is a volume question. If the work is a few prompts a day, use a hosted API. If it is thousands of prompts a day, or a service your users hit constantly, a vLLM server on your own GPU is usually cheaper and always more private. If you want that server built, monitored, and kept up on hardware you own instead of a laptop that sleeps, book a free infra audit with DevOpsBY. We run GPU and node infrastructure for Web3 teams, and an inference server needs the same uptime, alerting, and runbooks as any other production service.

Disclaimer: This post is educational only, not financial advice. Figures are as reported by the sources on 2026-11-12 and may change - always confirm version numbers, requirements, and model sizes in the official docs before acting on them.

Related Articles

DevOps & Infrastructure Services

Need hands-on help? We build and manage the infrastructure behind the concepts covered in this article.

DevOpsBy Assistant

How can we help?

Select a topic to get started: