Hosting OpenAI-Compatible Local API Servers with vLLM

Tutorial · 9 min read · Updated July 2026

One of the main advantages of vLLM is its native support for the OpenAI Chat Completions protocol. This allows developers to swap proprietary APIs with locally hosted open-weights models (like DeepSeek-V3 or Qwen3) with zero code modifications.

1. Launching the vLLM OpenAI Server

Assuming you have Docker and NVIDIA Container Toolkit installed, run this single CLI command to serve a model:

python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-Coder-32B-Instruct \
    --port 8000 \
    --tensor-parallel-size 1

2. Connecting your Application

Update your application's SDK client parameters to point to the local instance:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="token-not-needed-for-local"
)

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-Coder-32B-Instruct",
    messages=[{"role": "user", "content": "Write a fast binary search."}]
)