How to Self-Host DeepSeek-R1: A Hardware and Software Guide

DeepSeek-R1 has set a new standard for open-weights reasoning models, matching frontier closed models in math, science, and coding tasks. Because it is released under a permissive open license, developers and enterprises can host it locally or on private cloud servers. This guide explains the hardware sizing, quantization options, and software tools required to successfully run DeepSeek-R1.

The Scaling Dilemma: Which Model Size is Right?

DeepSeek-R1 is available in several versions, ranging from distilled models (based on Llama and Qwen architectures) to the full-weight Mixture-of-Experts (MoE) 671-billion parameter model. Your hosting strategy depends on your hardware constraints and performance requirements.

Hardware Sizing Guide for the Full 671B Model

Running the full 671B parameter Mixture of Experts model requires professional ML-ops infrastructure due to its sheer scale. Although the active parameters per token are only about 37 billion, the entire 671 billion parameter weight matrix must reside in memory for fast inference.

1. Multi-GPU Cloud Node Setup

To achieve real-time inference speeds (15–30 tokens/sec), the full model is typically run in FP8 (8-bit floating point precision) or Int4/Int8. This requires:

• FP8 Precision: A minimum of 720 GB VRAM. This is typically deployed on a single node with 8x H100 (80GB) or 8x A100 (80GB) connected via NVLink.
• Int4 Quantized Precision: A minimum of 380 GB VRAM. This can be run on 8x RTX 4090 (24GB) GPUs (totaling 192GB, but offloading parts or using aggressive quantization) or 4x A100 (80GB) GPUs.

2. Apple Silicon Mac Clusters (The Budget Alternative)

For small teams or local testing, high-end Apple Mac Studios are an excellent unified-memory alternative. A Mac Studio with an M3 Ultra and 192GB unified memory can run R1-Distill-70B at full speed, or a network-clustered setup of multiple Macs using frameworks like Exo can run the full 671B model in quantized format.

Quantization: Balancing Speed and Precision

Quantization compresses the model weights to smaller numerical formats, saving memory at the cost of slight precision loss. For DeepSeek-R1, we recommend these formats:

• FP8 (8-bit float): Currently the production standard. Offers near-zero quality loss and leverages native hardware acceleration on newer GPUs like H100, RTX 40-series, and L40S.
• GGUF (Q4_K_M / Q8_0): Best for CPU + GPU hybrid offloading (e.g., local desktops, Apple Macbooks). A 4-bit quantized version (Q4) reduces RAM requirements by ~70% while retaining over 95% of R1's reasoning capability.
• EXL2 (ExLlamaV2): The preferred choice for dedicated Linux GPU servers running consumer card rigs (like multiple RTX 3090/4090s) due to its high-speed token generation.

Software Selection & Setup

Several software stacks exist to host DeepSeek-R1, depending on whether you are running a single local machine or building a scalable API endpoint.

1. Ollama (Best for Local Desktop & Prototyping)

Ollama is the easiest way to host distilled versions of DeepSeek-R1 on Windows, macOS, or Linux. It automatically configures GPU acceleration and offloading.

To run the 32B version, open your terminal and execute:

ollama run deepseek-r1:32b

Ollama exposes a local OpenAI-compatible API at http://localhost:11434/v1, allowing you to easily hook it up to IDE extensions like Cursor or Continue.

2. vLLM (Best for Multi-GPU Production APIs)

vLLM is a high-throughput, easy-to-use LLM serving engine. It is ideal for serving DeepSeek-R1 on cloud GPU instances (like RunPod, Lambda Labs, or AWS).

To run the FP8 quantized version of DeepSeek-R1 on an 8x H100 node with vLLM, use the following docker command:

docker run --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model deepseek-ai/DeepSeek-R1-FP8 \
  --tensor-parallel-size 8 \
  --max-model-len 32768

This command runs tensor parallelism across all 8 GPUs and exposes an endpoint on port 8000 with a 32K token context window.

Optimizing DeepSeek-R1 Performance

Because DeepSeek-R1 generates a long "thinking process" before producing the final answer, latency can be high. Apply these configurations to improve performance:

• Enable Chunked Prefill: Helps manage large prompts (system prompts + context) by processing them in chunks, reducing GPU memory spikes.
• Enable FlashAttention-2: Crucial for maintaining speed as context lengths exceed 16K tokens.
• System Prompt Tuning: Ensure your system instructions don't suppress the <think> tags. R1's reasoning capability is hard-coded to trigger inside these XML tags. Suppressing them lowers answer quality.