Build your own AI lab — the complete 2026 self-hosting guide (GLM-5.2, DeepSeek V4, DGX Spark)

A real, end-to-end plan to build your own AI lab in 2026 — pick the hardware, buy the right parts, install the stack, and actually run today's open-weights giants (GLM-5.2, DeepSeek V4) plus the smaller models you'll live in day to day. We start from the one law that decides everything (VRAM capacity and memory bandwidth), walk the full hardware ladder (single RTX 4090 → Mac Studio → NVIDIA DGX Spark → multi-GPU workstation → an 8× H200 node), give a parts list per tier, the exact install + vLLM/Ollama commands, the honest self-host-vs-API cost math, and a long troubleshooting section built from what people actually hit (OOM, NCCL, the CPU-offload cliff, the DGX Spark bandwidth trap, KV-cache blow-ups). Numbers come from vendor specs and 2026 build write-ups; sources at the end.

This is the "read one thing and understand the whole problem" guide. No assumed background: I explain why a 744-billion-parameter model needs a server rack but a 14-billion one runs on a gaming GPU, what "memory bandwidth" actually buys you, and where every dollar goes. If you're a researcher, a tinkerer, or a small team deciding whether to build a box or just call an API — this maps the whole terrain.

What an "AI lab" actually is

Strip the glamour and an AI lab is a small number of concrete jobs done on hardware you control:

Inference — run a trained model to get answers, code, or agent actions. This is 90% of what most people want.
Serving — expose that model as an API so your apps, editors, and agents can call it (often the OpenAI-compatible API shape).
Fine-tuning — nudge an open model on your own data (LoRA / QLoRA for most people; full fine-tunes need real iron).
Experiment + research — poke at the model itself: routing, attention, quantization, long-context recall.

You build a lab for a few honest reasons: privacy / data sovereignty (your data never leaves the building), cost at high volume, latency control, no rate limits, the ability to fine-tune freely, and plain learning. If none of those apply to you, the punchline of this entire guide — said up front so you can't miss it — is: just use the API. Self-hosting frontier models is rarely cheaper than the API until you are pushing enormous, steady volume. We'll prove that with numbers later. Everything below is for when you have a real reason to own the hardware anyway.

The one law that governs everything: VRAM & bandwidth

Before a single purchase, internalize the two walls every local-LLM setup runs into. Get these and the whole hardware market suddenly makes sense.

Wall 1 — the capacity wall (will it even fit?)

To run a model, its weights must fit in fast memory (GPU VRAM, or Apple/Grace unified memory). The size is simple: memory ≈ number of parameters × bytes per parameter. The bytes depend on precision:

Precision	Bytes / param	Quality	70B model weighs…
FP16 / BF16	2	Full, no loss	~140 GB
FP8 / INT8	1	Near-lossless	~70 GB
INT4 / Q4	~0.5	Small, usually OK	~40 GB

Then you add two more costs that beginners always forget: the KV cache (memory for the conversation/context — grows with how long your context is and how many requests run at once) and ~10–20% runtime overhead. A rough working formula:

Total VRAM ≈ (params × bytes_per_param)   # weights
            + KV cache (grows with context length × concurrency)
            + 10–20% overhead

The #1 sizing mistake. People size for weights only and get a surprise out-of-memory crash the moment context gets long. At very long context the KV cache can rival or exceed the weights — one source measured the KV cache at 3.4× the model weights at 128K context on a large model. Always budget weights + KV + overhead, never weights alone.

The #2 MoE mistake. For a Mixture-of-Experts model, you must hold all the parameters in memory, not just the active ones. GLM-5.2 runs ~40B active per token but you still need room for all ~744B stored. "Active params" tells you about speed; "total params" tells you about VRAM. Never size a MoE model off its active count.

Wall 2 — the bandwidth wall (how fast will it be?)

Fitting the model is only half the battle. Token generation speed is set by memory bandwidth, not raw compute. Here's the why in one sentence: to produce each single token, the hardware must read the model's active weights out of memory. More bytes to read per token, or slower memory, means fewer tokens per second.

This is the single most misunderstood thing in local AI, and it is exactly why the NVIDIA DGX Spark gets mixed reviews (more on that below): it has plenty of memory capacity (128 GB) but modest memory bandwidth (273 GB/s), so big models fit but generate slowly. A stack of three old RTX 3090s, with far less total capacity but much higher aggregate bandwidth, generates a 120B model at ~124 tokens/sec versus the Spark's ~38.

Fig 1 — capacity decides if it runs; bandwidth decides how fast. Hardware shopping is just buying past both.

Quick rule of thumb. "Will it fit?" → look at VRAM in GB. "Will it be fast?" → look at memory bandwidth in GB/s. A 24 GB RTX 4090 has ~1,000 GB/s; a Mac/Grace unified-memory box may have 100s of GB of capacity but only 200–800 GB/s. That trade — big-but-slower unified memory vs small-but-fast GPU memory — is the whole story of which box suits which job.

The 2026 open-model lineup and what each one demands

What are we even trying to run? Here are the headline open-weights models of 2026 and the hardware reality each one implies. (The smaller ones are where most real work happens — don't fixate on the giants.)

Model	Total / active params	Context	Realistic self-host
DeepSeek V4-Pro	1.6T / 49B	1M (384K out)	Datacenter only — 8× H200 (FP8) or 16× H100 (BF16)
GLM-5.2	~744B / ~40B	1M (128K out)	8× H200 (FP8) / ~4–5 GPUs at INT4 / else cloud
DeepSeek V4-Flash	284B / 13B	1M	2× H200 or 2× RTX Pro 6000 or 4× A100 — feasible for a serious lab
Mid models (~70B)	70B dense	128K	~48 GB VRAM (1× RTX 6000 / 2× 4090 / Mac Studio)
Workhorse (~32B)	32B dense	128K	Single RTX 4090 24 GB (Q4) — the sweet spot
Small (7–14B)	7–14B dense	32–128K	Any 16 GB GPU, a laptop, or a Mac

The pattern: the two flagship MoE giants (DeepSeek V4-Pro, GLM-5.2) are fundamentally datacenter-class at full precision — you do not fit ~744B–1.6T parameters in a desk machine. But DeepSeek's V4-Flash (284B) is the model the "serious home/lab" tier can actually serve, and the 70B / 32B / 14B classes run on hardware you can buy off a shelf. Plan the lab around the model you'll truly use 95% of the time, then decide separately whether to also reach a flagship via the cloud.

Why DeepSeek V4 is cheaper to run than its size suggests. V4's hybrid attention (Compressed Sparse Attention + Heavily Compressed Attention) slashes the KV cache to roughly 7–10% of the previous generation's footprint at 1M context — V4-Flash needs only ~10 GB of KV cache for the full million tokens. The weights still dominate, but long context stops being the budget-killer it is on older designs. GLM-5.2's IndexShare sparse attention does the same kind of job. Modern flagships are explicitly engineered to be cheaper to serve at long context.

The hardware ladder — five tiers, pick your rung

This is the heart of the guide. Five tiers, cheapest to most serious. Each lists what it runs, what it costs, and who it's for.

Fig 2 — the five rungs. Most people should stand on Tier 1 or 2 and rent Tier 5 when they truly need a flagship.

Tier 1 — single consumer GPU (the default starting point)

One NVIDIA gaming card in a normal PC. This is where almost everyone should begin. The card is the lab; everything else is a box to hold it.

RTX 4090 (24 GB) — the long-standing sweet spot. Runs ~32B models at Q4 comfortably, ~1,000 GB/s bandwidth so it's fast. ~$1,600–2,000.
RTX 5090 (32 GB) — Blackwell, first 32 GB consumer card, handles ~45B-class models, higher bandwidth still.
RTX 4060 Ti (16 GB) — budget entry, ~14B models, great for learning.

Runs: 7B, 14B, 32B (Q4) snappily; a 70B only via aggressive offload (slow). For: individuals, learning, coding-assistant duty, LoRA fine-tuning of small models. This single card plus Ollama covers most people's entire needs.

Tier 2 — Apple Silicon (Mac Studio / Mac mini)

Apple's unified memory is a cheat code for capacity: the CPU and GPU share one big pool, so a Mac Studio M3 Ultra with 96–512 GB can load models a 24 GB GPU never could — 70B easily, even up to ~180B at heavy quantization. Bandwidth is high for unified memory (M3 Ultra ~800 GB/s) but still below a dedicated GPU, so big models run but aren't blazing.

Mac Studio M3 Ultra, 96–256 GB — runs 70B in-memory, quiet, sips power (~$4k–10k by RAM).
M4 Max MacBook Pro, 64–128 GB — a portable lab; 70B models in a backpack.

For: people who want big-model capacity, silence, and low power over raw speed; mixed dev machines; anyone already in the Apple ecosystem. Pair with LM Studio or Ollama (MLX backend, ~69 tok/s measured on small models).

Tier 3 — NVIDIA DGX Spark (the desk "supercomputer")

The DGX Spark is NVIDIA's GB10 Grace Blackwell desktop unit: 128 GB unified LPDDR5x, 1 petaFLOP FP4 compute, 20-core Arm CPU, 4 TB NVMe, full CUDA stack, ~$3,000–4,699. It can run models up to 200B parameters locally, and two units linked over its built-in ConnectX networking reach 405B. It ships with NVIDIA's whole AI software stack preloaded and works with essentially every open-source LLM framework because it's real CUDA.

The DGX Spark reality check — read before you buy. Its memory bandwidth is 273 GB/s, shared between CPU and GPU. Because generation speed is bandwidth-bound, big models fit but feel slow: under-20B run at interactive 15–40 tok/s, 20–70B feel sluggish at 3–10 tok/s, and 70B+ technically load but are too slow for real-time agent chat. On a 120B model it does ~38 tok/s — slower than three older RTX 3090s (~124 tok/s) whose aggregate bandwidth beats it. NVIDIA even signals a future generation will raise bandwidth. The Spark shines as a development / prototyping / fine-tuning / agent-building appliance with a huge memory pool and a real datacenter software stack on your desk — not as a fast big-model chat server. Buy it for what it is.

For: developers who want the genuine NVIDIA/CUDA datacenter workflow locally, big-memory prototyping, fine-tuning, and multi-model agent development, and who value the unified 128 GB + software stack over peak tokens/sec.

Tier 4 — multi-GPU workstation / small server (the serious lab)

This is where you cross from "personal" into "serves a team and runs a 284B model." You put 2–4 professional GPUs in a workstation/server chassis and use tensor parallelism to split a model across them.

RTX PRO 6000 Blackwell (96 GB each) — the modern prosumer powerhouse. 2× = 192 GB (runs DeepSeek V4-Flash); 4× = 384 GB.
A100 80 GB / H100 80 GB — datacenter classics; 4× A100 = 320 GB (a common V4-Flash setup, since vLLM likes power-of-two GPU counts).

Runs: 70B at full precision, DeepSeek V4-Flash (284B), GLM-5.2 at INT4 if you stack enough cards. For: startups, research groups, anyone serving a real model to multiple users. Budget $20k–60k+ and plan seriously for power and cooling (next sections).

Tier 5 — the frontier node (8× H200) — usually rent, don't buy

To run GLM-5.2 or DeepSeek V4-Pro at full/FP8 precision you need an 8× H200 node: 8 × 141 GB = ~1,128 GB aggregate VRAM, enough for the ~744 GB of FP8 weights plus KV cache and overhead. Full BF16 roughly doubles that and wants ~16 GPUs. This is a $250k+ purchase, or — far more sanely — an AWS p5.48xlarge (8× H100) at ~$55/hr on-demand (~$33/hr reserved). For: almost nobody owns this; you rent it by the hour for a fine-tune or a burst, or you just use the model's API.

The shopping list — what to actually buy (per tier)

A GPU alone is not a lab. Here's the full bill of materials, so nothing surprises you at checkout. The non-GPU parts matter more than beginners expect — an undersized PSU or a hot case will quietly wreck performance.

Part	Tier 1 (single GPU)	Tier 4 (multi-GPU)	Why it matters
GPU	1× RTX 4090/5090	2–4× RTX PRO 6000 / A100	The whole game — VRAM + bandwidth.
CPU	Ryzen 7 / i7 (8c+)	Threadripper / EPYC / Xeon (many PCIe lanes)	Multi-GPU needs lots of PCIe lanes.
Motherboard	B650 / Z790	WRX90 / workstation, ≥2 PCIe x16 slots	Slot count & spacing limit GPU count.
System RAM	32–64 GB	128–512 GB (≥ total VRAM)	Model load, offload buffer, OS headroom.
PSU	1000 W 80+ Gold	1600 W+ (or dual PSU)	A 4090 pulls ~450 W; 4 GPUs need serious supply.
Storage	2 TB NVMe	4 TB+ NVMe (models are 100s of GB)	Big weights download & load fast off NVMe.
Cooling	Good airflow case	High static-pressure fans / blower GPUs	Throttled GPUs lose tokens/sec silently.
Networking	—	NVLink / 10–100 GbE / InfiniBand for multi-node	Inter-GPU bandwidth caps tensor-parallel speed.
UPS	Optional	Recommended	A power blip mid-fine-tune wastes hours.

One-line buying advice from the build community: prefer one bigger card over two smaller ones for inference. Two GPUs split the model but add inter-card communication latency, so a single 48 GB card usually beats 2× 24 GB for pure speed. Go multi-GPU when you need capacity a single card can't give (to fit a bigger model), not to chase speed on a model that already fits one card.

The software stack — OS, drivers, and the engine zoo

Hardware bought, here's the software ladder. On NVIDIA the base is always the same: Ubuntu Linux → NVIDIA driver → CUDA → (optional) Docker + NVIDIA Container Toolkit → an inference engine. The interesting choice is the engine, and the right one depends entirely on whether you're a person at a keyboard or a service under load.

Engine	Best for	Notes
Ollama	Solo use, fast start	One command to pull + run; OpenAI-compatible API; great on Mac (MLX) & single GPU.
LM Studio	GUI users, Mac	Desktop app + headless CLI; KV-cache checkpointing for agents.
llama.cpp	Tinkerers, CPU/Mac, GGUF	Maximum control; the GGUF quant ecosystem; runs almost anywhere.
vLLM	Serving many users	The production default. PagedAttention KV cache, continuous batching, tensor parallelism. Prefers power-of-two GPU counts.
SGLang / TensorRT-LLM	Max throughput	Squeeze the last drops of performance; more setup effort.

Choosing in one line. Just you, just chatting → Ollama (or LM Studio on a Mac). Serving a model to apps / many users / an agent fleet → vLLM. The quantization format follows the engine: GGUF for llama.cpp/Ollama/single-GPU/Apple; AWQ or GPTQ for vLLM serving (better throughput).

Step-by-step — building the lab from a bare Ubuntu box

Here's the concrete path on a fresh NVIDIA machine. Two tracks: the quick solo path (Ollama) and the serving path (vLLM).

1. Base system — driver + CUDA

# Fresh Ubuntu 24.04. Install the NVIDIA driver + CUDA toolkit.
sudo apt update && sudo apt install -y nvidia-driver-560 build-essential
sudo reboot
# After reboot, confirm the GPU is seen:
nvidia-smi          # shows GPU, VRAM, driver/CUDA version

If nvidia-smi prints your card and memory, the hard part is done. If it errors, that's a driver/kernel mismatch — see troubleshooting.

2. Quick solo path — Ollama

curl -fsSL https://ollama.com/install.sh | sh
ollama run qwen3:32b          # pulls a ~32B model (Q4) and chats
# It now serves an OpenAI-compatible API on http://localhost:11434
curl http://localhost:11434/v1/chat/completions \
  -d '{"model":"qwen3:32b","messages":[{"role":"user","content":"hello"}]}'

That's a working lab. On a 24 GB RTX 4090 a 32B Q4 model runs at interactive speed. Point your editor/agent at that local endpoint and you're done.

3. Serving path — vLLM

pip install "vllm>=0.8.0"
# Serve a model with tensor parallelism across GPUs (power-of-two count):
python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen3-32B-AWQ \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --port 8000

Key flags you'll tune constantly: --tensor-parallel-size (number of GPUs — keep it 1/2/4/8), --max-model-len (context length — the lever that controls KV-cache VRAM), and --gpu-memory-utilization (how much VRAM vLLM may claim; lower it if you OOM).

Running GLM-5.2 specifically

Reality first: GLM-5.2 is ~744B total parameters, so the honest options are (a) rent/own an 8× H200 node and serve FP8 weights (~744 GB) with room for KV cache, (b) run a community INT4 quant (~372 GB) on ~4–5 high-VRAM GPUs and validate quality on your tasks, or (c) just use the Z.ai API. There is no desktop that runs full GLM-5.2 fast. The serving shape on a node looks like vLLM with a high tensor-parallel size:

# Conceptual — on an 8×H200 node, FP8 weights
python -m vllm.entrypoints.openai.api_server \
  --model zai-org/GLM-5.2-FP8 \
  --tensor-parallel-size 8 \
  --max-model-len 131072 \
  --trust-remote-code \
  --port 8000
# vLLM prefers power-of-two TP sizes (1,2,4,8). Validate the exact GPU count
# against the model card's layer/KV-head values + your target concurrency.

Don't size GLM-5.2 off its 40B active count. You need memory for all ~744B. At FP8 that's ~744 GB of weights before KV cache — squarely a multi-GPU node, not a workstation. If you only have a workstation, run a smaller model locally and reach GLM-5.2 through its API for the few tasks that genuinely need it.

Running DeepSeek V4 specifically

This is the flagship a serious lab can self-host — but pick the right variant. V4-Pro (1.6T) is datacenter-only (8× H200 FP8, or 16× H100 BF16). V4-Flash (284B) is the realistic one: ~158 GB of FP4+FP8 weights, only ~10 GB KV cache for the full 1M context (thanks to its compressed attention), ~170–175 GB total — which fits 2× H200, 2× RTX PRO 6000, or 4× A100 80 GB.

# Download V4-Flash
pip install huggingface_hub
huggingface-cli download deepseek-ai/DeepSeek-V4-Flash --local-dir ./deepseek-v4-flash

# Serve on 4× A100 80GB (use --tensor-parallel-size 2 for 2× H200)
pip install "vllm>=0.8.0"
python -m vllm.entrypoints.openai.api_server \
  --model ./deepseek-v4-flash \
  --tensor-parallel-size 4 \
  --max-model-len 131072 \
  --trust-remote-code \
  --port 8000

Parallelism for MoE — two knobs. Tensor parallelism (TP) splits each layer across GPUs (use when weights exceed one card). Expert parallelism (EP) spreads different experts across GPUs (MoE-specific efficiency). For V4-Flash, TP alone is enough; for V4-Pro you combine TP + EP. And resist over-quantizing: pushing V4 to INT4 measurably hurts math, reasoning, and agentic tasks — stick to the official checkpoints for production.

The silent VRAM killer — context length & KV cache

The mistake that turns a working setup into a crashing one: cranking context length. The KV cache grows with context_length × concurrency, and at long context it can dwarf the weights. Doubling context from 4K to 32K adds 2–4 GB on a 7B model and 6–10 GB on a 32B model; at 128K it can be several times the weight memory.

Practical context discipline. Start --max-model-len modest (e.g. 32K) and raise it only as far as your VRAM and real workload need. Use an engine with paged KV cache (vLLM's PagedAttention cuts KV waste from 60–80% to near zero and enables 2–4× more concurrent requests). Prefer models with compressed attention (DeepSeek V4's CSA/HCA, GLM-5.2's IndexShare) when you genuinely need 1M context — they're built to keep the KV cache small.

The honest money math — self-host vs API

Here's the part that saves you the most money: a clear-eyed break-even. The formula is simple — break-even tokens/day = daily self-host cost ÷ blended API price per token. Run it for DeepSeek V4-Flash on a rented 8× H100 node:

Item	Number
Rented node (AWS p5.48xlarge, reserved)	~$33/hr → ~$790/day
DeepSeek API blended price	~$0.21 / 1M tokens
Break-even volume	~3.8 billion tokens/day
Can one node actually push that?	No — a single node can't physically sustain it for V4-Flash

The uncomfortable conclusion. For all but the very largest, steadiest serving operations, the API is cheaper than self-hosting a flagship — often by a lot. Self-hosting wins when non-cost factors dominate: data sovereignty / regulatory residency, custom fine-tuning, guaranteed latency, air-gapped operation, or no rate limits. Decide on those grounds, not on a fantasy of beating API pricing on raw tokens. Size the lab to your reason, not to a spreadsheet that usually says "just call the API."

Networking the lab — when one box isn't enough

Cross into multiple machines and the network becomes the bottleneck. Tensor parallelism makes GPUs talk constantly, so inter-GPU bandwidth caps your speed. The hierarchy, fastest to slowest: NVLink (GPUs in one box) > InfiniBand / NVIDIA ConnectX (between boxes, the DGX Spark's trick to pair two units for 405B) > 100/25/10 GbE > ordinary 1 GbE (too slow for model parallelism). Keep a tensor-parallel model inside one box where possible; only go multi-node when a model won't fit a single machine, and then use the fastest interconnect you can afford.

The physical stuff people forget — power, heat, noise, electricity

Power draw: an RTX 4090 pulls ~450 W; four pro GPUs plus CPU can exceed 2 kW. Size the PSU with headroom (1600 W+ for multi-GPU, or dual PSUs) and check your wall circuit — a 15 A / 120 V circuit tops out near 1.8 kW.
Heat: that power becomes heat in your room. A multi-GPU box is a space heater; plan ventilation/AC. Throttled GPUs silently lose tokens/sec — watch temps.
Noise: blower/server GPUs and high-static-pressure fans are loud. Don't put a 4-GPU server in your bedroom.
Electricity cost: a 2 kW box running 24/7 is ~48 kWh/day — a real monthly bill. Factor it into the self-host-vs-API math above.
UPS: a power blip during a multi-hour fine-tune wastes the run; a UPS pays for itself the first time.

The Apple/DGX Spark counterpoint. This is exactly why unified-memory boxes (Mac Studio, DGX Spark) are attractive despite lower bandwidth — they fit big models in a quiet, low-power, desk-friendly package. If "no jet-engine server in my flat" matters to you, you're trading peak speed for livability, and that's a legitimate trade.

Quantization, deeper — the lever that decides what fits

Quantization is the single biggest knob you control, so it deserves its own section. The idea: model weights are stored as numbers, and you can store those numbers with fewer bits. Fewer bits → less memory → fits on cheaper hardware → but, past a point, worse answers. The whole game is finding the lowest precision that still passes your quality bar.

Format	Bits	Where it's used	Quality
BF16 / FP16	16	Training, "no compromise" serving	Reference — zero loss.
FP8	8	Modern serving baseline (vLLM, H100/H200)	Near-lossless; the sane default for big models.
INT8 / GPTQ-8	8	vLLM serving	Very close to full; safe for production.
AWQ / GPTQ 4-bit	4	vLLM serving on tight VRAM	Usually good; validate on reasoning/math.
GGUF (Q4_K_M etc.)	2–8	llama.cpp, Ollama, Mac, single GPU	The consumer default; Q4_K_M is the popular sweet spot.

Two practical truths the build community keeps relearning. First, format follows engine: if you serve with vLLM to many users, AWQ or GPTQ give better throughput; if you run llama.cpp/Ollama on a single GPU or a Mac, GGUF is the default and has the richest ready-made quant library. Second, 4-bit is where quality risk begins. For chat and general tasks INT4/Q4 is often indistinguishable from full precision; for math, multi-step reasoning, and agentic coding it can measurably degrade — which is exactly why the DeepSeek team tells you to stick to the official checkpoints for V4 in production rather than pushing to community INT4. The rule: quantize as far as your hardware needs, then run your own eval before you trust it.

How to pick precision in practice. Start at the model's official quant (often FP8). If it doesn't fit, drop to 4-bit AWQ/GGUF and run a small evaluation set from your real workload. If quality holds, ship it. If reasoning slips, you've found your floor — go back up one level or add VRAM. Never assume; always measure on tasks you care about.

Fine-tuning your own models — what it really takes

Inference is renting the model's intelligence; fine-tuning is reshaping it on your data. Most people don't need it (good prompting + retrieval beats fine-tuning for most tasks), but when you do, the hardware story is different from inference, and it's important to know which kind you're doing.

Full fine-tuning — update every weight. Needs memory for weights plus optimizer states and gradients — roughly 3–4× the inference memory. A 70B full fine-tune is a multi-GPU-node job. Rarely the right tool outside well-funded labs.
LoRA (Low-Rank Adaptation) — freeze the big model, train tiny "adapter" matrices bolted onto it. You only store and update a few million extra parameters, so memory barely rises above inference. This is how a single RTX 4090 can fine-tune models far larger than full fine-tuning would allow on it.
QLoRA — LoRA on top of a 4-bit quantized base model. The base sits in 4-bit (small), adapters train in higher precision. This is the democratizing trick: it pushes fine-tuning of surprisingly large models onto a single consumer or prosumer GPU.

The practical ladder: a Tier 1 RTX 4090 or a Tier 2 Mac can do LoRA/QLoRA on small-to-mid models; the DGX Spark's 128 GB unified memory makes it a genuinely good fine-tuning prototyping appliance (capacity matters more than peak bandwidth when you're training adapters); full fine-tunes of big models belong on rented Tier-5 nodes. Tools: Hugging Face peft + transformers for LoRA/QLoRA, axolotl or Unsloth for friendlier recipes, and trl for preference tuning. Keep your dataset small and clean before you reach for more GPUs — data quality beats parameter count for most fine-tunes.

Decision shortcut. Want the model to know your documents? Use retrieval (RAG), not fine-tuning. Want it to adopt a style, format, or narrow skill? LoRA/QLoRA. Need to change the model's fundamental behavior at scale? Full fine-tune on rented iron. Most "we need to fine-tune" instincts are actually "we need RAG and better prompts."

What real builders report — patterns from 2026 write-ups

The request was to learn from how people actually did this, including where it bit them. A few consistent patterns show up across 2026 build reviews and self-host guides — worth internalizing because they're the lessons people paid for in time and money.

"It fit but it was slow" is the most common disappointment. People buy capacity (a 128 GB unified box, or a Mac with lots of RAM), load a 70B–120B model, and are let down by single-stream speed. The fix is expectation-setting: capacity boxes are for fitting and developing, fast GPUs are for serving. The DGX Spark reviews crystallize this — reviewers praise it as a CUDA dev appliance while flagging that a 120B model crawls at ~38 tok/s, beaten by a few old RTX 3090s on aggregate bandwidth.
Concurrency changes the verdict. Single-stream benchmarks make capacity boxes look bad, but some builders measured very different numbers under many concurrent requests — throughput-oriented engines (vLLM, continuous batching) recover a lot of the apparent loss when you're serving an agent fleet rather than one chat. Benchmark the workload you'll actually run, not a single prompt.
The CPU-offload cliff surprises everyone once. The first time a model is even slightly too big for VRAM and silently spills to system RAM, speed collapses from 30–60 tok/s to 2–5. Builders learn to size so the model fits entirely in fast memory, with KV-cache headroom — never "almost fits."
Power and heat are underestimated by first-timers. Multi-GPU builders consistently report tripping breakers, needing dedicated circuits, and turning rooms into saunas. The people who enjoy their labs planned the electrical and thermal side before buying the fourth GPU.
Most end up using the API anyway for the frontier. A recurring arc: build a capable local lab for day-to-day 7–70B work and privacy, and call GLM-5.2 / DeepSeek V4-Pro's API for the handful of tasks that genuinely need a flagship. The hybrid — local for volume and sovereignty, API for peak capability — is where many thoughtful builders land.

The meta-lesson. Almost every "I regret this purchase" story traces to buying for a headline number (parameters it can hold, petaFLOPS) instead of for the actual workload (tokens/sec at your model size, under your concurrency, within your power budget). Define the workload first; let it pick the hardware.

Security, privacy & data sovereignty — the real reason to self-host

Since cost rarely justifies self-hosting a flagship, the strongest reason is usually control of your data. If you build a lab for this, do it properly — owning the GPUs is necessary but not sufficient.

Network isolation. For true data sovereignty, run the model on a machine that can't phone home — air-gapped, or behind a strict egress firewall. Verify the inference stack and any "telemetry" are off. Open weights help here: nothing leaves unless you let it.
Treat model output as untrusted. If your lab runs agents that execute code or call tools, sandbox that execution (containers/microVMs), broker secrets so the model never sees raw credentials, and default-deny network egress from the agent. A local model is still an untrusted code generator.
Validate downloaded weights. Pull models from official repositories, check hashes, and prefer safetensors over pickle formats (pickle can execute arbitrary code on load). A poisoned checkpoint is a supply-chain risk.
Access control on the endpoint. Your local OpenAI-compatible API is a powerful tool — don't expose it unauthenticated on a shared network. Put it behind auth and bind it to localhost or a trusted subnet.
Patch and isolate. The inference server, drivers, and OS are attack surface like any other service. Keep them updated; don't run the lab flat on your main workstation if it handles sensitive data.

Sovereignty is a property of the whole pipeline, not just the box. A self-hosted model that quietly ships logs to a cloud dashboard, or an agent that exfiltrates data through an unsandboxed tool call, defeats the entire purpose. If privacy is your reason to build, audit the data flow end to end — that audit is the deliverable.

Troubleshooting — the problems people actually hit

Built from real 2026 build write-ups and the failure modes that recur. Skim it now; you'll be back when something breaks.

Symptom	Likely cause	Fix
CUDA out of memory on load	Sized for weights only; KV cache + overhead overflowed; or used active-param count for a MoE	Lower `--max-model-len`, lower `--gpu-memory-utilization`, quantize more, or add VRAM. Budget weights + KV + 15%.
Painfully slow (2–5 tok/s)	Model spilled from GPU to system RAM (CPU offload) — the biggest perf cliff	Use a model that fits VRAM fully; quantize down a level; the moment it offloads, speed collapses from 30–60 to 2–5 tok/s.
Fits but still feels slow	Bandwidth wall (e.g. DGX Spark 273 GB/s on a big model)	Run a smaller model, or move to higher-bandwidth GPUs. Capacity ≠ speed.
vLLM won't start / TP error	Tensor-parallel size not a power of two, or doesn't divide attention heads	Use TP size 1/2/4/8; match it to GPU count and the model's head count.
NCCL / multi-GPU hang	Inter-GPU comm misconfig, mismatched drivers, bad PCIe topology	Align driver/CUDA across GPUs; check NVLink/PCIe; set NCCL env vars; verify all GPUs in `nvidia-smi`.
nvidia-smi fails after reboot	Driver/kernel mismatch (kernel updated, module didn't rebuild)	Reinstall the matching driver / DKMS; pin kernel; reboot. Don't mix distro + .run drivers.
Quantized model gives worse answers	Too-aggressive quant (INT4) on reasoning/math/agent tasks	Move up to FP8/INT8 or the official checkpoint; always validate quality on your tasks after quantizing.
OOM only at long context / many users	KV cache scales with context × concurrency	Cap context, cap concurrency, use paged-KV engine, pick a compressed-attention model.
GPU throttles under load	Thermal — inadequate cooling/airflow	Improve airflow, undervolt/limit power, space the cards, add fans. Watch temps in `nvidia-smi`.
AMD GPU won't work	Wrong/old ROCm	Install ROCm v7 directly from AMD's page — kernel-bundled versions lack features. (NVIDIA is the smoother path for a first lab.)

The single most useful debugging habit. Keep watch -n1 nvidia-smi open while you load and run. You'll see VRAM fill (catch OOM before it crashes), GPU utilization (catch CPU-offload — low GPU use + slow output = it spilled to RAM), and temperature (catch throttling). Ninety percent of local-LLM problems are visible right there in real time.

Right-sizing — stop reaching for the biggest model

The most expensive mistake in lab design is running a model far bigger than the job needs. Bigger models cost more VRAM, more power, and more latency for every single request — forever. A huge amount of real work is done perfectly well by 7B–32B models, and matching model size to task is the highest-leverage cost decision you'll make.

Job	Often enough	Why
Autocomplete, simple extraction, classification	7–14B	Narrow, well-defined tasks don't need frontier reasoning. Fast and cheap.
General coding assistant, summarization, RAG answers	14–32B	The day-to-day workhorse band; a single RTX 4090 covers it.
Hard reasoning, multi-step agents, tricky refactors	70B–284B	Where bigger genuinely helps; multi-GPU or DeepSeek V4-Flash.
Frontier multi-step engineering, the very hardest tasks	GLM-5.2 / V4-Pro	Datacenter or API only; reach for it sparingly.

A useful architecture that mirrors how GLM-5.2 itself works internally: route by difficulty. Run a small fast model locally for the bulk of requests, and escalate only the hard ones to a big local model or a flagship API. You get most of the quality at a fraction of the average cost, and your lab's expensive resources are spent only where they change the answer. Don't run a 70B to do a 7B's job thousands of times a day.

The right-sizing test. Before provisioning for a big model, take 20 real examples of your task and try them on a 14B and a 32B model. If the smaller one passes your bar, you just saved yourself a tier of hardware. Only buy capacity for the tasks that provably need it.

Your first week — a concrete plan

Theory is cheap; here's an actual sequence to go from nothing to a working, useful lab without overspending. Each step is reversible and teaches you what the next purchase should be.

Day 1 — borrow before you buy. Before spending anything, rent a GPU by the hour (any cloud GPU provider) or use the free tier of a model API. Run a 7B and a 32B model. Feel the difference. This tells you what size you actually need before you commit hardware money.
Day 2 — install the base on whatever you have. Even a modest GPU or a Mac. Ubuntu + NVIDIA driver + CUDA (or just Ollama on a Mac). Get nvidia-smi green. Run ollama run on a small model. You now have a lab, however humble.
Day 3 — wire it into your real workflow. Point your code editor, a chat UI, or a small script at the local OpenAI-compatible endpoint. Using it for real work — not benchmarks — is the only way to learn what's missing.
Day 4 — measure your two walls. Watch VRAM and tokens/sec with nvidia-smi while you work. Note where it gets slow (bandwidth) or won't load (capacity). This is the data that justifies any upgrade.
Day 5 — try the serving path. Stand up vLLM with a quantized model and hit it with a few concurrent requests. See how throughput behaves under load versus single-stream. Decide whether you need it.
Day 6 — run the money math on your real volume. Take your measured tokens/day and plug it into the break-even formula. Be honest about whether self-hosting beats the API for your usage, or whether your reason is sovereignty/latency instead.
Day 7 — decide the one upgrade that matters. By now you know your model size, your bottleneck wall, your workload, and your reason. Now buy — a bigger GPU for bandwidth, a unified-memory box for capacity, or nothing because the API was the right answer. Buying last, after measuring, is how you avoid the regret stories.

The whole plan in one sentence. Rent → install small → use it for real → measure the two walls → run the cost math → buy exactly the one thing your measurements proved you need. Spending money is the last step, not the first.

Learning / solo / coding assistant: one RTX 4090 (24 GB) in a normal PC + Ollama. Runs 32B models well. ~$2k all-in. Start here.
Big-model capacity, quiet, low power: Mac Studio M3 Ultra (128–256 GB) + LM Studio. Runs 70B+ in-memory.
NVIDIA/CUDA dev appliance, prototyping & fine-tuning: DGX Spark — eyes open about the 273 GB/s bandwidth limit; great toolbox, not a fast big-model chat server.
Serious lab serving a team / DeepSeek V4-Flash: 2× RTX PRO 6000 or 4× A100 workstation + vLLM. Plan power/cooling.
Need a flagship (GLM-5.2 / V4-Pro) occasionally: rent an 8× H200/H100 node by the hour, or just use the API. Don't buy $250k of GPUs for bursty use.

The whole thing in one breath

Build-your-own-AI-lab in one breath. Two walls decide everything: capacity (VRAM in GB — weights + KV cache + ~15% overhead; size MoE models off total not active params) and bandwidth (GB/s — sets tokens/sec, which is why the 128 GB DGX Spark at 273 GB/s fits big models but runs them slowly). Five tiers: single RTX 4090 (~$2k, 32B, start here) → Mac Studio unified memory (70B+, quiet) → DGX Spark (200B capacity, bandwidth-bound) → 2–4× RTX PRO 6000 / A100 workstation (70–284B, serves DeepSeek V4-Flash) → 8× H200 node for GLM-5.2 / V4-Pro (rent it). Stack: Ubuntu + NVIDIA driver + CUDA, then Ollama for solo, vLLM for serving (power-of-two tensor-parallel; AWQ/GPTQ quant). The flagship MoE giants are datacenter-class; DeepSeek V4-Flash (284B, ~170 GB) is the realistic self-host. And the money truth: the API is usually cheaper than self-hosting a flagship — own the hardware for sovereignty, fine-tuning, latency, or air-gapping, not to beat token prices. Keep nvidia-smi open; most failures (OOM, the CPU-offload cliff, thermal throttle) are visible there live.

FAQ

Can I run GLM-5.2 or DeepSeek V4-Pro on a desktop?

No. Both are 744B–1.6T-parameter MoE models needing hundreds of GB to over a terabyte of VRAM — datacenter nodes (8× H200 and up). On a desktop, run a smaller model (DeepSeek V4-Flash 284B on a multi-GPU workstation, or a 32–70B model on one or two cards) and reach the flagships through their API.

Is the NVIDIA DGX Spark worth it?

It depends on the job. 128 GB unified memory, full CUDA stack, ~$3–4.7k, runs up to 200B params (405B with two linked units) — excellent for development, prototyping, fine-tuning, and agent-building. But its 273 GB/s bandwidth makes big-model generation slow (≈38 tok/s on a 120B model, beaten by three RTX 3090s). Buy it as a CUDA dev appliance with a big memory pool, not as a fast big-model chat server.

How much VRAM do I need for a model?

Roughly params × bytes-per-param (FP16=2, FP8=1, INT4≈0.5), plus KV cache (grows with context length and concurrency) plus ~10–20% overhead. A 70B model is ~140 GB at FP16, ~70 GB at FP8, ~40 GB at INT4 — before KV cache. For MoE models, use total parameters, not active.

Mac or NVIDIA for a home lab?

NVIDIA (CUDA) is the smoothest, fastest, best-supported path and the only realistic option for serious multi-GPU serving and fine-tuning. Apple Silicon's unified memory is unbeatable for loading big models cheaply, quietly, and at low power — capacity over peak speed. Many people keep both.

Will self-hosting save me money vs the API?

Usually not, for a flagship — break-even is often billions of tokens per day, which a single node can't sustain. Self-host for data sovereignty, fine-tuning, guaranteed latency, air-gapping, or no rate limits — not to undercut API token pricing.

Which inference engine should I start with?

Ollama for solo use (one command, OpenAI-compatible API, great on a single GPU or Mac). vLLM when you're serving a model to apps or many users (PagedAttention, continuous batching, tensor parallelism). llama.cpp/LM Studio if you want GGUF control or a GUI.

References & extra reads

NVIDIA — DGX Spark product page · official specs (128 GB, 1 petaFLOP FP4, up to 200B / 405B linked)
Tom's Hardware — DGX Spark review · real-world performance vs alternatives
BetterClaw — DGX Spark memory bandwidth (273 GB/s) analysis · why big models run slow
Lushbinary — Self-host GLM-5.2 (hardware & vLLM) · FP8/INT4 VRAM tiers, 8× H200
Lushbinary — Self-host DeepSeek V4 (vLLM, hardware) · V4-Flash 284B setup & commands
Morph — DeepSeek V4 (1.6T MoE) architecture & benchmarks · CSA/HCA attention, KV savings
Kunal Ganglani — Local LLM hardware guide 2026 (tested) · GPU tiers, the CPU-offload cliff, budgets
AIMultiple — Self-hosted LLM & VRAM calculator · sizing math
GPU-Mart — Self-hosted LLM: GPU selection & VRAM · quantization & engine choices

Build your own AI lab — the complete 2026 guide to self-hosting GLM-5.2, DeepSeek V4 & friends.