Self-hosted LLM serving in the EU: runtime choice, GPU sizing, and the sovereignty argument

Self-hosting an open-weights model in the EU is an infrastructure decision with a compliance consequence, and the two usually get argued in the wrong order. This runbook fixes the order: confirm the regulatory driver is real, pick a runtime, size the hardware from the model's own config file, then write the serving command and choose a host. Every figure below is either read off a vendor page named in the sentence, or a formula you fill in with your own numbers.

The question survives because "EU region" and "EU operated" are different claims: a managed endpoint served from a European region under a US-parent processor satisfies most audits and fails a few.

Prerequisites

• A written regulatory driver naming the supervisor and the clause. "The board prefers it" will not survive the cost model below.
• A token volume estimate split into prefill (input) and decode (output). Cost is dominated by decode.
• One EU host with a published data centre location and a data processing agreement you have read.
• Linux, plus a pinned NVIDIA driver, CUDA runtime and serving-runtime version. Any of them can change numeric output.
• Object storage for the weights, the checksum, and the exact commit revision. Without a revision pin there is no rollback.
• A reverse proxy you control. The runtime's own authentication is one shared bearer token, with no per-consumer quota.

Deciding before you buy

Self-hosting moves obligations from a vendor onto you, and buys neither speed nor privacy by itself. The drivers that genuinely favour it are narrow.

Choosing the runtime

The field narrowed. Hugging Face's Text Generation Inference documentation now opens with a maintenance-mode notice stating the project accepts only minor bug fixes and lightweight maintenance, and points readers to vLLM and SGLang. A new deployment should not start on TGI.

vLLM is the default for sustained multi-tenant traffic: tensor parallelism across cards and an OpenAI-compatible surface. llama.cpp covers CPU, Apple Silicon and edge devices where no GPU is permitted. Ollama belongs on workstations and small internal tools; its FAQ documents parallel requests through OLLAMA_NUM_PARALLEL, so the usual dismissal on concurrency grounds is out of date, and what keeps it off a production path is missing tensor parallelism, a thin metrics surface and no per-consumer quota.

Sizing from the model config

Published sizing tables go stale and most omit the only column that matters, headroom. Compute it from the parameter count on the model card and three fields in the repository's config.json: num_hidden_layers, num_key_value_heads, and head_dim where present, otherwise hidden_size divided by num_attention_heads.

Weights. Parameter count multiplied by bytes per parameter. FP16 and BF16 are 2 bytes, FP8 is 1 byte, four-bit schemes are roughly 0.5 bytes plus scale and zero-point overhead.

KV cache, per token. Two (keys and values) multiplied by layers, by key/value heads, by head dimension, by the bytes of the KV cache dtype. Qwen2.5-32B-Instruct publishes 64 layers and 8 key/value heads, carries no head_dim key, and gives hidden_size 5120 over 40 attention heads, so head dimension is 128. At 2 bytes: 2 x 64 x 8 x 128 x 2 = 262,144 bytes, 256 KiB per token, so one 32,768-token sequence needs 8 GiB. --kv-cache-dtype fp8_e4m3 halves both.

Headroom. Card memory multiplied by --gpu-memory-utilization, minus weight bytes, minus one to two gigabytes for CUDA context and activations. Divide that by the per-token figure and by your average served sequence length for the concurrency ceiling.

Run the same arithmetic against cards you can rent in the EU. Hetzner's GPU server matrix lists the GEX44 with an RTX 4000 SFF Ada at 20 GB and the GEX131 with an RTX PRO 6000 Blackwell Max-Q at 96 GB, and no SXM-class part. Take the 96 GB card at 0.90 with a BF16 32B checkpoint: 86 GB addressable, 65 GB of weights, under 2 GB of CUDA context, leaving roughly 19 GB of KV cache, on the order of 140,000 tokens at 128 KiB under fp8_e4m3. FP8 weights roughly triple that. Scaleway, by contrast, lists H100 PCIe and SXM instances.

Quantization, briefly

FP8 is the sensible default on Hopper parts, which support it in the tensor cores per NVIDIA's H100 product page, and on the Blackwell generation that succeeded them, per NVIDIA's Blackwell architecture page. On Hopper, fp8_e4m3 beats fp8_e5m2 for the KV cache: it keeps mantissa bits, and KV values do not need the wider exponent range.

FP8 weights arrive one of two ways, and this is the step most write-ups skip. Either the checkpoint is already published in FP8, or vLLM's online quantization converts a BF16 or FP16 checkpoint's linear weights at load time, which its documentation states needs no pre-quantized checkpoint and no calibration data. The scheme names accepted by --quantization have changed across engine versions, so read that page for the version you pin. The command below passes no quantization argument, which is why the sizing above counts weights at 2 bytes. At any precision, re-measure quality on your own evaluation set, in the language you will serve.

The serving command

This starts a single-GPU deployment bound to loopback, with a proxy in front doing TLS and per-key rate limiting. Flag names and defaults come from the vLLM engine arguments page; check them against the version you pin.

# Pin the runtime version and the model revision. Both move independently.
export VLLM_API_KEY="$(openssl rand -hex 32)"

vllm serve Qwen/Qwen2.5-32B-Instruct \
  --revision 5ede1c9 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768 \
  --max-num-seqs 32 \
  --kv-cache-dtype fp8_e4m3 \
  --served-model-name house-model \
  --host 127.0.0.1 \
  --port 8000 \
  --api-key "$VLLM_API_KEY"

vllm serve is the documented entry point on the vLLM online-serving page; the older python -m vllm.entrypoints.openai.api_server path signals a stale playbook. --served-model-name is what clients send in the model field, so holding it constant across upgrades makes a rollback invisible to callers.

One flag you will meet in older playbooks is --enforce-eager false. The engine arguments page documents --enforce-eager as a boolean with a paired --no-enforce-eager and a default of False, so a trailing word is a parse error. Prefix caching has the same shape, its default has changed across versions, and any published throughput gain belongs to somebody else's workload.

Expected output

$ curl -s http://127.0.0.1:8000/v1/models \
    -H "Authorization: Bearer $VLLM_API_KEY" | jq -r '.data[].id'
house-model

$ curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8000/v1/models
401

$ nvidia-smi --query-gpu=memory.used,memory.total --format=csv,noheader

The first command must echo back exactly the string passed to --served-model-name. The second must return 401 without the header, the cheapest proof that the key is enforced. The third is the check most people skip: vLLM pre-allocates the KV cache at start-up, so used memory should sit close to total memory multiplied by --gpu-memory-utilization. Well below that, the cache did not get the block count you assumed and your concurrency ceiling is lower than planned.

The cost model

Do this arithmetic slowly. A month is about 2.63 million seconds (86,400 x 30.44), so sustained decode throughput of R tokens per second, aggregated across every concurrent user, produces R x 2.63 million tokens per month. At R = 30 that is roughly 79 million a month; the per-day figure is 2.59 million. Cost per million tokens is your total monthly cost, hardware plus storage plus observability plus the fraction of an engineer, divided by that count. Take a machine and its operations at EUR 2,000 a month at a genuine 30 tokens per second: 2,000 divided by 79 gives about EUR 25 per million. Both inputs are illustrative; substitute your own.

Anthropic's published pricing page lists Claude Haiku 4.5 at USD 1 per million input tokens and USD 5 per million output, and Claude Sonnet 4.6 at USD 3 and USD 15. Self-hosted cost is dominated by decode, so compare against the output rate: about EUR 25 per million against USD 5 and USD 15 is a gap no exchange rate closes.

Invert the formula to find the bar your workload has to clear. Reaching EUR 5 per million on that EUR 2,000 needs 400 million tokens a month: 400,000,000 divided by 2,630,000 is about 152 tokens per second, continuously. A workload busy eight hours on weekdays runs at a duty cycle near 0.24, so it needs roughly 640 tokens per second during business hours to average that.

The regulatory record you will be asked for

Self-hosting removes a processor from the chain and relocates that processor's obligations onto you. DORA entered into application on 17 January 2025, the date EIOPA records on its DORA page. Articles 28(3) and 30 of Regulation (EU) 2022/2554 require financial entities to maintain a register of information covering contractual arrangements for the use of ICT services, and set out the key provisions those arrangements must carry. Self-hosting shortens that register to the infrastructure provider without deleting the entry, and the exit strategy still has to be written. Record which hosting pattern you chose, owned rack, EU bare metal, managed sovereign host or a major cloud's European region; the register asks for it.

NIS2, as the European Commission describes it, requires medium and large entities in the covered sectors to take appropriate cybersecurity risk-management measures and makes top management accountable. Model weights are supply chain: record the source repository, the pinned revision and the verified checksum.

The AI Act applies regardless of where the model runs. The Commission's framework page sets out four risk tiers and dates the high-risk obligations from 2 December 2027, with governance and general-purpose AI rules applicable since 2 August 2025. Those obligations include activity logging and technical documentation, which self-hosting makes easier to satisfy, because you hold the logs, and harder to avoid, because no provider carries them.

If retrieval over your own documents is the actual requirement, that is a separate design with its own permission model, which is what DECKLOG, the private retrieval product addresses.

Side effects

• vLLM pre-allocates GPU memory, so dashboards show the card near-full at idle. Alert on queue depth and time-to-first-token.
• --api-key is one shared secret for all callers. Per-consumer identity, quota and revocation exist only if the proxy provides them.
• Prefix caching shares KV blocks between requests, a timing-observable channel in a multi-tenant deployment. vLLM offers a per-request cache salt; set it deliberately.
• Pinning driver, CUDA and runtime versions freezes security fixes as well as numeric behaviour. Schedule the unpin with a staging pass.
• Weights are tens to hundreds of gigabytes, so cold starts are minutes and per-request autoscaling is out.
• Logs of prompts and completions are personal data in most deployments. The trace store needs the same retention and access control as the model host.

Rollback

1. Keep the previous model revision on local disk, not only in object storage. Restoring from a bucket at model scale is a bandwidth event.
2. Keep the previous runtime image tagged and present on the host.
3. Because --served-model-name is stable, reverting is a restart with the old revision and the old image. No client changes.
4. Re-run the three checks under Expected output against values recorded before the change. Record them first; a comparison without a baseline is a guess.
5. If the host is unrecoverable, the OpenAI-compatible surface lets a managed EU endpoint stand in behind the same proxy. Decide in advance whether your regulatory driver permits that.

Limitations

No throughput figure here was measured. The sizing arithmetic is exact given the config values you supply, and every performance claim is attributed to the page that published it. The EUR 2,000 a month and the 30 tokens per second are illustrative inputs, not quoted prices or benchmarks; measure your own before using the cost model in a decision.

This runbook assumes a single node, NVIDIA hardware and Linux. Multi-node parallelism, AMD ROCm and Apple Silicon each change the sizing arithmetic and the operational shape. Mixture-of-experts models change it most: routing reduces active compute per token while resident weight memory stays, so a sparse model still needs enough aggregate memory to hold every expert. Fine-tuning and training are out of scope.

The regulatory summary is a reading of published Commission, ESA and EUR-Lex material, not legal advice, and it stops applying the moment your sectoral supervisor states a position of its own. Prices, SKUs and flag defaults moved during 2026 and will move again; the cited vendor pages are the authority.

Last verified 2026-07-28. Checked against every source in the citation list, including the published config.json for Qwen2.5-32B-Instruct, the vLLM engine-arguments and online-quantization pages, Hetzner's GPU matrix, and EUR-Lex Regulation (EU) 2022/2554.