Self-managed endpoints

What works out of the box

Any server that speaks OpenAI's /v1/chat/completions over HTTP: vLLM, TGI, LM Studio, LocalAI, and llama.cpp server are all covered by the same code path already used for cloud providers.

Fields on the card

• Base URL — the /v1 root; Aarmos appends /chat/completions.
• Model id — whatever id your server was launched with (HF repo id, alias, quantized tag).
• Bearer token — optional. Leave blank if the endpoint runs without auth.
• Custom headers — for proxies that require extras (e.g. a non-standard OAuth header).

vLLM — copy-paste starter

Launch the server with tool calling and a bearer:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 --port 8000 \
  --api-key sk-local-$(openssl rand -hex 16) \
  --allowed-origins '["https://aarmos.io","https://www.aarmos.io"]' \
  --enable-auto-tool-choice --tool-call-parser llama3_json

Pick a --tool-call-parser that matches your model's chat template (hermes, llama3_json, mistral, granite). Wrong pairing = malformed tool-call JSON.

TLS in front of it (recommended)

The Aarmos PWA runs on HTTPS, so plain-HTTP endpoints on a non-loopback host get blocked as mixed content. Terminate TLS with Caddy in one line:

gpu.example.com {
  reverse_proxy localhost:8000
}

Or a Cloudflare Tunnel, or nginx with a Let's Encrypt cert. Then point the card at https://gpu.example.com/v1.

The three errors you'll actually hit

• Network / CORS error on Test — the browser refused to reach the endpoint. Check --allowed-origins, that the URL is reachable from your machine, and that HTTPS/HTTP match.
• HTTP 401 / 403 — bearer token is missing, wrong, or your proxy expects a different header. Try the Custom headers row.
• HTTP 404 (model not found) — the model id in the card doesn't match what the server was launched with. Hit GET /v1/models yourself to confirm.

Other servers

• TGI — expose the /v1 messages route: text-generation-launcher --model-id ... --port 8080 then use base URL http://host:8080/v1.
• LM Studio — enable "Start Server" in the app; base URL http://localhost:1234/v1, no bearer.
• LocalAI — default base URL http://localhost:8080/v1.
• llama.cpp server — ./llama-server -m model.gguf --port 8080, base URL http://localhost:8080/v1.

Governance still applies

The endpoint sits behind the same policy gate, redaction rules, and AVAR receipts as every other provider. Nothing about "self-managed" bypasses the control plane — it just changes where the tokens are computed.