Claude Code on a Local Model Stack

Claude Code on a Local Model Stack

Overview

Claude Code can run fully offline against models served on the same Mac, but a plain local model server rejects the requests it sends. Claude Code adds Anthropic-specific beta headers, a system role message, and a ?beta=true query parameter. A small stripping proxy sits between Claude Code and the model server to remove those, and three model tiers (frontier, middle, fast) map onto Claude Code's opus, sonnet, and haiku slots. Every component binds to loopback only.

When to Use

• You want Claude Code to run without cloud calls on Apple Silicon with enough unified memory for at least one large model.
• Claude Code fails with HTTP 422 messages[N].role or 401 errors when pointed at a local server.
• A Hermes agent must launch Claude Code as a subprocess against local models.
• Claude Code's opus, sonnet, and haiku slots should resolve to different local models.

Features

• A stdlib-only proxy that drops system messages, strips the ?beta=true parameter, and forwards only an allowlist of headers plus its own bearer token.
• Server-sent-event pass-through, so streamed tokens reach Claude Code as they are generated.
• Three-tier routing through ANTHROPIC_DEFAULT_OPUS_MODEL, ANTHROPIC_DEFAULT_SONNET_MODEL, and ANTHROPIC_DEFAULT_HAIKU_MODEL.
• Per-model server settings: context window, sampling, idle TTL, pinning, KV-cache quantization, and reasoning output off.
• A memory budget and cache sizing rule tied to unified memory.
• A preflight checklist and an end-to-end Claude Code test.

Benefits

• Coding sessions work with no data leaving the machine and no per-token cloud cost.
• The three-tier split keeps the large model for planning and sends routine calls to a fast model.
• Known failure modes (422 role errors, 401 key errors, silent tool-loop stalls) each map to one setting.
• The whole stack is reproducible from one settings file and one proxy script.

Procedure

1. Check versions. Tested with oMLX 0.3.12 and Claude Code 2.1.156; confirm your versions:

   omlx --version
   claude --version
   python3 --version   # 3.11 or later

2. Configure the model server in ~/.omlx/settings.json. Bind to loopback, enable the memory guard, and set the cache limits:

   {
     "server": {"host": "127.0.0.1", "port": 8000},
     "memory": {"memory_guard_tier": "aggressive", "soft_threshold": 0.87, "hard_threshold": 0.96},
     "scheduler": {"max_concurrent_requests": 4, "chunked_prefill": true},
     "cache": {"enabled": true, "ssd_cache_max_size": "<size>", "hot_cache_max_size": "<size>"},
     "auth": {"api_key": "<local-api-key>", "skip_api_key_verification": false},
     "claude_code": {"mode": "local", "context_scaling_enabled": true, "target_context_size": 131072,
                     "opus_model": "<frontier-model>", "sonnet_model": "<middle-model>", "haiku_model": "<fast-model>"}
   }

3. Set per-model rules in ~/.omlx/model_settings.json. Each key must equal the model's folder name under the model directory:

   {
     "version": 1,
     "models": {
       "<middle-model>": {"max_context_window": 131072, "temperature": 0.6, "top_p": 0.95,
                          "enable_thinking": false, "ttl_seconds": -1, "is_pinned": true,
                          "forced_ct_kwargs": ["enable_thinking"]},
       "<fast-model>": {"max_context_window": 65536, "enable_thinking": false, "ttl_seconds": 120},
       "<frontier-model>": {"max_context_window": 131072, "temperature": 1.0, "enable_thinking": false,
                            "turboquant_kv_enabled": true, "turboquant_kv_bits": 4.0,
                            "ttl_seconds": 60, "is_pinned": false}
     }
   }

   Use the sampling values from each model's card. Only the middle tier is pinned.

4. Start the proxy from the script in the Proxy Script section below. Export the local key from your secret store, run the script in its own terminal (or under launchd, step 7), and confirm it with a probe. The proxy has no GET route, so use a POST:

   export LOCAL_LLM_API_KEY=<key-from-your-secret-store>
   UPSTREAM_PORT=<server-port> LISTEN_PORT=<proxy-port> python3 -u <proxy-dir>/proxy.py
   curl -s -X POST http://127.0.0.1:<proxy-port>/v1/messages -H 'Content-Type: application/json' \
     -d '{"model":"<fast-model>","max_tokens":16,"messages":[{"role":"user","content":"Say ok."}]}'

5. Point Claude Code at the proxy. Write this to $HOME/.claude/settings.json, merging into any existing env block:

   {
     "env": {
       "ANTHROPIC_BASE_URL": "http://127.0.0.1:8002",
       "ANTHROPIC_AUTH_TOKEN": "<local-api-key>",
       "ANTHROPIC_DEFAULT_OPUS_MODEL": "<frontier-model>",
       "ANTHROPIC_DEFAULT_SONNET_MODEL": "<middle-model>",
       "ANTHROPIC_DEFAULT_HAIKU_MODEL": "<fast-model>",
       "CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS": "1",
       "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1",
       "DISABLE_TELEMETRY": "1"
     }
   }

6. Set the GPU wired-memory limit below total RAM, so macOS keeps headroom for itself and its other processes:

   sudo sysctl iogpu.wired_limit_mb=<MB>

7. (Optional) Run the proxy under launchd so it survives reboots. Create a LaunchAgent plist with RunAtLoad and KeepAlive set to true, ProgramArguments of /usr/bin/python3 -u <proxy-dir>/proxy.py, and log paths for stdout and stderr. Load it with launchctl load <plist-path>.
8. Start Claude Code from an agent with an explicit child environment. Copy the parent environment, remove ANTHROPIC_API_KEY, add the proxy variables and the key from the secret store, and launch the CLI with a timeout. Do not pass the key through a shell string.

Proxy Script

Save this as <proxy-dir>/proxy.py. It drops system messages, removes the beta=true query parameter, forwards only a short allowlist of headers, and adds its own bearer token. It handles non-streaming responses; check streamed responses before relying on them.

#!/usr/bin/env python3
"""Claude Code to local model server shim. Stdlib only; loopback only."""
import http.server, json, os, re
from http.client import HTTPConnection

UP, LISTEN = int(os.environ.get("UPSTREAM_PORT", "8000")), int(os.environ.get("LISTEN_PORT", "8002"))
KEY, KEEP = os.environ["LOCAL_LLM_API_KEY"], {"accept", "content-type", "user-agent", "anthropic-version"}

class Handler(http.server.BaseHTTPRequestHandler):
    def do_POST(self):
        body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
        body["messages"] = [m for m in body.get("messages", []) if m.get("role") != "system"]
        data = json.dumps(body).encode()
        headers = {k: v for k, v in self.headers.items() if k.lower() in KEEP}
        headers.update({"Authorization": f"Bearer {KEY}", "Content-Length": str(len(data))})
        conn = HTTPConnection("127.0.0.1", UP, timeout=300)
        conn.request("POST", re.sub(r"[?&]beta=true", "", self.path), body=data, headers=headers)
        resp = conn.getresponse()
        self.send_response(resp.status)
        self.send_header("Content-Type", resp.getheader("Content-Type", "application/json"))
        self.end_headers()
        while chunk := resp.read(64):
            self.wfile.write(chunk)
        conn.close()

http.server.ThreadingHTTPServer(("127.0.0.1", LISTEN), Handler).serve_forever()

Pitfalls

• **Never export ANTHROPIC_API_KEY to Claude Code.** It is sent as an x-api-key header, which the local server ignores, so every request fails with 401. Use ANTHROPIC_AUTH_TOKEN, which becomes a bearer token.
• **Point ANTHROPIC_BASE_URL at the proxy, not the model server.** Going straight to the model server returns 422 on every request because of the system-role message and beta headers.
• **Confirm $HOME before writing settings.** Claude Code reads $HOME/.claude/settings.json. Isolated agent profiles often set a different HOME, so the file lands where Claude Code never looks.
• Keep thinking output off for tool-calling models. Reasoning tokens consume the budget before any visible output, so tool calls look hung. Set enable_thinking to false everywhere and force it with the forced-kwargs list where supported.
• **Never set ttl_seconds to 0.** The model unloads after every request, adding a 1-2 second load to each tool call. Use -1 for the always-hot model and a finite TTL for large or rarely used ones.
• Pin only the model that must stay loaded. Loading all three tiers at once can exceed unified memory. Add up the resident sizes before you load a second large model. Pinning the frontier model prevents the other tiers from loading.
• Quantize the KV cache for very large models. A 128K-token KV cache for a large model can need about 126 GB unquantized but about 24 GB at 4-bit, which is the difference between fitting and an out-of-memory crash.
• Respect the model card's sampling values. Some frontier models require temperature 1.0; their model cards warn that temperature 0 causes endless thinking loops.
• Keep the version field an integer. "version": "1" as a string silently disables every per-model setting. Model keys are case-sensitive and must match the folder name.
• Keep every listener on loopback. Bind both the model server and the proxy to 127.0.0.1, never 0.0.0.0, and do not expose either port through a tunnel. Keep skip_api_key_verification false and keep the key out of shell history, logs, and commits.

Verification

• Model server answers with auth: curl -s http://127.0.0.1:8000/v1/models -H "Authorization: Bearer <local-api-key>" lists each tier's model id.
• The probe request in step 4 returns a message object, not an error.
• Proxy inference works: POST a one-message request with max_tokens 16 to /v1/messages through the proxy and get a message object back.
• Claude Code works end to end: claude --print "say hi" returns text, and claude --print "what model are you running right now?" names the routed tier.
• Log check: the proxy log shows a 200 from the model server for each request, with no system role left in the messages.

Attribution

• oMLX by jundot, the local model server this stack is built on: https://github.com/jundot/omlx
• Claude Code gateway and authentication docs: https://docs.anthropic.com/en/docs/claude-code/llm-gateway and https://docs.anthropic.com/en/docs/claude-code/authentication