6.2. Model serving
The model runs on one GPU host. llama-server exposes an OpenAI-compatible
API on 127.0.0.1 only, and clients reach it through an SSH tunnel. How to
build that host is Setting up an inference host; this page explains what it serves
and why. What the client does with the endpoint — the backend it selects,
the ports, the sampling it sends, the smoke test — is Model backends.
6.2.1. Current profile
The profile is server/runtime/manifest.json: Qwen3-Coder-Next, Q8_0, in
four shards (85 GB, 79 GiB), on a 96 GB card. server/inference/serve.sh turns the
deployment’s config/server.conf into this command line:
llama-server
--model <runtime>/models/gguf/Qwen3-Coder-Next-Q8_0-00001-of-00004.gguf
--ctx-size 524288
--parallel 1
--n-gpu-layers 99
--flash-attn on
--cache-type-k q8_0 --cache-type-v q8_0
--host 127.0.0.1 --port 8010
--jinja
--threads 16
--rope-scaling yarn --rope-scale 2 --yarn-orig-ctx 262144
--override-kv qwen3next.context_length=int:524288
6.2.2. The flags that matter
--jinjaRequired. It enables the model’s chat template, and with it native tool calling. Without it the assistant loses structured tool calls entirely.
--n-gpu-layers 99, and no--n-cpu-moeEverything on the card. On a smaller GPU,
SPEAR_SERVER_NCPUMOEkeeps the mixture-of-experts weights in system RAM while attention and the KV cache stay on the GPU — the hybrid offload that makes this model usable on a laptop-class card, at a large cost in speed.--cache-type-k q8_0 --cache-type-v q8_0Quantised KV cache. Buys context length at a small quality cost.
--ctx-size 524288and the YaRN flagsTwice the 262 144 tokens the model was trained at. Past that length the RoPE positions are rescaled with YaRN, and the GGUF’s
context_lengthis overridden becausellama-serverotherwise caps each slot at it. Both are added byserve.shonly when the window exceeds the trained length, since YaRN is static and would otherwise touch every short prompt too. Measured: a fact 40 % into a 311 744-token prompt was recalled exactly, and generation speed did not change. The harness still truncates long tool outputs atmax_output_chars(10 000 by default) so a single verbose build log cannot evict the conversation.--parallel 1One slot. This is a single-user assistant; concurrency would only fragment the KV cache.
6.2.3. The launcher holds no flag values
serve.sh contains no context size, no model path and no thread count,
and its port fallback (8080) is not the profile’s. Those come from <runtime>/config/server.conf, or from
SPEAR_SERVER_* in the environment; server/config/server.conf.example
documents every key. The script itself contributes only what is a property of
the build rather than of a deployment:
--cache-type-k q8_0 --cache-type-v q8_0the quantised KV cache;
--flash-attnunconditional;
--jinjarequired for the chat template and therefore for tool calling;
--n-cpu-moefor a mixture-of-experts model, unless a deployment says otherwise;
- the YaRN flags
derived from
SPEAR_SERVER_CTX,SPEAR_SERVER_NATIVE_CTXandSPEAR_SERVER_ARCHas above.
Context size has no built-in default at all: it must be stated by the deployment rather than inherited from whichever launcher happens to run.
6.2.4. Changing the model or the adapter
Both are keys of server.conf — SPEAR_SERVER_MODEL and
SPEAR_SERVER_LORA (none for the base model) — followed by a restart of
the service. serve.sh refuses to start on an unreadable model, and serves
the base model, saying so, when the adapter is unreadable.
6.2.5. Service placement
On the host the server is spear-inference.service, a system unit
(Setting up an inference host, step 6), so it sits in system.slice:
$ cat /proc/$(systemctl show spear-inference -p MainPID --value)/cgroup
0::/system.slice/spear-inference.service
This placement is load-bearing for the security model when the harness runs on
the same machine. Sandboxed tool commands sit in transient scopes under
user.slice/…/app.slice. The two trees are disjoint, so no memory limit,
OOM kill or task-limit hit inside a tool call can reach the model server.
Resource control verifies this explicitly.
6.2.6. Checking it from a client
The smoke test sends one request through the client’s backend and confirms that serving and the backend contract are both healthy.