Zai-GLM5-TileRT

Serve layer for TileRT GLM-5.x deployments: tilert/serve_sglang/ (decode server, PD router, KV receiver/converter, metrics) plus the sglang prefill overlay, Docker recipes, and delivery scripts.

SGLang Prefill + TileRT Decode disaggregated serving for GLM-5.2 with MTP speculative decoding:

• SGLang handles prefill (layer-split + EAGLE/MTP, fp8 KV)
• TileRT handles decode with MTP speculative decoding (bs=1), fp8 KV, up to 1M context
• Mooncake transfers KV cache GPU-to-GPU via SGLang's native disagg framework
• Prefill runs layer-split so it emits the per-layer KV (+ the MTP/nextn-layer KV and last-hidden aux) that the TileRT decode consumes

Install

The runtime ships as a single wheel built from the TileRT-GLM5 engine with this repo's tilert/serve_sglang injected. Install the wheel attached to the GitHub Release for your version:

pip install <release-asset-url>/tilert-<version>+glm.g<sha>-<tags>.whl

The wheel embeds its provenance: python -c "import tilert; print(tilert.__build_info__)".

Repo layout

• tilert/serve_sglang/ — the ONLY python maintained here; every other tilert.* module comes from the engine wheel (no code copies, by policy).
• sglang/ — prefill-side patch overlay.
• docker/, entrypoint.sh — container recipes.
• delivery/, tests/ — benchmark and validation scripts.

Architecture

Client
  │
  ▼
PD Router (pd_router.py)
  │  1. Pick free decode (router-side busy tracking)
  │  2. Inject bootstrap fields into request
  │  3. Send to decode + prefill concurrently
  │  4. Proxy response (streaming or not)
  │
  ├──────────────────────────┐
  ▼                          ▼
TileRT Decode              SGLang Prefill Cluster
(one per node, bs=1)       (layer-split, TP=8, EP=8, CP=8)
  │                          │
  │  ◄── Mooncake RDMA ──── │
  │      (per-layer KV + KI + MTP-layer KV + metadata)
  │
  ▼
MTP decode → response

The standalone PD router manages prefill-decode mapping. Decode nodes don't need to know about prefill at startup — the router injects bootstrap fields at request time and tracks each decode's in-flight count in-memory.

Weight Directory Integrity

Since post12 the loader resolves every tensor from the file designated by model.safetensors.index.json (weight_map), visiting shards in sorted order. Directories that carry duplicate keys across shards (e.g. after re-exporting a subset of layers) now load deterministically — the index-designated copy always wins. Still, keeping a single copy per key is recommended; stale shadowed tensors waste disk and confuse other tools that read whole files.

Weight Conversion

Convert HuggingFace/SGLang GLM-5.2 weights to TileRT format before serving:

# Step 1: main model weights
python -m tilert.models.preprocess.weight_converter \
    --model_type glm-5_2 \
    --model_dir <HF_MODEL_PATH> \
    --save_dir <TILERT_WEIGHTS>

# Step 2: append the MTP (speculative) weights into the same converted dir
python -m tilert.models.preprocess.weight_converter \
    --model_type glm-5_2 \
    --model_dir <HF_MTP_MODEL_PATH> \
    --save_dir <TILERT_WEIGHTS> \
    --append_mtp

Quick Start

1. SGLang Prefill (layer-split + EAGLE/MTP, fp8 KV)

The prefill must run layer-split (--enable-nsa-cache-layer-split + --nsa-prefill-cp-mode round-robin-split) so it produces the per-layer KV, the MTP/nextn-layer KV, and the last-hidden aux that TileRT decode receives. --kv-cache-dtype=fp8_e4m3 unlocks the native 1M context.

# key env
export NCCL_GIN_ENABLE=1 NCCL_CUMEM_ENABLE=1 NCCL_IB_MERGE_NICS=0 NVSHMEM_IB_TRAFFIC_CLASS=98
export SGLANG_DEEPEP_USE_V2=1 SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=8192
export SGLANG_NSA_PREFILL_DENSE_ATTN_KV_LEN_THRESHOLD=0 SGLANG_NSA_MQA_LOGITS_CHUNK_ROWS=512
export SGLANG_PIPELINED_KV_TRANSFER=1 SGLANG_EAGLE_RAW_PAGE_RADIX_CACHE=1
export GLM_USE_OPTIMIZED_RADIX_PREFIX_MATCHING=1 GLM_FORCE_INDEXER_LOGITS_CHUNK=1
export GLM_USE_HICACHE_SYNC_KERNEL=1 GLM_NSA_CHUNK_MEM_RATIO=0.2

python3 -m sglang.launch_server \
    --model-path=<GLM5.2_MODEL> --trust-remote-code \
    --tp=8 --ep=8 --moe-a2a-backend=deepep --deepep-mode normal \
    --host=0.0.0.0 --port=30000 --skip-server-warmup \
    --max-prefill-tokens=1048576 --context-length=1048576 \
    --mem-fraction-static=0.78 --max-running-requests=8 \
    --attention-backend=nsa --nsa-prefill-backend=flashmla_sparse --nsa-decode-backend=flashmla_kv \
    --kv-cache-dtype=fp8_e4m3 --chunked-prefill-size=65536 --page-size=64 \
    --reasoning-parser=glm5 --tool-call-parser=glm5stream \
    --disaggregation-transfer-backend=mooncake --disaggregation-mode=prefill \
    --disaggregation-bootstrap-port=8998 --disaggregation-ib-device=<IB_DEVICES> \
    --enable-nsa-prefill-context-parallel --nsa-prefill-cp-mode round-robin-split \
    --attn-cp-size=8 --enable-dp-attention \
    --enable-nsa-cache-layer-split \
    --enable-hierarchical-cache --hicache-size=1280 --glm-nsa-shared-layer-group-hicache --hicache-io-backend=kernel \
    --disable-overlap-schedule --watchdog-timeout=1200 --soft-watchdog-timeout=300 \
    --speculative-algorithm=EAGLE --speculative-num-steps=3 --speculative-eagle-topk=1 --glm-stream-speculated-tokens \
    --speculative-draft-model-path=<GLM5.2_MTP> --glm-special-token-escape-seed=42 \
    --json-model-override-args '{"architectures": ["GlmMoeDsaForCausalLM"]}'

--json-model-override-args loads the checkpoint as GlmMoeDsaForCausalLM (the GLM-5.2 arch). --enable-nsa-cache-layer-split + --nsa-prefill-cp-mode round-robin-split are required for TileRT decode — a non-layer-split prefill will not emit the KV layout the decode expects. --attn-cp-size=8 --enable-dp-attention are shown explicitly to match the deployed command, but --enable-nsa-prefill-context-parallel already forces both (it sets attn_cp_size = tp_size // dp_size = 8, enable_dp_attention=True, moe_dense_tp_size=1, and requires tp=8, dp=1), so they are redundant on a single 8-GPU node.

The prefill draft depth must match the decode: MTP-4 = prefill --speculative-num-steps=3 ↔ decode --num-mtp 3 (the default). For MTP-8 use num-steps=7 on prefill and --num-mtp 7 on decode. Do not mix depths.

2. TileRT Decode (one per node)

python3 -m tilert.serve_sglang.decode_server \
    --model-weights-dir <TILERT_WEIGHTS> \
    --port 30000 --bootstrap-port 8998 --max-seq-len 1048576 \
    --fp8-kv --prefill-fp8-kv --with-mtp --num-mtp 3 \
    --transfer-backend mooncake --ib-device <IB_DEVICES> \
    --reasoning-parser glm5 --tool-call-parser glm5stream \
    --glm-special-token-escape-seed 42

• --fp8-kv --prefill-fp8-kv: fp8 KV cache; unlocks GLM-5.2 native 1M decode context (bf16 KV caps ~202752). Must be paired with an fp8-KV prefill (--kv-cache-dtype=fp8_e4m3, above).
• --num-mtp 3 = MTP-4 (default); --num-mtp 7 = MTP-8.
• MTP speculative decode covers the full configured context up to --max-seq-len (1M). There is no seq-len fallback to the non-speculative path — TILERT_SHOW_HANDS_MAX_SEQ_LEN defaults to max_seq_len and can be set lower only if an operator wants to cap it.
• --with-mtp is on by default; queue depth via TILERT_DECODE_QUEUE_DEPTH (default 2).
• --reasoning-parser glm5 --tool-call-parser glm5stream: with the escape seed set (below), the tokenizer renders every control token as <literal><hash> — the glm5-family parsers read sglang's escape mapping and split on the full escaped strings. The older glm45 / glm47 parsers split on plain literals and leak the hash marker into parsed fields (tool names come back as <0c7dc7cb>get_weather, content starts with <5e9c7f0d>); the server logs a warning if they are combined with an escape seed.
• --glm-special-token-escape-seed 42: MUST match the prefill's --glm-special-token-escape-seed. The tokenizer appends a per-seed hash suffix to special tokens; if the decode seed differs from the prefill's, the NSA-state KV page counts diverge and KV transfer fails with Failed to get kvcache from prefill instance. Use the same value on both sides (the deployed value is 42).

No --prefill-url needed — the PD router injects bootstrap fields at request time.

Constrained decoding (xgrammar)

The decode server can constrain generation to a grammar via the xgrammar backend. Two kinds, with different <think> behavior:

• Template structural constraint — set TILERT_GLM_DECODING_CONSTRAINT=1 to apply the GLM chat-template EBNF (thinking block / answer text / tool-call structure) to every request. It is active from the first generated token: its grammar encodes the thinking block itself, shaping only the template structure (e.g. excluding literal </think> / tool pseudo-tags from free text) while leaving the reasoning content otherwise unrestricted.
• Per-request constraints (sglang-compatible) stay dormant inside the <think> block and activate after </think>, so reasoning is never restricted — only the answer is constrained. Selected in priority order json_schema > regex > ebnf > structural_tag:
  • response_format: {"type": "json_schema", "json_schema": {"schema": { ... }}}
  • response_format: {"type": "json_object"}
  • response_format: {"type": "structural_tag", ...}
  • regex: "<pattern>" (top-level field)
  • ebnf: "<grammar>" (top-level field)

Requires the xgrammar backend in the decode image. A malformed or uncompilable grammar returns 400; a missing backend returns 500. Constraints fail closed — a requested grammar is never silently dropped.

Example — constrain the answer to a JSON schema (the model reasons in <think>, then the constraint activates on the answer after </think>):

curl http://<ROUTER_HOST>:23340/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model":"default",
    "messages":[{"role":"user","content":"Extract the person info: Alice is 30 years old and lives in Paris."}],
    "response_format":{"type":"json_schema","json_schema":{"name":"person","schema":{
      "type":"object",
      "properties":{"name":{"type":"string"},"age":{"type":"integer"},"city":{"type":"string"}},
      "required":["name","age","city"],"additionalProperties":false}}},
    "max_tokens":2048,"temperature":0
  }'
# → message.content: {"name": "Alice", "age": 30, "city": "Paris"}   (schema-valid)
#   message.reasoning_content holds the thinking; finish_reason "stop"

3. PD Router

# 1P:ND — single prefill, multiple decode
python3 -m tilert.serve_sglang.pd_router \
    --prefill http://<PREFILL_HOST>:30000 8998 \
    --decode http://<TILERT_1>:30000 \
    --decode http://<TILERT_2>:30000 \
    --host 0.0.0.0 --port 23340

# MP:ND — multiple prefills with per-prefill bootstrap ports
python3 -m tilert.serve_sglang.pd_router \
    --prefill http://<PREFILL_1>:30000 8998 \
    --prefill http://<PREFILL_2>:30000 8999 \
    --decode http://<TILERT_1>:30000 \
    --decode http://<TILERT_2>:30000 \
    --host 0.0.0.0 --port 23340

Features: per-node in-flight/capacity tracking (capacity = each decode's queue_depth, synced from /server_info; the router dispatches while in_flight < capacity, so a queue_depth>1 decode pipelines receive/compute), 429 when every node is at capacity, raw JSON passthrough (no field loss), SSE streaming, per-prefill bootstrap port, round-robin for both prefill and decode. Health checks probe decode via /health and prefill via /get_model_info (prefill's /health runs a generation that a layer-split prefill rejects).

Test

# Quick verify
curl http://<ROUTER_HOST>:23340/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"default","messages":[{"role":"user","content":"Say hello"}],"max_tokens":32}'

Performance & accuracy (8×B200, GLM-5.2, fp8 KV, layer-split, MTP-4: num-steps=3 / num-mtp 3)

Validated on AIME2026 (30 questions), 128k-gen, PD path (1 prefill + 3 tilert decode):

Metric	Value
AIME2026 accuracy	~88–91% (on par with the full-sglang baseline)
Server decode TPS	~490–560 tokens/s (≈4.6× sglang decode per stream)
MTP accepted / step	~3.0–3.5
KV recv (layer-split, long ctx)	~6 s/req at ~1M input

File Structure

tilert/
  serve_sglang/             # the only tilert python maintained in this repo
    pd_router.py           # Standalone PD router (asyncio + aiohttp)
    decode_server.py       # HTTP server + decode logic + CLI
    kv_receiver.py         # KVManager setup, buffer allocation, RDMA receive
    kv_converter.py        # KV split + KI FP8 (de)quant + TileRT inject
  # tilert.models, tilert.benchmark, etc. come from the installed engine wheel
sglang/                     # Prefill-side patch overlay
docker/                     # Container recipes
docs/                       # Design docs and testbed setup
delivery/                   # Delivery scripts
tests/                      # Import-surface audit, pressure tests, and benchmarks

Heterogeneous Decode Pools

TileRT decode and SGLang decode can share the same (layer-split) prefill cluster:

              Prefill Cluster (shared, layer-split)
                     │
         ┌───────────┴───────────┐
         │                       │
  PD Router (TileRT pool)  sglang_router (SGLang pool)
  pd_router.py              PD mode (standard)
  :23340                    :23334

License

MIT