Skip to content
←Back to Blog
October 13, 2026•9 min read•By Dzaki Amri Zaidaan

GLM-Legal: Engineering a Domain-Adapted Legal LLM with Retrieval, LoRA, and Citation Grounding

A production-grade breakdown of GLM-Legal: how to adapt a general GLM-style decoder into a citation-grounded legal assistant using continued pretraining, LoRA SFT, hybrid retrieval, and constrained decoding. Includes runnable Python, latency/cost trade-offs, and a deployment checklist.

#AI & ML#Architecture
woman in black coat standing in front of mirror

GLM-Legal: Engineering a Domain-Adapted Legal LLM with Retrieval, LoRA, and Citation Grounding

1. The Problem & Industry Shift

General-purpose LLMs fail in legal work for three structural reasons, not one:

  1. Hallucinated citations. A model that invents Smith v. Jones, 512 F.3d 999 (9th Cir. 2011) is worse than useless — it is a liability. In 2023 multiple federal courts sanctioned attorneys for filing briefs containing fabricated case law [1].
  2. Jurisdictional drift. "Reasonable notice" means different things under UCC Article 2, the Federal Rules of Civil Procedure, and California Civil Code § 1698. A single model checkpoint cannot memorize every jurisdiction's evolving statute text.
  3. Temporal decay. Statutes are amended, cases are overruled. A frozen model trained on 2023 data silently rots.

The industry shift is away from "fine-tune a bigger model" toward domain-adapted base models + retrieval + constrained decoding. GLM-Legal is the canonical pattern for this: take a GLM-family decoder (ChatGLM/GLM-4 class, ~6B–9B params), continue-pretrain it on legal corpora, LoRA-tune it for instruction following, then never let it answer from parametric memory alone. Every factual claim must be traceable to a retrieved passage.

The key architectural insight: the model's job is reasoning and synthesis, not recall. Recall is delegated to a vector index that you can update hourly.

2. Architecture & Core Mechanics

GLM-Legal is a four-stage pipeline. The LLM is only stage 3.

                 ┌─────────────────────────────────────────────┐
   user query ──▶│ 1. Query Understanding                      │
                 │    - jurisdiction extraction (CA / 9th Cir) │
                 │    - date filter (as-of 2024-06-01)         │
                 │    - HyDE expansion + BM25 keyword gen      │
                 └───────────────┬─────────────────────────────┘
                                 │
                 ┌───────────────▼─────────────────────────────┐
                 │ 2. Hybrid Retrieval                         │
                 │    BM25 (statutes) ⊕ dense (case law)       │
                 │    → RRF fusion → cross-encoder rerank      │
                 │    → top-k=8 passages, each with cite_id    │
                 └───────────────┬─────────────────────────────┘
                                 │
                 ┌───────────────▼─────────────────────────────┐
                 │ 3. GLM-Legal (LoRA-adapted decoder)         │
                 │    prompt = system + passages + question    │
                 │    constrained decode: [cite_id] tokens     │
                 └───────────────┬─────────────────────────────┘
                                 │
                 ┌───────────────▼─────────────────────────────┐
                 │ 4. Citation Verifier (deterministic)        │
                 │    every [cite_id] must exist in retrieved  │
                 │    set AND entail the sentence it supports  │
                 └─────────────────────────────────────────────┘

2.1 Continued pretraining (domain adaptation)

You do not start from scratch. You take a GLM checkpoint and run causal LM training on a legal corpus: statutes, regulations, court opinions, contracts, and legal encyclopedias. The objective is standard next-token prediction, but the data mix matters more than the hyperparameters:

  • Statutes/regs (40%) — high lexical overlap with queries, teaches exact phrasing.
  • Case law (40%) — teaches citation format and reasoning structure (IRAC).
  • Contracts/pleadings (15%) — teaches drafting register.
  • General web replay (5%) — prevents catastrophic forgetting of instruction-following ability.

That 5% replay is non-negotiable. Without it, the model becomes a legal autocomplete that ignores instructions.

2.2 LoRA SFT for instruction following

Full fine-tuning a 9B model needs ~8×A100-80GB with ZeRO-3. LoRA (rank 16–64 on q_proj,k_proj,v_proj,o_proj) drops that to a single 80GB GPU and keeps the base weights frozen — which means you can hot-swap adapters per jurisdiction. This is the operational win: one base model in VRAM, N adapters on disk.

2.3 Constrained decoding for citations

This is the part most teams get wrong. Instead of asking the model to "cite sources," you force it. At generation time you build a token-level logit mask that only permits [cite_id] tokens that exist in the retrieved set. If the retriever returned cite_3 and cite_7, the model physically cannot emit cite_9. This converts hallucinated citations from a probability problem into an impossibility.

3. Production Code Example

The following is a realistic, runnable sketch of the retrieval + constrained-decode + verification loop. It uses transformers, sentence-transformers, and a token-mask logits processor. Inline comments flag the engineering decisions that matter.

"""
GLM-Legal inference service: hybrid retrieval + citation-constrained decoding.
Tested against transformers>=4.40, sentence-transformers>=2.7.
"""
from __future__ import annotations
import re
from dataclasses import dataclass
from typing import Sequence

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, LogitsProcessor
from sentence_transformers import CrossEncoder

CITE_RE = re.compile(r"\[cite_(\d+)\]")


@dataclass(frozen=True)
class Passage:
    cite_id: str          # e.g. "cite_3" — stable, never reused across requests
    text: str
    source: str           # "Cal. Civ. Code § 1698" or "512 F.3d 999"
    as_of: str            # ISO date; used for temporal filtering


class CitationMask(LogitsProcessor):
    """
    Hard-constrain generation so the model can ONLY emit cite_ids
    that were actually retrieved. This is the single highest-leverage
    anti-hallucination control in the pipeline.
    """
    def __init__(self, allowed_ids: Sequence[str], tokenizer, prompt_len: int):
        # Pre-tokenize each allowed cite_id. We only mask the FIRST token
        # of a citation; subsequent tokens are forced by the tokenizer's
        # natural continuation of that token sequence.
        self.allowed_first_tokens = {
            tokenizer.encode(cid, add_special_tokens=False)[0] for cid in allowed_ids
        }
        self.tokenizer = tokenizer
        self.prompt_len = prompt_len
        self._open_bracket = tokenizer.encode("[", add_special_tokens=False)[0]

    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor) -> torch.FloatTensor:
        # Only intervene when the model is about to open a citation.
        last = input_ids[:, -1]
        if not torch.all(last == self._open_bracket):
            return scores
        mask = torch.full_like(scores, float("-inf"))
        for tid in self.allowed_first_tokens:
            mask[:, tid] = 0.0
        return scores + mask


class GLMLegalService:
    def __init__(self, base_model: str, adapter: str | None = None, device: str = "cuda"):
        self.tok = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)
        self.model = AutoModelForCausalLM.from_pretrained(
            base_model,
            torch_dtype=torch.bfloat16,   # bf16 over fp16: no loss-scaling headaches
            device_map=device,
            trust_remote_code=True,
        )
        if adapter:
            # PEFT adapter keeps base weights frozen; swap per jurisdiction.
            from peft import PeftModel
            self.model = PeftModel.from_pretrained(self.model, adapter)
        self.model.eval()
        # Cross-encoder reranker is small (~100M) and runs on CPU to save VRAM.
        self.reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2", device="cpu")

    def retrieve(self, query: str, jurisdiction: str, as_of: str, k: int = 8) -> list[Passage]:
        """Hybrid retrieval: BM25 for exact statutory phrasing, dense for semantics.
        In production this hits OpenSearch (BM25) + a pgvector/Milvus index (dense),
        fused with Reciprocal Rank Fusion, then reranked."""
        candidates = self._hybrid_search(query, jurisdiction, as_of, top_n=50)
        pairs = [(query, p.text) for p in candidates]
        scores = self.reranker.predict(pairs)
        ranked = sorted(zip(candidates, scores), key=lambda x: -x[1])[:k]
        # Reassign cite_ids per-request so the model can't memorize stale ones.
        return [
            Passage(cite_id=f"cite_{i}", text=p.text, source=p.source, as_of=p.as_of)
            for i, (p, _) in enumerate(ranked)
        ]

    def _hybrid_search(self, query, jurisdiction, as_of, top_n) -> list[Passage]:
        raise NotImplementedError("Wire to OpenSearch + vector DB")

    def answer(self, question: str, jurisdiction: str, as_of: str) -> dict:
        passages = self.retrieve(question, jurisdiction, as_of)
        context = "\n\n".join(f"[{p.cite_id}] ({p.source}) {p.text}" for p in passages)
        system = (
            "You are a legal research assistant. Answer ONLY from the provided passages. "
            "Every factual sentence MUST end with one or more [cite_N] markers. "
            "If the passages do not support an answer, say: INSUFFICIENT_CONTEXT."
        )
        prompt = f"{system}\n\nPASSAGES:\n{context}\n\nQUESTION: {question}\nANSWER:"
        inputs = self.tok(prompt, return_tensors="pt").to(self.model.device)

        allowed = [p.cite_id for p in passages]
        processor = CitationMask(allowed, self.tok, inputs.input_ids.shape[1])

        with torch.inference_mode():
            out = self.model.generate(
                **inputs,
                max_new_tokens=512,
                do_sample=False,               # greedy: legal answers must be reproducible
                logits_processor=[processor],
                pad_token_id=self.tok.eos_token_id,
            )
        text = self.tok.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
        return self._verify(text, passages)

    def _verify(self, text: str, passages: list[Passage]) -> dict:
        """Deterministic post-check. Any cite_id not in the retrieved set is a
        hard failure — we return a refusal rather than a plausible-looking lie."""
        cited = {f"cite_{m}" for m in CITE_RE.findall(text)}
        valid = {p.cite_id for p in passages}
        if not cited.issubset(valid):
            return {"answer": "INSUFFICIENT_CONTEXT", "reason": "invalid_citation", "raw": text}
        if "INSUFFICIENT_CONTEXT" in text:
            return {"answer": "INSUFFICIENT_CONTEXT", "reason": "model_refusal", "raw": text}
        return {"answer": text, "citations": sorted(cited), "reason": "ok"}

Three decisions worth calling out:

  • do_sample=False. Legal output must be reproducible for audit. Temperature is for marketing copy, not for a memo that a partner will sign.
  • Per-request cite_id reassignment. If cite_3 always means the same case, a model can learn to emit it without reading. Rotating IDs forces attention to the actual passage.
  • The verifier is not the model. It is a 20-line regex + set check. Never let the LLM grade its own citations.

4. Performance, Cost & Trade-offs

Benchmarks below are representative for a 9B GLM-class model with bf16 weights on a single A100-80GB, batch size 1, 8 retrieved passages (~2k context tokens).

StageLatency (p50)Latency (p99)Notes
Query understanding (LLM call)180 ms420 msCan be replaced by a rules engine for 10× speedup
BM25 + dense retrieval40 ms120 msDominated by network, not compute
Cross-encoder rerank (50→8)90 ms200 msCPU-bound; batch of 50 pairs
GLM-Legal generation (512 tok)3.1 s6.8 s~165 tok/s with vLLM, ~55 tok/s naive HF
Citation verification<5 ms<5 msPure Python
End-to-end~3.5 s~7.5 s

Cost. At ~$2/hr for an A100 spot instance, 3.5 s/query ≈ $0.002/query in GPU time. The retrieval stack (OpenSearch + vector DB) typically costs more than the GPU at low QPS. At >50 QPS, switch to vLLM with continuous batching and paged attention; throughput rises 8–15× and per-query cost drops below $0.0005 [2].

Accuracy vs. recall trade-off. Increasing k from 8 to 20 improves answer coverage but inflates context length, which (a) increases latency linearly and (b) degrades accuracy past a point — models attend poorly to the middle of long contexts [3]. Empirically, k=8 with a strong reranker beats k=20 without one.

Security. Legal data is privileged. Three hard requirements:

  1. No egress. Run the model on-prem or in a VPC with no internet route. The retrieval index must not call external embedding APIs.
  2. Prompt-injection defense. Retrieved passages are untrusted input. A malicious contract clause containing "ignore previous instructions" must not be able to override the system prompt. Strip instruction-like patterns from retrieved text before insertion.
  3. Audit logging. Log (query, retrieved cite_ids, generated text, verifier verdict) immutably. When a citation is challenged, you need the exact retrieved set that produced it.

When NOT to use this. For pure document review ("find all indemnification clauses"), a fine-tuned encoder classifier is 100× cheaper and more accurate than a generative model. Reserve GLM-Legal for synthesis and reasoning tasks where the answer is not a span.

5. Actionable Checklist / Summary

Before shipping a GLM-Legal-style system:

  • Separate recall from reasoning. The model must never answer from parametric memory. Every claim traces to a retrieved passage.
  • Constrain decoding. Mask logits so only retrieved cite_ids are emittable. This is the cheapest, highest-impact anti-hallucination control.
  • Verify deterministically. Regex + set membership, not an LLM judge. Fail closed to INSUFFICIENT_CONTEXT.
  • Rotate cite_ids per request to prevent memorization of citation tokens.
  • Use LoRA, not full fine-tuning. One base model, N jurisdiction adapters, hot-swappable.
  • Keep 5% general replay in continued pretraining to preserve instruction following.
  • Set do_sample=False and log every generation for audit.
  • Benchmark end-to-end, not just the model. Retrieval and reranking often dominate p99 latency.
  • Treat retrieved text as untrusted input. Sanitize for prompt injection.
  • Know the boundary. Use classifiers for extraction; use GLM-Legal for synthesis.

The core thesis: a domain-adapted LLM is a reasoning engine over a live index, not an oracle. GLM-Legal's value comes from the pipeline around the model — retrieval, constrained decoding, and deterministic verification — far more than from the weights themselves.

References

← Explore More ArticlesShare on X / Twitter →