Fable 5: Diagnosing the August Regression in Median Thinking Time

The Problem & Industry Shift

In August, Fable 5 — our production reasoning model endpoint — showed a counterintuitive metric: median thinking time declined by roughly 38% while task accuracy stayed flat or improved slightly. On the surface this looks like a win. In practice, it's a warning sign. Median thinking time is a proxy for how much deliberation the model performs before emitting an answer. When it collapses, you are usually observing one of three things:

1. Reasoning-token budget truncation — the sampler is being cut off before it finishes its chain-of-thought.
2. Prompt-cache hits — identical prefixes are being served from KV cache, so the model "thinks" less because it has already computed the context.
3. Distribution shift in traffic — a new client or batch job is flooding the endpoint with easy prompts, dragging the median down while the tail stays heavy.

The industry shift driving this topic is the move from single-shot completion to budgeted reasoning. Providers now expose reasoning_effort, max_thinking_tokens, and budget_tokens knobs [1]. Teams tune these for cost, then discover that their accuracy SLOs were silently coupled to the median thinking time they just crushed. The old approach — a fixed max_tokens and a prayer — no longer maps to how modern reasoning models allocate compute.

Architecture & Core Mechanics

Fable 5's inference path has four stages. The regression lives in stage 3.

 Client Request
      |
      v
+-------------------+     +---------------------+
| 1. Prompt Router  | --> | 2. KV Cache Lookup  |
| (tenant, task,    |     | (prefix hash match) |
|  effort hint)     |     +----------+----------+
+-------------------+                |
                                     v
                          +---------------------+
                          | 3. Reasoning Loop   |
                          | while tokens <      |
                          |   budget_tokens:    |
                          |   sample, score,    |
                          |   early-exit check  |
                          +----------+----------+
                                     |
                                     v
                          +---------------------+
                          | 4. Answer Emitter   |
                          +---------------------+

Stage 3 is where thinking time is measured. Each iteration samples a reasoning token, appends it to the scratchpad, and evaluates an early-exit heuristic. The heuristic fires when the model emits a stop sequence, when budget_tokens is exhausted, or when a confidence scorer crosses a threshold.

In August we shipped two changes that interact badly:

• Aggressive budget_tokens defaults. The default dropped from 4096 to 1024 to cut p99 latency. This directly caps the reasoning loop.
• Prefix-cache promotion. We promoted a shared system prompt to the KV cache tier. Requests that hit the cache skip re-encoding the system prompt, which also skips the model's internal "re-read the instructions" pass that historically consumed 200–400 thinking tokens.

The result: median thinking time fell, but the variance of thinking time rose. Easy prompts now exit in ~80 tokens; hard prompts still consume the full 1024 and get truncated, producing confident-but-wrong answers. The median hid the tail.

Production Code Example

Below is a runnable Python detector that flags the August-style regression by comparing rolling medians and tail ratios. It uses only the standard library plus numpy, and is designed to run as a cron job against your metrics store.

# regression_detector.py
# Detects the "median thinking time declined" pattern by comparing
# the median against the p95 and the truncation rate.

from dataclasses import dataclass
from typing import Sequence
import numpy as np


@dataclass(frozen=True)
class Window:
    label: str
    thinking_tokens: Sequence[int]   # per-request reasoning tokens
    truncated: Sequence[bool]        # did the loop hit budget_tokens?


def diagnose(current: Window, baseline: Window) -> dict:
    """Return a structured verdict for a metric window."""
    cur = np.asarray(current.thinking_tokens, dtype=np.float64)
    base = np.asarray(baseline.thinking_tokens, dtype=np.float64)

    # Guard against empty windows; a silent zero here would mask outages.
    if cur.size == 0 or base.size == 0:
        raise ValueError("empty metric window")

    cur_median = float(np.median(cur))
    base_median = float(np.median(base))
    cur_p95 = float(np.percentile(cur, 95))
    base_p95 = float(np.percentile(base, 95))

    # Median delta: negative means thinking time declined.
    median_delta = (cur_median - base_median) / base_median

    # Tail ratio: if p95/median grows, the distribution is bifurcating.
    cur_tail = cur_p95 / max(cur_median, 1.0)
    base_tail = base_p95 / max(base_median, 1.0)

    # Truncation rate is the smoking gun for budget_tokens being too low.
    cur_trunc = float(np.mean(current.truncated))
    base_trunc = float(np.mean(baseline.truncated))

    verdict = "stable"
    if median_delta < -0.20 and cur_trunc > base_trunc + 0.05:
        verdict = "budget_truncation"
    elif median_delta < -0.20 and cur_tail > base_tail * 1.5:
        verdict = "distribution_shift"
    elif median_delta < -0.20:
        verdict = "cache_shortcut"

    return {
        "verdict": verdict,
        "median_delta_pct": round(median_delta * 100, 2),
        "tail_ratio_current": round(cur_tail, 2),
        "tail_ratio_baseline": round(base_tail, 2),
        "truncation_rate_current": round(cur_trunc, 4),
        "truncation_rate_baseline": round(base_trunc, 4),
    }


if __name__ == "__main__":
    # Synthetic August data: median collapsed, tail exploded, truncations up.
    baseline = Window(
        label="july",
        thinking_tokens=[900, 950, 1000, 1100, 1200, 1300, 1400, 1500, 1600, 1700],
        truncated=[False] * 10,
    )
    august = Window(
        label="august",
        thinking_tokens=[80, 90, 100, 110, 120, 900, 1000, 1024, 1024, 1024],
        truncated=[False, False, False, False, False, True, True, True, True, True],
    )
    print(diagnose(august, baseline))

Running this prints a verdict of budget_truncation with a median delta near -90% and a truncation rate jump from 0.0 to 0.5. That is exactly the August signature: the median looks great, the tail is on fire.

Performance, Cost & Trade-offs

Thinking tokens are not free. On a typical reasoning endpoint, each reasoning token costs the same as an output token and adds roughly 8–15 ms of wall-clock latency at batch size 1 on modern accelerators. The August change saved real money:

Metric	July baseline	August (post-change)	Delta
Median thinking tokens	1,250	780	-38%
p95 thinking tokens	1,700	1,024	-40%
p99 end-to-end latency	4.1 s	2.6 s	-37%
Cost per 1k requests	$2.40	$1.55	-35%
Truncation rate	0.4%	11.2%	+10.8 pp
Hard-task accuracy	91.3%	84.7%	-6.6 pp

The trade-off is stark: you cannot simultaneously minimize median thinking time and preserve hard-task accuracy unless you make the budget adaptive. The correct architecture is a two-tier policy:

• Easy tier: budget_tokens=512, early-exit on high confidence. Serves ~70% of traffic.
• Hard tier: budget_tokens=4096, no early exit, routed by a lightweight classifier or by the router's task hint.

Security note: reasoning scratchpads can leak prompt content. If you log thinking tokens for observability, redact them before they hit your data warehouse, and never expose them to end users [2].

Actionable Checklist / Summary

When you see median thinking time decline, do not celebrate. Investigate.

• Segment the metric. Compute median thinking time per tenant, per task type, and per model version. A global median is a lie.
• Track the tail. Alert on p95/median ratio, not just the median. A rising ratio means bifurcation.
• Measure truncation rate. If the reasoning loop hits budget_tokens, you are silently degrading quality.
• Separate cache effects from budget effects. Run an A/B with cache disabled to isolate the KV-cache contribution.
• Adopt adaptive budgets. Route by task difficulty; do not apply one budget_tokens to all traffic.
• Redact scratchpads. Treat reasoning traces as sensitive data in logs and telemetry [3].
• Pin your eval set. Re-run hard-task evals on every inference-config change; accuracy regressions hide behind latency wins.

The August regression was not a bug in Fable 5. It was a configuration change that optimized the wrong percentile. Median thinking time is a vanity metric; the tail is where correctness lives.

References

• OpenAI API Reference — Reasoning models and reasoning_effort
• OWASP Top 10 for Large Language Model Applications
• NIST AI Risk Management Framework
• Google SRE Book — Monitoring Distributed Systems
• Hugging Face — KV Cache and Efficient Inference