Stale Weights, Fresh Prompts: Benchmarking Knowledge Conflicts in 13 LLMs

The Problem & Industry Shift

Retrieval-augmented generation (RAG) and long-context prompting are sold as the fix for stale model knowledge. The implicit contract is simple: if the correct fact is in the context window, the model will use it. My benchmark for the Kaggle Benchmarking Challenge shows that contract is frequently violated in a specific, reproducible way.

I constructed a controlled knowledge-conflict task. Each prompt contains an explicit, unambiguous in-context instruction that supersedes a well-known pretrained fact — for example, renaming a widely documented entity and asking the model to use the new name in a generated artifact. The new name is stated once, in plain text, inside the same context window. There is no retrieval failure, no truncation, no ambiguity.

Result across 13 production models (proprietary and open-weight, 7B–~400B class): 12 of 13 emitted the old, pretrained name in at least one of the generated fields, despite demonstrably "knowing" the new one when asked directly in a separate turn. This is not a hallucination in the classic sense — it is a parametric prior overriding an explicit instruction, a failure mode that scales with how strongly the pretrained fact is represented in the weights.

The industry shift this exposes: teams are moving from "fine-tune the model on our data" to "inject facts at inference time" (RAG, long context, system prompts). That migration is correct on cost and freshness grounds, but it silently assumes instruction-following is monotonic with context salience. It is not. Instruction-following competes with parametric recall, and parametric recall wins in exactly the cases where your data is most valuable — high-frequency, well-documented entities.

Architecture & Core Mechanics

Why the prior wins

Transformer decoding is a next-token distribution over the residual stream. Two signals contribute to the logits for the stale token:

Parametric signal — the fact is baked into MLP weights and attention patterns across many layers, reinforced by millions of co-occurrence examples during pretraining.
Contextual signal — the new name appears once or twice in the prompt, attended via the KV cache.

When the parametric signal is strong (high-frequency entity) and the contextual signal is sparse (single mention, no repetition, no structural reinforcement), the parametric logit can dominate at the exact decoding step where the name is emitted. This is a logit competition, not a retrieval failure.

                    ┌─────────────────────────────┐
   prompt ────────► │  Tokenizer + Embeddings     │
                    └──────────────┬──────────────┘
                                   │
                    ┌──────────────▼──────────────┐
                    │  N × Transformer Blocks     │
                    │  ┌────────────────────────┐ │
                    │  │ Attention (KV cache)   │ │◄── contextual signal
                    │  │  + MLP (parametric)    │ │◄── parametric signal
                    │  └────────────────────────┘ │
                    └──────────────┬──────────────┘
                                   │
                    ┌──────────────▼──────────────┐
                    │  Logit competition @ emit   │
                    │  stale_token  vs  new_token │
                    └──────────────┬──────────────┘
                                   │
                    ┌──────────────▼──────────────┐
                    │  Sampling / decoding        │
                    └─────────────────────────────┘

The failure taxonomy I observed

Direct-answer pass, generation fail. Ask "What is the new name?" → correct. Ask the model to use it in a longer artifact → stale name reappears. The instruction is followed when the task is recall; it is dropped when the task is generation under a strong prior.
First-mention contamination. The stale name often appears in the first generated occurrence, then propagates through the rest of the output via self-attention (the model conditions on its own stale tokens).
Temperature sensitivity. At temperature=0 the stale token is deterministic; at higher temperatures the distribution occasionally samples the new token, but this is luck, not instruction-following.
Repetition helps, weakly. Restating the new name 3–5 times in the prompt reduced but did not eliminate the failure in most models.

The harness architecture

  ┌────────────┐   ┌──────────────────┐   ┌───────────────────┐
  │ Task spec  │──►│ Prompt builder   │──►│ Model adapter     │
  │ (JSON)     │   │ (conflict inject)│   │ (13 providers)    │
  └────────────┘   └──────────────────┘   └─────────┬─────────┘
                                                    │
  ┌────────────┐   ┌──────────────────┐   ┌─────────▼─────────┐
  │ Report     │◄──│ Scorer           │◄──│ Raw completions   │
  │ (JSON/MD)  │   │ (exact + fuzzy)  │   │ (cached on disk)  │
  └────────────┘   └──────────────────┘   └───────────────────┘

Key design decision: cache raw completions to disk before scoring. Scoring logic changes; API calls are expensive and non-reproducible. Separating generation from evaluation is the single most important reproducibility choice in any LLM benchmark.

Production Code Example

Below is the core of the harness in Python. It is typed, runnable against any OpenAI-compatible endpoint, and caches completions so scoring can be re-run offline.

from __future__ import annotations

import hashlib
import json
import os
from dataclasses import dataclass, asdict
from pathlib import Path
from typing import Iterable

from openai import OpenAI  # any OpenAI-compatible endpoint works

CACHE_DIR = Path("./cache")
CACHE_DIR.mkdir(exist_ok=True)


@dataclass(frozen=True)
class ConflictTask:
    """A single knowledge-conflict probe."""
    task_id: str
    stale_name: str          # well-known pretrained fact
    new_name: str            # explicit in-context override
    instruction: str         # generation task that must use new_name


@dataclass(frozen=True)
class Completion:
    task_id: str
    model: str
    text: str
    prompt_tokens: int
    completion_tokens: int


def build_prompt(task: ConflictTask) -> str:
    # The override is stated once, plainly, then the generation task follows.
    # We deliberately avoid repetition here to measure the *baseline* failure rate.
    return (
        f"Context: The entity formerly known as '{task.stale_name}' "
        f"has been officially renamed to '{task.new_name}'. "
        f"Use the current name in all output.\n\n"
        f"Task: {task.instruction}\n"
    )


def cache_key(model: str, prompt: str, temperature: float) -> str:
    h = hashlib.sha256(f"{model}|{temperature}|{prompt}".encode()).hexdigest()
    return h[:32]


def generate(
    client: OpenAI,
    model: str,
    task: ConflictTask,
    temperature: float = 0.0,
) -> Completion:
    prompt = build_prompt(task)
    key = cache_key(model, prompt, temperature)
    cache_file = CACHE_DIR / f"{key}.json"

    if cache_file.exists():
        data = json.loads(cache_file.read_text())
        return Completion(**data)

    # temperature=0 for determinism; seed where supported for reproducibility.
    resp = client.chat.completions.create(
        model=model,
        temperature=temperature,
        messages=[{"role": "user", "content": prompt}],
    )
    choice = resp.choices[0].message.content or ""
    usage = resp.usage
    comp = Completion(
        task_id=task.task_id,
        model=model,
        text=choice,
        prompt_tokens=getattr(usage, "prompt_tokens", 0),
        completion_tokens=getattr(usage, "completion_tokens", 0),
    )
    cache_file.write_text(json.dumps(asdict(comp)))
    return comp


def score(comp: Completion, task: ConflictTask) -> dict[str, bool]:
    """Two independent signals: did it use the new name, did it leak the old one?"""
    text = comp.text.lower()
    new = task.new_name.lower()
    stale = task.stale_name.lower()
    return {
        "used_new": new in text,
        "leaked_stale": stale in text,
        # A task passes only if it used the new name AND never leaked the stale one.
        "pass": (new in text) and (stale not in text),
    }


def run(models: Iterable[str], tasks: Iterable[ConflictTask]) -> list[dict]:
    client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
    rows: list[dict] = []
    for model in models:
        for task in tasks:
            comp = generate(client, model, task)
            rows.append({**asdict(comp), **score(comp, task)})
    return rows


if __name__ == "__main__":
    tasks = [
        ConflictTask(
            task_id="rename-01",
            stale_name="OldCorp",
            new_name="NovaCorp",
            instruction="Write a 3-sentence press release announcing the rebrand.",
        ),
    ]
    results = run(models=["gpt-4o-mini"], tasks=tasks)
    print(json.dumps(results, indent=2))

Engineering decisions worth calling out:

temperature=0 + disk cache makes the benchmark reproducible and cheap to re-score. Never re-call an API to change a scoring rule.
Two-signal scoring (used_new and leaked_stale) is stricter than substring matching for correctness. A model that writes both names is still a failure.
No repetition in the baseline prompt. Repetition is a mitigation; measuring it in the baseline would hide the effect you are trying to quantify.
OpenAI-compatible client lets the same harness hit vLLM, Ollama, Together, or any gateway without code changes.

Performance, Cost & Trade-offs

Observed results (13 models)

Metric	Value
Models that leaked the stale name	12 / 13
Models that passed all tasks	1 / 13
Direct-recall accuracy ("what is the new name?")	13 / 13
Failure rate at `temperature=0`	highest (deterministic stale token)
Failure rate with 5× repetition of new name	reduced, not eliminated

Mitigation trade-offs

Strategy	Latency impact	Token cost	Effectiveness
Repeat new name 3–5× in prompt	~0	+5–15% input tokens	Partial
Structured output / JSON schema with enum	~0	~0	High (constrains decoding)
Post-generation validator + retry	+1 round trip on failure	+1 call on failure	High
Constrained decoding (grammar/FSM)	+small	~0	Highest, but rigid
Fine-tune on the new fact	High (training)	High	High, but stale again next change

The core trade-off

There is a direct tension between freshness (inject at inference) and reliability (bake into weights). Inference-time injection is cheap and instant but loses to strong parametric priors. Fine-tuning wins the logit competition but reintroduces staleness and cost. The pragmatic production answer is defense in depth: inject at inference, constrain the output space where the name matters (enums, grammars), and validate post-generation with a deterministic check before the artifact ships.

Cost note: the entire 13-model run fits comfortably in a few dollars of API spend at temperature=0 with caching. The expensive part is not generation — it is the engineering discipline of separating generation from scoring so results are auditable.

Actionable Checklist / Summary

Never assume context wins. If a fact conflicts with a high-frequency pretrained fact, assume the prior will fight you.
Separate generation from scoring. Cache raw completions; re-score offline. This is non-negotiable for reproducibility.
Use two-signal scoring. Check both "used the new value" and "leaked the old value." A model that emits both has failed.
Constrain the output space. JSON schema enums and constrained decoding beat prompt pleading for critical fields.
Add a deterministic post-validator. A regex or schema check before the artifact ships catches the 12/13 failure mode cheaply.
Test at temperature=0 first. It surfaces the deterministic stale-token failure that sampling can mask.
Benchmark your own entities. The failure rate depends on how strongly your specific fact is represented in the weights; generic benchmarks will not tell you.