Skip to content
←Back to Blog
October 2, 2026•9 min read•By Dzaki Amri Zaidaan

Benchmarking Frontier LLMs on Indian Regional Knowledge: A 123-Respondent Human Eval Pipeline

A reproducible methodology for building a human-graded benchmark of frontier LLMs on Indian regional knowledge, covering stratified sampling, inter-annotator agreement, cost-aware inference, and the statistical limits of a 123-respondent study.

#AI & ML
green and black van on road during daytime

The Problem & Industry Shift

Frontier model leaderboards (MMLU, GPQA, MMLU-Pro) are dominated by English-language, Western-centric knowledge. When a model scores 88% on MMLU, that number tells you almost nothing about whether it can correctly answer "Which state's legislative assembly passed the 2023 amendment to the Karnataka Sarana Sahaya scheme?" or "Is 'Bhojpuri' a scheduled language under the Eighth Schedule of the Indian Constitution?" (it is not — it's a recognized but non-scheduled language).

The failure mode is not that models are dumb. It's that benchmark contamination and distributional skew mean the eval set no longer measures what practitioners actually deploy against. If you're building a support bot for a fintech operating in 12 Indian states, or a RAG system over regional legal corpora, the aggregate leaderboard score is a vanity metric.

This is the gap the Kaggle Benchmarking Challenge targets: build a small, high-signal, human-verified eval set rather than another 10k-row scraped MCQ dump. The 123-respondent survey approach is deliberately low-N and high-fidelity — the opposite of the "scrape and pray" pipeline. The engineering challenge is making a 123-sample study statistically defensible and reproducible.

Architecture & Core Mechanics

The pipeline has four stages: elicitation → normalization → inference → adjudication. The critical design decision is that humans generate the questions and gold answers, and a separate human pass adjudicates model outputs — never the same person for both.

┌──────────────┐   ┌───────────────┐   ┌──────────────┐   ┌──────────────┐
│  Elicitation │   │ Normalization │   │  Inference   │   │ Adjudication │
│  (123 humans)│──▶│  (dedupe +    │──▶│  (N models,  │──▶│  (blind      │
│  Google Form │   │   stratify)   │   │   temp=0)    │   │   grading)   │
└──────────────┘   └───────────────┘   └──────────────┘   └──────────────┘
       │                   │                   │                  │
   raw Q+A pairs      canonical set       raw completions    labels + IAA
   (n≈380)            (n=123)             (n=123 × M)        (κ, F1)

Stage 1 — Elicitation. Respondents submit a question, a gold answer, and a self-reported confidence + region tag. This is the noisiest stage; expect ~40% of submissions to be unusable (ambiguous, duplicate, or factually wrong gold answers).

Stage 2 — Normalization. Deduplicate via embedding cosine similarity (threshold ~0.92 on text-embedding-3-small), then stratify into buckets: History/Civics, Language/Linguistics, Geography, Law/Policy, Current Affairs. Current Affairs is a trap — it decays. Cap it at 15% of the set or your benchmark has a half-life of six months.

Stage 3 — Inference. Run every model at temperature=0, fixed max_tokens, and log the raw completion plus token usage. Determinism matters: without it, your error bars are noise.

Stage 4 — Adjudication. Two annotators grade each completion blind (model identity stripped). Compute Cohen's κ. If κ < 0.6, your rubric is broken, not your annotators.

Production Code Example

The following is the normalization + inference harness. It's the part most people get wrong: they skip dedup and they don't persist raw completions, making re-grading impossible.

"""
benchmark_harness.py — normalize elicited Q/A pairs and run frontier inference.
Requires: openai>=1.30, numpy, pandas
"""
from __future__ import annotations

import hashlib
import json
import time
from dataclasses import dataclass, asdict
from typing import Iterable

import numpy as np
from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY from env

EMBED_MODEL = "text-embedding-3-small"
DEDUP_COSINE_THRESHOLD = 0.92


@dataclass(frozen=True)
class QAItem:
    item_id: str
    question: str
    gold_answer: str
    bucket: str          # History | Language | Geography | Law | CurrentAffairs
    region: str          # e.g. "KA", "TN", "national"
    contributor_conf: float  # 0.0-1.0 self-reported


def _stable_id(question: str) -> str:
    # Deterministic ID so re-runs are idempotent — critical for reproducibility.
    return hashlib.sha256(question.strip().lower().encode()).hexdigest()[:12]


def embed(texts: list[str]) -> np.ndarray:
    # Batch embeddings; 123 items is one call, but keep batching for scale.
    resp = client.embeddings.create(model=EMBED_MODEL, input=texts)
    return np.array([d.embedding for d in resp.data], dtype=np.float32)


def deduplicate(items: list[QAItem]) -> list[QAItem]:
    """Greedy dedup: keep the highest-confidence contributor per near-dup cluster."""
    if not items:
        return []
    vecs = embed([it.question for it in items])
    # L2-normalize so dot product == cosine similarity.
    vecs /= np.linalg.norm(vecs, axis=1, keepdims=True) + 1e-9
    sim = vecs @ vecs.T

    kept: list[QAItem] = []
    suppressed: set[int] = set()
    # Sort by contributor confidence desc so we keep the best gold answer.
    order = sorted(range(len(items)), key=lambda i: items[i].contrib_conf, reverse=True)
    for i in order:
        if i in suppressed:
            continue
        kept.append(items[i])
        # Suppress everything near-identical to the kept item.
        for j in range(len(items)):
            if j != i and sim[i, j] >= DEDUP_COSINE_THRESHOLD:
                suppressed.add(j)
    return kept


def run_inference(items: Iterable[QAItem], model: str) -> list[dict]:
    """Deterministic inference. Persist raw output — never re-generate for grading."""
    results = []
    for it in items:
        t0 = time.perf_counter()
        resp = client.chat.completions.create(
            model=model,
            temperature=0.0,          # determinism: required for reproducible error bars
            top_p=1.0,
            max_tokens=512,
            messages=[
                {"role": "system", "content": "Answer concisely and factually."},
                {"role": "user", "content": it.question},
            ],
        )
        latency_ms = (time.perf_counter() - t0) * 1000
        results.append({
            "item_id": it.item_id,
            "model": model,
            "completion": resp.choices[0].message.content,
            "prompt_tokens": resp.usage.prompt_tokens,
            "completion_tokens": resp.usage.completion_tokens,
            "latency_ms": round(latency_ms, 1),
        })
    return results


if __name__ == "__main__":
    raw = [QAItem(**r) for r in json.load(open("elicited.json"))]
    canonical = deduplicate(raw)
    print(f"raw={len(raw)} canonical={len(canonical)} "
          f"dedup_rate={1 - len(canonical)/len(raw):.2%}")
    out = run_inference(canonical, model="gpt-4o-2024-08-06")
    json.dump(out, open("completions.json", "w"), indent=2)

Two engineering decisions worth calling out. First, _stable_id makes the pipeline idempotent — re-running dedup produces identical IDs, so you can diff runs. Second, we persist prompt_tokens/completion_tokens per item, which lets you compute cost-per-correct-answer, the metric that actually matters for production routing.

Performance, Cost & Trade-offs

Statistical power. With n=123, your 95% confidence interval on an accuracy estimate is roughly ±8.8 percentage points at p=0.5 (Wilson interval). That means a model scoring 71% vs 76% is not a distinguishable difference on this set. If you need to resolve 5-point gaps, you need n≈400+. Be honest about this in your writeup — reviewers will check.

Inter-annotator agreement. Target Cohen's κ ≥ 0.7 for factual buckets. Language and Current Affairs buckets routinely land at κ≈0.5 because "correct" is genuinely contested. Report per-bucket κ, not a single pooled number.

Cost. At ~150 prompt tokens and ~200 completion tokens per item, 123 items × 5 models ≈ 615 calls. On a mid-tier frontier model at $2.50/1M input and $10/1M output, that's roughly $0.14 total for inference. The real cost is human time: 123 items × 2 annotators × ~90s = ~6 annotator-hours, plus elicitation. Budget the humans, not the tokens.

Latency vs. accuracy. Log latency_ms per call. In practice you'll find a 3-5x latency spread between the fastest and slowest frontier models for a ~5-point accuracy delta — which is exactly the trade-off a production router needs to encode.

Contamination risk. Because questions were elicited from 123 individuals in 2024-2025, contamination is low but non-zero. Hold back ~20% of items as a private split and never publish them.

Actionable Checklist

  • Freeze the rubric before grading. Write the accept/reject criteria for each bucket, then grade. Retro-fitting a rubric to results is p-hacking.
  • Blind the graders. Strip model names from completions; randomize order per annotator.
  • Report κ per bucket. A single pooled κ hides the buckets where your benchmark is actually unreliable.
  • Cap Current Affairs at ≤15%. Otherwise your benchmark decays within two quarters.
  • Persist raw completions + token counts. Never re-generate for grading; you lose reproducibility and cost attribution.
  • Publish the Wilson interval, not just the point estimate. n=123 demands it.
  • Hold out a private split. ~20% of items, never released, for contamination checks.
  • Version the dataset. Tag releases (v1.0.0) so downstream comparisons are pinned.

References

← Explore More ArticlesShare on X / Twitter →