Evaluating LLM Output Without a Golden Dataset

Evaluating an LLM’s output when you have no reference answer is the norm in production, not the exception. You might be summarizing internal documentation that has no “correct” summary, generating creative marketing copy, or writing code for a problem that has never been solved before. Traditional metrics like BLEU and ROUGE require a reference text, so they are useless here. The problem becomes: how do you build a proxy for human judgment that is fast, cheap, and repeatable?

Why ground truth isn't always available

Real-world LLM applications rarely come with a golden dataset. Internal document summarization, for example, has no canonical summary – the acceptable output varies by audience and purpose. Creative text generation, from product descriptions to story drafts, is inherently subjective. Code generation for novel problems (e.g., “write a function that parses this custom log format”) has no pre-existing solution to compare against. Even when a reference exists, it may be outdated or incomplete.

Traditional metrics depend on n-gram overlap with a reference: BLEU for machine translation, ROUGE for summarization. Without a reference, they cannot compute a score. The evaluation task therefore shifts from “measure distance to a known answer” to “proxy human judgment through automated signals.” That is a harder problem, but one that practical systems must solve.

Self-consistency as a signal

A simple but effective reference-free technique is to sample the same prompt multiple times with a non-zero temperature and measure agreement. If the model produces the same answer (or semantically similar answers) across runs, that is evidence the output is reliable – at least, it is not random.

Example: factual QA.
For a question like “What is the capital of Bhutan?”, generate 5–10 responses with temperature 0.7. Compute the fraction of responses that agree on the city name. An agreement rate of 80%+ suggests the model is confident, even if you cannot verify the fact externally.

For structured outputs, you can use exact match. For free‑form text, use semantic similarity via embeddings:

import numpy as np
from sentence_transformers import SentenceTransformer
 
model = SentenceTransformer('all-MiniLM-L6-v2')
 
def self_consistency_score(responses: list[str]) -> float:
    if len(responses) < 2:
        return 0.0
    embeds = model.encode(responses)
    # pairwise cosine similarity
    sims = []
    for i in range(len(embeds)):
        for j in range(i+1, len(embeds)):
            sim = np.dot(embeds[i], embeds[j]) / (np.linalg.norm(embeds[i]) * np.linalg.norm(embeds[j]))
            sims.append(sim)
    return float(np.mean(sims))

The choice of embedding model affects the quality of similarity scores; see Selecting an Embedding Model for Code Search for guidance on picking one.

Trade-off. High self-consistency does not mean the answer is correct – the model could be consistently wrong. For example, if the model always answers “42” to any question, self-consistency is perfect but useless. This signal must be combined with others.

LLM-as-judge: using another model for evaluation

A more powerful approach is to use a strong model (e.g., GPT-4, Claude, or a fine‑tuned judge model) to score outputs on dimensions like relevance, coherence, factuality, and instruction‑following. The judge model is given a rubric and optionally a few examples, then asked to assign a score or a preference.

Prompt design. The rubric must be explicit. Avoid vague criteria like “is it good?” – instead define specific scales. For example:

JUDGE_PROMPT = """You are an expert evaluator. Rate the following response on a scale of 1-5 for each criterion:
 
- Relevance: Does the response directly address the user's request?
- Coherence: Is the response logically structured and easy to follow?
- Factuality: Does the response contain verifiable information? (Assume you can fact-check)
 
User request: {prompt}
Response: {response}
 
Output a JSON object with keys "relevance", "coherence", "factuality" and integer scores.
"""

Position bias. LLM judges often prefer the first or second output in a pair. Mitigate by shuffling outputs when comparing two candidates, or by using a pairwise comparison prompt that asks for a rationale before the choice.

Comparison: open‑source vs. proprietary judge models.
Open‑source judges like Prometheus (Kim et al. 2024) are cheaper and can be run locally, but may have lower agreement with human judges than proprietary models. Proprietary models (GPT-4, Claude) are more expensive and introduce latency, but often correlate better with human ratings. A practical compromise: use a small open‑source judge for frequent, low‑stakes evaluations, and a proprietary judge for periodic calibration or high‑stakes decisions.

Task-specific proxy metrics

When you can define a concrete success criterion for the task, use a proxy metric that captures part of the quality. These metrics are cheap to compute and can be tracked continuously.

• Code generation. Unit test pass rate is the gold standard. Also measure compilation success, linting violations, and cyclomatic complexity. For example, a code snippet that compiles and passes all provided tests is likely correct, even if there is no reference solution.
• Summarization. Length ratio (summary length / source length), keyword coverage (fraction of important terms from the source that appear in the summary), and readability scores like Flesch‑Kincaid. These capture conciseness and coverage, but not deeper understanding.
• Classification. Confidence scores (e.g., softmax probabilities) and calibration (expected calibration error). A well‑calibrated model’s confidence matches its accuracy, which is a useful proxy for reliability.

Limit. Proxy metrics only capture partial quality. A code snippet may pass unit tests but contain a logic error that only appears in edge cases. A summary may have high keyword coverage but be incoherent. Always combine multiple proxies.

Human evaluation at scale: crowdsourcing and sampling

When automated metrics are insufficient – especially for subjective tasks like tone, creativity, or user satisfaction – human raters remain the gold standard. The challenge is doing this at scale while keeping costs manageable.

Sampling strategy. Evaluate a random subset of outputs (e.g., 5% of production traffic) rather than every single one. Use stratified sampling by prompt type or model version to ensure coverage. Compute confidence intervals around the aggregate score to detect regressions.

Tools. Platforms like LabelStudio (open‑source) or Amazon SageMaker Ground Truth allow you to define rubrics, manage raters, and track inter‑rater agreement. A clear rubric with examples and a Likert scale (e.g., 1–5) reduces noise.

Cost vs. accuracy. Human evaluation is expensive and slow. Use it sparingly, mainly to calibrate automated metrics or to validate a new model version before a full rollout.

Pitfalls and failure modes in reference-free evaluation

Each technique has failure modes that can mislead you.

• LLM-as-judge. The judge model can hallucinate – it may invent a factuality error that does not exist. It also tends to favor longer, more verbose outputs (length bias) and outputs that match its own writing style. These biases can be mitigated by using a separate judge model, shuffling, and asking for explanations.
• Self-consistency. Systematic errors are invisible to this metric. If the model always outputs a plausible‑sounding but wrong answer, consistency will be high but output quality low. This is especially dangerous for factual questions where the model is confidently wrong.
• Proxy metrics. They may not correlate with user satisfaction. A code snippet that compiles but has a wrong algorithm will pass tests but frustrate users. A summary with perfect keyword coverage may be unreadable.
• Human evaluation. Inter‑rater reliability is often low for subjective criteria. Even with a rubric, two raters may disagree on what “coherent” means. Scaling human evaluation to thousands of samples is logistically challenging.

Building a practical evaluation pipeline

A robust pipeline combines several signals, tracks them over time, and uses the feedback to improve the system.

1. Collect multiple signals. For each output, compute:

   • Self-consistency score (3–5 samples)
   • LLM judge score (using a cheap judge for most, a strong judge for a sample)
   • Task‑specific metric (e.g., test pass rate for code)
   • Human rating on a small random subset (e.g., 1% of outputs)

2. Normalise and aggregate. Combine scores into a composite quality index. For example, a weighted average: 0.3 × self-consistency + 0.4 × LLM judge + 0.2 × task metric + 0.1 × human rating (when available). The weights depend on the task.

3. Set regression thresholds. Monitor the composite index over time. If it drops below a threshold (e.g., 0.1 standard deviations from the running mean), trigger an alert. This catches model regressions after a deployment or a prompt change.

4. Iterate. Use evaluation results to improve prompts, fine‑tune the model, or select a different model. For example, if the LLM judge consistently flags low coherence, rewrite the prompt to include a structure constraint. If self-consistency is low, try reducing temperature or adding reasoning steps.

A practical example of this pipeline is described in Building an agent loop: tool calls, retries, failure modes, where evaluation of each step’s output is critical for deciding whether to retry.

Key takeaways

• Without a golden dataset, evaluation must combine multiple proxy signals: self-consistency, LLM-as-judge, task-specific metrics, and occasional human review.
• Self-consistency is cheap to compute but does not guard against systematic errors; always pair it with a factuality check when possible.
• LLM-as-judge is powerful but suffers from biases (length, style, position). Mitigate with shuffling, explicit rubrics, and separate judge models for different criteria.
• Task-specific metrics (unit test pass rate, readability scores, calibration) capture only part of quality; use them as one component in a composite score.
• Human evaluation is still the ultimate arbiter for subjective tasks, but sample strategically and track inter‑rater reliability to avoid noise.
• Build a pipeline that tracks a composite quality index over time, with thresholds for regression alerts, and use the evaluations to drive prompt and model improvements.