Don't Train a Classifier. Hallucinate One.

Skip the labeled dataset: have an LLM hallucinate documents for each class, embed them, and classify by similarity. Works shockingly well.

Dafallah Ibrahim

— 3 min read
Share

Your PM wants every support ticket auto-routed by Monday. The textbook path — export tickets, hand-label a few thousand, fine-tune a classifier — costs weeks. A post trending on Hacker News right now, "Don't classify, hallucinate," argues for a shortcut: let an LLM hallucinate the training data.

The trick inverts the usual pipeline. Instead of collecting real documents and learning their classes, you describe each class and generate fake documents that fit it. Embeddings do the rest.

Why this matters

Labels, not modeling, are the expensive part of every classification system. This removes the cold-start problem entirely: no training loop, no GPU, no retraining pipeline when a category changes. You edit a prompt.

The insight is that a frontier LLM already carries a strong prior about what a "billing complaint" or "refund request" looks like in writing. You're distilling that prior into a small classifier you fully own — one that runs for fractions of a cent per item.

How it works

Three steps:

  • Hallucinate. For each label, write a one-line description and ask an LLM for 5–10 diverse hypothetical documents at high temperature. "Write five short tickets a customer with a billing problem would send."
  • Embed. Run every hypothetical through the same embedding model you'll use at inference. Consistency here matters more than which model you pick.
  • Classify. Embed the incoming text and take the label with the highest cosine similarity — either against any single hypothetical (max-sim) or against the class centroid.

The lineage is HyDE — hypothetical document embeddings, from search research in 2022: generate a fake answer, embed it, retrieve with it. Same move, pointed at classification instead of retrieval.

Two upgrades once it works: use the hallucinated docs as few-shot examples in a direct LLM classification prompt, or distill everything into a proper classifier once real labels accumulate.

Where this helps

  • Ticket triage: route billing/bugs/features on day one, before you have a single labeled ticket.
  • Feedback tagging: bucket app-store reviews or NPS verbatims into themes you define in plain English.
  • Agent routing: decide which sub-agent handles a request; change your taxonomy by editing prompts, not retraining.
  • Log triage: cluster noisy free-text error messages into families nobody has labeled yet.

Watch out

The generator's blind spots become your classifier's blind spots. If it doesn't know your internal jargon, the prototypes come out generic and accuracy sinks. Overlapping classes — billing vs. refunds — bleed into each other; force the generator to state what's unique to each label.

Centroids average away multimodal classes; max-sim is noisier. Generate several hypotheticals per class and test both scoring modes. A fine-tuned classifier still wins once you have thousands of clean labels — treat this as the cold-start tool, not the endgame.

And regenerate carefully: new prototypes silently change live decisions. Version those prompts like code, and think twice before shipping sensitive class definitions to a third-party API.

Try it yourself

# pip install openai
import math
from openai import OpenAI
client = OpenAI()

LABELS = {
    "billing": "5 short support tickets about billing problems",
    "bug":     "5 short support tickets describing software bugs",
    "feature": "5 short support tickets requesting new features",
}

def embed(texts):
    r = client.embeddings.create(model="text-embedding-3-small", input=texts)
    return [d.embedding for d in r.data]

# 1. hallucinate hypothetical documents per class, then embed them
prototypes = {}
for label, desc in LABELS.items():
    out = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user",
                   "content": f"Write {desc}. One per line, varied phrasing."}],
    ).choices[0].message.content.splitlines()
    prototypes[label] = embed([l.strip("- ").strip() for l in out if l.strip()])

# 2. classify new text by max cosine similarity to any hypothetical
def cos(a, b):
    return sum(x*y for x, y in zip(a, b)) / (
        math.sqrt(sum(x*x for x in a)) * math.sqrt(sum(y*y for y in b)))

def classify(text):
    v = embed([text])[0]
    return max(((lbl, max(cos(v, p) for p in protos))
                for lbl, protos in prototypes.items()),
               key=lambda t: t[1])[0]

print(classify("I was charged twice for my subscription this month"))
# -> billing

TL;DR

  • What: A front-page post argues for generative classification — hallucate documents per class with an LLM instead of labeling training data.
  • Why it matters: It deletes the labeling bottleneck; you ship a working classifier in an afternoon and change its behavior by editing prompts.
  • Try today: Generate 5 hypothetical docs per label, embed everything, classify by cosine similarity — then compare max-sim vs. centroid scoring.

Read more