Route LLM Calls by Complexity, Not by Habit

Kimi K3 just landed at the top of Hacker News with over 1,200 upvotes. It's an open-weight model that claims frontier-level intelligence. Whether or not it dethrones GPT-5 on every benchmark, the signal is clear: open models are now good enough for production use in many scenarios.

That changes the math on your AI bill.

Why this matters

Most AI-powered apps treat one API as their entire backend. Every request — from "what's your return policy?" to "refactor this distributed system" — hits the same premium model. You're paying frontier prices for tasks a smaller model could handle.

As open-weight models close the quality gap, keeping everything on one expensive endpoint becomes a choice, not a necessity.

How it works

The pattern is called LLM cascading or model routing. You place a lightweight decision layer in front of your model endpoints. It evaluates each request's complexity and routes accordingly:

• Simple, repetitive queries go to a self-hosted open model (like Kimi K3 running on your own infra).
• Complex reasoning tasks escalate to a premium API.
• Ambiguous cases get a confidence score, and anything below a threshold gets escalated.

The router itself can be a tiny classifier, a rules engine, or even a fast small model that costs fractions of a cent per call.

Where this helps

• Customer support bots: most queries are FAQ lookups that don't need frontier reasoning.
• Code review automation: simple style checks on a local model, deep architectural feedback on a premium one.
• RAG pipelines: retrieval does the heavy lifting, so the model just needs to synthesize clearly.
• Content classification: spam detection and moderation rarely need frontier-level reasoning.

Watch out

The routing layer adds latency — you're making two calls instead of one for ambiguous requests. You also need to maintain multiple model endpoints, which means more infrastructure, more monitoring, and more failure modes. And some requests look deceptively simple but actually require deep reasoning. Your escalation threshold needs tuning.

Compatibility is another concern: different models produce different output formats. Your downstream parsing needs to handle that variation gracefully.

Try it yourself

A minimal router in Python — note that vLLM, Ollama, and LM Studio all expose OpenAI-compatible endpoints, so you can swap base URLs without changing your client code:

import openai

def route_request(prompt: str) -> str:
    # Heuristic: short prompts with common keywords go local
    cheap_keywords = ["return policy", "hours", "location",
                      "price", "faq", "contact"]
    
    if len(prompt) < 200 and any(
        k in prompt.lower() for k in cheap_keywords
    ):
        return call_local_model(prompt)
    
    # Everything else hits the premium API
    return openai.chat.completions.create(
        model="gpt-5",
        messages=[{"role": "user", "content": prompt}]
    ).choices[0].message.content

def call_local_model(prompt: str) -> str:
    # Point to your self-hosted Kimi K3 via vLLM or Ollama
    return openai.chat.completions.create(
        model="kimi-k3",
        messages=[{"role": "user", "content": prompt}],
        base_url="http://localhost:8000/v1",
        api_key="not-needed"
    ).choices[0].message.content

TL;DR

• What happened: Kimi K3, an open-weight frontier-level model, signals that self-hostable AI is production-ready.
• Why it matters: You no longer need to send every request to a premium API — routing by complexity cuts costs dramatically.
• What to try today: Add a simple classifier in front of your LLM endpoint and route low-complexity requests to a local model.