Not Every AI Problem Needs an LLM

I've been working with AI agents around tool calling, MCP integrations, and multi-step workflows.

I've noticed how often we use general-purpose LLMs when a task doesn't need language generation.

Think about tool selection.

A user sends a request; an LLM interprets it and selects a tool.

Sometimes all we need is that decision, not a paragraph or reasoning trace.

Recently, I came across three interesting approaches: Jev from TypeSafe AI, Constrained Decoding Models (CDMs) from ScaleDown, and Next-Concept Modeling (NCM) from Stanford's Hazy Research group.

Together they made me rethink how we design AI systems.

Jev by TypeSafe AI: Models That Make Decisions

TypeSafe AI's Jev evaluates application state and returns typed decisions rather than open-ended text.

It supports three operations:

• Choice: Select an option from predefined choices.
• Score: Rate something against ordered, described levels.
• Noul: Return the probability that the answer to a yes/no question is yes.

For example, imagine an agent needs to select a tool:

User: "Find the function that handles authentication."

Available tools:
- search_code
- read_file
- search_web

Decision:
search_code

A general-purpose LLM can handle this too. Jev treats it as a decision problem: for a Choice, it returns the selected option and a probability distribution across all available options. Its separate confidence value summarizes how concentrated that distribution is. A Noul returns one yes probability, without a separate confidence value.

That uncertainty could help an agent decide whether to continue, ask for clarification, or use another approach. But a confident answer can still be wrong. Calibration means checking whether predicted probabilities match observed outcomes across examples; I'd evaluate that on my own tasks before trusting a threshold.

CDMs by ScaleDown: Models Built for Specific Jobs

ScaleDown's documentation describes a suite of task-specific models, including Constrained Decoding Models for classification and structured decisions. Its broader suite also covers other focused tasks:

• Classify: Categorize information using chosen labels or rubrics.
• Extract: Retrieve structured fields from unstructured text.
• Summarize: Produce concise summaries.
• Compress: Retain context relevant to a particular task.

Context compression particularly caught my attention.

Long-running agents accumulate conversation history, tool responses, retrieved documents, and previous decisions. Not all of it remains relevant.

Imagine an agent retrieving 20,000 tokens of documentation to answer a question that only requires two paragraphs.

Instead of forwarding everything to another LLM, a compression model could keep the relevant context. It could also drop a crucial detail. I'd like to evaluate that tradeoff.

Fine-tuning is a technique for adapting a pretrained model to a task. ScaleDown's CDMs are specialized models within its suite. The concepts are related, but they aren't interchangeable; the documentation doesn't say every model in the suite was made by fine-tuning an LLM.

NCM by Stanford Hazy Research: Rethinking Text Generation

Jev and CDMs focus on specialized decisions and tasks, while Next-Concept Modeling explores a different architecture for language generation.

Traditional autoregressive LLMs generate one token at a time. NCM takes a coarse-to-fine approach.

Instead of directly predicting the next text token, it learns a hierarchy of discrete codes that represent text at increasingly detailed levels. These "concepts" are learned codes, not human-readable ideas.

Traditional LLM:

Token → Token → Token → Token → ...


NCM:

Coarse representation
         ↓
More detailed codes
         ↓
Fine-grained codes
         ↓
      Text

NCM generates one level at a time, predicting the codes within that level in parallel. A decoder then turns the completed hierarchy into text.

In the research paper, the authors report 10–31× faster generation than strong diffusion language model baselines on the evaluated LM1B, OpenWebText, and XSum tasks, at comparable or better reported quality.

That doesn't mean NCM is 31× faster than GPT or Claude.

Those research comparisons don't establish performance across real-world agent workloads or at frontier scale. They do challenge the assumption that text must be generated token by token.

Different problems, different models

Here's how I think about these approaches:

They're not direct replacements for one another. That's what makes this interesting.

A single agent might need to understand a request, select a tool, process a large response, and generate a final answer.

Using the same large model for every step is convenient, but it may not always be the most efficient choice.

A specialized model might work well for classification. Another approach might reduce the context passed to a general-purpose LLM. A capable language model could still handle the reasoning and final response.

Whether that architecture is actually better depends on accuracy, latency, cost, reliability, and integration overhead. It's something to measure, not assume.

What I'm taking away

What interests me is the broader direction: more ways to decide which work belongs with a general-purpose language model.

An agent isn't just a language model. It's an entire system involving decisions, state, tools, context, and execution.

Improving that system doesn't always mean using a bigger model.

Sometimes, it might mean using the right model for the right task.

References

• TypeSafe AI — Jev and its decision types
• TypeSafe AI — Choice and confidence
• ScaleDown — Documentation
• Hazy Research — Next-Concept Modeling
• Next-Concept Modeling — Research Paper