Low-latency decision routing with a small encoder

A small sentence encoder can answer the common cases of an LLM-backed decision in under a millisecond and pass the rest to the LLM.

Problem

Many applications call an LLM to decide what to do with each user message: which action to take, which template to use, which tool to call. A hosted model call puts a network round trip and model latency in front of every decision.

Many messages fall into a small set of common intents, such as "email me the details" or "how much does it cost". For those, the response is already known, and the decision is a selection, not a generation.

Approach

The encoder answers when it is confident and hands off when it is not (Figure 1).

Intents. Each common intent has a prepared response or action. The router selects an intent and does not generate text.

Examples. Each intent has a few example phrases. Their embeddings are averaged into one prototype per intent.

Encoder. A small MiniLM sentence encoder embeds the message. The cosine similarity to each prototype is the intent score.

Margin. The margin is the top score minus the second score. A large margin means one intent is clearly closest.

Gate. If the margin is at or above a threshold, the matched intent is used. Otherwise the message goes to the LLM. The threshold is the smallest margin at which at least 20 validation phrases are accepted and 95% of them are correct. The validation phrases are held out from both the examples and the test set.

Results

All numbers except Table 4 come from 30 intents and 60 test utterances of short customer messages, such as questions about price, objections and scheduling replies. The 10 training phrases per intent, the 3 validation phrases per intent and the test utterances are synthetic and were written by Claude. No phrase appears in more than one set.

Latency. The whole routing step takes a fraction of a millisecond at p50 (Table 1). On the same machine, PyTorch fp32 on the CPU took 1.5 ms (L3) and 4.2 ms (L6) at p50 for encoders of the same architecture. Loading the model took about 0.3 to 0.9 s in the benchmark runs, so it should happen at startup, not per request.

Table 1. Encoder size and latency. Apple M5 Mac, Core ML on the Neural Engine and CPU, 32 tokens, 1,000 runs. Embed is tokenization plus the encoder. Route adds the intent search. Python call overhead is included.

On device. The same models ran in a Swift app on an iPhone 13 Pro (A15, iOS 27.2). The tokenizer is a Swift WordPiece implementation. It produced the same token ids as the reference tokenizer for all 60 test utterances. The embeddings matched the Mac's (cosine above 0.999), and the routing decisions agreed on all 60 utterances. Core ML placed 74 of 102 operations (L3) and 146 of 174 (L6) on the Neural Engine. The route p50 stays under a millisecond (Table 2).

Table 2. On-device latency. iPhone 13 Pro, Core ML on the Neural Engine and CPU, int8 weights, 32 tokens. Median of three passes of 1,000 runs each. The device was at nominal thermal state and not in Low Power Mode.

Allowing the GPU as well (.all) raised the route p50 to 3.5 ms (L3) and 3.9 ms (L6). With the CPU alone, every output on this device was NaN, so the compute units should be set explicitly and the embeddings validated against a reference on each target.

Examples. Accuracy depends more on the number of labeled examples per intent than on the encoder (Figure 2). With 3 examples per intent, top-1 accuracy is 0.77 for the 6-layer encoder and 0.70 for the 3-layer one. With 10 per intent, it is 0.93 and 0.92.

Head. A linear classifier head, softmax regression on the frozen embeddings, differed from averaging the examples by at most two utterances (Table 3). The gain comes from the data.

Table 3. Top-1 accuracy with 10 examples per intent on the 60 test utterances. One utterance is 1.7 points.

Routing. The threshold is chosen on the validation phrases and then applied once to the test set. For the 6-layer encoder the threshold is 0.108. It answers 45 of 60 utterances (75%), and all 45 are correct. The other 15 go to the LLM. For the 3-layer encoder the threshold is 0.119. It answers 34 (57%), and all 34 are correct (Figure 3).

Public dataset. The same procedure ran on CLINC150, which has 150 intents and human-written messages. Each intent is represented by the average of k training phrases. The threshold is chosen on 10 validation phrases per intent plus the validation out-of-scope messages, then applied once to the test set of 4,500 in-scope and 1,000 out-of-scope messages. The 6-layer encoder with 10 phrases per intent answers 79% of the in-scope messages at 97% accuracy (Table 4).

Table 4. Routing on CLINC150 with the threshold chosen on validation. Means over 5 random draws of the k phrases per intent (a single draw for k = 100). The last column counts an accepted out-of-scope message as an error.

Out-of-scope. The gate also answers some out-of-scope messages, and each one is an error. Out-of-scope messages are 18% of the test set and 6% of the validation set, so accuracy on answered messages lands below the 95% target. Reweighting the test set to the validation mix gives about 0.95 in every row. The validation set should contain out-of-scope messages in the proportion expected in production.

Implementation

The encoder is a BERT model with mean pooling and L2 normalization, traced at a fixed input shape. Here it is converted to a Core ML ML program to run on the Apple Neural Engine of a Mac or an iPhone, and any runtime that can execute the model works the same way. Conversion used torch 2.7.0, transformers 4.x, numpy below 2.4 and coremltools 9.0. Newer combinations failed on an unsupported new_ones op or on a numpy scalar conversion.

import numpy as np
import torch
import coremltools as ct
from torch import nn
from torch.nn import functional as F
from transformers import BertModel


class Encoder(nn.Module):
    def __init__(self, bert):
        super().__init__()
        self.bert = bert

    def forward(self, input_ids, attention_mask):
        hidden = self.bert(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state
        mask = attention_mask.unsqueeze(-1).to(hidden.dtype)
        pooled = (hidden * mask).sum(dim=1) / mask.sum(dim=1).clamp(min=1e-9)
        return F.normalize(pooled, dim=-1)


bert = BertModel.from_pretrained(
    "sentence-transformers/all-MiniLM-L6-v2",
    add_pooling_layer=False,
    attn_implementation="eager",
).eval()
ids = torch.ones(1, 32, dtype=torch.int32)
mask = torch.ones(1, 32, dtype=torch.int32)
traced = torch.jit.trace(Encoder(bert).eval(), (ids, mask), strict=False)

model = ct.convert(
    traced,
    inputs=[
        ct.TensorType(name="input_ids", shape=(1, 32), dtype=np.int32),
        ct.TensorType(name="attention_mask", shape=(1, 32), dtype=np.int32),
    ],
    outputs=[ct.TensorType(name="embedding")],
    convert_to="mlprogram",
    compute_precision=ct.precision.FLOAT16,
    minimum_deployment_target=ct.target.iOS17,
)
config = ct.optimize.coreml.OptimizationConfig(
    global_config=ct.optimize.coreml.OpLinearQuantizerConfig(mode="linear_symmetric", dtype="int8")
)
model = ct.optimize.coreml.linear_quantize_weights(model, config=config)
model.save("encoder_int8.mlpackage")

The router builds a prototype per intent, scores a message against all of them, and returns an intent only when the margin clears the threshold. None means escalate to the LLM. The embeddings are already normalized, so a dot product is the cosine similarity.

import numpy as np
import coremltools as ct
from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file("tokenizer.json")
tokenizer.enable_truncation(max_length=32)
tokenizer.enable_padding(length=32, pad_id=0, pad_token="[PAD]")
encoder = ct.models.MLModel("encoder_int8.mlpackage", compute_units=ct.ComputeUnit.CPU_AND_NE)


def embed(text):
    encoding = tokenizer.encode(text)
    outputs = encoder.predict({
        "input_ids": np.asarray([encoding.ids], dtype=np.int32),
        "attention_mask": np.asarray([encoding.attention_mask], dtype=np.int32),
    })
    return outputs["embedding"].reshape(-1)


def build_prototypes(examples):
    names = list(examples)
    matrix = np.stack([np.mean([embed(text) for text in examples[name]], axis=0) for name in names])
    matrix /= np.linalg.norm(matrix, axis=1, keepdims=True)
    return names, matrix


def route(text, names, matrix, threshold):
    scores = matrix @ embed(text)
    first, second = np.argsort(-scores)[:2]
    margin = scores[first] - scores[second]
    return (names[first] if margin >= threshold else None), margin


def choose_threshold(validation, names, matrix, target=0.95, minimum=20):
    rows = []
    for intent, texts in validation.items():
        for text in texts:
            guess, margin = route(text, names, matrix, threshold=0.0)
            rows.append((margin, guess == intent))
    rows.sort()
    for margin, _ in rows:
        accepted = [correct for m, correct in rows if m >= margin]
        if len(accepted) >= minimum and np.mean(accepted) >= target:
            return margin
    return None