How to Cut Your LLM Classification Costs by 90%

Channel: Hamel Husain
Published: August 12, 2026
URL: https://www.youtube.com/watch?v=FWFjLF_VoVI

Join the AI Evals September 2026 cohort: https://maven.com/parlance-labs/evals?promoCode=yt-2026

Most people treat classification as a fixed cost. The task needs a smart model, so you run the smart model on every record and eat the bill. On a million support tickets, that's tens of thousands of dollars. But you can match that model's answers 95%+ of the time while cutting 90% or more of the cost, and you don't need to fine tune anything.

Transcript

Welcome everybody. Shreya is going to be talking about how to save money for LLM classification. It's actually like a very clever technique but you're going to want to pay attention to this because there is a way that you can sort of hill climb your way into having a cheaper model for classification without sacrificing any performance, without fine tuning. But I think it's really interesting and important. Classification is becoming more and more important, like especially with things like model routing you often want to do classification. And as token costs are going up, like model routing is really important. So I'll go ahead and kick it off.

Very excited to do this live whiteboard lecture on reducing costs for classification. Could be a total train wreck. So let's start with some basic motivation of broadly what is kind of classification or a classification problem statement. So the whole goal of classification, especially for LLMs, is if you have a collection of text records, say given a collection of text records, produce a label for each record. And this label is drawn from some sort of finite set. So let's make it really simple. Let's make it binary classification.

So for example, if I have a collection of customer support tickets, what do we want to classify? How do you want to think of like binary classification tasks? Customer support tickets. Yeah. Should it be escalated to human or not? Should the ticket be escalated to a human or not? And we can have various criteria for escalation. Like maybe it has to do with billing or it has to do with like some more sensitive topic. So let's be even more precise then. Ticket be escalated to human or not because the customer is complaining about something sensitive like billing. All right.

So hopefully people understand like this is a binary classification task right? It's just a yes or no task. So an LLM in the ideal world would be given the customer support ticket and it would be prompted: "Should we escalate to the human because it contains something sensitive?" and then it would produce yes or no.

So let's draw out what that, what we'll call Oracle plan looks like. So given, I'm going to call this the Oracle plan. So here I'm going to draw what I think is the customer's—that's not the right shape. So maybe this is like the ticket, customer support ticket. What happens to it? Well it gets sent to a large language model, LLM. The label inside this is the LLM API. And then outcomes some yes or no, yes or no. But this is what our pipeline is going to look like.

And let's be even more specific. So what LLM do we want to use? Ideally we want to use some LLM that has good quality. It's not like a bad LLM. So I'm going to say that this is GPT-5.6 Sol. So everybody on the same page. So this is what we call the Oracle plan. That is, we want to be able to execute this plan on every single ticket. Yeah, so that's what Oracle plan means for us here.

So ideally we execute pipeline on all. One naive thing that is probably top of mind for everybody who has to pay personally for these calls is "wow, this is going to be really expensive." Like let's say I am Walmart or let's say that I'm not even Walmart, I'm like a midsize company. GPT-5.6 Sol is quite expensive, right? If you do some back of envelope calculations, which I should have done in preparation for this lecture, I would guess that it would cost tens of thousands of dollars to be able to run this pipeline on, let's say, like a million customer support tickets. So depending on the length of the ticket, depending on whether you know you enable reasoning or not, basically you can imagine that the cost of this is going to blow up, especially if you do this at scale.

Now, again, don't quote me on the exact numbers. But let's all assume that we do not want to pay for this. However, what we do want is to be able to maintain the accuracy. We really like GPT-5.6 Sol because it's high quality.

Problem Statement

So now let me give us a more concrete problem statement: GPT-5.6 Sol is too expensive. How can we reduce the cost of this pipeline while maintaining the same accuracy as what GPT-5.6 Sol would give us? We want to reduce the cost of pipeline, try to find the cheapest pipeline. And maybe it's possible to get 100% accuracy with respect to GPT-5.6 Sol.

Remember, accuracy means we want to find a plan that agrees with GPT-5.6 Sol 100% of the time, or as much of the time as possible. So what we'll instead say is: while maintaining a target accuracy of 95% compared to with respect to what GPT-5.6 Sol would give us. We're not yet concerned about the accuracy of the GPT-5.6 Sol. We're just saying how can we mimic GPT-5.6 Sol exactly?

So this is now our problem statement. And now let's get into, at a very high level, what the architecture of the pipeline is that's going to get us this accuracy.

Model Cascade Architecture

I'm going to call this a model cascade. Okay, so our solution is called a model cascade. What does a model cascade architecture look like?

Instead of the expensive plan, what we're going to do is first send every ticket to a cheap LLM API. And let's say this is like a nano model. And then what's going to happen? So nano is going to tell us yes or no. So it's going to give us a prediction. I'm going to call this P. I'm going to call this ticket x. And I'm going to call this actually proxy LLM API.

So that way when we say prediction P, it's like the proxy, it's going to tell us the prediction as well as a confidence score. I'll get into that a little bit: C(x). And now we'll hit a decision point. So this is our decision triangle. And we're going to say: if C(x) is high, then we accept P(x). Otherwise, what we're going to do is we're going to ask the oracle. Oracle then will tell us—let's use notation here—but let's just say at a high level, you know, P(x) is going to give us either yes or no. C(x) is a score between zero and one, and O(x) is also yes or no.

Things to keep in mind:

• P(x): proxy model score, zero to one (probability score)
• C(x): confidence score between 0 and 1
• O(x): ground truth label (either true or false)

Ground truth in this context means what the oracle would say, not what is the truth necessarily. It's helpful to use that terminology so we can move forward. When we define accuracy, it makes sense with respect to O(x).

So this is the high level overview of a model cascade.

Finding the Threshold

Next thing I'm going to do is let's think about: okay, what are the ambiguous things in this cascade? Let's assume that the LLM API providers allow you to see the probabilities for each token. So you can get that directly from OpenAI or Anthropic or so forth. So let's assume that we know how to get C(x).

Now our question is: what does "high" mean? I said here our decision threshold is if C(x) is high, that's extremely vague. We need an actual number here to compare C(x) to. So then this gets into the question of how are we going to construct the model cascade.

Model Cascade Algorithm

The first step of the cascade algorithm is to find threshold, and let's call it τ (tau). And we will do this in an offline way. The second step is to actually do the inference with this architecture—run the cascade on all the tickets.

So these are the two steps to being able to classify all of your tickets.

How to Find the Threshold

At a very high level, the answer is we're going to use samples. Imagine that we have visualizing all of the proxy predictions. Here we are and we start out with zero and here is one. Okay, and this is supposed to be C(x). So this is a line that is visualizing all of C(x), so all the proxies' predictions. All these blue represent various different confidences.

Here I've drawn out like what plausibly could be some sample of proxy scores. The first step of the algorithm is going to be:

Step One: Take a sample of tickets (typically between 200 and 500—let's say 500 tickets) and run the proxy on each ticket. This is going to give us P(x_i) and C(x_i) for 500 tickets.

Step Two: Label each ticket in the sample with the oracle. This will give us O(x_i) for each i. So concretely, maybe the oracle goes through and says this one was good, this one was good, this one was bad, etc. When you say good and bad we're meaning like true or false. Should it be escalated to human? Let's say the oracle is saying the check is escalated, and then the x is not escalated. So it doesn't mean like the proxy was correct or not.

Step Three: Now that we have that, what we're going to do is we are going to essentially simulate what would happen in the sample if we had different values of our threshold τ. So for each possible C(x_i) in the sample, the first thing that we're going to do is simulate the model cascade with threshold τ = C(x_i), record accuracy and cost.

Specifically, what I'm saying is: we will run this model cascade, we will substitute τ for this decision diamond here, and then we will get all of the outputs from the cascade. So now we have all these C(x_i)s (we have 500 of them) and we can try for each one of these 500. What if we set τ to C(x_i)? What will our accuracy of the cascade be? And what will the cost of the cascade be?

To reframe it: you pick a threshold. So τ is some threshold, let's say 0.8. And then you say: "hey, if every time the confidence, the C(x), is over or greater than or equal to 0.8, what if we just route it to the small model?" How often does it agree with the oracle? Well, they all go through the small model first.

So we're imagining we've set τ to 0.9. Now let's simulate what would happen if τ is 0.9. Anytime the confidence is over 0.9, we just accept. Otherwise, we will route it to the oracle. And then we will take all of those final predictions and compute the accuracy with respect to O(x) on the sample.

Note that of course, anytime we query the oracle, we are essentially getting it correct. Like the oracle is always going to be correct by how we've defined it. So really what we're trying to see is making sure that the proxy is correct in the times that we accept that proxy's prediction.

You have your target accuracy, which is 95%, and you're trying to see: okay, what threshold—like how low can you go in the threshold—to still get to 95%?

Sanity Check

First, I want to give a sanity check for people to understand. There is a trivial cascade that will give you 100% accuracy. What would τ be in that case? In that case, we would just set τ to be 1. So if τ is 1, then confidence can never be over 1, and we would always route to the oracle. And then by definition we'll get 100% accuracy.

Okay, so I want people to really understand deeply here: this model cascade, we can set the threshold such that we meet the accuracy while we're trying to reduce costs. The lower we go—which we'll get into—the better. Here we can just assume that we will meet the accuracy. We can always find some τ that meets the accuracy, and trivially that is τ = 1.

Selecting the Optimal Threshold

So now let's get into the cost part. So now we've gotten the number of accuracies and costs for each possible C(x_i). And what we want to do at the end of this is return the τ = C(x_i) with minimum cost that meets the accuracy target.

Basically, remember: for each C(x_i), we've recorded what the accuracy and the cost could be on the sample. If the accuracy is below, like, 95% or whatever target we've set, then throw it out. We'll never choose that C(x_i) as a threshold. We'll just choose the threshold that gives us the minimum cost.

One thing that's very nice about this model cascade business is this: suppose I chose, and actually, one thing I want to mention is that I want to reconsider P(x_i) here. If it's greater than τ, I will definitely have it be returning true. And if it's less than τ, I will have the proxy return false.

Let's say that I've set this to be possible τ₁. And then I am setting this guy to be a τ₂. Now something that's very interesting about accuracy is that the accuracy basically—let's say that I have selected τ₂. My overall cost of my cascade when I select τ₂ is obviously going to be less than if I select τ₁.

And this is actually hard to realize. So I'm going to say this again: if I select τ₂ for my decision point here, the overall cost is going to be less than if I select τ₁. And that's because in τ₂, because there are more dots to the right of τ₂ and those dots all go to the smaller proxy. That's why it's cheaper!

Realize that this τ is like everything on the right of this τ is accepted by the proxy. Everything to the left of the τ is what gets routed to the oracle. By definition, right, the lower of a threshold that we could find, the happier that we will be, right? Because we are going to find something lower cost. Ideally we find something over here because that means we can accept all of these dots greater than the threshold.

However, in practice, this is not the case because the proxy is not always accurate, right? For many of these middle confidence dots, the proxy may not be accurate. We will not meet our target accuracy. The lower threshold the more cost saving that you'll have. Of course, you have to take care to make sure we meet the accuracy on the sample, but we want to find a lower threshold.

One thing that we can do to make this a more efficient algorithm is to sort C(x_i) in ascending order and then return first the x_i that meets accuracy.

(capture appears truncated)