Stop AI Overthinking: Save API Token Costs

Posted on In Views:

Suppress self-reflection tokens like “Wait” and “Hmm” to lower LLM API costs

A lot of frontier models have dropped recently. I noticed a clear pattern while using models like GLM-5.3 and Opus 5. They start to sound like they are talking to themselves during reasoning. They constantly self-correct using high-frequency trigger words like “hmm,” “wait,” and “actually” to signal a pivot.

The catch is that every output becomes longer and more expensive.

Taking my own workload as an example, here is a comparison based on the frequency per 1k characters.

The numbers tell a clear story. “Wait” doubled, while “hmm” exploded by nearly 10x. We can see this shift in another set of numbers: the median reasoning length. The median reasoning length for GLM-5.2 was 948, but GLM-5.3 jumped to 2621.

Simply put, these frontier models are not just more expensive per token. They also consume way more tokens.

Let’s look at another breakdown. I analyzed the actual impact of each “hmm” and “wait.”

When “hmm” appears, the semantic inversion ratio hits 57%. If a reasoning block contains three “hmm”s, the model is likely spinning its wheels in place.

Here is an example. An agent might reason like this: “I checked the evidence provided by the user and found a pattern for option A. Hmm, wait, am I trusting that evidence too much? Actually, I haven’t verified it myself. Wait, the user probably ran tools to fetch results beforehand for my analysis. Hmm, but I should still run through it once myself. Wait, would that contaminate the evidence collection process? Hmm, actually I should keep verifying the feasibility of option A.”

You can see how the agent enters a bizarre state of split personality dialectics after receiving an instruction. It talks to itself endlessly. Most of the time, it verifies the exact same point from different angles even when the answer is already obvious.

When our goal is simply asking the agent to analyze the feasibility of option A, it might overthink instead of executing the task.

This pattern gets worse on complex workloads. Most of my tasks are analytical. Seeing this self-reflective rumination really tests my patience.

Looking at the multi-fold increase in reasoning tokens, this drives up costs and severely degrades execution efficiency.

Can we fix this? Yes, a few ways exist.

First, I will mention a common approach that I personally advise against: reducing the thinking effort from high to low. The flaw here is obvious. Effort is a hard limit for the agent. Its behavior does not actually change. It just gets cut off directly.

Given enough time, an agent might bounce back and forth until it finds the right track. Setting a strict limit might cause the agent to execute after a single turn. It ends up heading in completely the wrong direction. In the example above, the agent would stop evaluating option A and pivot back to validating the evidence.

To solve this, we must prevent the agent from getting stuck in loops. Several papers have addressed this issue with solutions. Here is one with a direct approach.

Wait, We Don’t Need to “Wait”! Removing Thinking Tokens Improves Reasoning Efficiency

To sum up, the paper shows that words like “hmm,” “wait,” and “actually” are unnecessary for many common workloads. Removing them completely does not degrade response quality. It actually boosts both efficiency and answer quality.

The approach is called NoWait. It is simple and brute-force. The key mechanism is Logit Bias. By adjusting token probability, we reduce the probability of tokens like “hmm,” “wait,” and “actually” by 100%, preventing them from appearing at all.

This feature has been available for a while, but only OpenAI and select open-source models support it. The Claude family does not offer this option.

Let me demonstrate using Fireworks with GLM-5.3.

First, we need to find the specific token IDs for those words.

1
2
3
4
5
6
7
8
9
10
11
12
13
curl -s https://api.fireworks.ai/inference/v1/chat/completions \
  -H "Authorization: Bearer $FIREWORKS_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "accounts/fireworks/models/glm-5p3",
    "max_tokens": 600,
    "temperature": 0,
    "logprobs": true,
    "top_logprobs": 1,
    "messages": [
      {"role": "user", "content": "Repeat exactly, nothing else: Wait, hmm, wait, Hmm."}
    ]
  }'

My prompt is simple. It asks the model to repeat my words. Enabling logprobs and top_logprobs modifies the output to expose the token ID for each word. We get the following JSON.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
{
  "choices": [
    {
      "finish_reason": "stop",
      "message": {
        "role": "assistant",
        "content": "Wait, hmm, wait, Hmm.",
        "reasoning_content": "The user wants me to repeat exactly: \"Wait, hmm, wait, Hmm.\"\n\nLet me look at this carefully. The text to repeat is: \"..."
      },
      "logprobs": {
        "content": [
          {
            "token": "Wait", "token_id": 14181, "text_offset": 38,
            "logprob": -0.00012719, "sampling_logprob": 0.0,
            "bytes": [87, 97, 105, 116],
            "top_logprobs": [{"token": "Wait", "token_id": 14181, "logprob": -0.00012719, "bytes": [87, 97, 105, 116]}]
          },
          { "...": "..." },
          {
            "token": " hmm", "token_id": 86776, "text_offset": 43, "logprob": -9.54e-06,
            "bytes": [32, 104, 109, 109],
            "top_logprobs": [{"token": " hmm", "token_id": 86776, "logprob": -9.54e-06}]
          },
          {
            "token": " wait", "token_id": 3783, "text_offset": 48, "logprob": -5.72e-06,
            "bytes": [32, 119, 97, 105, 116],
            "top_logprobs": [{"token": " wait", "token_id": 3783, "logprob": -5.72e-06}]
          },
          {
            "token": " Hmm", "token_id": 87459, "text_offset": 54, "logprob": -3.219e-05,
            "bytes": [32, 72, 109, 109],
            "top_logprobs": [{"token": " Hmm", "token_id": 87459, "logprob": -3.219e-05}]
          }
        ]
      }
    }
  ]
}

Most OpenAI-compatible APIs allow adjusting token probabilities using logit_bias. For example, the request below suppresses the four variations of “hmm” and “wait” listed above.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
curl -s https://api.fireworks.ai/inference/v1/chat/completions \
  -H "Authorization: Bearer $FIREWORKS_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "accounts/fireworks/models/glm-5p3",
    "max_tokens": 600,
    "temperature": 0,
    "logprobs": true,
    "top_logprobs": 1,
    "logit_bias": {
      "14181": -100,
      "86776": -100,
      "3783": -100,
      "87459": -100
    },
    "messages": [
      {"role": "user", "content": "Repeat exactly, nothing else: Wait, hmm, wait, Hmm."}
    ]
  }'

Notice how casing matters, as well as leading spaces. To block these words completely, we must identify all possible token_ids for suppression. Token IDs are not shared across models. Applying GLM-5.3 IDs to GLM-5.2 will not work.

Wrap Up

That covers today’s paper overview. I hope I clearly explained the problem, why it matters, the solution, and the benefits. This is not just a theoretical issue. It directly impacts everyone’s API bills.

For me, the bill is secondary. My main concern is that current agent behavior is reaching the limit of my patience. It feels too annoying, too slow, and too dumb. This tweak offers clear benefits for my workflow.

I should mention that not every workload fits the NoWait approach.

As mentioned earlier, my primary tasks are analytical. Using NoWait directly reduces extrapolation and self-reflection, which might make the agent more narrow-minded. The ideal setup includes a router to categorize the incoming prompt. For data collection and exploration, NoWait is a perfect match. For data analysis, we should preserve the agent’s ability to reason out loud.

To improve agent capability and response quality, the key lies in harness design rather than raw model capacity, a point I have emphasized repeatedly in previous articles. This serves as another classic example.

Originally published on Medium