Logit bias: banning or boosting specific tokens

Logit bias is an API setting that adds a fixed amount to the scores of specific token IDs, making those tokens more likely, less likely, or effectively banned.

What logit bias is

Logit bias lets you reach into the model's output at the token level. You give a map of token IDs to numbers, and those numbers are added to the token's logit before sampling. Positive values make a token more likely; negative values make it less likely. In the OpenAI spec the range is -100 to 100, where -100 is effectively a ban and 100 effectively forces the token.

How it works

The bias is applied to raw scores, before softmax. A bias of -1 to -5 makes a token noticeably rarer without removing it. Values near -100 push it so low it essentially never appears. The parameter is keyed by token ID, not by word:

{"logit_bias": {"12345": -100, "6789": 5}}

That has two big consequences. First, a word often maps to several tokens, and the same word with a leading space, with a capital letter, or at the start of a line is usually a different token. Banning one form leaves the others. Second, token IDs are specific to each tokenizer. An ID that means "smirk" for one model means something unrelated for another.

Use in roleplay

People use logit bias to suppress pet phrases and words a model overuses, to discourage particular formatting such as a certain symbol, or to stop a model writing for the user's character by biasing against the user's name. SillyTavern has a Logit Bias panel that takes plain text, tokenizes it with the tokenizer you select, and sends the IDs. If the selected tokenizer does not match the model, the IDs are wrong and the bias hits random tokens.

Common mistakes

Banning only the first token of a multi-token word, which can also block unrelated words that start with that token.
Using -100 on common pieces and breaking ordinary sentences.
Moving a logit bias list from one model to another without re-tokenizing.
Expecting a ban on a word to stop an idea. The model will simply use a synonym.

Logit bias on Wild West API

logit_bias is a standard OpenAI parameter, so it is accepted in the request body. Because token IDs belong to each model's tokenizer, a list built for one model will not carry over to another, and whether the upstream server applies it should be confirmed by testing. For broad style problems, instructions in the system prompt or author's note are often more effective than token bans.

FAQ

Can logit bias ban a word completely?

It can ban specific token IDs with -100, but a word has several tokenized forms (with or without a leading space, capitalized), so you have to ban each one, and the model may still use synonyms.

Why does my logit bias do nothing or break the text?

Usually the token IDs came from a different tokenizer than the model uses, so the bias landed on the wrong tokens.

Tokenizers: how text becomes token IDsA tokenizer is the component that converts text into the numbered tokens a language model works with, and converts the model's tokens back into text. Tokens: the units LLMs read, write and bill byA token is a small chunk of text, often a word or part of a word, that a language model reads and writes one at a time. Stop sequences: telling the model where to endA stop sequence is a string that, when the model produces it, immediately ends the response so the text stops at that point. Repetition penalty: what it does and how to set itRepetition penalty is a sampler that makes tokens already present in the recent context less likely to be chosen again.