Mirostat sampling explained

Mirostat is an adaptive sampler that adjusts itself token by token to keep the text's average surprise close to a target value.

What Mirostat is

Most samplers use fixed rules: keep the top 40 tokens, keep 90 percent of the mass, drop anything under 5 percent of the leader. Mirostat instead sets a target for how surprising the text should be and steers toward it. If recent tokens were too predictable, it loosens; if they were too surprising, it tightens. The idea comes from a 2020 paper that tried to avoid both the boring loops of low-randomness decoding and the incoherence of high randomness.

How it works

"Surprise" here means the negative log probability of the chosen token, measured in bits. Two settings control it:

Tau: the target average surprise. Higher tau allows more unexpected tokens. A common default is 5.0.
Eta: the learning rate, how quickly Mirostat corrects after each token. A common default is 0.1.

Mirostat keeps a running value, called mu, that sets the current truncation. After each token it compares that token's surprise with tau and moves mu up or down by eta times the error. Mode 1 is the original algorithm, which estimates a top K value from the distribution. Mode 2 is a simpler variant that drops any token whose surprise exceeds mu. In llama.cpp style backends, enabling Mirostat replaces most of the usual truncation samplers, so top K, top P and min P stop mattering while it is on.

Effect on roleplay

Mirostat can give a steady texture over long outputs, avoiding the drift into repetition that fixed samplers sometimes show deep into a reply. Its weak points are that tau is not intuitive, results vary a lot between models, and it is harder to combine with other tools. Many roleplay presets moved to min P with DRY because those are easier to reason about.

Sensible values

Mode 2, tau 5, eta 0.1: a sane default to try.
Tau 3 to 4: more focused.
Tau 6 to 8: looser, more varied, more risk of drift.

Keep temperature near 1.0 when testing so you are only adjusting one thing.

Mirostat on Wild West API

Mirostat is backend specific. It exists in llama.cpp, KoboldCpp and similar local servers, and is not part of the OpenAI chat completions spec. The API may ignore Mirostat fields unless the upstream model server supports them, so for hosted use stick to temperature and top_p.

FAQ

What do tau and eta mean in Mirostat?

Tau is the target average surprise of the text; higher means more unexpected words. Eta is how fast Mirostat corrects toward that target after each token.

Can I use Mirostat with min P?

In llama.cpp style backends, Mirostat replaces most other truncation samplers while it is enabled, so min P has little or no effect.

Temperature: how it changes what the model writesTemperature is a sampling setting that makes a language model's word choices more predictable when lowered and more varied when raised. Min P sampling explainedMin P is a truncation sampler that removes every token whose probability is less than a set fraction of the most likely token's probability. Top K sampling explainedTop K is a sampler that only lets the model choose its next token from the K most probable candidates. Top P sampling, also called nucleus samplingTop P, or nucleus sampling, limits each token choice to the smallest group of candidates whose combined probability reaches a set threshold.