Top-k & top-p sampling
i
Top-k & top-p sampling
Two filters for next-token choices. Top-k keeps a fixed number of candidates; top-p keeps enough candidates to reach a probability total.
Trim the probability tail, renormalize what remains, then sample from the surviving tokens.
Top-k caps the number of choices. Top-p keeps the smallest set whose original probability reaches the threshold.
Once upon the
the43%
a26%
this18%
one13%
everycut
somecut
4 of 6 candidates surviveProbabilities renormalized to 100%