← Back to the parts catalog
Inference · Default Starter Code

Sampling Strategies

How the model actually picks its next word

Once a trained model has worked out a probability for every word it could say next — a number showing how likely each one is — something still has to pick one. That choice is called sampling. The plainest choice, "greedy," always picks whichever word has the highest probability. A more natural-sounding choice, "top-p," first narrows the list down to the smallest group of words whose probabilities add up to some target like 90 per cent, and only then picks randomly from that smaller group, weighted by how likely each one is.

The code

import random


def greedy_sample(probabilities):
    """Always pick the single most likely word."""
    return probabilities.index(max(probabilities))


def top_p_sample(probabilities, p=0.9):
    """Narrow down to the smallest group of words whose probabilities
    add up to at least p, then pick randomly from that group."""
    ranked = sorted(
        range(len(probabilities)),
        key=lambda i: probabilities[i],
        reverse=True,
    )
    pool, running_total = [], 0.0
    for i in ranked:
        pool.append(i)
        running_total += probabilities[i]
        if running_total >= p:
            break
    weights = [probabilities[i] for i in pool]
    return random.choices(pool, weights=weights, k=1)[0]

Try it — edit the code and run it for real

This runs real Python, right in your own browser. Nothing you type here is ever sent anywhere.

Checking what your browser can tell us about your computer…

Press Run to see the real output.

Log

A real, timestamped record of what this box has actually done, not a plan of what it will do.

Nothing has run yet.

What each part does

greedy_sample takes the list of probabilities and returns the position of the biggest one. It never varies, which makes it predictable but also makes it repeat itself in longer answers.

top_p_sample first sorts every word's position by how likely it is, from most to least likely. It then walks down that sorted list, keeping a running total, and stops as soon as the group it has kept adds up to at least p (90 per cent, by default). Finally, random.choices picks one position from that smaller group, giving more likely words a bigger chance without always forcing the single most likely one.

Why this shape

Any accepted submission in this category takes one list of probability numbers (all adding up to 1) and returns a single whole number: the position of the word that was chosen. That fixed shape is what lets any sampling piece plug into any model's output without either side needing to know how the other one works inside.