What is a token in NLP and how does tokenization affect prompt length and model input constraints?

• A. A token is only a character.
• B. A token is the model's basic input unit (word, subword, or character). Tokenization affects how many tokens a prompt uses, impacting max token budget and cost; longer prompts may reduce space for the answer.
• C. A token is always a word.
• D. Tokenization has no effect on input size.

Answer: (B)

In NLP, a token is the model’s basic input unit—it's not limited to a single word. A token can be a whole word, a subword piece, or even a character, depending on how the text is tokenized. Tokenization converts raw text into these tokens before the model processes it, and that choice of tokens directly changes how long the input feels to the model.

This matters because there is a fixed limit on how many tokens the model can handle in one prompt plus response. If your prompt uses many tokens, there’s less room left for the model’s answer. That’s why longer prompts can reduce the maximum possible length of the reply and can also affect the cost, since many APIs charge per token.

Also, token counts aren’t the same across models. The same sentence might yield a different number of tokens with different tokenizers, especially because subword tokenization splits words into smaller pieces. This variability is another reason tokenization shapes prompt length and input constraints.

So the best choice describes token as the model’s basic input unit (word, subword, or character) and notes that tokenization determines how many tokens a prompt uses, which in turn affects the max token budget and cost, with longer prompts leaving less space for the answer.