Define perplexity and explain its role as a language model evaluation metric.

• A. Perplexity measures how surprised the model is by a test set; derived from cross-entropy; lower is better.
• B. Perplexity measures how well the model's grammar is.
• C. Lower perplexity indicates worse predictions.
• D. Perplexity is unrelated to cross-entropy.

Answer: (A)

Perplexity tells you how surprised the model is by the data, quantified as the exponent of the average negative log-likelihood (cross-entropy) of the model’s predictions on a held-out test set. To compute it, you evaluate the model on each token of the test data, record the probability it assigns to the actual next token, take the logarithm of those probabilities, average the negatives, and then exponentiate. This makes perplexity the geometric mean of the inverse probabilities the model assigns to the true next tokens, so lower values mean the model is more confident and accurate about its predictions. Since perplexity is tied directly to cross-entropy, it serves as a natural and widely used measure of language model performance: as cross-entropy decreases, perplexity decreases as well, indicating better predictive ability. Remember to compare perplexities on the same dataset with the same preprocessing and vocabulary, because changes in data or vocabulary affect the metric. Perplexity focuses on prediction likelihood rather than grammar or fluency per se, so other metrics may be needed to assess those aspects of generation.