Explain ROUGE metrics and their typical use in summarization; what are their limitations?

• A. ROUGE uses automatic scoring with BLEU.
• B. ROUGE computes overlaps with reference summaries (ROUGE-N, ROUGE-L); limits include reliance on lexical overlap, not quality.
• C. ROUGE measures syntax correctness.
• D. ROUGE is used only for machine translation.

Answer: (B)

ROUGE evaluates a generated summary by comparing it to human reference summaries and using text overlap as a stand-in for how well the content is covered. The idea is to quantify how much of the important material in the reference is present in the system output.

Two common variants are ROUGE-N and ROUGE-L. ROUGE-N looks at shared n-grams between the candidate and reference(s); for example, ROUGE-1 counts matching unigrams and ROUGE-2 counts matching bigrams. This captures direct word-for-word overlap and gives a sense of content similarity. ROUGE-L uses the longest common subsequence, which measures not just exact word matches but also the ordering of content, rewarding summaries that preserve the sequence of ideas even if some wording differs. In practical work, you’ll often see precision, recall, and F1 reported, reflecting how much of the reference content is found in the summary and how much of the summary content is relevant to the reference.

ROUGE is typically used in summarization research as an automatic evaluation method. It lets researchers quickly compare models by scoring system outputs against one or more human references, and it’s common to optimize or select models based on ROUGE scores. Using multiple reference summaries helps account for variation in acceptable summaries and makes the evaluation more robust.

However, there are importantLimitations to keep in mind. Because ROUGE relies on lexical overlap, paraphrase, synonyms, or content expressed in markedly different wording may receive a low score even if the ideas are well captured. It doesn’t assess semantic correctness, factual accuracy, coherence, or readability, so a summary could be fluent and true while scoring poorly if it doesn’t reuse similar wording to the references. It also depends on the quality and representativeness of the reference set; a narrow or biased reference can skew results. Finally, the metrics can be sensitive to choices like which n in ROUGE-N to use, how to handle stopwords, and how much length bias is allowed.

To get a fuller picture of quality, researchers often pair ROUGE with human judgments or complement it with semantic or embedding-based metrics that better capture meaning beyond exact word matches.