What is dataset leakage in evaluation sets, and why is it problematic?

• A. Leakage occurs when evaluation data never overlaps with training data.
• B. Leakage refers to data leakage prevention measures.
• C. Leakage is when the model's predictions are leaked to the user.
• D. Leakage occurs when evaluation data overlaps with training data, inflating scores.

Answer: (D)

Dataset leakage in evaluation sets happens when information from the evaluation data ends up available during training. If some evaluation examples are present in the training data, or if preprocessing and feature engineering are computed using the full dataset (including the evaluation portion), the model can learn from data it should be tested on. This makes the model appear to perform better than it would on truly unseen data because it has effectively memorized or exploited information from the test set.

Why this is a problem is that evaluation is meant to measure how well the model generalizes to new, unseen data. Leakage creates an inflated performance estimate, which can mislead you into choosing models that won’t generalize in real use. It also masks the true difficulty of the task and undermines trust in the evaluation results.

To prevent leakage, keep a strict train/validation/test separation, ensure preprocessing and feature engineering are fitted only on the training data, and use time- or event-appropriate splits for sequential data. This helps ensure the evaluation reflects genuine generalization.