嵌套交叉验证与非嵌套交叉验证#

本示例比较了在 Iris 数据集分类器上使用非嵌套和嵌套交叉验证策略。嵌套交叉验证 (CV) 通常用于训练模型，其中超参数也需要优化。嵌套 CV 估计底层模型及其（超）参数搜索的泛化误差。选择最大化非嵌套 CV 的参数会使模型对数据集产生偏差，从而导致过于乐观的评分。

没有嵌套 CV 的模型选择使用相同的数据来调整模型参数和评估模型性能。因此，信息可能会“泄漏”到模型中并过度拟合数据。这种影响的大小主要取决于数据集的大小和模型的稳定性。有关这些问题的分析，请参见 Cawley 和 Talbot [1]。

为了避免这个问题，嵌套 CV 有效地使用了一系列训练/验证/测试集拆分。在内部循环（此处由 GridSearchCV 执行）中，通过将模型拟合到每个训练集来近似最大化评分，然后在验证集上选择（超）参数时直接最大化。在外部循环（此处在 cross_val_score 中）中，通过对多个数据集拆分平均测试集评分来估计泛化误差。

下面的示例使用具有非线性核的支持向量分类器来构建一个通过网格搜索优化超参数的模型。我们通过比较非嵌套和嵌套 CV 策略的评分差异来比较它们的性能。

另请参见

参考文献

Non-Nested and Nested Cross Validation on Iris Dataset

Average difference of 0.007581 with std. dev. of 0.007833.

import numpy as np
from matplotlib import pyplot as plt

from sklearn.datasets import load_iris
from sklearn.model_selection import GridSearchCV, KFold, cross_val_score
from sklearn.svm import SVC

# Number of random trials
NUM_TRIALS = 30

# Load the dataset
iris = load_iris()
X_iris = iris.data
y_iris = iris.target

# Set up possible values of parameters to optimize over
p_grid = {"C": [1, 10, 100], "gamma": [0.01, 0.1]}

# We will use a Support Vector Classifier with "rbf" kernel
svm = SVC(kernel="rbf")

# Arrays to store scores
non_nested_scores = np.zeros(NUM_TRIALS)
nested_scores = np.zeros(NUM_TRIALS)

# Loop for each trial
for i in range(NUM_TRIALS):
    # Choose cross-validation techniques for the inner and outer loops,
    # independently of the dataset.
    # E.g "GroupKFold", "LeaveOneOut", "LeaveOneGroupOut", etc.
    inner_cv = KFold(n_splits=4, shuffle=True, random_state=i)
    outer_cv = KFold(n_splits=4, shuffle=True, random_state=i)

    # Non_nested parameter search and scoring
    clf = GridSearchCV(estimator=svm, param_grid=p_grid, cv=outer_cv)
    clf.fit(X_iris, y_iris)
    non_nested_scores[i] = clf.best_score_

    # Nested CV with parameter optimization
    clf = GridSearchCV(estimator=svm, param_grid=p_grid, cv=inner_cv)
    nested_score = cross_val_score(clf, X=X_iris, y=y_iris, cv=outer_cv)
    nested_scores[i] = nested_score.mean()

score_difference = non_nested_scores - nested_scores

print(
    "Average difference of {:6f} with std. dev. of {:6f}.".format(
        score_difference.mean(), score_difference.std()
    )
)

# Plot scores on each trial for nested and non-nested CV
plt.figure()
plt.subplot(211)
(non_nested_scores_line,) = plt.plot(non_nested_scores, color="r")
(nested_line,) = plt.plot(nested_scores, color="b")
plt.ylabel("score", fontsize="14")
plt.legend(
    [non_nested_scores_line, nested_line],
    ["Non-Nested CV", "Nested CV"],
    bbox_to_anchor=(0, 0.4, 0.5, 0),
)
plt.title(
    "Non-Nested and Nested Cross Validation on Iris Dataset",
    x=0.5,
    y=1.1,
    fontsize="15",
)

# Plot bar chart of the difference.
plt.subplot(212)
difference_plot = plt.bar(range(NUM_TRIALS), score_difference)
plt.xlabel("Individual Trial #")
plt.legend(
    [difference_plot],
    ["Non-Nested CV - Nested CV Score"],
    bbox_to_anchor=(0, 1, 0.8, 0),
)
plt.ylabel("score difference", fontsize="14")

plt.show()

脚本的总运行时间：（0 分钟 6.571 秒）