RegressionEvaluator

class skeval.evaluators.regression.RegressionEvaluator(model: ~typing.Any, scorer: ~typing.Callable[[...], float] | ~typing.Dict[str, ~typing.Callable[[...], float]] = <function accuracy_score>, verbose: bool = False, meta_regressor: ~typing.Any | None = None)

Bases: BaseEvaluator

Regression-based evaluator for classification models.

This evaluator estimates the performance of a classification model (e.g., accuracy, precision) without requiring labeled test data. It works by training a meta-regressor (e.g., Random Forest) that learns to map meta-features, extracted from a classifier’s probability distributions, to its true performance score on unseen data.

To use this evaluator, it must first be fitted on a collection of diverse datasets using the same model type to learn this mapping robustly.

Parameters:

• model (object) – An unfitted classifier instance that will be used as the base model. Clones of this model will be trained during the fit process and the same model type must later be fitted manually before estimate.

• meta_regressor (object, default=None) – A regression model implementing fit and predict. If None, a RandomForestRegressor with 500 trees is used for each scorer.

• scorer (callable or dict of str -> callable, default=accuracy_score) – The performance metric(s) to estimate. If a dictionary is provided, a separate meta-regressor will be trained to estimate each metric.

• verbose (bool, default=False) – If True, prints informational messages during the training of the meta-regressors and during estimation.

meta_regressors_

A dictionary mapping each scorer’s name to its fitted meta-regressor instance. This attribute is populated after the fit method is called.

Notes

• The fit method expects lists of datasets (x and y), where each

  element is a dataset (features/labels) used to build the meta-dataset.

• After training the meta-regressors, fit will also fit self.model

  on the first dataset provided (x[0], y[0]). If you prefer to use a differently trained final model before calling estimate, manually set evaluator.model to your fitted estimator.

• The underlying classifier must implement predict_proba because the

  evaluator extracts meta-features from predicted probability distributions. This evaluator is designed for classification tasks.

Type:

dict

Examples

>>> # Authors: The scikit-autoeval developers
>>> # SPDX-License-Identifier: BSD-3-Clause
>>>
>>> # ==============================================================
>>> # RegressionEvaluator Example
>>> # ==============================================================
>>> import pandas as pd
>>> from sklearn.metrics import accuracy_score, f1_score
>>> from sklearn.ensemble import RandomForestClassifier
>>>
>>> from skeval.evaluators import RegressionEvaluator
>>> from skeval.utils import get_cv_and_real_scores, print_comparison
>>>
>>> def run_regression_eval(verbose=False):
>>>     # =====================================
>>>     # 1. Load datasets
>>>     # =====================================
>>>     geriatrics = pd.read_csv("./skeval/datasets/geriatria-controle-alzheimerLabel.csv")
>>>     neurology = pd.read_csv("./skeval/datasets/neurologia-controle-alzheimerLabel.csv")
>>>
>>>     # =====================================
>>>     # 2. Separate features and target
>>>     # =====================================
>>>     X1, y1 = geriatrics.drop(columns=["Alzheimer"]), geriatrics["Alzheimer"]
>>>     X2, y2 = neurology.drop(columns=["Alzheimer"]), neurology["Alzheimer"]
>>>
>>>     # =====================================
>>>     # 3. Define pipeline (KNNImputer + RandomForest)
>>>     # =====================================
>>>     model = RandomForestClassifier(n_estimators=180, random_state=42)
>>>
>>>     # =====================================
>>>     # 4. Define scorers and evaluator
>>>     # =====================================
>>>     scorers = {
>>>         "accuracy": accuracy_score,
>>>         "f1_macro": lambda y, p: f1_score(y, p, average="macro"),
>>>     }
>>>
>>>     evaluator = RegressionEvaluator(model=model, scorer=scorers, verbose=False)
>>>
>>>     # =====================================
>>>     # 5. Fit evaluator using multiple datasets
>>>     # =====================================
>>>     evaluator.fit([X1, X2], [y1, y2], n_splits=4)
>>>
>>>     # =====================================
>>>     # 6. Estimate scores for new dataset
>>>     # =====================================
>>>     estimated_scores = evaluator.estimate(X2)
>>>
>>>     # ======================
>>>     # 7. Cross-Validation and Real Performance
>>>     # ======================
>>>     train_data = X1, y1
>>>     test_data = X2, y2
>>>     scores_dict = get_cv_and_real_scores(
>>>         model=model, scorers=scorers, train_data=train_data, test_data=test_data
>>>     )
>>>     cv_scores = scores_dict["cv_scores"]
>>>     real_scores = scores_dict["real_scores"]
>>>
>>>     if verbose:
>>>         print_comparison(scorers, cv_scores, estimated_scores, real_scores)
>>>
>>>     return {"cv": cv_scores, "estimated": estimated_scores, "real": real_scores}
>>>
>>> if __name__ == "__main__":
>>>     results = run_regression_eval(verbose=True)

estimate(x_eval: Any) → Dict[str, float]

Estimates the performance of the current model on unlabeled data.

The model assigned to self.model must already be a fitted classifier (manually trained by the user). This method extracts meta-features from its predictions on the unlabeled data x_eval and uses the pre-trained meta-regressor(s) to predict the performance scores.

Parameters:

x_eval (array-like of shape (n_samples, n_features)) – The unlabeled input data.

Returns:

A dictionary with the estimated scores, where keys are the names of the scorers and values are the predicted performance scores.

Return type:

dict

Raises:

• RuntimeError – If the estimate method is called before the evaluator has been fitted with the fit method.

• ValueError – If self.model does not implement predict_proba.

fit(x: Sequence[Any], y: Sequence[Any], n_splits: int = 5) → RegressionEvaluator

Trains the internal meta-regressor(s) using a single model type.

This method builds a meta-dataset to train the evaluator. For each dataset, it performs multiple random splits to increase the number of meta-examples.

Parameters:

• x (list of array-like) – A list of datasets (features) used to train the meta-model.

• y (list of array-like) – A list of labels corresponding to x.

• n_splits (int, default=5) – Number of random splits per dataset to generate multiple meta-examples.

Returns:

self – The fitted evaluator instance.

Return type:

object