RegressionNoiseEvaluator

class skeval.evaluators.regression_noise.RegressionNoiseEvaluator(model: ~typing.Any, scorer: ~typing.Callable[[...], float] | ~typing.Dict[str, ~typing.Callable[[...], float]] = <function accuracy_score>, verbose: bool = False, meta_regressor: ~typing.Any | None = None)[source]

Bases: RegressionEvaluator

Regression-based evaluator for classification models.

This evaluator estimates the performance of a classification model (e.g., accuracy, precision) without requiring labeled test data. It works by training a meta-regressor (e.g., Random Forest) that learns to map meta-features, extracted from a classifier’s probability distributions, to its true performance score on unseen data.

To use this evaluator, it must first be fitted on a collection of diverse datasets using the same model type to learn this mapping robustly.

Parameters:

  • model (object) – An unfitted classifier instance that will be used as the base model. Clones of this model will be trained during the fit process and the same model type must later be fitted manually before estimate.

  • meta_regressor (object, default=None) – A regression model implementing fit and predict. If None, a RandomForestRegressor with 500 trees is used for each scorer.

  • scorer (callable or dict of str -> callable, default=accuracy_score) – The performance metric(s) to estimate. If a dictionary is provided, a separate meta-regressor will be trained to estimate each metric.

  • verbose (bool, default=False) – If True, prints informational messages during the training of the meta-regressors and during estimation.

meta_regressors_

A dictionary mapping each scorer’s name to its fitted meta-regressor instance. This attribute is populated after the fit method is called.

Type:

dict

Examples

>>> # Authors: The scikit-autoeval developers
>>> # SPDX-License-Identifier: BSD-3-Clause
>>>
>>> # ==============================================================
>>> # RegressionNoiseEvaluator Example
>>> # ==============================================================
>>> import pandas as pd
>>> from sklearn.metrics import accuracy_score, f1_score
>>> from sklearn.ensemble import RandomForestClassifier
>>>
>>> from skeval.evaluators import RegressionNoiseEvaluator
>>> from skeval.utils import get_cv_and_real_scores, print_comparison
>>>
>>> def run_regression_noise_eval(verbose=False):
>>>     # 1. Load datasets
>>>     geriatrics = pd.read_csv("./skeval/datasets/geriatria-controle-alzheimerLabel.csv")
>>>     neurology = pd.read_csv("./skeval/datasets/neurologia-controle-alzheimerLabel.csv")
>>>
>>>     # 2. Separate features and target
>>>     X1, y1 = geriatrics.drop(columns=["Alzheimer"]), geriatrics["Alzheimer"]
>>>     X2, y2 = neurology.drop(columns=["Alzheimer"]), neurology["Alzheimer"]
>>>
>>>     # 3. Define pipeline (Optional preprocessing + RandomForest)
>>>     model = RandomForestClassifier(n_estimators=180, random_state=42)
>>>
>>>     # 4. Define scorers and evaluator
>>>     scorers = {
>>>         "accuracy": accuracy_score,
>>>         "f1_macro": lambda y, p: f1_score(y, p, average="macro"),
>>>     }
>>>
>>>     evaluator = RegressionNoiseEvaluator(model=model, scorer=scorers, verbose=False)
>>>
>>>     # 5. Fit evaluator using multiple datasets (builds meta-dataset with label noise)
>>>     evaluator.fit([X1, X2], [y1, y2], n_splits=5)
>>>
>>>     # 6. Estimate scores for new dataset
>>>     estimated_scores = evaluator.estimate(X2)
>>>
>>>     # 7. Cross-Validation and Real Performance
>>>     train_data = X1, y1
>>>     test_data = X2, y2
>>>     scores_dict = get_cv_and_real_scores(
>>>         model=model, scorers=scorers, train_data=train_data, test_data=test_data
>>>     )
>>>     cv_scores = scores_dict["cv_scores"]
>>>     real_scores = scores_dict["real_scores"]
>>>
>>>     if verbose:
>>>         print_comparison(scorers, cv_scores, estimated_scores, real_scores)
>>>     return {"cv": cv_scores, "estimated": estimated_scores, "real": real_scores}
>>>
>>> if __name__ == "__main__":
>>>     results = run_regression_noise_eval(verbose=True)
fit(x: Sequence[Any], y: Sequence[Any], n_splits: int = 5, noise_cfg: Dict[str, int] | None = None) → RegressionNoiseEvaluator[source]

Trains the internal meta-regressor(s) using a single model type.

This method builds a meta-dataset to train the evaluator. For each dataset, it performs multiple random splits to increase the number of meta-examples.

Parameters:

  • x (list of array-like) – A list of datasets (features) used to train the meta-model.

  • y (list of array-like) – A list of labels corresponding to x.

  • n_splits (int, default=5) – Number of random splits per dataset to generate multiple meta-examples.

  • noise_cfg (dict, default=None) – Configuration for label noise levels: - ‘start’: int, starting percentage of label noise (inclusive). - ‘end’: int, ending percentage of label noise (inclusive). - ‘step’: int, step size for noise percentage increments.

Returns:

self – The fitted evaluator instance.

Return type:

object

Notes

  • This evaluator extends RegressionEvaluator by generating meta-examples

    that incorporate controlled label noise. The noise_cfg parameter configures the percentage(s) of examples to permute during meta-example generation.

  • The fit method expects x and y to be lists of datasets (as in

    RegressionEvaluator). Each dataset is processed with multiple splits and multiple noise levels to build a larger meta-dataset.

  • After training meta-regressors, fit will fit self.model on the first

    provided dataset (x[0], y[0]). If you wish to use a separately trained final model before calling estimate, set evaluator.model to your fitted estimator manually.