
.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/ensemble/plot_stack_predictors.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        Click :ref:`here <sphx_glr_download_auto_examples_ensemble_plot_stack_predictors.py>`
        to download the full example code

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_ensemble_plot_stack_predictors.py:


=================================
Combine predictors using stacking
=================================

.. currentmodule:: sklearn

Stacking refers to a method to blend estimators. In this strategy, some
estimators are individually fitted on some training data while a final
estimator is trained using the stacked predictions of these base estimators.

In this example, we illustrate the use case in which different regressors are
stacked together and a final linear penalized regressor is used to output the
prediction. We compare the performance of each individual regressor with the
stacking strategy. Stacking slightly improves the overall performance.

.. GENERATED FROM PYTHON SOURCE LINES 18-25

.. code-block:: default

    print(__doc__)

    # Authors: Guillaume Lemaitre <g.lemaitre58@gmail.com>
    #          Maria Telenczuk    <https://github.com/maikia>
    # License: BSD 3 clause









.. GENERATED FROM PYTHON SOURCE LINES 26-42

Download the dataset
##############################################################################

 We will use `Ames Housing`_ dataset which was first compiled by Dean De Cock
 and became better known after it was used in Kaggle challenge. It is a set
 of 1460 residential homes in Ames, Iowa, each described by 80 features. We
 will use it to predict the final logarithmic price of the houses. In this
 example we will use only 20 most interesting features chosen using
 GradientBoostingRegressor() and limit number of entries (here we won't go
 into the details on how to select the most interesting features).

 The Ames housing dataset is not shipped with scikit-learn and therefore we
 will fetch it from `OpenML`_.

 .. _`Ames Housing`: http://jse.amstat.org/v19n3/decock.pdf
 .. _`OpenML`: https://www.openml.org/d/42165

.. GENERATED FROM PYTHON SOURCE LINES 42-71

.. code-block:: default


    import numpy as np

    from sklearn.datasets import fetch_openml
    from sklearn.utils import shuffle


    def load_ames_housing():
        df = fetch_openml(name="house_prices", as_frame=True)
        X = df.data
        y = df.target

        features = ['YrSold', 'HeatingQC', 'Street', 'YearRemodAdd', 'Heating',
                    'MasVnrType', 'BsmtUnfSF', 'Foundation', 'MasVnrArea',
                    'MSSubClass', 'ExterQual', 'Condition2', 'GarageCars',
                    'GarageType', 'OverallQual', 'TotalBsmtSF', 'BsmtFinSF1',
                    'HouseStyle', 'MiscFeature', 'MoSold']

        X = X[features]
        X, y = shuffle(X, y, random_state=0)

        X = X[:600]
        y = y[:600]
        return X, np.log(y)


    X, y = load_ames_housing()




.. rst-class:: sphx-glr-script-out

.. code-block:: pytb

    Traceback (most recent call last):
      File "/build/scikit-learn-ZSX7SD/scikit-learn-0.23.2/examples/ensemble/plot_stack_predictors.py", line 68, in <module>
        X, y = load_ames_housing()
      File "/build/scikit-learn-ZSX7SD/scikit-learn-0.23.2/examples/ensemble/plot_stack_predictors.py", line 50, in load_ames_housing
        df = fetch_openml(name="house_prices", as_frame=True)
      File "/build/scikit-learn-ZSX7SD/scikit-learn-0.23.2/.pybuild/cpython3_3.10/build/sklearn/utils/validation.py", line 72, in inner_f
        return f(**kwargs)
      File "/build/scikit-learn-ZSX7SD/scikit-learn-0.23.2/.pybuild/cpython3_3.10/build/sklearn/datasets/_openml.py", line 738, in fetch_openml
        data_info = _get_data_info_by_name(name, version, data_home)
      File "/build/scikit-learn-ZSX7SD/scikit-learn-0.23.2/.pybuild/cpython3_3.10/build/sklearn/datasets/_openml.py", line 369, in _get_data_info_by_name
        json_data = _get_json_content_from_openml_api(url, error_msg, True,
      File "/build/scikit-learn-ZSX7SD/scikit-learn-0.23.2/.pybuild/cpython3_3.10/build/sklearn/datasets/_openml.py", line 161, in _get_json_content_from_openml_api
        return _load_json()
      File "/build/scikit-learn-ZSX7SD/scikit-learn-0.23.2/.pybuild/cpython3_3.10/build/sklearn/datasets/_openml.py", line 61, in wrapper
        return f(*args, **kw)
      File "/build/scikit-learn-ZSX7SD/scikit-learn-0.23.2/.pybuild/cpython3_3.10/build/sklearn/datasets/_openml.py", line 157, in _load_json
        with closing(_open_openml_url(url, data_home)) as response:
      File "/build/scikit-learn-ZSX7SD/scikit-learn-0.23.2/.pybuild/cpython3_3.10/build/sklearn/datasets/_openml.py", line 106, in _open_openml_url
        with closing(urlopen(req)) as fsrc:
      File "/usr/lib/python3.10/urllib/request.py", line 216, in urlopen
        return opener.open(url, data, timeout)
      File "/usr/lib/python3.10/urllib/request.py", line 519, in open
        response = self._open(req, data)
      File "/usr/lib/python3.10/urllib/request.py", line 536, in _open
        result = self._call_chain(self.handle_open, protocol, protocol +
      File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
        result = func(*args)
      File "/usr/lib/python3.10/urllib/request.py", line 1391, in https_open
        return self.do_open(http.client.HTTPSConnection, req,
      File "/usr/lib/python3.10/urllib/request.py", line 1351, in do_open
        raise URLError(err)
    urllib.error.URLError: <urlopen error [Errno -2] Name or service not known>




.. GENERATED FROM PYTHON SOURCE LINES 72-87

Make pipeline to preprocess the data
##############################################################################

 Before we can use Ames dataset we still need to do some preprocessing.
 First, the dataset has many missing values. To impute them, we will exchange
 categorical missing values with the new category 'missing' while the
 numerical missing values with the 'mean' of the column. We will also encode
 the categories with either :class:`sklearn.preprocessing.OneHotEncoder
 <sklearn.preprocessing.OneHotEncoder>` or
 :class:`sklearn.preprocessing.OrdinalEncoder
 <sklearn.preprocessing.OrdinalEncoder>` depending for which type of model we
 will use them (linear or non-linear model). To falicitate this preprocessing
 we will make two pipelines.
 You can skip this section if your data is ready to use and does
 not need preprocessing

.. GENERATED FROM PYTHON SOURCE LINES 87-139

.. code-block:: default



    from sklearn.compose import make_column_transformer
    from sklearn.impute import SimpleImputer
    from sklearn.pipeline import make_pipeline
    from sklearn.preprocessing import OneHotEncoder
    from sklearn.preprocessing import OrdinalEncoder
    from sklearn.preprocessing import StandardScaler


    cat_cols = X.columns[X.dtypes == 'O']
    num_cols = X.columns[X.dtypes == 'float64']

    categories = [
        X[column].unique() for column in X[cat_cols]]

    for cat in categories:
        cat[cat == None] = 'missing'  # noqa

    cat_proc_nlin = make_pipeline(
        SimpleImputer(missing_values=None, strategy='constant',
                      fill_value='missing'),
        OrdinalEncoder(categories=categories)
        )

    num_proc_nlin = make_pipeline(SimpleImputer(strategy='mean'))

    cat_proc_lin = make_pipeline(
        SimpleImputer(missing_values=None,
                      strategy='constant',
                      fill_value='missing'),
        OneHotEncoder(categories=categories)
    )

    num_proc_lin = make_pipeline(
        SimpleImputer(strategy='mean'),
        StandardScaler()
    )

    # transformation to use for non-linear estimators
    processor_nlin = make_column_transformer(
        (cat_proc_nlin, cat_cols),
        (num_proc_nlin, num_cols),
        remainder='passthrough')

    # transformation to use for linear estimators
    processor_lin = make_column_transformer(
        (cat_proc_lin, cat_cols),
        (num_proc_lin, num_cols),
        remainder='passthrough')



.. GENERATED FROM PYTHON SOURCE LINES 140-156

Stack of predictors on a single data set
##############################################################################

 It is sometimes tedious to find the model which will best perform on a given
 dataset. Stacking provide an alternative by combining the outputs of several
 learners, without the need to choose a model specifically. The performance of
 stacking is usually close to the best model and sometimes it can outperform
 the prediction performance of each individual model.

 Here, we combine 3 learners (linear and non-linear) and use a ridge regressor
 to combine their outputs together.

 Note: although we will make new pipelines with the processors which we wrote
 in the previous section for the 3 learners, the final estimator RidgeCV()
 does not need preprocessing of the data as it will be fed with the already
 preprocessed output from the 3 learners.

.. GENERATED FROM PYTHON SOURCE LINES 156-184

.. code-block:: default



    from sklearn.experimental import enable_hist_gradient_boosting  # noqa
    from sklearn.ensemble import HistGradientBoostingRegressor
    from sklearn.ensemble import RandomForestRegressor
    from sklearn.ensemble import StackingRegressor
    from sklearn.linear_model import LassoCV
    from sklearn.linear_model import RidgeCV


    lasso_pipeline = make_pipeline(processor_lin,
                                   LassoCV())

    rf_pipeline = make_pipeline(processor_nlin,
                                RandomForestRegressor(random_state=42))

    gradient_pipeline = make_pipeline(
        processor_nlin,
        HistGradientBoostingRegressor(random_state=0))

    estimators = [('Random Forest', rf_pipeline),
                  ('Lasso', lasso_pipeline),
                  ('Gradient Boosting', gradient_pipeline)]

    stacking_regressor = StackingRegressor(estimators=estimators,
                                           final_estimator=RidgeCV())



.. GENERATED FROM PYTHON SOURCE LINES 185-194

Measure and plot the results
##############################################################################

 Now we can use Ames Housing dataset to make the predictions. We check the
 performance of each individual predictor as well as of the stack of the
 regressors.

 The function ``plot_regression_results`` is used to plot the predicted and
 true targets.

.. GENERATED FROM PYTHON SOURCE LINES 194-253

.. code-block:: default



    import time
    import matplotlib.pyplot as plt
    from sklearn.model_selection import cross_validate, cross_val_predict


    def plot_regression_results(ax, y_true, y_pred, title, scores, elapsed_time):
        """Scatter plot of the predicted vs true targets."""
        ax.plot([y_true.min(), y_true.max()],
                [y_true.min(), y_true.max()],
                '--r', linewidth=2)
        ax.scatter(y_true, y_pred, alpha=0.2)

        ax.spines['top'].set_visible(False)
        ax.spines['right'].set_visible(False)
        ax.get_xaxis().tick_bottom()
        ax.get_yaxis().tick_left()
        ax.spines['left'].set_position(('outward', 10))
        ax.spines['bottom'].set_position(('outward', 10))
        ax.set_xlim([y_true.min(), y_true.max()])
        ax.set_ylim([y_true.min(), y_true.max()])
        ax.set_xlabel('Measured')
        ax.set_ylabel('Predicted')
        extra = plt.Rectangle((0, 0), 0, 0, fc="w", fill=False,
                              edgecolor='none', linewidth=0)
        ax.legend([extra], [scores], loc='upper left')
        title = title + '\n Evaluation in {:.2f} seconds'.format(elapsed_time)
        ax.set_title(title)


    fig, axs = plt.subplots(2, 2, figsize=(9, 7))
    axs = np.ravel(axs)

    for ax, (name, est) in zip(axs, estimators + [('Stacking Regressor',
                                                   stacking_regressor)]):
        start_time = time.time()
        score = cross_validate(est, X, y,
                               scoring=['r2', 'neg_mean_absolute_error'],
                               n_jobs=-1, verbose=0)
        elapsed_time = time.time() - start_time

        y_pred = cross_val_predict(est, X, y, n_jobs=-1, verbose=0)

        plot_regression_results(
            ax, y, y_pred,
            name,
            (r'$R^2={:.2f} \pm {:.2f}$' + '\n' + r'$MAE={:.2f} \pm {:.2f}$')
            .format(np.mean(score['test_r2']),
                    np.std(score['test_r2']),
                    -np.mean(score['test_neg_mean_absolute_error']),
                    np.std(score['test_neg_mean_absolute_error'])),
            elapsed_time)

    plt.suptitle('Single predictors versus stacked predictors')
    plt.tight_layout()
    plt.subplots_adjust(top=0.9)
    plt.show()


.. GENERATED FROM PYTHON SOURCE LINES 254-257

The stacked regressor will combine the strengths of the different regressors.
However, we also see that training the stacked regressor is much more
computationally expensive.


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** ( 0 minutes  0.030 seconds)


.. _sphx_glr_download_auto_examples_ensemble_plot_stack_predictors.py:


.. only :: html

 .. container:: sphx-glr-footer
    :class: sphx-glr-footer-example



  .. container:: sphx-glr-download sphx-glr-download-python

     :download:`Download Python source code: plot_stack_predictors.py <plot_stack_predictors.py>`



  .. container:: sphx-glr-download sphx-glr-download-jupyter

     :download:`Download Jupyter notebook: plot_stack_predictors.ipynb <plot_stack_predictors.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_
