Add column-wise transforms & refactor TableVectorizer #902
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jeromedockes
force-pushed
the
columnwise-transformations
branch
from
f8636a4 to
9f16bff
Compare
jeromedockes
changed the title
This reverts commit 87e5a3c.
TheooJ
approved these changes
May 27, 2024
Co-authored-by: Théo Jolivet <57430673+TheooJ@users.noreply.github.com>
|
I'll make a pass now. |
glemaitre
reviewed
May 28, 2024
| # | ||
| # Let's first retrieve the dataset: | ||
| # Let's first retrieve the dataset, using one of the downloaders from the :mod:`skrub.datasets` module. |
There was a problem hiding this comment.
Do you to make the example black complient now (less than 88 characters) or make an automatic pass of the tool in another PR?
Comment on lines
41
to
+42
| ############################################################################### | ||
| # A simple prediction pipeline | ||
| # ---------------------------- | ||
| # Easily encoding a dataframe |
There was a problem hiding this comment.
If we change the example, I would probably use the # %% delimiter nowadays.
Suggested change
| ############################################################################### | |
| # A simple prediction pipeline | |
| # ---------------------------- | |
| # Easily encoding a dataframe | |
| # %% | |
| # Easily encoding a dataframe |
|
|
||
| from skrub.datasets import fetch_employee_salaries | ||
|
|
||
| dataset = fetch_employee_salaries() | ||
| employees, salaries = dataset.X, dataset.y | ||
| employees | ||
|
|
||
| ############################################################################### |
There was a problem hiding this comment.
Suggested change
| ############################################################################### | |
| # %% |
|
|
||
| X = dataset.X | ||
| y = dataset.y | ||
| ############################################################################### |
There was a problem hiding this comment.
Suggested change
| ############################################################################### | |
| # %% |
Comment on lines
79
to
88
| ############################################################################### | ||
| # We observe diverse columns in the dataset: | ||
| # - binary (``'gender'``), | ||
| # - numerical (``'employee_annual_salary'``), | ||
| # - categorical (``'department'``, ``'department_name'``, ``'assignment_category'``), | ||
| # - datetime (``'date_first_hired'``) | ||
| # - dirty categorical (``'employee_position_title'``, ``'division'``). | ||
| # | ||
| # Using skrub's |TableVectorizer|, we can now already build a machine-learning | ||
| # pipeline and train it: | ||
| # From our 8 columns, the |TableVectorizer| has extracted 143 numerical | ||
| # features. Most of them are one-hot encoded representations of the categorical | ||
| # features. For example, we can see that 3 columns ``'gender_F'``, ``'gender_M'``, | ||
| # ``'gender_nan'`` were created to encode the ``'gender'`` column. | ||
|
|
||
| ############################################################################### | ||
| # By performing appropriate transformations on our complex data, the |TableVectorizer| produced numeric features that we can use for machine-learning: | ||
|
|
||
| from sklearn.ensemble import HistGradientBoostingRegressor |
There was a problem hiding this comment.
Suggested change
| ############################################################################### | |
| # We observe diverse columns in the dataset: | |
| # - binary (``'gender'``), | |
| # - numerical (``'employee_annual_salary'``), | |
| # - categorical (``'department'``, ``'department_name'``, ``'assignment_category'``), | |
| # - datetime (``'date_first_hired'``) | |
| # - dirty categorical (``'employee_position_title'``, ``'division'``). | |
| # | |
| # Using skrub's |TableVectorizer|, we can now already build a machine-learning | |
| # pipeline and train it: | |
| # From our 8 columns, the |TableVectorizer| has extracted 143 numerical | |
| # features. Most of them are one-hot encoded representations of the categorical | |
| # features. For example, we can see that 3 columns ``'gender_F'``, ``'gender_M'``, | |
| # ``'gender_nan'`` were created to encode the ``'gender'`` column. | |
| ############################################################################### | |
| # By performing appropriate transformations on our complex data, the |TableVectorizer| produced numeric features that we can use for machine-learning: | |
| from sklearn.ensemble import HistGradientBoostingRegressor | |
| # %% | |
| # From our 8 columns, the |TableVectorizer| has extracted 143 numerical | |
| # features. Most of them are one-hot encoded representations of the categorical | |
| # features. For example, we can see that 3 columns ``'gender_F'``, ``'gender_M'``, | |
| # ``'gender_nan'`` were created to encode the ``'gender'`` column. | |
| # | |
| # By performing appropriate transformations on our complex data, the |TableVectorizer| produced numeric features that we can use for machine-learning: | |
| from sklearn.ensemble import HistGradientBoostingRegressor |
Comment on lines
231
to
235
| ############################################################################### | ||
| # The simple pipeline applied on this complex dataset gave us very good results. | ||
| # We can see that this new pipeline achieves a similar score but is fitted much faster. | ||
| # This is mostly due to replacing |GapEncoder| with |MinHashEncoder| (however this makes the features less interpretable). | ||
|
|
||
| ############################################################################### |
There was a problem hiding this comment.
Suggested change
| ############################################################################### | |
| # The simple pipeline applied on this complex dataset gave us very good results. | |
| # We can see that this new pipeline achieves a similar score but is fitted much faster. | |
| # This is mostly due to replacing |GapEncoder| with |MinHashEncoder| (however this makes the features less interpretable). | |
| ############################################################################### | |
| # %% | |
| # We can see that this new pipeline achieves a similar score but is fitted much faster. | |
| # This is mostly due to replacing |GapEncoder| with |MinHashEncoder| (however this makes the features less interpretable). | |
| # |
| pipeline = make_pipeline(TableVectorizer(), regressor) | ||
| pipeline.fit(X, y) | ||
| pipeline = make_pipeline(vectorizer, regressor) | ||
| pipeline.fit(employees, salaries) | ||
|
|
||
| ############################################################################### |
There was a problem hiding this comment.
Suggested change
| ############################################################################### | |
| # %% |
| remainder="drop", | ||
| ) | ||
|
|
||
| X_enc = encoder.fit_transform(X) | ||
| pprint(encoder.get_feature_names_out()) | ||
| # pprint(encoder.get_feature_names_out()) |
| @@ -85,6 +85,7 @@ def cols(*columns): | |||
| >>> s.all() & ['kind', 'ID'] | |||
| (all() & cols('kind', 'ID')) | |||
|
|
|||
| # noqa | |||
| Here we can see the input to ``transform`` has been converted back to the | ||
| timezone used during ``fit`` and that we get the same result for "hour". | ||
|
|
||
| # noqa |
There was a problem hiding this comment.
OK so this is to avoid the check on the docstring. I assume that we can clean it afterwords
glemaitre
approved these changes
May 28, 2024
|
Oops so sorry @glemaitre I should have said so but I think @GaelVaroquaux was planning to review it as well ... @GaelVaroquaux , if you would like LMK if you want to review maybe the easiest way will be to revert the merge commit and open a new PR to un-revert |
|
Thanks a lot for the review @glemaitre ! |
|
No, no, it's good to have merged. I can give feedback via issues. Hurray for merge. Thanks a lot to everyone involved!! |
|
Thanks @GaelVaroquaux. We will address the subsequent issues. Let's roll ;) |
This was referenced May 28, 2024
Closed
9 tasks
ok, thanks. there will be a few follow-up PRs in any case, @TheooJ and I are going to open a couple of issues |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
closes #874, #886, #894, #877, #848, #904, #905, #830, #626, #870
This is the last part of the changes outlined in #877 (the first two parts have been merged in #895 and #888)
The main addition is
OnEachColumn, a transformer that applies a transformation independently to each column in a dataframe, and is used to refactor the TableVectorizer and ensure it does consistent operations across calls totransform.