Adding ignore_index to pandas_to_eland #154
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Parameter is useful when adding multiple pd.DataFrame's to the same index. Also, updated test module to pandas.testing for 1.0.x compliance,
sethmlarson
suggested changes
Mar 31, 2020
eland/utils.py
Outdated
| if es_dropna: | ||
| values = row[1].dropna().to_dict() | ||
| else: | ||
| values = row[1].to_dict() | ||
|
|
||
| # Use integer as id field for repeatable results | ||
| action = {"_index": es_dest_index, "_source": values, "_id": str(id)} | ||
| if ignore_index: |
There was a problem hiding this comment.
Maybe we should change the name of this parameter to be more verbose but also clear on what "index" is being ignored since the concept appears in both pandas and ES.
Maybe ignore_pandas_index or even flip the default and use use_pandas_index_as_id=True?
eland/utils.py
Outdated
| number of pandas.DataFrame rows to read before bulk index into Elasticsearch | ||
| Number of pandas.DataFrame rows to read before bulk index into Elasticsearch | ||
| ignore_index: bool, default 'False' | ||
| Ignore pandas.DataFrame.index when indexing into Elasticsearch? |
There was a problem hiding this comment.
Drop the ?, document the behavior for the true and false case
| @@ -65,6 +65,40 @@ def test_generate_es_mappings(self): | |||
|
|
|||
| assert_pandas_eland_frame_equal(df, ed_df_head) | |||
|
|
|||
| ES_TEST_CLIENT.indices.delete(index=index_name) | |||
|
|
|||
| def test_pandas_to_eland_ignore_index(self): | |||
There was a problem hiding this comment.
This unit test passes without the ignore_index=True, should add a that fails if not using the functionality.
Maybe adding this assert is enough:
# Ensure that index is populated by ES.
assert not (df.index == pd_df.index).any()
sethmlarson
reviewed
Mar 31, 2020
| pd_df = ed.eland_to_pandas(ed_df) | ||
|
|
||
| # Compare values excluding index | ||
| assert df.values.all() == pd_df.values.all() |
There was a problem hiding this comment.
Shouldn't this be (df.values == pd_df.values).all()?
|
Many thanks! Great comments - resolved in new commit. |
sethmlarson
approved these changes
Mar 31, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Parameter is useful when adding multiple pd.DataFrame's to
the same index.
Also, updated test module to pandas.testing for 1.0.x
compliance,