Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delete keys from Redis when tearing down online store #1965

Merged
merged 7 commits into from
Oct 24, 2021

Conversation

achals
Copy link
Member

@achals achals commented Oct 22, 2021

Signed-off-by: Achal Shah achals@gmail.com

What this PR does / why we need it:
When you run feast teardown on a feast project that uses redis as the online store, it leaves behind all the keys that're written into redis.

This is bad and is an obvious leak of memory/resources. In contrast, in dynamo and datastore, the tables and values are cleaned up.

Does this PR introduce a user-facing change?:

none

Signed-off-by: Achal Shah <achals@gmail.com>
@achals achals force-pushed the achal/delete-redis-keys branch from b0da50a to 57adc31 Compare October 22, 2021 20:33
@codecov-commenter
Copy link

codecov-commenter commented Oct 22, 2021

Codecov Report

Merging #1965 (e67e582) into master (b0635c3) will increase coverage by 0.04%.
The diff coverage is 96.42%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1965      +/-   ##
==========================================
+ Coverage   81.98%   82.03%   +0.04%     
==========================================
  Files         100      100              
  Lines        7926     7948      +22     
==========================================
+ Hits         6498     6520      +22     
  Misses       1428     1428              
Flag Coverage Δ
integrationtests 74.38% <96.42%> (+0.07%) ⬆️
unittests 59.43% <28.57%> (-0.11%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
sdk/python/feast/infra/online_stores/redis.py 92.85% <94.11%> (ø)
sdk/python/feast/feature_table.py 44.89% <100.00%> (ø)
sdk/python/feast/infra/key_encoding_utils.py 91.17% <100.00%> (+2.28%) ⬆️
sdk/python/feast/infra/online_stores/helpers.py 87.09% <100.00%> (+0.88%) ⬆️
...ts/integration/feature_repos/repo_configuration.py 93.58% <0.00%> (+1.18%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b0635c3...e67e582. Read the comment docs.

Signed-off-by: Achal Shah <achals@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>
@achals achals marked this pull request as ready for review October 22, 2021 22:34
sdk/python/feast/infra/key_encoding_utils.py Show resolved Hide resolved
client = self._get_client(config.online_store)
deleted_count = 0
for table in tables_to_delete:
pipline = client.pipeline()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo: pipeline

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

client = self._get_client(config.online_store)
deleted_count = 0
for table in tables:
pipline = client.pipeline()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo: pipeline

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

sdk/python/feast/infra/online_stores/redis.py Show resolved Hide resolved
Signed-off-by: Achal Shah <achals@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>
Copy link
Collaborator

@felixwang9817 felixwang9817 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@feast-ci-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: achals, felixwang9817

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [achals,felixwang9817]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@felixwang9817
Copy link
Collaborator

/kind bug

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants