Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DynamoDB support for online feature store #1409

Closed
blvp opened this issue Mar 24, 2021 · 2 comments · Fixed by #1483
Closed

DynamoDB support for online feature store #1409

blvp opened this issue Mar 24, 2021 · 2 comments · Fixed by #1483
Labels
kind/feature New feature or request

Comments

@blvp
Copy link

blvp commented Mar 24, 2021

Is your feature request related to a problem? Please describe.
We need to have an online serving backend with Dynamo to support customer deployments on AWS.

Describe the solution you'd like
Implementation of Provider similar to /sdk/python/feast/infra/local_sqlite.py with DynamoDB backend

@woop
Copy link
Member

woop commented Mar 24, 2021

Thanks @blvp.

As mentioned before, a complete implementation should contain

  1. A provider (aws), which is used to update infrastructure during feast apply
  2. Offline store support for this provider. Ideally Snowflake and/or Redshift
  3. Online store support for this provider. Dynamo makes sense

To be clear, we can consider (2) to be out of scope for this card, but there may be some work required in creating the aws provider. Our plan is for all our providers (local, gcp) to be split from our online stores and offline stores. So the object through which the FeatureStore object can update infrastructure based on FeatureView/Table changes, and also the object the FeatureStore object uses to retrieve an OnlineStore or OfflineStore object.

Right now we still have some work in flight. Splitting the OnlineStore and Provider being one. We also want to do some code refactoring to make sure that the FeatureStore class has a clean abstraction between itself and the Provider.

I'd recommend the following steps.

  1. Document the data model that we'd like to use for DynamoDB in this issue before starting development
  2. Create an isolated implementation of the DynamoDB I/O. With the idea that changes in the Feast code base wont force you to rewrite the connector.
  3. Once we are ready with the first release of Feast 0.10, we can integrate the connector. It remains to be seen how we will integrate the connector, because we'd want AWS users to have a complete experience (ideally with a production offline store). We may have to document this connector as experimental or contrib for the start.

@woop woop added the kind/feature New feature or request label Mar 24, 2021
@blvp
Copy link
Author

blvp commented Mar 24, 2021

Thank you @woop. Will update the ticket with the data model design.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants