Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Object store persistence in operator #4758

Merged
merged 7 commits into from
Nov 15, 2024
Merged

feat: Object store persistence in operator #4758

merged 7 commits into from
Nov 15, 2024

Conversation

dmartinol
Copy link
Contributor

What this PR does / why we need it:

This PR adds object store persistence configuration options to the FeatureStore CRD.

  • Support is provided only to the Registry service.
  • Additional S3 configurations are also supported
  • Follows definitions in RFC #42
  • Configuration examples are provided in config/samples
apiVersion: feast.dev/v1alpha1
kind: FeatureStore
metadata:
  name: sample-s3-registry
spec:
  feastProject: my_project
  services:
    onlineStore:
      persistence:
        file:
          path: /data/online_store.db
    offlineStore:
      persistence:
        file:
          type: dask
    registry:
      local:
        persistence:
          file:
            path: s3://bucket/registry.db
            s3_additional_kwargs:
              ServerSideEncryption: AES256
              ACL: bucket-owner-full-control
              CacheControl: max-age=3600

Which issue(s) this PR fixes:

Relates to #4561

@dmartinol
object store validation and additional field
dabe757 
Signed-off-by: Daniele Martinoli <dmartino@redhat.com>
@dmartinol
s3 additional kwargs
7bbfb72 
Signed-off-by: Daniele Martinoli <dmartino@redhat.com>
@dmartinol
adding extra line
96706c7 
Signed-off-by: Daniele Martinoli <dmartino@redhat.com>
@dmartinol dmartinol requested a review from a team as a code owner November 13, 2024 10:14
@dmartinol
Copy link
Contributor Author

@tchughesiv @redhatHameed FYI

redhatHameed
redhatHameed approved these changes Nov 14, 2024
View reviewed changes
Copy link
Contributor

@redhatHameed redhatHameed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

lokeshrangineni
lokeshrangineni reviewed Nov 14, 2024
View reviewed changes
infra/feast-operator/api/v1alpha1/zz_generated.deepcopy.go
@@ -452,6 +452,17 @@ func (in *RegistryFilePersistence) DeepCopyInto(out *RegistryFilePersistence) {
*out = new(PvcConfig)
(*in).DeepCopyInto(*out)
}
if in.S3AddtlKwargs != nil {
in, out := &in.S3AddtlKwargs, &out.S3AddtlKwargs
*out = new(map[string]string)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this has been reassigned in the deep copy using make. Do we need to initialize with empty map?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this file is generated by make generate, I assume it is corrrect 😉

tchughesiv
tchughesiv approved these changes Nov 14, 2024
View reviewed changes
Copy link
Contributor

@tchughesiv tchughesiv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@dmartinol
updated registry validation message
c1c86cf 
Signed-off-by: Daniele Martinoli <dmartino@redhat.com>
@dmartinol
renamed S3AddtlKwargs to S3AdditionalKwargs
b583ddf 
Signed-off-by: Daniele Martinoli <dmartino@redhat.com>
@dmartinol
generalized similar test functions
90bb904 
Signed-off-by: Daniele Martinoli <dmartino@redhat.com>
@lokeshrangineni lokeshrangineni enabled auto-merge (squash) November 15, 2024 18:07
@lokeshrangineni lokeshrangineni merged commit 0ae86da into feast-dev:master Nov 15, 2024
25 checks passed
shuchu pushed a commit to shuchu/feast that referenced this pull request Nov 21, 2024
@dmartinol @shuchu
feat: Object store persistence in operator (feast-dev#4758)
b53c379 
* object store validation and additional field

Signed-off-by: Daniele Martinoli <dmartino@redhat.com>

* s3 additional kwargs

Signed-off-by: Daniele Martinoli <dmartino@redhat.com>

* adding extra line

Signed-off-by: Daniele Martinoli <dmartino@redhat.com>

* updated registry validation message

Signed-off-by: Daniele Martinoli <dmartino@redhat.com>

* renamed S3AddtlKwargs to S3AdditionalKwargs

Signed-off-by: Daniele Martinoli <dmartino@redhat.com>

* generalized similar test functions

Signed-off-by: Daniele Martinoli <dmartino@redhat.com>

---------

Signed-off-by: Daniele Martinoli <dmartino@redhat.com>
franciscojavierarceo pushed a commit that referenced this pull request Dec 5, 2024
@feast-ci-bot
chore(release): release 0.42.0
aac334a 
# [0.42.0](v0.41.0...v0.42.0) (2024-12-05)

### Bug Fixes

* Add adapters for sqlite datetime conversion ([#4797](#4797)) ([e198b17](e198b17))
* Added grpcio extras to default feature-server image ([#4737](#4737)) ([e9cd373](e9cd373))
* Changing node version in release ([7089918](7089918))
* Feast create empty online table when FeatureView attribute online=False ([#4666](#4666)) ([237c453](237c453))
* Fix db store types in Operator CRD ([#4798](#4798)) ([f09339e](f09339e))
* Fix the config issue for postgres ([#4776](#4776)) ([a36f7e5](a36f7e5))
* Fixed example materialize-incremental and improved explanation ([#4734](#4734)) ([ca8a7ab](ca8a7ab))
* Fixed SparkSource docstrings so it wouldn't used inhereted class docstrings ([#4722](#4722)) ([32e6aa1](32e6aa1))
* Fixing PGVector integration tests ([#4778](#4778)) ([88a0320](88a0320))
* Incorrect type passed to assert_permissions in materialize endpoints ([#4727](#4727)) ([b72c2da](b72c2da))
* Issue of DataSource subclasses using parent abstract class docstrings ([#4730](#4730)) ([b24acd5](b24acd5))
* Operator envVar positioning & tls.SecretRef.Name ([#4806](#4806)) ([1115d96](1115d96))
* Populates project created_time correctly according to created ti… ([#4686](#4686)) ([a61b93c](a61b93c))
* Reduce feast-server container image size & fix dev image build ([#4781](#4781)) ([ccc9aea](ccc9aea))
* Removed version func from feature_store.py ([#4748](#4748)) ([f902bb9](f902bb9))
* Support registry instantiation for read-only users ([#4719](#4719)) ([ca3d3c8](ca3d3c8))
* Syntax Error in BigQuery While Retrieving Columns that Start wit… ([#4713](#4713)) ([60fbc62](60fbc62))
* Update release version in a pertinent Operator file ([#4708](#4708)) ([764a8a6](764a8a6))

### Features

* Add api contract to fastapi docs ([#4721](#4721)) ([1a165c7](1a165c7))
* Add Couchbase as an online store ([#4637](#4637)) ([824859b](824859b))
* Add Operator support for spec.feastProject & status.applied fields ([#4656](#4656)) ([430ac53](430ac53))
* Add services functionality to Operator ([#4723](#4723)) ([d1d80c0](d1d80c0))
* Add TLS support to the Operator ([#4796](#4796)) ([a617a6c](a617a6c))
* Added feast Go operator db stores support ([#4771](#4771)) ([3302363](3302363))
* Added support for setting env vars in feast services in feast controller  ([#4739](#4739)) ([84b24b5](84b24b5))
* Adding docs outlining native Python transformations on singletons ([#4741](#4741)) ([0150278](0150278))
* Adding first feast operator e2e test. ([#4791](#4791)) ([8339f8d](8339f8d))
* Adding github action to run the operator end-to-end tests. ([#4762](#4762)) ([d8ccb00](d8ccb00))
* Adding ssl support for registry server. ([#4718](#4718)) ([ccf7a55](ccf7a55))
* Adding SSL support for the React UI server and feast UI command. ([#4736](#4736)) ([4a89252](4a89252))
* Adding support for native Python transformations on a single dictionary  ([#4724](#4724)) ([9bbc1c6](9bbc1c6))
* Adding TLS support for offline server. ([#4744](#4744)) ([5d8d03f](5d8d03f))
* Building the feast image ([#4775](#4775)) ([6635dde](6635dde))
* File persistence definition and implementation ([#4742](#4742)) ([3bad4a1](3bad4a1))
* Object store persistence in operator ([#4758](#4758)) ([0ae86da](0ae86da))
* OIDC authorization in Feast Operator ([#4801](#4801)) ([eb111d6](eb111d6))
* Operator will create k8s serviceaccount for each feast service ([#4767](#4767)) ([cde5760](cde5760))
* Printing more verbose logs when we start the offline server  ([#4660](#4660)) ([9d8d3d8](9d8d3d8))
* PVC configuration and impl ([#4750](#4750)) ([785a190](785a190))
* Qdrant vectorstore support ([#4689](#4689)) ([86573d2](86573d2))
* RBAC Authorization in Feast Operator ([#4786](#4786)) ([0ef5acc](0ef5acc))
* Support for nested timestamp fields in Spark Offline store ([#4740](#4740)) ([d4d94f8](d4d94f8))
* Update the go feature server from Expedia code repo. ([#4665](#4665)) ([6406625](6406625))
* Updated feast Go operator db stores ([#4809](#4809)) ([2c5a6b5](2c5a6b5))
* Updated sample secret following review ([#4811](#4811)) ([dc9f825](dc9f825))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@redhatHameed redhatHameed redhatHameed approved these changes

@lokeshrangineni lokeshrangineni lokeshrangineni approved these changes

@tchughesiv tchughesiv tchughesiv approved these changes

Assignees
No one assigned
Labels
ok-to-test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@dmartinol @tchughesiv @lokeshrangineni @redhatHameed