feat: Remote offline Store

[redhatHameed]

What this PR does / why we need it:

This PR added feature for remote offline store contain below changes.

Added an Offline Server using Arrow Flight:

• Implement the remote OfflineServer similar to the one developed for the Remote Registry or Feature Server but using Arrow Flight Server instead of pure gRPC.
• Added the command into cli for starting the offline server using the feast serve_offline command.
• Implemented the the OfflineStore interface methods as arrow flight server methods.
• Added unit test and integration test functions for the OfflineServer class.

Added an Offline Remote Client:

• Implement the offline remote client using arrow flight remote client approach, which supports the type: remote configuration in the offline store configuration for example as seen below.

    type: remote 
    host: mystore.feast.svc.cluster.local
     port: 8815

• Develop the RemoteOfflineStore class, which implements the existing OfflineStore interface in Feast. Utilize the get_historical_features method and pass parameters such as entity definition, feature views, offline store config to interact with the Arrow server functions using the Arrow Flight client.
• Client implemented the other methods like pull_latest_from_table_or_query, write_logged_features , offline_write_batch
• added unit test function to test the remote client.

Updated Helm Chart for Deploying as a service on Kubernetes:

• updated chart deployment to install offline server by using feast_mode property based on the deployment choice user can set the property. The online mode is the default and maintains backward compatibility with previous Feast Feature Server implementations.

Helm install examples:

helm install feast-feature-server feast-charts/feast-feature-server --set feature_store_yaml_base64=$(base64 > feature_store.yaml)
helm install feast-offline-server feast-charts/feast-feature-server --set feast_mode=offline  --set feature_store_yaml_base64=$(base64 > feature_store.yaml)
helm install feast-ui-server feast-charts/feast-feature-server --set feast_mode=ui  --set feature_store_yaml_base64=$(base64 > feature_store.yaml)
helm install feast-registry-server feast-charts/feast-feature-server --set feast_mode=registry  --set feature_store_yaml_base64=$(base64 > feature_store.yaml)

Added documentation for remote offline support

Added example for remote offline

Which issue(s) this PR fixes:

Fixes #4032

Fixes

[tokoko]

@redhatHameed just curious, are the tests failing only in gh workflows? Do they pass locally?

[dmartinol]

Just a note, if we decide in the future to use this server for remote materialization engine as well, the name might no longer be a good fit. Just pointing it out.. I don't have any better ideas.

Maybe it's too early to kickoff a thread, but in that case, shouldn't we have a separate materialization server with its own codebase? (if you have an issue we can move the discussion there)

[tokoko]

Maybe it's too early to kickoff a thread, but in that case, shouldn't we have a separate materialization server with its own codebase? (if you have an issue we can move the discussion there)

maybe, that would certainly be a cleaner approach, but imho wouldn't really serve any other practical purpose. I'll open a ticket and we probably should pick this up later once remote offline is a bit more fleshed out.

[dmartinol]

@redhatHameed just curious, are the tests failing only in gh workflows? Do they pass locally?

@tokoko Yes, apart some of them that fail locally. Do you have any idea why from the GH action execution we don't even see the log of the Test local integration tests step? Do you have any chance to re-run it and enable the debugging option?

[tokoko]

I don't have any privileges there. @franciscojavierarceo @jeremyary any ideas?

[redhatHameed]

@redhatHameed just curious, are the tests failing only in gh workflows? Do they pass locally?

Yes it does - apart from the one test case -> test_spark_materialization_consistency with error - py4j.protocol.Py4JJavaError: An error occurred while calling o114.count. but if I skip it all are passing locally see the here results

[lokeshrangineni]

it is not a mandatory to give the types, but it makes it more readable to have types unless it is expected to have any data type.

[tmihalac]

@lokeshrangineni
You mean in the methods parameters or the return value or both?
I guess you mean for all the methods and not only this specific one?

[lokeshrangineni]

I was specifically about method parameters. but if there is no change in functionality better to add types to return variable also.

[redhatHameed]

@lokeshrangineni the change implemented under commit thanks @tmihalac addressing it

[redhatHameed]

@jeremyary any thoughts I am getting incorrect credentials error one of integration test it was running fine previously ERROR sdk/python/tests/integration/offline_store/test_push_features_to_offline_store.py::test_push_features_and_read[AWS:Redshift:redis:python_fs:False] - feast.errors.RedshiftCredentialsError: Redshift API failed due to incorrect credentials ->https://github.com/feast-dev/feast/actions/runs/9504367262/job/26196850053?pr=4262

[tokoko]

@redhatHameed It's a rate limit error, says An error occurred (ValidationException) when calling the DescribeTable operation: Rate exceeded if you scroll up. Redshift is being a little dramatic in the final error. A rerun should fix it.

[redhatHameed]

@tokoko can we get your approval on this if no more comments. Also who will approve this for merging ?

[tokoko]

Yup, lgtm. great work, guys. @jeremyary or @franciscojavierarceo can approve and merge