Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement JobService API calls & connect it to SDK #1129

Merged
merged 13 commits into from
Nov 4, 2020

Conversation

tsotnet
Copy link
Collaborator

@tsotnet tsotnet commented Nov 4, 2020

What this PR does / why we need it:

  • Implement start_offline_to_online_ingestion, start_historical_feature_retrieval_job, start_stream_to_online_ingestion SDK calls with JobService (including branching logic depending on whether job service is enabled or not)
  • Set correct environment for JobService (Helm Charts & Dockerfile)
  • Update JobService protos as necessary (according to Slack discussion)
  • Add e2e tests for JobService

Does this PR introduce a user-facing change?:

Users can now use Feast Job Service as a proxy from Python SDK to get historical features, start batch ingestion, or start streaming ingestion jobs. When running the corresponding SDK methods from the Python client, the actual logic will run on Job Service. Also, listing jobs and getting a job by job_id will go through Job Service.

Job Service can be enabled by setting FEAST_JOB_SERVICE_URL environment variable.

Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
…ne mode

* New API calls (start_offline_to_online_ingestion, start_stream_to_online_ingestion) now return Remote Jobs instead of job ids
* Implement list_jobs & get_job for standalone mode (looks like Spark is running in local mode and we can't get job statuses so we have to keep cache in memory)
* Wire up list_jobs & get_job on client side with job service
* Tested locally on Feast 101 notebook, everything works

Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
…mode

Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
@feast-ci-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tsotnet
To complete the pull request process, please assign zhilingc
You can assign the PR to them by writing /assign @zhilingc in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
@woop
Copy link
Member

woop commented Nov 4, 2020

/kind feature

@feast-ci-bot feast-ci-bot added kind/feature New feature or request and removed needs-kind labels Nov 4, 2020
@tsotnet
Copy link
Collaborator Author

tsotnet commented Nov 4, 2020

/test "complete / tests-docker-compose (pull_request)"

@woop woop merged commit 4068b3e into feast-dev:master Nov 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature New feature or request size/L
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants