Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Task]: Support embeddings using ML models in MLTransform #29356

Closed
1 of 16 tasks
AnandInguva opened this issue Nov 8, 2023 · 1 comment · Fixed by #29564 or #29938
Closed
1 of 16 tasks

[Task]: Support embeddings using ML models in MLTransform #29356

AnandInguva opened this issue Nov 8, 2023 · 1 comment · Fixed by #29564 or #29938
Assignees
@AnandInguva
Labels
done & done Issue has been reviewed after it was closed for verification, followups, etc. P2 python task
Milestone
2.54.0 Release

Comments

@AnandInguva
Copy link
Contributor

AnandInguva commented Nov 8, 2023 •
edited
Loading

What needs to happen?

doc - https://docs.google.com/document/d/1En4bfbTu4rvu7LWJIKV3G33jO-xJfTdbaSFSURmQw_s/edit#heading=h.wskna8eurvjv discusses on supporting generation of embeddings in MLTransform.

This would be an umbrella issue to track different features/tasks required for embeddings in MLTransform.

Issue Priority

Priority: 2 (default / most normal work should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
@AnandInguva AnandInguva self-assigned this Nov 8, 2023
@AnandInguva AnandInguva mentioned this issue Nov 21, 2023
Closed
3 tasks
@AnandInguva AnandInguva changed the title [Task]: Support generating text embeddings using ML models in MLTransform [Task]: Support embeddings using ML models in MLTransform Dec 8, 2023
@AnandInguva
Copy link
Contributor Author

AnandInguva commented Dec 8, 2023 •
edited
Loading

Tasks needs to be done after PR #29564 is merged.

  • Add set_model_handler method
  • Support Dead letter queue for RunInference and asses its compatibility with tft transforms.
  • Add inference_fn to the embedding config
  • Add support for artifact locations other than local system and GCS.
  • Integrate ArtifactFetcher with MLTransform.

@AnandInguva AnandInguva linked a pull request Jan 5, 2024 that will close this issue
Support Embeddings in mltransform #29564
Merged
3 tasks
@AnandInguva AnandInguva mentioned this issue Jan 5, 2024
Merged
3 tasks
@AnandInguva AnandInguva linked a pull request Jan 5, 2024 that will close this issue
Support DLQ in MLTransform #29938
Merged
3 tasks
@github-actions github-actions bot added this to the 2.54.0 Release milestone Jan 12, 2024
@damccorm damccorm added the done & done Issue has been reviewed after it was closed for verification, followups, etc. label Jan 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@AnandInguva AnandInguva

Labels
done & done Issue has been reviewed after it was closed for verification, followups, etc. P2 python task
Projects
None yet
Milestone
2.54.0 Release
2 participants
@AnandInguva @damccorm