Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Embeddings in mltransform #29564

Merged
merged 58 commits into from
Dec 11, 2023
Merged

Support Embeddings in mltransform #29564

Show file tree
Hide file tree
Changes from 55 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
80e5c4a
Make base.py framework agnostic and add helper transforms
AnandInguva Nov 29, 2023
0d34847
Add tests for base.py
AnandInguva Nov 29, 2023
58b24f6
Add sentence-transformers
AnandInguva Nov 29, 2023
88f9ceb
Add tensorflow hub
AnandInguva Nov 29, 2023
23f7027
Add vertex_ai
AnandInguva Nov 29, 2023
04ebdb0
Make TFTProcessHandler a PTransform
AnandInguva Nov 29, 2023
f86c259
raise RuntimeError in ArtifactsFetcher when it is used for embeddings
AnandInguva Nov 29, 2023
fc4ec00
Add JsonPickle to requirements
AnandInguva Nov 29, 2023
3da5ce8
Add tox tests
AnandInguva Nov 29, 2023
4b4ee58
Mock frameworks in pydocs
AnandInguva Nov 29, 2023
01ba217
Add Row type check
AnandInguva Dec 4, 2023
f080c25
Remove requires_chaining
AnandInguva Dec 4, 2023
6111c31
change name of PTransformProvider to MLTransformProvider
AnandInguva Dec 4, 2023
ba24e81
remove batch_len in utility fun
AnandInguva Dec 4, 2023
d690aec
Change type annotation and redundant comments
AnandInguva Dec 4, 2023
af7496b
Remove get_transforms method
AnandInguva Dec 4, 2023
d713555
remove requires_chaining from tft
AnandInguva Dec 4, 2023
50450f3
add tests to sentence-transformers
AnandInguva Dec 4, 2023
c2b691f
Merge remote-tracking branch 'origin/master' into embeddings_mltransform
AnandInguva Dec 5, 2023
8823a75
Pass inference_args to RunInference
AnandInguva Dec 5, 2023
a7e2bd3
Add TODO GH issue
AnandInguva Dec 5, 2023
519b3ed
Merge branch 'embeddings_mltransform' of https://github.com/AnandIngu…
AnandInguva Dec 5, 2023
f77ae60
refactor variables in vertex_ai embeddings
AnandInguva Dec 5, 2023
95ed3c5
remove try/catch and throw error if options is empty for GCS artifact…
AnandInguva Dec 5, 2023
c235499
Refactor NotImplementedError message
AnandInguva Dec 5, 2023
6eebfa4
remove tensorflow hub from this PR
AnandInguva Dec 5, 2023
c27aabb
Add _validate_transform method
AnandInguva Dec 5, 2023
422a86a
add more tests
AnandInguva Dec 5, 2023
08b3665
fix test
AnandInguva Dec 6, 2023
91255ad
Fix test
AnandInguva Dec 6, 2023
c7237c3
Add more tests in sentence-transformer
AnandInguva Dec 6, 2023
a942885
use np.max instead of max
AnandInguva Dec 6, 2023
89c19fb
round to 2 decimals
AnandInguva Dec 6, 2023
2db4a20
Remove gradle command action
AnandInguva Dec 6, 2023
b7a48d5
Refactor throwing dataflow client exception
AnandInguva Dec 6, 2023
eb46e08
Merge branch 'embeddings_mltransform' of https://github.com/AnandIngu…
AnandInguva Dec 6, 2023
bad1b3b
skip the test if gcp is not installed
AnandInguva Dec 6, 2023
b850cee
remove toxTests for hub
AnandInguva Dec 6, 2023
ffff21a
remove toxTests for hub
AnandInguva Dec 6, 2023
88412ea
Fix values in assert for sentence_transformer_test
AnandInguva Dec 7, 2023
617f9d6
rename sentence_transformers to huggingface
AnandInguva Dec 7, 2023
5cae04b
fix pydocs
AnandInguva Dec 7, 2023
489200f
Change the model name for tests since it is getting different results…
AnandInguva Dec 7, 2023
816174a
Fix pydoc in vertexai
AnandInguva Dec 7, 2023
cfb1883
add suffix to artifact_location
AnandInguva Dec 8, 2023
2cb6f03
Revert "add suffix to artifact_location"
AnandInguva Dec 8, 2023
cd7050e
add no_xdist
AnandInguva Dec 8, 2023
98cd949
Try fixing pydoc for vertexai
AnandInguva Dec 8, 2023
8ea0906
change tox.ini to use pytest directly
AnandInguva Dec 8, 2023
5187b0e
Merge remote-tracking branch 'origin/master' into embeddings_mltransform
AnandInguva Dec 8, 2023
6f83d3c
raise FileExistError if Attribute file is already present
AnandInguva Dec 8, 2023
c9ddb25
Merge branch 'embeddings_mltransform' of https://github.com/AnandIngu…
AnandInguva Dec 8, 2023
9dce3cf
modify build.gradle to match tox task names
AnandInguva Dec 8, 2023
539c9ad
Add note to CHANGES.md
AnandInguva Dec 8, 2023
b967cd8
change gcs bucket to gs://temp-storage-for-perf-tests
AnandInguva Dec 8, 2023
f1bb42c
Add TODO GH links
AnandInguva Dec 11, 2023
8d0b47d
Merge remote-tracking branch 'origin/master' into embeddings_mltransform
AnandInguva Dec 11, 2023
c173d6a
Update CHANGES.md
AnandInguva Dec 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@
* Python GCSIO is now implemented with GCP GCS Client instead of apitools ([#25676](https://github.com/apache/beam/issues/25676))
* Adding support for LowCardinality DataType in ClickHouse (Java) ([#29533](https://github.com/apache/beam/pull/29533)).
* Added support for handling bad records to KafkaIO (Java) ([#29546](https://github.com/apache/beam/pull/29546))
* Add support for generating text embeddings in MLTransform for Vertex AI and Huggingface hub models.([#29564](https://github.com/apache/beam/pull/29564))
AnandInguva marked this conversation as resolved.
Show resolved Hide resolved

## New Features / Improvements

Expand Down
Loading
Loading