Updated characters, underscore and comma preprocessors to be TorchScriptable. #3602
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
justinxzhao
reviewed
Sep 14, 2023
| if isinstance(v, torch.Tensor): | ||
| raise ValueError(f"Unsupported input: {v}") | ||
|
|
||
| inputs: List[str] = [] |
There was a problem hiding this comment.
I see that you are adapting an existing implementation, though this seems more complicated than I would expect (for example, why do we have a get_tokens() function that returns its own input?).
@geoffreyangus, ooc does this also look strange to you, or is this imposed on us by torchscript?
There was a problem hiding this comment.
It looks like NgramTokenizer, which subclasses SpaceStringToListTokenizer (which in turn subclasses the new StringSplitTokenizer), seems to override get_tokens: https://github.com/ludwig-ai/ludwig/pull/3602/files#diff-5cbace55f4f4fd07725c061b9f981b83fe43cb53b0045cf1257c9fb5d4931f0dR132-R142
justinxzhao
approved these changes
Sep 14, 2023
martindavis
deleted the
update-basic-preprocessors-to-be-torchscriptable
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The
comma,underscoreandcharacterspreprocessors are not currently TorchScriptable because they do not implement the TorchScript Module methods. As part of this PR, we now have a generalStringSplitTokenizerthat can be re-used across the various defaults we support. Unfortunately, since TorchScript only supports basic data types,CharactersToListTokenizerhad to be written as a separate class instead of using Lamda Function.Will help users run into this error less often:
This covered by the
tests/integration_tests/test_torchscript.py::test_torchscript_e2e_texttest because this PR updatesTORCHSCRIPT_COMPATIBLE_TOKENIZERSto include the new TorchScriptable tokenizers.