-
Notifications
You must be signed in to change notification settings - Fork 81
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[HF][fix] Use ParameterizedModelParser instead of ModelParser #877
+9
−6
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rossdanlm
added a commit
that referenced
this pull request
Jan 11, 2024
Add model setting completion params for Image2Text prompt schema TSIA, pretty simple actually ## Test Plan Follow the README from AIConfig Editor https://github.com/lastmile-ai/aiconfig/tree/main/python/src/aiconfig/editor#dev, then run these commands: Make sure also that you don't have the `python-aiconfig-test` installed: `pip3 uninstall python-aiconfig-test` ```bash aiconfig_path=/Users/rossdancraig/Projects/aiconfig/cookbooks/Gradio/huggingface.aiconfig.json parsers_path=/Users/rossdancraig/Projects/aiconfig/cookbooks/Gradio/hf_model_parsers.py alias aiconfig="python3 -m 'aiconfig.scripts.aiconfig_cli'" aiconfig edit --aiconfig-path=$aiconfig_path --server-port=8080 --server-mode=debug_servers --parsers-module-path=$parsers_path ``` <img width="1261" alt="Screenshot 2024-01-11 at 01 21 21" src="https://github.com/lastmile-ai/aiconfig/assets/151060367/63ef4830-1163-4229-90ea-d49b914d1ec2"> --- Stack created with [Sapling](https://sapling-scm.com). Best reviewed with [ReviewStack](https://reviewstack.dev/lastmile-ai/aiconfig/pull/875). * #877 * __->__ #875
rossdanlm
changed the title
[fix] Use ParameterizedModelParser instead of ModelParser
[HF][fix] Use ParameterizedModelParser instead of ModelParser
Jan 11, 2024
Fixing this error: ``` Exception: HuggingFaceImage2TextTransformer.run() got an unexpected keyword argument 'callback_manager' ``` This seems like a pretty brittle bug. If we use `ModelParser` isntead of `ParameterizedModelParser`, we get a bug because `callback_manager` is not a supported kwargs for the `run_inference()` method. This gets filtered out by the `ParameterizedModelParser.run()` method (https://github.com/lastmile-ai/aiconfig/blob/0ceb17636bae2b5416e7b415cdc87fd71b5ba3b0/python/src/aiconfig/default_parsers/parameterized_model_parser.py#L48-L54) but these kwags are not accepted by the `ModelParser.run()` method (https://github.com/lastmile-ai/aiconfig/blob/0ceb17636bae2b5416e7b415cdc87fd71b5ba3b0/python/src/aiconfig/model_parser.py#L62-L69) ## Test Plan Before https://github.com/lastmile-ai/aiconfig/assets/151060367/a23a5d5c-d9a2-415b-8a6e-9826da56e985 After https://github.com/lastmile-ai/aiconfig/assets/151060367/8899c527-7496-410b-b6b6-9e5a9525b1b5
rossdanlm
requested review from
saqadri,
rholinshead,
suyoglastmileai,
Ankush-lastmile and
jonathanlastmileai
as code owners
January 11, 2024 07:11
rossdanlm
pushed a commit
that referenced
this pull request
Jan 11, 2024
Getting this error: ``` Exception: Cannot get prompt template string from prompt input: attachments=[Attachment(data='https://s3.amazonaws.com/lastmileai.aiconfig.public/uploads/2024_1_10_18_41_31/1742/bear_with_honey.png', mime_type='image/png', metadata=None), Attachment(data='https://s3.amazonaws.com/lastmileai.aiconfig.public/uploads/2024_1_10_18_41_31/7275/fox_in_forest.png', mime_type='image/png', metadata=None)] data=None ``` When Sarmad did this fix in #866, he missed a callsite in the `parameterized_model_parser`. This was missed because at the time the image_2_text model parser was of type `ModelParser` instead of `ParameterizedModelParser`, which had to be converted in here to fix a bug: #877 Before
rossdanlm
pushed a commit
that referenced
this pull request
Jan 11, 2024
Getting this error: ``` Exception: Cannot get prompt template string from prompt input: attachments=[Attachment(data='https://s3.amazonaws.com/lastmileai.aiconfig.public/uploads/2024_1_10_18_41_31/1742/bear_with_honey.png', mime_type='image/png', metadata=None), Attachment(data='https://s3.amazonaws.com/lastmileai.aiconfig.public/uploads/2024_1_10_18_41_31/7275/fox_in_forest.png', mime_type='image/png', metadata=None)] data=None ``` When Sarmad did this fix in #866, he missed a callsite in the `parameterized_model_parser`. This was missed because at the time the image_2_text model parser was of type `ModelParser` instead of `ParameterizedModelParser`, which had to be converted in here to fix a bug: #877 ## Test Plan Before <img width="1295" alt="Screenshot 2024-01-11 at 02 52 42" src="https://github.com/lastmile-ai/aiconfig/assets/151060367/c0c99f1e-2996-41f3-8274-c13ecc43f07e"> After https://github.com/lastmile-ai/aiconfig/assets/151060367/7a10932c-8580-4584-8942-4c287f23fc82
rossdanlm
pushed a commit
that referenced
this pull request
Jan 11, 2024
See #877 for context, and #880 for why that original fix needed to be reverted Just a quick hack to get unblocked. I tried originally to make the ASR and Image2Text ParameterizedModel but that caused other errors. ## Test Plan Before https://github.com/lastmile-ai/aiconfig/assets/151060367/a23a5d5c-d9a2-415b-8a6e-9826da56e985
rossdanlm
pushed a commit
that referenced
this pull request
Jan 11, 2024
See #877 for context, and #880 for why that original fix needed to be reverted Just a quick hack to get unblocked. I tried originally to make the ASR and Image2Text ParameterizedModel but that caused other errors. ## Test Plan Before https://github.com/lastmile-ai/aiconfig/assets/151060367/a23a5d5c-d9a2-415b-8a6e-9826da56e985 After https://github.com/lastmile-ai/aiconfig/assets/151060367/f29580e9-5cf6-43c5-b848-bb525eb368f2
rossdanlm
pushed a commit
that referenced
this pull request
Jan 11, 2024
See #877 for context, and #880 for why that original fix needed to be reverted Just a quick hack to get unblocked. I tried originally to make the ASR and Image2Text ParameterizedModel but that caused other errors. ## Test Plan Before https://github.com/lastmile-ai/aiconfig/assets/151060367/a23a5d5c-d9a2-415b-8a6e-9826da56e985 After https://github.com/lastmile-ai/aiconfig/assets/151060367/f29580e9-5cf6-43c5-b848-bb525eb368f2
rossdanlm
pushed a commit
that referenced
this pull request
Jan 11, 2024
See #877 for context, and #880 for why that original fix needed to be reverted Just a quick hack to get unblocked. I tried originally to make the ASR and Image2Text ParameterizedModel but that caused other errors. ## Test Plan Before https://github.com/lastmile-ai/aiconfig/assets/151060367/a23a5d5c-d9a2-415b-8a6e-9826da56e985 After https://github.com/lastmile-ai/aiconfig/assets/151060367/f29580e9-5cf6-43c5-b848-bb525eb368f2
Will leave a comment in other diff as well, but it's important that models that can't have paramrterized inputs NOT extend that ParameterizedModelParser class |
saqadri
added a commit
that referenced
this pull request
Jan 11, 2024
#881) Revert "[HF][fix] Use ParameterizedModelParser instead of ModelParser" Reverts #877 This caused a bug because image and audio inputs can't be parametrized. See #882 for a better fix instead --- Stack created with [Sapling](https://sapling-scm.com). Best reviewed with [ReviewStack](https://reviewstack.dev/lastmile-ai/aiconfig/pull/881). * #885 * #883 * #882 * __->__ #881 * #879
saqadri
added a commit
that referenced
this pull request
Jan 11, 2024
[HF][fix] Allow `kwargs` in `ModelParser.run()` See #877 for context, and #880 for why that original fix needed to be reverted Just a quick hack to get unblocked. I tried originally to make the ASR and Image2Text ParameterizedModel but that caused other errors. ## Test Plan Before https://github.com/lastmile-ai/aiconfig/assets/151060367/a23a5d5c-d9a2-415b-8a6e-9826da56e985 After https://github.com/lastmile-ai/aiconfig/assets/151060367/f29580e9-5cf6-43c5-b848-bb525eb368f2 --- Stack created with [Sapling](https://sapling-scm.com). Best reviewed with [ReviewStack](https://reviewstack.dev/lastmile-ai/aiconfig/pull/882). * #885 * #883 * __->__ #882 * #881 * #879
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[HF][fix] Use ParameterizedModelParser instead of ModelParser
Fixing this error:
This seems like a pretty brittle bug. If we use
ModelParser
isntead ofParameterizedModelParser
, we get a bug becausecallback_manager
is not a supported kwargs for therun_inference()
method. This gets filtered out by theParameterizedModelParser.run()
method (aiconfig/python/src/aiconfig/default_parsers/parameterized_model_parser.py
Lines 48 to 54 in 0ceb176
ModelParser.run()
method (aiconfig/python/src/aiconfig/model_parser.py
Lines 62 to 69 in 0ceb176
Test Plan
Before
Screen.Recording.2024-01-11.at.02.00.14.mov
After
Screen.Recording.2024-01-11.at.01.59.36.mov