Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changed transformers_client design so it can support most (all?) HuggingFace models #208

Open
wants to merge 36 commits into
base: main
Choose a base branch
from

Conversation

adjahossoualexandre
Copy link

New design

TransformerClient into 3 client classes:

  1. TransformerEmbeddingModelClient
  • Merger of TransformerEmbedder and TransformerClient
  1. TransformerRerankerModelClient
  • Merger of TransformerReranker and TransformerClient
  1. TransformerLLMModelClient
  • Merger of TransformerLLM and TransformerClient

Pros:

  • Abstracts model/tokenizer initialisation as well as inference code
  • Hence, most of the time, the only customization needed is to change some parameters during client instantiation
  • Works with many different model architectures (example here)
  • Make the code easier to read and more maintainable since each class takes care of only one client
  • Easier to customize for users as it removes the need for model SDKs
  • Removes the need to check whether a model is supported or not

Cons:

  • The tradeoff here would be to couple inference code with the response and api kwargs handling.

Added 'tokenizer_kwargs' in TransformerLLMModelClient constructor for more flexibility.

Added chat template argument in constructor for more flexibility.

Added pad token check.

Added tokenizer in '_infer_from_pipeline()' when chat_template is used (required).

Fixed _handle_input() for 'apply_chat_template'==True.

Not sure: ficed message in convert_inputs_to_api_kwargs().
Moved get_device andclean_device_cache at top of file.

Allow user to specify autoclasses for  Reranker models.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant