Applying custom Huggingface model as an encoder

[efima-ai]

Hi,

The documentation says

The available encoders include encoders used for Sequence Features as well as encoders from the huggingface transformers library: bert, gpt, gpt2, xlnet, xlm, roberta, distilbert, ctrl, camembert, albert, t5, xlmroberta, flaubert, electra, longformer and auto-transformer.

Does this mean that Ludwig does not support a model that is found on Huggingface but not listed above, e.g. https://huggingface.co/TurkuNLP/bert-base-finnish-cased-v1, as an encoder for the text features?

Thanks 😊

[jppgks]

Hi @efima-ai, you can use arbitrary HuggingFace models using the auto_transformer encoder. You can try something like this for your case:

import pandas as pd
import yaml

from ludwig.api import LudwigModel

config = """
input_features:
    - name: text
      type: text
      encoder: auto_transformer
      pretrained_model_name_or_path: 'TurkuNLP/bert-base-finnish-cased-v1'
output_features:
    - name: category
      type: category
trainer:
    epochs: 1
"""
model = LudwigModel(yaml.load(config), backend="local")

df = pd.DataFrame(
    {
        "text": ["Suomessa vaihtuu kesän aikana sekä pääministeri että valtiovarain"],
        "category": ["Suomi"],
    }
)
model.train(df)
model.predict(df)

[efima-ai]

Thanks Joppe. Finally got time to try this out - and it works fantastically!