BAAI/bge-m3 Model: data did not match any variant of untagged enum PreTokenizerWrapper

[iprovalo]

Description

Runtime won't load a converted model for bge-m3

Expected Behavior

No errors.

Error Message

Exception in thread "main" java.lang.RuntimeException: data did not match any variant of untagged enum PreTokenizerWrapper at line 73 column 3
at ai.djl.huggingface.tokenizers.jni.TokenizersLibrary.createTokenizerFromString(Native Method)
at ai.djl.huggingface.tokenizers.HuggingFaceTokenizer.newInstance(HuggingFaceTokenizer.java:176)
at ai.djl.huggingface.tokenizers.HuggingFaceTokenizer.newInstance(HuggingFaceTokenizer.java:138)
at ai.djl.huggingface.tokenizers.HuggingFaceTokenizer$Builder.build(HuggingFaceTokenizer.java:817)
at ai.djl.huggingface.translator.TextEmbeddingTranslatorFactory.newInstance(TextEmbeddingTranslatorFactory.java:64)
at ai.djl.translate.DeferredTranslatorFactory.newInstance(DeferredTranslatorFactory.java:69)
at ai.djl.repository.zoo.BaseModelLoader.loadModel(BaseModelLoader.java:167)
at ai.djl.repository.zoo.Criteria.loadModel(Criteria.java:172)

How to Reproduce?

1. Convert the model:

djl-convert -m BAAI/bge-m3 -f OnnxRuntime

2. Then run in Java:

    public static void main(String[] args) throws TranslateException, ModelNotFoundException, MalformedModelException, IOException {
        System.out.println(getEmbeddings("test").length);
    }

    public static float[] getEmbeddings(String text) throws ModelNotFoundException, MalformedModelException, IOException, TranslateException {
        Criteria<String, float[]> criteria =
                Criteria.builder()
                        .setTypes(String.class, float[].class)
                        .optModelPath(Paths.get("bge-m3"))
                        .optEngine("OnnxRuntime")
                        .optTranslatorFactory(new DeferredTranslatorFactory())
                        .optProgress(new ProgressBar())
                        .build();

        ZooModel<String, float[]> model = criteria.loadModel();
        Predictor<String, float[]> predictor = model.newPredictor();
        return predictor.predict(text);
    }

3. Java project gradle:

    //DJL
    implementation 'ai.djl:api:0.29.0'
    implementation 'ai.djl.pytorch:pytorch-engine:0.29.0'

    implementation("ai.djl.onnxruntime:onnxruntime-engine:0.29.0") {
        exclude group: "com.microsoft.onnxruntime", module: "onnxruntime"
    }
    implementation "com.microsoft.onnxruntime:onnxruntime:1.18.0"

Environment Info - Conversion step

Please run the command ./gradlew debugEnv from the root directory of DJL (if necessary, clone DJL first). It will output information about your system, environment, and installation that can help us debug your issue. Paste the output of the command below:

> Task :integration:debugEnv
----------- System Properties -----------
java.specification.version: 21
sun.jnu.encoding: UTF-8
java.class.path: /dev/djl/integration/build/classes/java/main:/dev/djl/integration/build/resources/main:/Users/XXX/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.8.0/41a4bff12057eecb6daaf9c7f36c237815be3da1/commons-cli-1.8.0.jar:/Users/XXX/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j2-impl/2.23.1/c3ffee33404c3a178f026fd8c7ef0e058b01b01c/log4j-slf4j2-impl-2.23.1.jar:/dev/djl/basicdataset/build/libs/basicdataset-0.30.0-SNAPSHOT.jar:/dev/djl/model-zoo/build/libs/model-zoo-0.30.0-SNAPSHOT.jar:/dev/djl/testing/build/libs/testing-0.30.0-SNAPSHOT.jar:/dev/djl/engines/pytorch/pytorch-model-zoo/build/libs/pytorch-model-zoo-0.30.0-SNAPSHOT.jar:/dev/djl/engines/pytorch/pytorch-jni/build/libs/pytorch-jni-2.4.0-0.30.0-SNAPSHOT.jar:/dev/djl/engines/tensorflow/tensorflow-model-zoo/build/libs/tensorflow-model-zoo-0.30.0-SNAPSHOT.jar:/dev/djl/engines/ml/xgboost/build/libs/xgboost-0.30.0-SNAPSHOT.jar:/dev/djl/engines/ml/lightgbm/build/libs/lightgbm-0.30.0-SNAPSHOT.jar:/dev/djl/engines/onnxruntime/onnxruntime-engine/build/libs/onnxruntime-engine-0.30.0-SNAPSHOT.jar:/dev/djl/extensions/tokenizers/build/libs/tokenizers-0.30.0-SNAPSHOT.jar:/Users/XXX/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.23.1/905802940e2c78042d75b837c136ac477d2b4e4d/log4j-core-2.23.1.jar:/Users/XXX/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.23.1/9c15c29c526d9c6783049c0a77722693c66706e1/log4j-api-2.23.1.jar:/dev/djl/engines/pytorch/pytorch-engine/build/libs/pytorch-engine-0.30.0-SNAPSHOT.jar:/dev/djl/engines/tensorflow/tensorflow-engine/build/libs/tensorflow-engine-0.30.0-SNAPSHOT.jar:/dev/djl/api/build/libs/api-0.30.0-SNAPSHOT.jar:/Users/XXX/.gradle/caches/modules-2/files-2.1/org.testng/testng/7.10.2/30742acada21960d4333a4204039fbdc6a92083a/testng-7.10.2.jar:/Users/XXX/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/2.0.13/80229737f704b121a318bba5d5deacbcf395bc77/slf4j-api-2.0.13.jar:/Users/XXX/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-csv/1.11.0/8f2dc805097da534612128b7cdf491a5a76752bf/commons-csv-1.11.0.jar:/Users/XXX/.gradle/caches/modules-2/files-2.1/ml.dmlc/xgboost4j_2.12/2.0.3/db511d04d1ca1364cde79a6c8238a2694e31c592/xgboost4j_2.12-2.0.3.jar:/Users/XXX/.gradle/caches/modules-2/files-2.1/commons-logging/commons-logging/1.3.3/580ad1a4f34876c4f964c083361de31b3d60be68/commons-logging-1.3.3.jar:/Users/XXX/.gradle/caches/modules-2/files-2.1/com.microsoft.ml.lightgbm/lightgbmlib/3.2.110/f6c85e5d7cc44d49c4544240ea5c96004680007b/lightgbmlib-3.2.110.jar:/Users/XXX/.gradle/caches/modules-2/files-2.1/com.microsoft.onnxruntime/onnxruntime/1.18.0/bacf73dc2e1d92941744c6a3f8c01fc674189d36/onnxruntime-1.18.0.jar:/Users/XXX/.gradle/caches/modules-2/files-2.1/com.google.code.gson/gson/2.11.0/527175ca6d81050b53bdd4c457a6d6e017626b0e/gson-2.11.0.jar:/Users/XXX/.gradle/caches/modules-2/files-2.1/net.java.dev.jna/jna/5.14.0/67bf3eaea4f0718cb376a181a629e5f88fa1c9dd/jna-5.14.0.jar:/Users/XXX/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-compress/1.26.2/eb1f823447af685208e684fce84783b43517960c/commons-compress-1.26.2.jar:/Users/XXX/.gradle/caches/modules-2/files-2.1/commons-io/commons-io/2.16.1/377d592e740dc77124e0901291dbfaa6810a200e/commons-io-2.16.1.jar:/Users/XXX/.gradle/caches/modules-2/files-2.1/commons-codec/commons-codec/1.17.0/dbe8eef6e14460e73da07f7b11bf994d6626355/commons-codec-1.17.0.jar:/Users/XXX/.gradle/caches/modules-2/files-2.1/com.beust/jcommander/1.82/a7c5fef184d238065de38f81bbc6ee50cca2e21/jcommander-1.82.jar:/Users/XXX/.gradle/caches/modules-2/files-2.1/org.webjars/jquery/3.7.1/42088e652462c40a369b64d87e18e825644acfab/jquery-3.7.1.jar:/dev/djl/engines/tensorflow/tensorflow-api/build/libs/tensorflow-api-0.30.0-SNAPSHOT.jar:/Users/XXX/.gradle/caches/modules-2/files-2.1/org.scala-lang.modules/scala-collection-compat_2.12/2.10.0/bf81785e892f4185f470bddd205b011237aab553/scala-collection-compat_2.12-2.10.0.jar:/Users/XXX/.gradle/caches/modules-2/files-2.1/com.google.errorprone/error_prone_annotations/2.27.0/91b2c29d8a6148b5e2e4930f070d4840e2e48e34/error_prone_annotations-2.27.0.jar:/Users/XXX/.gradle/caches/modules-2/files-2.1/org.tensorflow/tensorflow-core-api/1.0.0-rc.1/ea1878fb8e289742237e5a0ba6f15398f3e9b7ef/tensorflow-core-api-1.0.0-rc.1.jar:/Users/XXX/.gradle/caches/modules-2/files-2.1/org.tensorflow/tensorflow-core-native/1.0.0-rc.1/62b5fa3283865cc696dfbebf073ca2116b18f327/tensorflow-core-native-1.0.0-rc.1.jar:/Users/XXX/.gradle/caches/modules-2/files-2.1/org.bytedeco/javacpp/1.5.10/afb6ae145e7563c66b677cb4896dd0197d49fce6/javacpp-1.5.10.jar:/Users/XXX/.gradle/caches/modules-2/files-2.1/com.google.protobuf/protobuf-java/3.25.3/d3200261955f3298e0d85c9892201e70492ce8eb/protobuf-java-3.25.3.jar:/Users/XXX/.gradle/caches/modules-2/files-2.1/org.tensorflow/ndarray/1.0.0-rc.1/4a96a398ad87bec32be9177b1441b9880c04d822/ndarray-1.0.0-rc.1.jar
java.vm.vendor: Amazon.com Inc.
sun.arch.data.model: 64
user.variant: 
java.vendor.url: https://aws.amazon.com/corretto/
user.timezone: America/Los_Angeles
java.vm.specification.version: 21
os.name: Mac OS X
user.country: US
sun.java.launcher: SUN_STANDARD
sun.boot.library.path: /Users/XXX/Library/Java/JavaVirtualMachines/corretto-21.0.3/Contents/Home/lib
sun.java.command: ai.djl.integration.util.DebugEnvironment
jdk.debug: release
sun.cpu.endian: little
user.home: /Users/XXX
user.language: en
java.specification.vendor: Oracle Corporation
java.version.date: 2024-04-16
java.home: /Users/XXX/Library/Java/JavaVirtualMachines/corretto-21.0.3/Contents/Home
file.separator: /
java.vm.compressedOopsMode: Zero based
line.separator: 

java.vm.specification.vendor: Oracle Corporation
java.specification.name: Java Platform API Specification
apple.awt.application.name: DebugEnvironment
sun.management.compiler: HotSpot 64-Bit Tiered Compilers
java.runtime.version: 21.0.3+9-LTS
user.name: XXX
stdout.encoding: UTF-8
path.separator: :
os.version: 14.5
java.runtime.name: OpenJDK Runtime Environment
file.encoding: UTF-8
java.vm.name: OpenJDK 64-Bit Server VM
java.vendor.version: Corretto-21.0.3.9.1
java.vendor.url.bug: https://github.com/corretto/corretto-21/issues/
java.io.tmpdir: /var/folders/x2/bs_p9trx49nbbqfntcn0fncm0000gn/T/
java.version: 21.0.3
user.dir: /dev/djl/integration
os.arch: aarch64
java.vm.specification.name: Java Virtual Machine Specification
native.encoding: UTF-8
java.library.path: /Users/XXX/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
java.vm.info: mixed mode, sharing
stderr.encoding: UTF-8
java.vendor: Amazon.com Inc.
java.vm.version: 21.0.3+9-LTS
sun.io.unicode.encoding: UnicodeBig
java.class.version: 65.0

--------- Environment Variables ---------
HOMEBREW_PREFIX: /opt/homebrew
rvm_script_name: 
rvm_gemstone_package_file: 
rvm_pretty_print_flag: 
rvm_docs_type: 
rvm_bin_flag: 
HOMEBREW_REPOSITORY: /opt/homebrew
VIRTUAL_ENV_PROMPT: (env) 
CONDA_PREFIX: /Users/XXX/miniconda
PATH: /dev/.venv/bin:/Users/XXX/miniconda/bin:/Users/XXX/miniconda/condabin:/Library/Frameworks/Python.framework/Versions/3.10/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/opt/nflx:/opt/nflx/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/usr/local/munki:/Users/XXX/.cargo/bin:/Users/XXX/.rvm/bin
LOGNAME: XXX
rvm_ruby_make: 
__CF_USER_TEXT_ENCODING: 0x1F5:0x0:0x0
XPC_FLAGS: 0x0
rvm_niceness: 
rvm_prefix: /Users/XXX
_CE_M: 
rvm_only_path_flag: 
VIRTUAL_ENV: /dev/.venv
CONDA_SHLVL: 1
rvm_gemstone_url: 
rvm_version: 1.29.12 (latest)
IJ_RESTARTER_LOG: /Users/XXX/Library/Logs/JetBrains/PyCharm2024.1/restarter.log
CONDA_EXE: /Users/XXX/miniconda/bin/conda
XPC_SERVICE_NAME: 0
CONDA_DEFAULT_ENV: base
rvm_use_flag: 
HOMEBREW_CELLAR: /opt/homebrew/Cellar
TERMINAL_EMULATOR: JetBrains-JediTerm
CONDA_PROMPT_MODIFIER: (base) 
COMMAND_MODE: unix2003
SHLVL: 1
INFOPATH: /opt/homebrew/share/info:
rvm_alias_expanded: 
rvm_nightly_flag: 
SHELL: /bin/zsh
__CFBundleIdentifier: com.jetbrains.pycharm
TMPDIR: /var/folders/x2/bs_p9trx49nbbqfntcn0fncm0000gn/T/
rvm_silent_flag: 
rvm_bin_path: /Users/XXX/.rvm/bin
HOME: /Users/XXX
LaunchInstanceID: F6696F0F-FEFC-4AF3-8782-C043A69635CF
rvm_ruby_mode: 
TERM: xterm-256color
_CE_CONDA: 
APP_ICON_74380: /dev/djl/media/gradle.icns
IDEA_INITIAL_DIRECTORY: /
LC_CTYPE: UTF-8
rvm_path: /Users/XXX/.rvm
rvm_proxy: 
rvm_quiet_flag: 
rvm_ruby_file: 
rvm_hook: 
SSH_AUTH_SOCK: /private/tmp/com.apple.launchd.OxzXduTzuf/Listeners
rvm_sdk: 
OLDPWD: /dev/djl
rvm_ruby_bits: 
USER: XXX
SECURITYSESSIONID: 186b3
PWD: /dev/djl
CONDA_PYTHON_EXE: /Users/XXX/miniconda/bin/python
rvm_ruby_make_install: 

-------------- Directories --------------
temp directory: /var/folders/x2/bs_p9trx49nbbqfntcn0fncm0000gn/T
DJL cache directory: /Users/XXX/.djl.ai
Engine cache directory: /Users/XXX/.djl.ai

------------------ CUDA -----------------
GPU Count: 0

----------------- Engines ---------------
DJL version: 0.30.0-SNAPSHOT
[INFO ] - Downloading https://publish.djl.ai/pytorch/2.4.0/cpu/osx-aarch64/native/lib/libc10.dylib.gz ...
[INFO ] - Downloading https://publish.djl.ai/pytorch/2.4.0/cpu/osx-aarch64/native/lib/libomp.dylib.gz ...
[INFO ] - Downloading https://publish.djl.ai/pytorch/2.4.0/cpu/osx-aarch64/native/lib/libtorch_cpu.dylib.gz ...
[INFO ] - Downloading https://publish.djl.ai/pytorch/2.4.0/cpu/osx-aarch64/native/lib/libtorch.dylib.gz ...
[INFO ] - Extracting jnilib/osx-aarch64/cpu/libdjl_torch.dylib to cache ...
[INFO ] - PyTorch graph executor optimizer is enabled, this may impact your inference latency and throughput. See: https://docs.djl.ai/master/docs/development/inference_performance_optimization.html#graph-executor-optimization
[INFO ] - Number of inter-op threads is 16
[INFO ] - Number of intra-op threads is 12
Default Engine: PyTorch:2.4.0, capabilities: [
        OPENMP,
]
PyTorch Library: /Users/XXX/.djl.ai/pytorch/2.4.0-cpu-osx-aarch64
Default Device: cpu()
Rust: 4
PyTorch: 2
XGBoost: 10
LightGBM: 10
OnnxRuntime: 10
TensorFlow: 3

--------------- Hardware --------------
Available processors (cores): 16
Byte Order: LITTLE_ENDIAN
Free memory (bytes): 1065250856
Maximum memory (bytes): 17179869184
Total memory available to JVM (bytes): 1082130432
Heap committed: 1082130432
Heap nonCommitted: 31916032
GCC: 
Apple clang version 15.0.0 (clang-1500.3.9.4)
Target: arm64-apple-darwin23.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

BUILD SUCCESSFUL in 1m 46s
52 actionable tasks: 52 executed

[frankfliu]

@iprovalo
I'm not able to reproduce your issue. I converted the model on mac, and everything works as expected.

Here is the files that after I converted to onnx:

-rw-r--r--   1 lufen  staff         736 Aug 26 21:25 config.json
-rw-r--r--   1 lufen  staff      683420 Aug 26 21:26 model.onnx
-rw-r--r--   1 lufen  staff  2266886160 Aug 26 21:26 model.onnx_data
-rw-r--r--   1 lufen  staff     5069051 Aug 26 21:25 sentencepiece.bpe.model
-rw-r--r--   1 lufen  staff         162 Aug 26 21:26 serving.properties
-rw-r--r--   1 lufen  staff         964 Aug 26 21:25 special_tokens_map.json
-rw-r--r--   1 lufen  staff    17082799 Aug 26 21:25 tokenizer.json
-rw-r--r--   1 lufen  staff        1173 Aug 26 21:25 tokenizer_config.json

[iprovalo]

@frankfliu thank you for trying to reproduce. I will continue debugging. I will report back on this thread if I find the issue with my set up.