Translations produced by test_page are (very, hilariously) wrong

[lhk]

I've followed your instructions to the letter, regarding the building of wasm, hosting the test page and running chrome with the correct command line arguments.

But when I open the test translation page and try it on some simple sentences, it outputs translations that are not even close to anything reasonable. In case you wonder, "schwanger" means "pregnant":

Similar problems appear for the other languages. At least "schwanger" is actually a German word. English to Spanish translates that phrase as "engineering engineering engineering engineering ...."

[XapaJIaMnu]

This is an issue with the GEMM, because either the WASM wormhole was not enabled (have you checked if it's enabled?) or if you are running apple M1 hardware, which is not supported.

[jerinphilip]

For what it's worth, I had previously reported this on var, which is not M1 hardware. Haven't managed to fix this till now. I downloaded a nightly launched locally and spawned via ssh -X and set prefs accordingly. I remember double-checking the instructions. A reproducible example should be available on a verbose run of a Makefile here, hopefully (make server, after configuring a few parameter variable names).

#87 (comment)

[lhk]

@XapaJIaMnu how do I check whether its enabled?

I did run the patch script. And this is executed in chromium, with the command line arguments that are given in the readme.

[lhk]

Also, it's not on M1

[XapaJIaMnu]

Did you use the Chrome canary 90+ with the appropriate flag:

https://github.com/browsermt/bergamot-translator/tree/main/wasm#demo

--js-flags="--experimental-wasm-simd"

[abhi-agg]

@lhk Currently, this feature doesn't work on Google Chrome browser (it used to). We are working on making it run there again.

In the meanwhile, you can try running it on Firefox Nightly where the translation speeds will be higher compared to running it on Chrome.

I will update the README to reflect the latest status 👍

[kpu]

Root cause is WebAssembly. Specifically it does not support 8-bit integer multiplication anywhere near efficiently WebAssembly/relaxed-simd#9 .
To make WebAssembly slightly less slow, we bypassed the standard to allow the Intel 8-bit instruction in Firefox https://bugzilla.mozilla.org/show_bug.cgi?id=1672160 . We're upgrading this to native 8-bit matrix multiply on Intel and ARM https://bugzilla.mozilla.org/show_bug.cgi?id=1715704 .

As mitigations, we should check that the 8-bit instruction is working first (by running it) and bail with an error when that happens, ideally one presented to the user. We could emulate with 16-bit integer instructions though this will make it even slower.

Full resolution entails not using WebAssembly at all for 12x speedup: https://translatelocally.com/

[lhk]

@XapaJIaMnu I'm running chromium 90+, over the command line and specifying that flag.

@abhi-agg can I run the code without this feature then? What if I don't apply the patch script?

@kpu interesting, thanks a lot for the details.

[abhi-agg]

@abhi-agg can I run the code without this feature then? What if I don't apply the patch script?

@lhk Unfortunately not because this feature was statically included for a faster experimentation. However, as I mentioned, we are working on a solution (should be landed soon) that will allow it to perform inference on Chrome as well (without this feature).

[kpu]

If you want a working version but care even less about speed, compile all the WASM stuff with https://github.com/kpu/intgemm/blob/master/CMakeLists.txt#L82-L85 with -DWORMHOLE=OFF