MD5-WASM is a fast asynchronous md5 calculator, optimized for large files. It returns a Promise which resolves to the md5 string. WebAssembly is seamlessly applied to calculate values for files above a certain size threshold.
● 30x faster than the most popular md5 utility
● Server-side (NodeJS) or client-side (browser)
● Non-blocking, uses Promise syntax
Our md5 hashing was initially performed using this simple and popular utility: https://www.npmjs.com/package/md5 (called "MD5" herein) However, MD5 is synchronous, blocking code execution, and slow — impractically slow for video files. (On our low-powered server platform, we clock it at about 1 second per megabyte.)
On larger files, yes. Here are the benchmarks, comparing MD5 to MD5-WASM, run on our (slow) production server platform using NodeJS (v10.18.1) on Ubuntu.
ELAPSED MILLISECONDS MEGABYTES PER SECOND
MD5 MD5-WASM MD5 MD5-WASM
0.2 Mbytes 260 90 1.05 2.2
0.3 Mbytes 390 300 0.98 1.4
0.5 Mbytes 520 360 0.98 1.4
1 Mbytes 1,000 170 1.00 8.5
2 Mbytes 2,000 240 1.00 8.5
4 Mbytes 4,000 330 1.00 12
8 Mbytes 7,600 400 1.05 20
12 Mbytes 12,400 490 0.96 24
24 Mbytes 23,600 700 1.02 34
37 Mbytes 38,500 990 0.96 37
On our benchmark system, MD5-WASM gives up 150 ms to complete WebAssembly instantiation. After that, the relative performance gap between the two keeps growing, reaching 30x for a 37Mbyte file.
It would not be surprising to see a 3x improvement up to 5x improvement from WebAssembly but 30x is definitely surprising. For md5 calculation, WebAssembly holds one other big advantage. Any JavaScript implementation does a lot of number format conversion during md5 calculation, while WebAssembly implementations need not.
JavaScript runs native with 64-bit floating point numbers but all bitwise operations are done with 32-bit integers. Since calculating a checksum is just scads of bitwise operations, Javascript implementations spend more time converting between number formats than they do on the checksum itself.
You need do nothing different to accomodate WebAssembly — MD5-WASM loads in a browser or Node environment just like a pure JavaScript utility would. Unlike MD5, MD5-WASM does not take parameters in a string format — you must convert the string before injecting it into MD5-WASM. There is no synchronous version; you must use a promise instead of a simple blocking function call.
let data = contentsOfAFile(); // Get the data any which way you can
// 'data' must be a Buffer, ArrayBuffer or Uint8Array
md5WASM(data) // Our function
.then( hash => console.log(hash) )
.catch( err => console.log(err) )At less than 32K, the code file does not justify minification. It is all-inclusive and has no external dependencies.
<script type="text/javascript" src="path/md5-wasm.js"></script>You will find the function at window.md5WASM
md5WASM = require("md5-wasm");Please open an issue at the GitHub repo.