Best Practices for Implementing Machine/Deep Learning Models

[grazder]

Hello! I am currently working on implementing a deep learning model in WebAudio for real-time inference. My model is basically written in "pytorch" and I want it to work inside an AudioWorklet. Do you have experience or best practices for this? I would like to hear your advices

[padenot]

There's essentially two ways to do it, and it depends on the workload you're attempting to run.

If your workload is real-time safe, you can run the code in the AudioWorkletProcessor's process method. This has very hard constraints, and any failure to meet the deadline with result in problems of various nature. I'd suggest you to read https://blog.paul.cx/post/profiling-firefox-real-time-media-workloads/ and the classic article linked in the second footnote, to get a feel for what this all means. This article also shows a way to measure your workload to understand what's going on, in Firefox. There is also a way to measure this in Chrome, I'll let @hoch outline the steps or point to a document or something, I know they have something cool in their devtools for performance assessment of real-time audio workloads.

Generally, this means that your workload can process and return synchronously in less than 128 / AudioContext.sampleRate seconds, always (not 99% of the time, 100% of the time). The result of the computation then needs to be communicated outside the AudioWorkletProcessor, which you can do in a manner similar to what the rest of this message explains, or using postMessage if it's infrequent. Again, measuring is the only way to know what's best, and we've made it easy to do because it's so important.

If it isn't possible to guarantee those properties for your particular problem (synchronicity + consistent processing time), the solution is to send the real-time audio data to a Web Worker, and to perform the analysis there. The correct way to do this is to not use postMessage, but to use a wait-free single-producer single-consumer ring-buffer. Because this is such a common thing we do in real-time audio programming, I've written and released a library that implements this data structure, geared towards audio developers needs: https://github.com/padenot/ringbuf.js. https://blog.paul.cx/post/a-wait-free-spsc-ringbuffer-for-the-web/ is an introductory blog post that explain the why and the how.

A mini-site (https://ringbuf-js.netlify.app/) has the documentation, important links and two examples. The second example, named "Getting audio out of an AudioWorkletNode" is exactly what you want to do: get the data from an AudioWorkletProcessor and send it to a worker, for processing. In my example, the processing is simply to turn it into a WAV file, but it could be anything. This library works well in all browsers and is liberally licensed.

With this setup, to have a lot more flexibility, there is no real-time constraint, but you still have very low latency. You can for example wait and combine multiple 128-frames chunks of audio to process 1024 frames of audio at a time, for efficiency, if your problem is ok with adding a bit of latency (in this example, 1024 / 44100., about 23ms or so, which is acceptable in most scenarios).

[grazder]

Thank you for information, i also saw that comment - #1503 (comment)

Do you also have information about supported targets for DL models inference inside audioworklet? Like onnxruntime-web, tfjs, tflite.

[padenot]

I'm sorry, I am absolutely not knowledgeable about any of this, but if you try something out, you can get Firefox Profiles (using the tooling mentioned above) and assess the situation.

[grazder]

ok, thank you

[grazder]

@padenot Hello! Thanks a lot for your explanation, it helped a lot. Scheme with ringbuffer worked great, but we meet some problems with necessity of turning on headers ("Cross-Origin-Embedder-Policy": "require-corp", "Cross-Origin-Opener-Policy": "same-origin") for SharedArrayBuffer. And right now we want to try worse scheme with postMessages.

Do you have an example of Worklet - Worker conversation with postMessages. Can they communicate directly without sending postMessages to main thread? Now we only have the scheme working with worklet -> main -> worker -> main -> worklet.

[padenot]

No, but I just wrote one:

<script type="worklet">
registerProcessor('example-worklet',
  class Processor extends AudioWorkletProcessor {
  constructor() {
    super();
    this.port.onmessage = (e) => {
      // 7. e.data is the port from the Worker, that can be used to directly
      // communicate with it.
      var portToWorker = e.data;
      // 8. Send a message to the Worker
      portToWorker.postMessage("Hi from the worklet, using MessagePort from the Worker");
    };
    this.port.postMessage("AudioWorklet constructed");
  }
  process(input, output, parameters) {
    /* DSP here */
    return true;
  }
});
</script>
<script type="worker">
onmessage = function(e) {
  if (e.data == "send port") {
    // 3. If the message is "send port", create a MessageChannel and send the
    // second half of the port over
    console.log("onmessage worker ", e.data);
    channel = new MessageChannel();
    // 4. On the first part, just print out what is received.
    channel.port1.onmessage = function(e) {
      // 9. Message received, from the AudioWorkletProcessor.
      console.log("From Worker MessageChannel:", e.data);
    }
    // 5. Actually send the port, transfering it to the main thread.
    postMessage(channel.port2, {transfer: [channel.port2]})
  }
}
</script>
<script>
// Create a Worker
var source = document.querySelector("script[type=worker]")
var text2 = source.innerText;
const blob2 = new Blob([text2], {type: "application/javascript"});
var worker = new Worker(URL.createObjectURL(blob2));

// Create and AudioWorkletProcessor
var ac = new AudioContext;
var source = document.querySelector("script[type=worklet]")
var text = source.innerText;
const blob = new Blob([text], {type: "application/javascript"});

// 1. AudioWorkletProcessor and Worker initialization initiated.
ac.audioWorklet.addModule(URL.createObjectURL(blob)).then(() => {
  var node = new AudioWorkletNode(ac, 'example-worklet');
  worker.onmessage = function(e) {
    // 6. When the worker sends a message, it's the port. Forward it to the
    // AudioWorkletProcessor. This is the last time the main thread is involved.
    console.log("From Worker: ", e);
    node.port.postMessage(e.data, {transfer: [e.data]});
  }
  node.port.onmessage = function(e) {
    // 2. AudioWorkletProcessor constructed.
    // Ask the worker to send a MessagePort over
    console.log("From AudioWorkletProcessor: ", e.data);
    worker.postMessage("send port")
  }
});
</script>

The comments with numbers are to be followed in order to understand the flow. It's not going to be very performant compared to ringbuf.js, but it's going to work. In this example, the main thread is used to send a MessagePort the worker has created to the AudioWorkletProcessor, during a setup phase, and then the AudioWorkletProcessor and the Worker can communicate directly.

[grazder]

Thank you very much!

[grazder]

@padenot Hello! Can you tell about AudioWorklet internal buffering? I'm profiling AudioWorklet + Worker + postMessages scheme (in Chrome) and seeing the following picture:

Timings are activated when AudioWorklet runs the process() method.

Initially I though that AudioWorklet run process() method every 2.6ms for 48kHz. But I see here, that it store multiple frames and sequentially runs process() for each stored frame. It can sequentially process up to 6 frames and i've got two sequential Worker calls. And I don't understand why is this happening.

And because of this, there may not be enough samples in the buffer for the AudioWorklet to take frames from there in time. So, I've created starting latency about 3 * 480 samples (When we take first 128 samples from outputBuffer, there are already 3 * 480 processed samples there), that fixes this problem. But I'm trying to understand how to make my implementation better, using knowledge about frames buffering.

More about implementation:
I'm saving up 480 frames in AudioWorklet and sending it to Worker using postMessage for audio processing.
One worker audio processing takes for about 2.8ms here.
In Firefox i don't face this error

[padenot]

I can only talk about Firefox's implementation (because that's what I know and have worked on, although I didn't write it myself), and it never buffers. @hoch will know more for the Chromium side.

Generally, it's wise to never make assumptions about relative timing of anything, though. It's perfectly legal (and in fact sometimes the only solution) to call process multiple times in a row. For example, if the OS-level system audio callback works with buffer sizes of 512 frames, I would expect to have process(...) called four times synchronously, then a pause, then called four times again, etc. Maybe this is what Chrome is doing, I don't know.

If the number of frames of the system-level audio callback is not a multiple of 128, then it's going to be irregular. If it's e.g. 192 frames (meaning 250 callbacks per second at 48000Hz), you'll have 2 callbacks, then one, then two, then one, etc.

#2450 was about this, and was recently merged. It's going to be implemented next year in browsers (I think).