Midi support

[robclouth]

Summary: I've started a basic implementation of MIDI support to get things started

Problem
Pedalboard doesn't currently support plugin instruments because there is no way of sending MIDI to the plugins

Solution
I've added very basic MIDI support using juce::MidiMessageSequence and juce::MidiBuffer.

Result
This is mostly to get the ball rolling with MIDI support, it's definitely not ready for prime-time and haven't rigorously followed the contribution guide. It works for single processors, but not chains because the note off doesn't seem to work for chains. I haven't written any tests because I'm reckless. It's got to where I need it so I'll give it a rest for while. If someone wants to pick up the baton feel free!

For now I've done the following:

• Passed juce::MidiMessageSequence to the top-level process methods and based on the block time, passed MidiBuffers to the plugin process methods. This causes some complications with the Chains. My ugly non-working solution is to overload the process method and convert the MidiBuffer back into MidiMessageSequence
• Added note on and off generation with mido
• Added a second parameter to the plugin.process and pedalboard process methods that takes midi notes with time offsets in seconds. This should be optional but I couldn't quickly figure out how to do that. If passing in MIDI then the input audio should be optional too I guess.

E.g.

from pedalboard import Pedalboard, load_plugin
from pedalboard.io import AudioFile
from pedalboard.midi import note_on, note_off

import numpy as np

# Load a VST3 or Audio Unit plugin from a known path on disk:
vst = load_plugin("TAL-NoiseMaker.vst3")

print(vst.parameters.keys())

output_audio = np.zeros((44100, 2))
board = Pedalboard([vst])

effected = board(output_audio, [note_on(0.0, 72, 127), note_off(0.2, 72, 0)], 44100)

with AudioFile('processed-output.wav', 'w', 44100, effected.shape[1]) as f:
  f.write(effected)

[psobot]

Thanks so much @robclouth! This looks like a great starting point - I'll take some time to review the changes in more detail soon. Very glad to see that it's possible to add basic MIDI message support in under 200 lines changed.

[psobot]

Unfortunately I don't think we can change this API like this; lots of user code passes sample_rate as the second argument to process (or __call__) without specifying its keyword argument name. If we merge this, all of that user code will break as it will interpret the second argument as midi_messages.

This change also makes it mandatory to pass MIDI messages every time a plugin is invoked. If we move this to the last argument, we can also make it optional and provide None or [] as a default value, to avoid adding complexity for those who don't use this feature.

[psobot]

Two things:

• I don't think this import is actually used (we seem to import mido instead of import midi elsewhere)
• I'm pretty hesitant to add another runtime dependency if we can avoid it; MIDI messages are fairly simple to represent (as a five-tuple of (type, note, channel, velocity, time)) and we could use duck typing to accept objects from the mido package (or the midi package) without adding them as a runtime dependency.

[psobot]

This helper seems to re-interpret floating-point data into integer data, which seems a bit risky and confusing. Given that MIDI messages are probably going to be sparse (i.e.: not nearly as large as audio data) we could probably just use a std::vector<std::tuple<int, int, int, float>> or could create our own thin wrapper class and use that (i.e.: std::vector<pedalboard.MIDIMessage>).

[psobot]

Suggested change

DBG( byte1 );

[psobot]

Suggested change

;

[rhelsing]

@psobot Hey there, whats the current status on this PR?

[psobot]

Hi @rhelsing! The ball is in @robclouth's court at the moment - although anybody else is also welcome to create a similar branch (optionally based on this one) and make a new pull request. I'm not directly working on adding MIDI at the moment, but as usual, would be happy to review others' contributions.

[robclouth]

Hey. I won't be continuing on this PR. I got it to where I needed it and put it here in case anyone else found it useful. If anyone wants to continue feel free.

[TylerMclaughlin]

hi @psobot et al.

i'm trying to get @robclouth's MIDI branch working with the example rob posted.

Seems like a holy grail for people like me who want to render midi to audio faster than real time. ( I'm not a pybind programmer so don't get ur hopes up. )

Sadly, i'm getting an error on this branch at the load_plugin phase (upstream of the MIDI input).

'ValueError: Plugin 'Vital' does not accept audio input. It may be an instrument plug-in and not an audio effect processor.

This... would make sense given Vital is a VST MIDI instrument, not an audio effect VST. But I also get the same error for loading a reverb:

load_plugin("/Library/Audio/Plug-Ins/VST3/ValhallaShimmer.vst3")

The issue is similar to #118 , but I'm on Mac OS 10.15, not Windows.

Anyway, these are the steps I used to build Rob's branch on my machine. Maybe I'm doing something wrong??

#basic build, but with the midi branch
git clone --recurse-submodules --shallow-submodules https://github.com/robclouth/pedalboard.git
cd pedalboard
pip3 install pybind11
pip3 install .
# required for this implementation
pip3 install mido
# not sure if needed
git fetch
# necessary to get to the branch with midi support
git switch midi-support

git log confirms i'm at rob's latest 'note on' commit, with SHA 2559a40.

any ideas or workarounds?

[psobot]

@TylerMclaughlin Note that the issue you mentioned (#118) was fixed on the master branch as of July 28th, 2022 (#118 (comment)) - but @robclouth's branch does not include that fix.

I'd recommend either merging from master or rebasing off of master to include the ~6 months of fixes that have happened since then. (Or, more work: starting a new branch from master and opening a new PR. I'm sure @robclouth is not a fan of being pinged over and over about this abandoned PR. 😉)

[psobot]

Closing in favour of #225, which has been merged into Pedalboard v0.7.4.