Transcribe MIDI to tablature automatically / semi-interactively

[1j01]

Scope

For a first pass, audio transcription can be left to other tools that convert audio to MIDI, like Basic Pitch.
That said, Basic Pitch is open source, so it could be integrated directly, eventually (especially if tensorflow.js is supported for in-browser inference; I think it requires a Python backend currently).

Problem Statement

I feel like the fundamental shortcoming of transcription systems is their all-or-nothing/once-off approach to automation. Transcription tools are either are all manual (a digital equivalent of writing sheet music by hand) or they are a one-off process that you can trigger, but which isn't interactive.

With automatic transcription systems that aren't interactive, if the result isn't quite right, you can go back and change settings (if there are any) and try again, but the settings don't necessarily work well across the whole song, especially if it has tempo changes, and if the measures don't line up properly, usually none of the note and rest durations chosen during quantization will make sense.

Also, with fretted instrument transcription, most programs seem to output tablature that is impossible to play, not really taking into consideration hand geometry. In general, a MIDI file may indeed be impossible to play, but, it feels like they're not even trying, idk.

Solution Strategy

• A slider at the top of the fretboard lets you adjust the hand position, to anywhere along the fretboard, and the transcription system will try to find fingerings close to this hand position.
• The hand position can be recorded at many points during the song.
• Capo position, tuning, and tempo changes can also be recorded anywhere in the song.
• (The transcription system should also try to pick fingerings that are easy to play in succession, ideally, but that seems more complicated. For instance, a diagonal arrangement of fingers may be slid to parallel easily, but the opposite diagonal often warrants switching which fingers are used for which string.)
• If it doesn't detect how measures should be broken up right, you can manually move bars around, that separate the measures, and it will recompute the transcription/quantization for just the affected measures. If you adjust tempo change events, it recomputes the bars after the tempo change.
• Tempo changes (and even tuning changes) can be gradual, so support for curves could be useful, but that's pretty advanced. That said, when designing the "change events" system, it may be better to go ahead and design it as a "curve editor", since it would need some sort of timeline anyway.
• When you select a note, you should be able to pick any equivalent note to replace it with, with exact matches on other strings highlighted, and different-octave matches highlighted differently.
• (Ideally, when adjusting settings that affect the automation, changes you've made already should be persisted if possible, such as deleting a note or picking a different fingering for it, but this is pretty complicated in general. For instance, adjusting settings may generate different numbers of notes, so if you adjusted the fingering of a note, where there are now two, should it affect both? Or if there were two and you adjusted one and there's now only one, should the adjustment apply to it, or should it be left, like the originally-unadjusted note?)
  • (It could at least let you know if changes you made will be overwritten.)

[1j01]

Tempo changes (and even tuning changes) can be gradual, so support for curves could be useful, but that's pretty advanced. That said, when designing the "change events" system, it may be better to go ahead and design it as a "curve editor", since it would need some sort of timeline anyway.

If the curve editor UI would be complicated to use enough that a simplified event list UI would be useful to have as well, that negates the notion that implementing a curve editor UI right off the bat would save effort compared to implementing an event list UI first.
That said, I like the mojs curve editor. It would be nice to have an excuse to use it. Although it might not be extensible enough. I'd want labels like note names on some axes for tuning automation, for instance.
Could probably show all the strings in one graph, since they usually shouldn't be tuned the same as each other. (An exception is a 12 string guitar, but that just uses 6 string tablature, with pairs of strings typically tuned in unison.) Which mojs curve editor wouldn't support.
Hm, there's also an experimental mojs-timeline-editor, but probably rolling a new implementation would be the way to go, to fit in with the UI, and undo system (if any), etc.