Incorporating neural networks in the material

[tombackstrom]

I'd like to start a major improvement task of importing baselines of neural networks in all applicable sections. Before starting, I'd like to have opinions on how to best include these models.
Some of my own preferences:

• I would like to easily update any models when the state-of-the-art changes.
• I would like to be able to train the models myself. That is, no giant models.
• Training of models should not be triggered when compiling the book! Training can take hours, days or weeks, and we'd have many neural models, which leads to much too much for compilation.
• Models should be pedagogical in the sense that we can readily understand how they work.

The main consequence of these preferences is that I think that each neural model should be in a separate repository. The idea is that the notebooks in the book would then load the external, pre-trained model for demonstrations. This leads to a follow-up question; should we have a single repo for all models in the book, or have a single repo for every model? I'm slightly in favour of the latter approach since this would make it easier to track library requirements and when the state-of-the-art changes, we need to keep track only of the current ones, not all of history.

Another question is whether we should primarily use our own implementations, or try to incorporate existing pre-trained models.

Especially hoping for comments from @orasanen

[orasanen]

I think the general idea sounds laudable, but I see some challenges.

In my view, a basic problem is that state-of-the-art does not necessarily mean pedagogical, and current or future state-of-the-art models might not work particularly well on manageable data sizes. Moreover, even a moderately ok performing neural network might not be very useful for understanding the phenomenon or the problem, but would still just be a bunch of layers and a loss function. I don't know which direction you want to take the book on the long run, but I would seriously consider

1. having DSP and basic ML examples as embedded code,
2. having more complex systems with their readings and source codes behind links.
   If you want to have state of the art systems in the book just to demonstrate their performance, importing models from HuggingFace is probably the easiest way to go. However, viewing and understanding the code of these systems is way beyond textbook level both from the content and reader needs perspective (IMHO), not least because there's a lot of code and dependencies that are particular to the HuggingFace ecosystem (or whatever code base and libraries are used). In my experience, even PhD students with ML background prefer to re-implement modern algorithms themselves instead of trying to understand implementations of others', or at least it takes days or weeks for them to get acquinted with a new model and its associated scripts.

Also, I have no idea how to do this in a practical way, and I'm not familiar with the way how the current code execution of the book platform even works (i.e., is it locally when you read the book, locally when you compile the repository as a book for viewing, or on some server in either of the two cases?).

[tombackstrom]

Perhaps it does not have to be an either-or but both. I do agree that we should present very pedagogical and basic models which give reasonable output. We should be able to train ourselves. I was thinking that we could then also have some advanced models as a comparison. Perhaps it can be just a link to a website. If possible, we should try to run the models ourselves to be able to compare them with the baseline.

This leads to a format where we have simple models trained ourselves.

As background, Jupyter always needs a server somewhere. I typically use my desktop as the server. It's easy to set up. The compilation with jupyterbook is then a wrapper for all the individual Jupyter notebooks. When starting compilation from scratch, each notebook would then be run through (on the server) and the output goes to html-format. All the outputs are then collected and glued together into a book. JupyterBook does this automatically, and if you download the git, then this should be straightforward. There is an option that only changed notebooks are compiled, which makes compilation much faster. This means that in the worst case, all notebooks will be compiled sequentially. This takes quite a bit of time already now. If we would add training of modules in the loop, then the compilation time will increase dramatically.

[tombackstrom]

Here is a simple GRU-based speech enhancement module that could fit the format of NN modules that we want to have in terms of complexity and pedagogy. https://jmvalin.ca/demo/rnnoise/
Jean-Marc's stuff in general might be good starting points.