Integrating projectM in strawberry

[jonaski]

I'm replying to this: https://forum.strawberrymusicplayer.org/topic/1286/reintegrate-projectm-visualizer/13

I've started porting the old projectm code from Clementine here:
strawberrymusicplayer/strawberry#1467
One major issue is that clementine was using the deprecated QGLWidget class, and with the new QOpenGLWidget class only some of the projectm visualizations works.

I've also tried to port to projectm version 4 with the C API, but I don't see any replacement for projectM_resetGL which is used here:
https://github.com/strawberrymusicplayer/strawberry/blob/5a4c00a30093883d9cc3c2e8e0564e5408c8aae1/src/visualisations/projectmvisualisation.cpp#L212

[kblaschke]

The name of the projectM_resetGL function wasn't really a great choice, as it didn't "reset" anything. It just changed the viewport size. Thus, the replacement in the new API is projectm_set_window_size(), which takes the same parameters.

In the current master branch, there's a new render-to-FBO method which might work a bit better with Qt, but I was never successful in getting Qt and projectM to work well together. There are no OpenGL errors, but some objects projectM allocates in OpenGL (VBOs, FBOs and textures for example) randomly vanish after the first frame, as if the GL API forgets about them. I've checked different things like using the same thread, making the Qt GL context current manually everywhere, late-initializing projectM in the first draw call and not in the setup method, plus other things, but the GL objects still went missing. I haven't seen this behaviour with any other OpenGL integration - just in Qt. So this might be some specialty within Qt's OpenGL wrapper that may cause this. Or some mismatch between the actual GL functions called from Qt and projectM.

The only way of getting projectM to work is using a plain QOpenGLWindow, and drawing directly into the native surface. This makes it harder to integrate projectM into any UI though.

If you can find out anything that goes beyond that - please let us know! I'm highly interested to find out what exactly breaks projectM within Qt.

[jonaski]

Here is how it looks, just to confirm we are talking about the same thing:

[kblaschke]

Depending on what you're drawing, you get different outputs, but basically all are broken.
I was thinking that it's more a Qt bug than anything in projectM's renderer, which isn't perfect, but works everywhere else including GLES platforms.

We might just be forced to use QOpenGlWindow instead, drawing directly into the window. Not great if you want to render widgets as well though...

[jonaski]

I was thinking of opening a Qt bug report with my test application as a reproducer, but you might be more qualified to provide the needed information.

[kblaschke]

I can at least double-check if there's really no issue on the projectM side or if we've missing something in Qt, e.g. properly configuring the GL context. I certainly was too lazy to check all GL function return values for errors, so that's what I'm going to add in some places. RenderDoc lists all errors as well, so for testing that was enough.

[kblaschke]

Found several forum posts about the same issue, e.g.:

• https://forum.qt.io/topic/64260/using-qopenglwidget-without-using-qopenglfunctions/3
• https://www.qtcentre.org/threads/71155-Raw-OpenGL-not-working-in-QOpenGLWidget-with-QT-5-11

So it's a rather long story of "doesn't work, and nobody has answer" - well, except for using Qt's GL functions exclusively, which of course is a no-go for projectM. Not sure if a bug report would change anything - looks like they did that by design.

[kblaschke]

Regarding the linked discussion about projectM in Linux distros: our main issue is that there are no package maintainers for Debian-based distros who could update it. Ubuntu for example still ships projectM 2.1.0 in their latest 24.04 release, which is an ancient version. Debian updated to 3.1.12 in Debian 12, so there seems to be someone who cared about it. Here's a nice overview of projectM versions in all major distros.

The old 3.1.12 release has many unfixed crashes and is visually far behind the current 4.1.x release of libprojectM, is heavier on the CPU and can't even run a major part of the available Milkdrop presets. That said, I'd strongly recommend using the current release, and use it as some kind of leverage to push distro maintainers to update the package (or at least add a new libprojectm package, as the old projectm one included two or three different frontend applications as well).

[jonaski]

I'm still having no luck with ProjectM4, it's not showing anything.
Any idea what's wrong? I'm calling projectm_opengl_render_frame() here: https://github.com/strawberrymusicplayer/strawberry/blob/6c93bc39afc0b61a9fb899e442b118064dc1f478/src/visualizations/projectmvisualization.cpp#L193

[kblaschke]

I'll have a look at it if I've got some time to spare! Maybe RenderDoc gives some helpful clues.

[kblaschke]

Built your minimal example, and Qt is definitely acting weird. glGetError() returns 1280 (Invalid enum) even before entering paintGL(), but only from the second frame onwards. According to RenderDoc, the error happens when Qt calls glClear(), but the pipeline state looks okay, it uses the default FBO and has no buffer objects bound.

Since so many people report issues with this widget, I'd suggest not bothering with it, but using QOpenGLWindow instead and draw into this one. If you really need to display the results inside another widget, even QOpenGLWidget, you may have more luck creating a shared GL context between both, render projectM into the hidden window and then use Qt's GL drawing functions to copy over the native window surface into the widget, e.g. via glBlitFramebuffer or a simple draw call. Maybe QPainter can also perform this operation.

[jonaski]

Using QOpenGLWindow doesn't change much for me as far as I can tell. With projectm version 4 most of the presets are blank, only a few works. With projectm version 3 most show animations, some bogus, some blank.
I used the same test application, made VisualizationOpenGLWidget inherit QOpenGLWindow, in VisualizationContainer I removed setCentralWidget and added:

  QOpenGLWindow *visualizations = new VisualizationOpenGLWidget(projectm_visualization_);
  visualizations->show();

[kblaschke]

You'll have to configure the GL profile via the QSurface properly to a 3.3 core profile before showing the window. That could fix some issues. At normally creates a 1.5 compatibility profile or so by default, something that's supported basically everywhere.

[jonaski]

I tried that, still no luck.

[kblaschke]

Fixed a bug in master (see PR #827) causing the GL_INVALID_ENUM error. That might have caused some problems in Qt's rendering, so could be worth a try and check again with the fix applied.

[guprobr]

Hello,

I'm really trying hard to help integrate this for almost two months.

But now I'm more than completely lost, because it does not make any sense. After a lot of effort I've created a class to spawn an SDL2 OpenGL 3.3 core profile window, put all components like the buffer consumer, interface etc, in the same class and thread, trying to follow the sdl2 frontend specs at all times.

The "VisualizerConteiner" from Strawberry instantiates this "VisualizationSDL2" class, which is kinda like a mini-sdl2 frontend with all components inside. I made this way to guarantee no race conditions or OpenGL context issues, etc.

• ConsumeBuffer works fine, I can notice the interaction of the songs with the graphics, and if you stop the song, interaction stops, so this is not the issue.
• On the graphical point of view, the way I see it, SDL2 should have worked or at least display different results! But thats not what happens thats why I'm asking for a little orientation here, please.

Certainly I must be missing something, because the results are the same as with several other attempts I made with QOpenGLWindow:

• majority of corrupted presets and/or black screen;
• The behaviour is the same for all kinds of attempts, the same presets show the same results etc.
• Some seem to render the first frames fine but then they fade to black. :(

So this is not a QT bug, its something else we are missing, because SDL2 also does not work. Here is the code, sorry for posting here, but its not big at all. Its a little mess because I was constantly debugging and changing stuff..... I would really appreciate if anyone had the time to look into it just to figure something I cannot understand because of my own limitations. TKS

#ifndef VISUALIZATIONSDL2_H
#define VISUALIZATIONSDL2_H

#include <QObject>
#include <SDL2/SDL.h>
#include <SDL2/SDL_opengl.h>
#include <projectM-4/projectM.h>
#include <projectM-4/playlist.h>
#include <QThread>
#include <QMutex>
#include <gst/gst.h>
#include "engine/gstbufferconsumer.h"

class VisualizationSDL2 : public QObject, public GstBufferConsumer {
    Q_OBJECT

public:
    VisualizationSDL2();
    ~VisualizationSDL2();

    void initialize();
    void renderFrame();
    void resize(int width, int height);
    void start(); // Starts the event loop in a separate thread
    void Init(size_t w, size_t h);
    void handleKeyPress(SDL_Keycode);

    projectm_handle projectm_instance_ = nullptr;
    projectm_playlist_handle projectm_playlist_instance_;

    void ConsumeBuffer(GstBuffer *buffer, const int pipeline_id, const QString &format) override;


private:
    void run();  // Event loop, now private since it's managed by a thread
    SDL_Window* window;
    SDL_Renderer* sdlRenderer;
    SDL_GLContext glContext;
    bool running;
    QThread* sdlThread;  // Thread for SDL2 event loop
};

#endif

#include <projectM-4/projectM.h>
#include <projectM-4/playlist.h>

#include <SDL2/SDL.h>
#include <SDL2/SDL_opengl.h>
#include <QDebug>
#include <QThread>
#include <QMutex>
#include <QTimer>

#include "visualizationopenglwidget.h"
#include "engine/gstbufferconsumer.h"

QMutex renderMutex;  // Global mutex for rendering

VisualizationSDL2::VisualizationSDL2()
    : window(nullptr), glContext(nullptr), running(true), sdlThread(nullptr) { 
    qDebug() << "VisualizationSDL2 init";
    start();
    qDebug() << "VisualizationSDL2 started";
}

VisualizationSDL2::~VisualizationSDL2() {
    running = false;

    if (sdlThread) {
        sdlThread->quit();
        sdlThread->wait();
        delete sdlThread;
    }

    if (glContext) {
        SDL_GL_DeleteContext(glContext);
    }
    if (window) {
        SDL_DestroyWindow(window);
    }
    SDL_Quit();
}

void VisualizationSDL2::run() {
    qDebug() << "VisualizationSDL2::run() - Running in thread" << QThread::currentThread();

    // Initialize SDL and create OpenGL context
    if (SDL_Init(SDL_INIT_VIDEO) < 0) {
        qDebug() << "SDL could not initialize! SDL_Error:" << SDL_GetError();
        return;
    }

    SDL_GL_SetAttribute(SDL_GL_CONTEXT_MAJOR_VERSION, 3);
    SDL_GL_SetAttribute(SDL_GL_CONTEXT_MINOR_VERSION, 3);
    SDL_GL_SetAttribute(SDL_GL_CONTEXT_PROFILE_MASK, SDL_GL_CONTEXT_PROFILE_CORE);

    window = SDL_CreateWindow("OpenGL Window", SDL_WINDOWPOS_CENTERED, SDL_WINDOWPOS_CENTERED, 800, 600, SDL_WINDOW_OPENGL | SDL_WINDOW_RESIZABLE | SDL_WINDOW_SHOWN |  SDL_WINDOW_ALLOW_HIGHDPI );
    if (!window) {
        qDebug() << "Window could not be created! SDL_Error:" << SDL_GetError();
        return;
    }
    qDebug() << "SDL window created successfully";

    glContext = SDL_GL_CreateContext(window);
    if (!glContext) {
        qDebug() << "OpenGL context could not be created! SDL_Error:" << SDL_GetError();
        return;
    }
    qDebug() << "OpenGL context created successfully";

    if (SDL_GL_MakeCurrent(window, glContext) != 0) {
        qDebug() << "Failed to make OpenGL context current! SDL_Error:" << SDL_GetError();
        return;
    }
    qDebug() << "OpenGL context made current";

    ////auto glewError = glewInit();

    // Initialize ProjectM
    Init(800, 600);
    qDebug() << "ProjectM initialized";

    // Main loop
    bool isRunning = true;
    SDL_Event e;

    while (isRunning) {
        while (SDL_PollEvent(&e) != 0) {
            if (e.type == SDL_QUIT) {
                isRunning = false;
            } else if (e.type == SDL_KEYDOWN) {
                handleKeyPress(e.key.keysym.sym);
            }
        }

        // Render the frame
        renderFrame();

        // Sleep for a short period to prevent high CPU usage
        QThread::msleep(16); // Approx. 60 FPS
    }
    qDebug() << "SDL2 main loop exited";
}

void VisualizationSDL2::renderFrame() {
    QMutexLocker locker(&renderMutex);

    if (SDL_GL_MakeCurrent(window, glContext) != 0) {
        qDebug() << "Failed to make OpenGL context current during renderFrame! SDL_Error:" << SDL_GetError();
        return;
    }

    if (projectm_instance_) {
        projectm_opengl_render_frame(projectm_instance_);
        SDL_GL_SwapWindow(window);
    }
}

void VisualizationSDL2::start() {
    qDebug() << "VisualizationSDL2::start() - Starting thread";

    if (sdlThread != nullptr) {
        qDebug() << "VisualizationSDL2::start() - Thread already running!";
        return;
    }

    sdlThread = new QThread();
    this->moveToThread(sdlThread);

    connect(sdlThread, &QThread::started, this, &VisualizationSDL2::run);
    connect(sdlThread, &QThread::finished, sdlThread, &QThread::deleteLater);
    
    sdlThread->start();
}

void VisualizationSDL2::resize(int width, int height) {
    SDL_SetWindowSize(window, width, height);
    if (projectm_instance_) {
        projectm_set_window_size(projectm_instance_, width, height);
    }
}

void VisualizationSDL2::handleKeyPress(SDL_Keycode key) {
    switch (key) {
        case SDLK_n:  // Handle 'N' key press
            qDebug() << "Handle NEXT key press";
            projectm_playlist_play_next(projectm_playlist_instance_, true);
            break;
        case SDLK_p:  // Handle 'P' key press
            qDebug() << "Handle PREVIOUS key press";
            projectm_playlist_play_previous(projectm_playlist_instance_, true);
            break;
        case SDLK_l:  // Handle 'L' key press

            if ( !projectm_get_preset_locked(projectm_instance_) ) {
                qDebug() << "LOCKED preset";
                projectm_set_preset_locked(projectm_instance_, true);
            }
            else {
                qDebug() << "unLOCKED preset";
                projectm_set_preset_locked(projectm_instance_, false);
            }
        break;
        default:
            break;  // Other keys can be handled here if needed
    }
}

void VisualizationSDL2::Init(size_t w, size_t h) {
    if (projectm_instance_) {
        return;
    }

    Q_ASSERT(projectm_instance_ == nullptr);
    Q_ASSERT(projectm_playlist_instance_ == nullptr);
    projectm_instance_ = projectm_create();
        
    projectm_set_window_size(projectm_instance_, w, h);
    projectm_set_fps(projectm_instance_, 60);
    projectm_set_mesh_size(projectm_instance_, 220, 125);
    projectm_set_aspect_correction(projectm_instance_, false);
    projectm_set_preset_locked(projectm_instance_, false);
    
    projectm_set_preset_duration(projectm_instance_, 10.0);
    projectm_set_soft_cut_duration(projectm_instance_, 3.0);
    projectm_set_hard_cut_enabled(projectm_instance_, false);
    projectm_set_hard_cut_duration(projectm_instance_, 20.0);
    projectm_set_hard_cut_sensitivity(projectm_instance_, 1.0);
    projectm_set_beat_sensitivity(projectm_instance_, 1.0);

    const char *texture_search_paths[] = { "/usr/local/share/projectM/textures" };
    projectm_set_texture_search_paths(projectm_instance_, texture_search_paths, 1); 

    projectm_playlist_instance_ = projectm_playlist_create(projectm_instance_);
    projectm_playlist_add_path(projectm_playlist_instance_, "/usr/share/projectM/presets", true, true);

    projectm_playlist_set_shuffle(projectm_playlist_instance_, false);
}

void VisualizationSDL2::ConsumeBuffer(GstBuffer *buffer, const int pipeline_id, const QString &format) {
    Q_UNUSED(pipeline_id);
    Q_UNUSED(format);

    GstMapInfo map;
    gst_buffer_map(buffer, &map, GST_MAP_READ);

    if (projectm_instance_) {
        unsigned int samples_per_channel = map.size / sizeof(size_t) / 2;
        projectm_pcm_add_int16(projectm_instance_, reinterpret_cast<const int16_t*>(map.data), samples_per_channel, PROJECTM_STEREO);
    }

    gst_buffer_unmap(buffer, &map);
    gst_buffer_unref(buffer);
}

[kblaschke]

@guprobr I don't see a main() or a QApplication anywhere in your code, but you are using Qt classes, wouldn't it make sense to just try SDL without Qt? How are you sure the problem is not Qt-specific, do you start just a QCoreApplication or a full QApplication or QGuiApplication? Also, as far as I know ProjectM should show animations even when not fed any data, or am I wrong?

To clear that up, it's a plug-in for the Strawberry music player (which is a fork of Clementine, which in turn once was a fork of Amarok 1.4), so the actual application just loads these classes on demand.

Ideally, the visualizer would also use Qt-based windows and classes, but Qt changed their OpenGL implementation at some point which causes all kinds of issues if you don't use the Qt-provided GL function pointers, like resources (VBOs, textures etc.) getting lost. We'd probably have to build a Qt-specific version of projectM for this, which would be kinda complicated and not really make it reusable, e.g. installed as a system-wide shared library on Linux. I've not thoroughly checked it, but I suspect that Qt is using EGL instead of GLX for rendering on Linux, and it uses ANGLE on Windows - both of these backends are currently not supported.

It gets even more complicated if a Qt app is configured to use Vulkan as the 3D backend (or Metal on macOS).

As as long as the visualizer doesn't have to be embedded into the main window and no Qt UI overlay is needed, this window can just use SDL2 to create the GL context. In this case, the OpenGL functions used by projectM will work fine.

[guprobr]

Exactly. That one I put on visualizations2 of Strawberry branch I believe, @jonaski
But I can make a small Qt program too that spawns the SDL2 window. The idea was to bypass any QQpenGL stuff and let SDL2 handle GL stuff for ProjectM, but it didnt work.

[jonaski]

Maybe we need to create a context using glXCreateNewContext (https://registry.khronos.org/OpenGL-Refpages/gl2.1/xhtml/glXCreateNewContext.xml), use that in QGLXContext::fromNative() (https://doc.qt.io/qt-6/qnativeinterface-qglxcontext.html#fromNative) (assuming it's on X11), then in the QOpenGLWidget or QOpenGLWindow class, we use that context like context->makeCurrent(this);?

[guprobr]

nice job! it makes sense given the issue. I was avoiding to go into GLX because of portability.. also, BTW, it actually occurred to me now, that we only tested on Linux. What could be the behaviour on Windows with the regular approach? :D

[kblaschke]

From my experience, using OpenGL on Windows is actually easier, as there's basically only one GL desktop implementation one could use. projectM is using GLEW to look up the GL function pointers, so they should end up being the same ones as those used by Qt. On Linux, this isn't guaranteed though, as multiple GL vendor libraries and APIs might be available (GLX, EGL, GLES) which export the same functions, but are located in different libraries - mixing GL calls from different APIs, e.g. EGL with GLX, will cause issues as the allocated resource handles aren't compatible etc.

This issue lives on the back of my head for a long time now, especially with Wayland the GLX/EGL problems only get worse. Currently on Linux, libprojectM uses build-time linking for GL functions, linking either against GLES2/3 or the vendor (MESA) GLX library. Since projectM doesn't really care which GL function is actually called (despite a few actual API differences between desktop and embedded GL APIs), we could -optionally- leave the GL function pointer resolving to the embedding application using a callback. Like, if the callback is set, the application must respond with a valid (non-null) pointer for each function requested to work. This would happen when a projectM instance is being created. Without a callback, libprojectM would then default to some built-in strategy, e.g. trying to detect the type of context it works in (checking for being within an EGL context or not via eglQueryContext()) and try loading the proper library and functions for this API.

These changes will require some major refactoring of the renderer, moving all GL function calls into wrapper classes. Luckily, this is something we're already planning anyways, as we want to support other rendering APIs like Vulkan and Metal natively in a future (5.0) release of libprojectM. Having this in the current major version to add EGL support is certainly possible though, but probably happening in libprojectM 4.3 or 4.4 at the earliest. Vulkan and Metal both are easier to handle in this regard, as Vulkan comes with its own loader library and Metal is an OS-native API Apple (hopefully) cares for in regards of compatibility.