Integrate new preset parser and rewrite/refactor preset rendering. #716

kblaschke · 2023-04-26T16:25:21Z

First of all, big sorry for creating such a huge PR.

Sadly, just replacing the preset and expression parser with the newly written one wasn't exactly easy, as the old code was highly integrated in basically all major effect classes. It worked on a few assumptions no longer compatible with the new expression compiler, so I had to reorganize code. This lead to the decision to reorganize the whole Milkdrop preset renderer into an overall better structure, removing the "ping pong" way of how the preset parts were rendered previously and replacing it with a more straightforward approach, effectively containing the rendering of a preset into a single function call.

Having to restructure classes, I've also went fully down into the rabbit hole and checked every rendering step for Milkdrop compatibility, fixed issues and added missing code. Also added framebuffer usage to the rendering process for various reasons listed below.

Changes

Here's a list of changes, improvements and other notable things contained in this PR, in no particular order:

Removed all existing preset file parser and expression-related code from libprojectM and replaced it with a new, simpler file parser class and the newly written projectM-eval expression compiler library.
Renamed target "MilkdropPresetFactory" to "MilkdropPreset".
Moved all Milkdrop-specific rendering code to "MilkdropPreset".
Moved and refactored code from "Renderer" and "ShaderEngine" classes into other places, e,g, new classes. This includes blur textures, comp/warp shader code and some other math for generating the composite mesh.
Added lazy texture loading to TextureManager, so it doesn't load all images into VRAM on startup.
Added unloading logic for unused textures, very close to the code used in Milkdrop to save additional VRAM.
Moved warp mesh texture coordinate calculation code to a vertex shader, relieving the CPU of some work.
Properly implemented the motion vector grid effect, was only a simple grid of static points before. Now it properly reverse-propagates the warp mesh "motion" of the last frame and draws lines as it should. In contrast to Milkdrop, the actual motion vector calculation is completely done on the GPU inside the vertex shader, using the U/V coordinate color attachment of the previous frame.
Properly implemented the "Darken Center" effect.
Consolidated code for classic 1.x filter effects like "Darken", "Invert" and "Solarize".
Made preset rendering per-frame aware of external parameter changes like screen resolution and warp mesh size, only reallocating the textures and meshes which actually have changed since the last frame. This makes the process of resizing the window way faster.
Blur textures are now rendered in their own (smaller) framebuffer instead of directly drawing into the main image texture.

Work left for later

Will fix or add the following features and bugs later, so we can get this huge changeset merged ASAP:

Fix "upside-down" rendering issues and, if possible, pass the correct U/V coordinates to the HLSL shaders to prevent math errors.
Implement random texture selection.
Implement simple smooth preset blending using shaders.
Audio-related issues, especially the spectrum values, which are off the charts, causing a multitude of rendering issues in many presets.
Proper framebuffer/render-to-texture support for rendering a frame.
User-configurable blending shaders and options.

kblaschke · 2023-07-24T10:06:55Z

Emscripten version builds again, a crash was fixed and cleanups are done so far.
I'd say the branch is in a good enough state to be merged to master, even if there's some work left to get it into a releasable state.

Please test extensively, review the changes and give your opinions.

revmischa · 2023-08-02T16:57:05Z

src/libprojectM/Audio/BeatDetect.cpp

@@ -64,6 +60,28 @@ auto BeatDetect::GetPCMScale() const noexcept -> float
 }


+auto BeatDetect::GetFrameAudioData() const -> FrameAudioData


my knowledge of modern C++ is very poor. how does data get garbage-collected here? are we returning a struct copy? if so, since this is a hot path, can we return a pointer to a struct that frees itself when there are no more references or something? or is it not a big deal to copy 8 floats around?

Yes, it's an object that's - in theory - copied. The compiler optimizes such return values as implicit moves, so when the code actually runs, no copy is created, but the object instance created in the function is moved to the caller's context without creating another object.

The audio data is only requested once per frame, and changes every frame, so using shared pointers here would just add overhead. The presets will get a const reference of this object, so ideally, only this single instance exists during a render call.

revmischa · 2023-08-02T17:14:20Z

src/libprojectM/MilkdropPreset/BlurTexture.cpp

+        -1.0, -1.0, 0.0, 0.0,
+        1.0, -1.0, 1.0, 0.0,
+        -1.0, 1.0, 0.0, 1.0,
+        1.0, 1.0, 1.0, 1.0};


I had ChatGPT explain this to me

This Vertex Buffer Object (VBO) represents a square (technically, two triangles) that cover the entire screen in Normalized Device Coordinates (NDC), with texture coordinates associated with each vertex.

Here's a breakdown of the values:

The first two values in each row (-1.0, -1.0, 1.0, -1.0, -1.0, 1.0, 1.0, 1.0) represent the x and y coordinates of four vertices in a 2D space. This is a square that covers the entire viewport in NDC, where (-1, -1) is the bottom left corner and (1, 1) is the top right corner.
The third and fourth values in each row (0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 1.0, 1.0) represent the texture coordinates (u and v) associated with each vertex. They range from (0, 0) at the bottom left to (1, 1) at the top right of the texture.
The drawing order of the vertices forms two triangles that together cover the entire viewport. The first triangle is formed by the vertices at (-1,-1), (1,-1), and (-1,1), and the second triangle is formed by the vertices at (-1,1), (1,-1), and (1,1).

That's a very sophisticated explanation of "what is a quad?" 😅

Yeah, the blur textures are rendered as a simple quad, drawn with a specific gaussian blur shader. The quad also doesn't always fill the whole screen (or framebuffer, in this case), only in the first iteration. Subsequent blur steps will use only half or a quarter of the texture to create even more blurry images.

yeah I just didn't get the texture coords part of it

revmischa

great stuff! thank you

revmischa · 2023-08-02T17:33:53Z

src/libprojectM/MilkdropPreset/MilkdropPreset.cpp

+    assert(renderContext.textureManager);
+    m_state.renderContext = renderContext;
+
+    // Initialize variables and code now we have a proper render state.


Suggested change

// Initialize variables and code now we have a proper render state.

// Initialize variables and code now that we have a proper render state.

revmischa · 2023-08-02T17:40:54Z

src/libprojectM/MilkdropPreset/MilkdropPreset.hpp

+public:
+    /**
+     * @brief LoadCode a MilkdropPreset by filename with input and output buffers specified.
+     * @param factory The factory class that created this preset instance.


Suggested change

* @param factory The factory class that created this preset instance.

revmischa · 2023-08-02T17:41:10Z

src/libprojectM/MilkdropPreset/MilkdropPreset.hpp

+    /**
+     * @brief LoadCode a MilkdropPreset from an input stream with input and output buffers specified.
+     * @param presetData an already initialized input stream to read the MilkdropPreset file from
+     * @param presetOutputs initialized and filled with data parsed from a MilkdropPreset


Suggested change

* @param presetOutputs initialized and filled with data parsed from a MilkdropPreset

revmischa · 2023-08-02T17:45:29Z

src/libprojectM/MilkdropPreset/MotionVectors.hpp

+/**
+ * @brief Draws a flexible motion vector field.
+ *
+ * This is broken right now, as it only renders a relatively static 1px point grid, with no apparent motion trails.


is this still true?

No, not anymore. It's now properly implemented, I perform it purely on the GPU using a second color attachment and a vertex shader. Some presets like "Radar" and many others now actually look as they should.

JohannesKauffmann · 2023-08-03T19:27:09Z

First of all, big sorry for creating such a huge PR.

Forgive me if this may come across as bikeshedding, but some CMake parts of bf344b6 (the PROJECTM_STATIC_DEFINE parts) could be extracted to make this PR a bit smaller :)

kblaschke · 2023-09-09T19:00:23Z

Replaced the color masking with attaching/detaching the U/V texture for use with motion vectors, as GLES only supports per-buffer color masking with v3.2.

kblaschke · 2023-09-10T20:35:11Z

First of all, big sorry for creating such a huge PR.

Forgive me if this may come across as bikeshedding, but some CMake parts of bf344b6 (the PROJECTM_STATIC_DEFINE parts) could be extracted to make this PR a bit smaller :)

What exactly are you proposing there? Can't remember that I've done anything with the linking stuff, besides additing the evaluation library. Also unsure how this would make the PR substantially smaller, as the overwhelming part was rewriting code in the Renderer and MilkdropPreset[Factory] targets.

If we can make the static linking part easier, that's definitely a good thing to do. I'd do that separately though, as this PR already changes enough things, or better said, too many to even properly review. Could you create an issue please, detailing the proposed changes? This way I won't forget about that!

kblaschke · 2023-09-10T20:51:22Z

I'd personally say this changeset is as good as it gets without adding even more stuff to it, so I'd like anyone to pick up the branch, build and test it for stability. I've not seen any serious issues besides a few remaining rendering problems noted in the description, but these will be addressed later. Overall, it's a huge improvement over the current master, visually and performance-wise.

Now rendering everything upside down to make up for the difference in UV coordinates in HLSL shaders.

Should fix visuals in about half of the presets.

Main issue was a wrong -, replaced with + in the rotation calculation. Plus we shouldn't negate the Y pos, as this may mess up UV calculations done in HLSL.

Idle preset doesn't render the same way as before, as it was presumably adapted to projectM's previous math. It's now a bit more complex, with the M tilting, a sine waveform at the bottom and some other tweaks. The filter mesh was broken, so the effects didn't render at all.

…t more.

…oved ShaderEngine class. Milkdrop shaders are now read from .vert/.frag and .inc files, then copied into a generated code file by CMake. Makes editing easier, as many IDEs can properly syntax-highlight the shader code. The class was also moved into MilkdropPreset, as it's the only preset type which will ever make use of those shaders. Deleted the now fully "dissolved" ShaderEngine class.

… failed. Another option would be skipping the rendering if there's no active preset instead of displaying the idle preset.

…uffer.

…attachments and remove existing ones.

Only GLES 3.2 supports glColorMaski, which would exclude any platforms only supporting GLES 3.0 or 3.1, so we simply change the color attachment accordingly. This change also improves VRAM usage, as now there's only a single u/v texture used for both framebuffers as required.

kblaschke · 2023-09-11T08:21:17Z

FYI, last push was a rebase on latest master to prepare for merging.

JohannesKauffmann · 2023-09-11T19:24:13Z

What exactly are you proposing there?

The commit A few Windows build fixes from this PR changed both CMakeLists.txt such that PROJECTM_STATIC_DEFINE is defined for a wider range/scope of targets. My proposal was to move these changes specifically to a separate commit/PR, as these CMakeLists.txt changes didn't seem related to the new preset parser/rendering rewrite, but would apply more globally. It'd make this PR a bit smaller (not much smaller admittedly).

Not sure it is worth changing now, however. PR looks good either way.

kblaschke · 2023-09-11T21:13:06Z

Ah, I see. Yes, I had to do this because otherwise the class definitions in the PCM class headers were different between the compile units, which the MSVC compiler doesn't like (different declspec attributes), causing linker errors. The MilkdropPreset target uses the PCM class to retrieve the frame audio data, and I didn't also want to refactor this class structure as it needs more attention later due to the broken/high spectrum and beat detection values. Could've done this in an earlier PR, but it's really just a small fix for a typo and adding missing defines to other targets. Proper cleanup will be done in issue #708.

kblaschke · 2023-09-15T17:55:21Z

Merging now, as the current branch state is stable enough for most uses. Emscripten might still have some issues left on top of the other things on the to-do list, so @Blaquewithaq please create an issue if there are any problems running on Emscripten, I'll tackle these in a later fix.

kblaschke mentioned this pull request Apr 27, 2023

Texture loading: projectM loads all textures/images into VRAM on startup #498

Closed

kblaschke linked an issue Apr 27, 2023 that may be closed by this pull request

Fails to build with -DENABLE_SYSTEM_GLM=ON: fatal error: 'omptl/omptl' file not found #713

Closed

kblaschke self-assigned this Apr 27, 2023

kblaschke added this to the 4.1 milestone Apr 27, 2023

kblaschke added the important label Apr 27, 2023

kblaschke force-pushed the rewrite-preset-parser branch 3 times, most recently from 648f8ca to b08a1ae Compare April 29, 2023 13:58

kblaschke mentioned this pull request Jul 19, 2023

Visitor pattern in renderitem implemented in waveform #719

Closed

kblaschke force-pushed the rewrite-preset-parser branch from daf7bff to e4cda2b Compare July 24, 2023 09:55

kblaschke changed the title ~~[DRAFT] Integrate new preset parser and rewrite/refactor preset rendering.~~ Integrate new preset parser and rewrite/refactor preset rendering. Jul 24, 2023

kblaschke marked this pull request as ready for review July 24, 2023 10:03

kblaschke requested review from revmischa and Blaquewithaq July 24, 2023 10:07

revmischa reviewed Aug 2, 2023

View reviewed changes

revmischa approved these changes Aug 2, 2023

View reviewed changes

Blaquewithaq approved these changes Sep 10, 2023

View reviewed changes

hartwork mentioned this pull request Sep 10, 2023

[master] Class attributes ignored in compilation? #701

Closed

kblaschke and others added 23 commits September 11, 2023 10:19

Fill both spectrum channels with data.

df7cd08

Finished final composite implementation, more fixes and cleanups.

0ac9fe2

Now rendering everything upside down to make up for the difference in UV coordinates in HLSL shaders.

Fix uv_orig coordinates in warp shader.

2707a1d

Should fix visuals in about half of the presets.

Fixed more UV coordinate issues in the warp shader.

75fe0f0

Main issue was a wrong -, replaced with + in the rotation calculation. Plus we shouldn't negate the Y pos, as this may mess up UV calculations done in HLSL.

Added projectM-Eval as submodule, flipped rendering stuff around a bi…

4b3226a

…t more.

A few Windows build fixes

feb7a33

Add omptl to Renderer linked library list.

ef62ddf

Check out submodules (for projectM-eval) in GitHub Actions workflows.

d2780c5

Renamed simple textured/untextured shaders to be more readable.

e84acbc

Filename fix in comment.

486c86c

Deleted lots of unused files and code.

e9438ee

Check active preset after the preset switch event, as this might have…

b07163d

… failed. Another option would be skipping the rendering if there's no active preset instead of displaying the idle preset.

Fix two Emscripten/GLES compile issues.

a294b8a

Removed a few unused files.

1c93302

Use glDrawBuffers instead of glDrawBuffer with GLES.

b7572ea

Added missing float precision specifiers in vertex/fragment shaders.

494718d

GLES: Use GL_BACK instead of GL_COLOR_ATTACHMENT0 with default frameb…

b05a8cc

…uffer.

Added methods to Framebuffer class to add externally created texture …

561ea8c

…attachments and remove existing ones.

Restore framebuffer bindings after modifying texture attachments.

48b8d6a

Properly initialize u/v texture size for motion vectors.

c08de98

kblaschke force-pushed the rewrite-preset-parser branch from 1403e62 to c08de98 Compare September 11, 2023 08:20

kblaschke merged commit c08de98 into projectM-visualizer:master Sep 15, 2023

kblaschke deleted the rewrite-preset-parser branch September 15, 2023 18:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrate new preset parser and rewrite/refactor preset rendering. #716

Integrate new preset parser and rewrite/refactor preset rendering. #716

kblaschke commented Apr 26, 2023 •

edited

Loading

kblaschke commented Jul 24, 2023

revmischa Aug 2, 2023

kblaschke Aug 2, 2023 •

edited

Loading

revmischa Aug 2, 2023

kblaschke Aug 2, 2023

revmischa Aug 2, 2023

revmischa left a comment

revmischa Aug 2, 2023

revmischa Aug 2, 2023

revmischa Aug 2, 2023

revmischa Aug 2, 2023

kblaschke Aug 2, 2023

JohannesKauffmann commented Aug 3, 2023

kblaschke commented Sep 9, 2023

kblaschke commented Sep 10, 2023 •

edited

Loading

kblaschke commented Sep 10, 2023

kblaschke commented Sep 11, 2023

JohannesKauffmann commented Sep 11, 2023

kblaschke commented Sep 11, 2023

kblaschke commented Sep 15, 2023

		@@ -64,6 +60,28 @@ auto BeatDetect::GetPCMScale() const noexcept -> float
		}


		auto BeatDetect::GetFrameAudioData() const -> FrameAudioData

	// Initialize variables and code now we have a proper render state.
	// Initialize variables and code now that we have a proper render state.

Integrate new preset parser and rewrite/refactor preset rendering. #716

Integrate new preset parser and rewrite/refactor preset rendering. #716

Conversation

kblaschke commented Apr 26, 2023 • edited Loading

Changes

Work left for later

kblaschke commented Jul 24, 2023

Choose a reason for hiding this comment

kblaschke Aug 2, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

revmischa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JohannesKauffmann commented Aug 3, 2023

kblaschke commented Sep 9, 2023

kblaschke commented Sep 10, 2023 • edited Loading

kblaschke commented Sep 10, 2023

kblaschke commented Sep 11, 2023

JohannesKauffmann commented Sep 11, 2023

kblaschke commented Sep 11, 2023

kblaschke commented Sep 15, 2023

kblaschke commented Apr 26, 2023 •

edited

Loading

kblaschke Aug 2, 2023 •

edited

Loading

kblaschke commented Sep 10, 2023 •

edited

Loading