Add interactive music support #64488
Conversation
|
A new PROPERTY_HINT_FLAG_LIST? Where is it used on in here? |
|
@Mickeon Its used for the transitions |
|
After this is merged, it would be nice to add an |
|
First few findings (will test more) Case 1 Case 2 Proposals
|
|
Case 3 Case 4 Proposal 1 Proposal 2 NOTE
|
|
Case 5 Case 6 Proposal 3
So in case name is not adjusted by user, re-generate clip name using audio file name. |
|
Case 7 (might also be a security feature to avoid breaking everything.)
will test more another day…Until then, what does |
|
@MJacred thanks so much for testing! Transitions have the hold option so they remember which clip they come from. Return to hold just returns to that clip. This is useful for when you have the main theme playing and you want to transition to an incidental music clip, battle clip, etc. then go back to wherever you were before. The PreviousPosition option also helps to return to the previous saved state in case you want to continue from where it was (very common for the main theme). |
|
Glad to be of help and thanks for the explanation! Will check it out another day and do more testing 😪 💤 Ah, one last thing today: Another feature I really missed in the Inspector: the current TIME, i.e. the progress in seconds:milliseconds of the currently playing clip(s). A must have, especially if you add a EDIT2 |
|
Will it be available on 3.5? |
No, as this PR relies on backwards-incompatible changes to AudioServer. |
|
Is there any way to build this for Windows? I've been trying since the pull request came out and it consistently returns Error 1 in the last stage of compilation. |
|
@IntangibleMatter This pull request seems to be a work in progress right now. If you look at the checks it's failing for almost all platforms. |
|
@reduz I provide some feedback. Is it possible to set a property like Global BPM in AudioStreamInteractive and set a switching point for such like every bar or every 4 bars? Especially in music with a clear pattern of beats, there are several points suitable for switching, like in a DJ Mix. It is better to keep the current Stream until the point after the switching command, and switch when the point is reached.
There is already exist as AudioStreamRondomizer. |
The time signature / bpm can be different per audio file, so having one for all of them would be detrimental in other cases… so this would be a new
|
|
proposal 5: mass loader mass-loader.mp4 |
|
case 8: Auto Advance
|
I honestly think what a good approach would be (that I've been hesitating to suggest because it would require changing some of the fundamental code of this pr, as far as I can tell) would be to be able to set what multiple of beats/bars is transitions on. I.E. a song could be able to transition on every OTHER bar because it would better match the pattern of the song. |
|
case 9:
click This happens only on FADE_IN and FADE_CROSS What happens: stream continues playing and reduces volume dB to reeeeally quiet. no error/warning BUT! If you do a step 3 and switch back from clip 1 to 0, the volume dB is reset. This does only work if step 3 is a switch back to the clip where volume dB was working correctly. Switching to any other clip and then sometime back to clip 0 (where volume dB was working correctly at play start), the volume dB won't be fixed. EXCEPT you let it play on, wait for its auto-advance to trigger to another clip, then it suddenly plays volume dB correctly again. Confusing, I know.
|
|
case 10:
this way does NOT work, see code highlighted below in my review:
Might have found root cause, see code review But it works, if you immediately manually switch from clip 0 to 1. I foresee loads of misunderstandings concerning
Will continue testing in the coming days…still leftover
|
|
|
||
| protected: | ||
| static void _bind_methods(); | ||
| void _validate_property(PropertyInfo &property) const; |
There was a problem hiding this comment.
| void _validate_property(PropertyInfo &property) const; | |
| void _validate_property(PropertyInfo &p_property) const; |
(This isn't virtual, it's internal)
|
|
||
| protected: | ||
| static void _bind_methods(); | ||
| void _validate_property(PropertyInfo &property) const; |
There was a problem hiding this comment.
| void _validate_property(PropertyInfo &property) const; | |
| void _validate_property(PropertyInfo &p_property) const; |
modules/interactive_music/editor/audio_stream_interactive_editor_plugin.cpp
Outdated
Show resolved
Hide resolved
|
Thanks for checking it out! will give it a last spin tonight so we can merge tomorrow. |
This PR adds 3 types of audio streams used for interactive music support. * AudioStreamInteractive: Allows setting several sub-streams and transition between them with many options. * AudioStreamPlaylist: Allows sequential or shuffled playback of a list of streams. * AudioStreamSynchronized: Allows synchronous playback of several streams, the volume of each can be controlled. Theese three stream types can be combined to create complex, layered interactive music and transitions between them, similar to software such as WWise.
reduz
force-pushed
the
interactive-music-take3
branch
from
81e2622 to
43b78cd
Compare
|
Alright, shuffle is now working and applied the suggested changes. Can't believe this feature will finally get merged after 5 years in the works! |
|
Thanks! Great work Juan, and @DanielMatarov for the initial work on this feature during GSoC 2019! |
|
The editor AudioStreamInteractive editor window doesn't work properly on high-DPI displays. |
|
This may not be the right place or time but I've been thinking about this for a while, it's pretty important to me and maybe I should propose this somewhere else but I didn't have as much time to prepare for the interactive music support train as I thought. After playing with the new features in beta, I feel they are a bit lacking when an open source solution has existed for quite some time, but doesn't seem to get the love it deserves IMO. Proposal for Enhancing Godot's Interactive Audio Capabilities Current Limitations: -Multiple AudioStreamInteractive Tracks: Can we have these play in sync? It seems more functional to have simultaneous tracks with independent buses for effects. Suggestions: -FMOD Studio-like Concept, good: While this might attract modern developer workflow users, Godot's past support for tracker formats was a missed opportunity. Embracing something like the libopenmpt addon could be more beneficial for interactive music. Potential Benefits: -Enhanced Interactivity: Official support for tracker files would simplify syncing and transitioning music tracks in Godot. Conclusion: Leveraging open-source, free solutions like libopenmpt would offer Godot a unique, interactive audio solution born from game music concepts, aligning better with developers' needs. For reference on licensing: https://lib.openmpt.org/libopenmpt/license/ and have a nice day. |
I am not sure we can distribute Microsoft Foundation Classes for OpenMPT and also support our 5-10 platforms. |
Pretty sure that is only for OpenMPT application, the actual tracker/daw used to make the songs. The libopenmpt library is standalone from that. Rather OpenMPT uses libopenmpt and MFC, but libopenmpt is a separate software library. I changed OpenMPT to read libopenmpt in my post. Dependancies for libopenmpt are here: https://lib.openmpt.org/doc/dependencies.html#md_doc_libopenmpt_dependencies and I think it's pretty basic for cross-platforms, console systems aside, but they always want you to buy their DevKits as well. The plugin at DudeJoe's repo is the best source to see and test an extension implementation with code. He was concerned that it wouldn't be accepted to the asset library due to only having a windows implementation so far, but I think the community could help with that if enough people knew what it was and what it did. https://github.com/Dudejoe870/godot-openmpt Also, to avoid any confusion OpenMPT (the tracker that uses MFC libs) and libopenmpt are maintained together in the same repo to avoid project fragmentation, so if you look for the repo it does include the OpenMPT project as well, but the libopenmpt library source is just maintained along with it and free from other licenses other than it's own open source. PS, also wanted to be clear that since this works as a stream resource, it can exist parallel to the new implementations which I would consider higher level simplified interactive music and sound, while libopenmpt would be a lower level alternative to something like midi that would require both midi and a soundfont reader or General Midi of the platform. Everything is self-contained in the one file type and handled by the library. |
|
@GeminiSquishGames the reality is that most people composing music for games don't use tracker software and are unlikely to want to switch. Additionally, most devs these days aren't used to using trackers for music. While there are some benefits to the approach, its applications are far less broad than the ones that have been introduced in this PR. I think tracker support makes more sense to be implemented as a GDExtension/Module than as core to the engine. |
I don't think the current perception reflects reality, just the popular opinion. Do most game music composers know they could be using tracker software? Have they been presented with it as an option and learned about its features? They should have the chance to try it. Tracker software scales well with modern tech and is user-friendly once you get the workflow down. If Godot made it an official resource, it would give the engine a unique feature and appeal to retro developers who love its standalone 2D rendering engine. It could lead to unique developments and innovations, which would be great for Godot. But just bringing up some points may help to make what we have here better, if it doesn't find a way to convince getting libopenmpt more support officially, and maybe simply just drumming up support for the plugin. I've been testing beta 2, and while the features are nice, they're not groundbreaking. They're things that people have probably been doing in GDScript already. I'd certainly like to use these features and built-in support will be convenient, but it mainly organizes playlists, syncs without manual math, and transitions between stems. While pretty fundamental benefits, I don't think I would say they are broad. I can't send each stem's sequence through its own effects bus. When I master a final song, it’s fine because the music is linear and polished, but with stems, I need effects for each track to master during dynamic playback and game. Otherwise, it sounds bad. The editors are less than basic, and I can't drag and drop multiple resources into the transition or sync stem places, or organize them well. The inspector's data layout isn't conducive to a good composition workflow. In a tracker format, the resource type or file is the actual file the DAW creates. It can contain multiple songs, shared stems, samples, or instruments with many samples and has subtractive/FM-type synthesis via OPL. A composer can make precise changes, control panning and volume per channel, and adjust playback speed and instrument transposition without recording more tracks for that effect. They would just change the tuning or tempo without affecting each other, though long samples don't get the benefit that instruments do. The current PR lets devs/composers use stems in very nice ways, but doesn't allow them to create procedural melodies or music on the fly with instruments. It's more like DJing than composing or allowing a composer who also has programming skills to set their imagination free. How it is can work sometimes or be made to work regardless, but the editor isn't ready for a good composition workflow or DJing without spending time making specific tools. I am having to make my own exports to help make it easier to look at and keep track of my composition. I've got the cart but have to make the pony. Trackers have established methods for creation and are easy to use, and converting MIDI to tracker format is straightforward too. I know my passion might be hard for others to share, especially without the same exposure. I’m working on a tutorial and presentation on using the plugin DudeJoe made, which could help others see my perspective. I also need to test beta 2 more and compare its features and overhead with the official methods here and also show how they play well together. DudeJoe’s plugin works well and complements the PR so far. I just want to see tracker software thrive and be used to create amazing things in Godot and see it do more outside of the box than what major corporate engines do. Godot is a special thing and an opportunity to show what can really be done. Please support DudeJoe if you can. His work has motivated me to chase my dreams and be ambitious again. Thanks for your feedback and time; it means a lot to me. Also I hope GDExtention gets easier to work with. We both had trouble getting things to compile at times. |
I'm sorry to say, but I think you're too deep in the open source community. Happens to the best of us. While there are benefits to the tracker approach, very few composers, even when presented with it as an option, have an interest in changing from their current software. Programmers are rather unique in how much they change the tech they use. Most people have very little interest in ever learning new software. Ever tried to get an artist to change art programs? It's a difficult task. Additionally, Godot emphasizes keeping things as minimal as possible. QOI support was first proposed all the way back in 2021, and it's only getting added now because it's only started to be a relevant format, and because of its advantages for the movie writer feature. Trackers just aren't popular enough in the modern day, nor do they solve enough general problems for them to be implemented as part of the engine itself, sorry to say.
Yeah, I've been following this PR since before it was officially created, and I've been advocating for a more FMOD-like system to work for general parametric audio. It's nice that it's added by default, but it's only the most basic functionality.
Yeah, this wouldn't be solved if trackers were added as an AudioStream, it's part of the core design of the Audio Engine.
Nah, I entirely get it. I've also got technologies that I love and wish were better supported! There's no harm in being enthusiastic, but part of being part of an Open Source community in my experience is not only recognizing what your needs are, but also recognizing the goals of the project, and how well those align with any given thought. If you think it would be really beneficial, create a proposal! Though I suspect that one may already exist :p |
|
I was able to find a proposal for a dynamic music feature that includes OpenMPT support here: godotengine/godot-proposals#10077. The proposal seems to be a good place to make this discussion highly visible to interested parties. |
Heh, yeah probably. I have been for some time and I get it.
FMOD is pretty great and having used it before I know there is a plugin for Godot also. Having a nice compact resource type with a DAW like editor would seem a really good idea. I also used FMOD before with tracker formats since the Core features support it but not the mptm format ATM. Edit: But also, having less external/3rd party dependencies can be good. Part of my love of the tracker, midi, or generated synth is having small data footprints.
I think my point was that I can put effects on each channel in the tracker that are supported by libopenmpt, at least with MPTM files, so I have some amount of mastering rather than just one master track's effects. It might help if the actual Audio
I probably should have split up my ideas between what I think would work better here and in that other proposal, and I see that one existsts...
Thank you for taking time to address my ideas in length. Hopefully, we can work more on the plugin and get some more features into it that make it unique. I've been a dev/composer off and on for 20 years and have done some audio direction work with indies that make retro games and I keep trying to evangelize for Godot at conventions, and have converted a few devs for neo-retro projects. A lot of the work is with old 8 and 16 bit consoles though and getting them on cartridges. That's a whole other world that probably has a bit of influence on me too. ;) Thanks again. |
|
As music composer I'd say I'd prefer a Wwise approach rather than a FMOD one, but it's a detail (I think it fit much more the interface logic of Godot). On the tracker implementation and as teacher in a videogame school, I agree with @IntangibleMatter : trackers aren't really used by many people in the industry. Same for pure data implementation for instance, it feels better to put this into a plugin. Great work on this! It's nice to see audio getting some love :) |
|
Are there plans to add icons for AudioStreamPlaylist, AudioStreamInteractive, and AudioStreamSynchronized? I'm on 4.3 beta 3 and they appear to use the default Resource icon, unlike the rest of the AudioStream types which all have pretty custom icons:
Obviously this isn't some game- or editor-breaking issue, but matching icons may be nice to have for their debut in 4.3 :) |
|
Transitions is not Working?? |
Please open a new issue with a minimal reproduction project attached. |
|
I'm new to HitHub, how can I do that? |
|
|
|
Also I would like to have some fadeIn/fadeOut between AudioStreamPlayers |
|
@marcos-vallas Select the "issues" tab and fill out the form as provided |
Since you are new, welcome! And to make it helpful for us to help you, try to edit your original post rather than make new posts when you wish to add to your original thoughts or questions. It helps keep things easier for everyone to read and keep track of in an already big thread. 😉 If you have a good example I'm sure we can help better with that too. This sounds like something you can do with GDScript already, you just have to time things accordingly, and I have found you can have several AudioStreamPlayers working together, as it's the only way to have separate buses on different stems for a dynamic piece of music if you need to real-time master a dynamic piece. You may want to look a bit into how timing works so you can get things working in sync better. https://docs.godotengine.org/en/stable/tutorials/audio/sync_with_audio.html. It's something I've kinda struggled with at first, but once you get used to it you can time things better knowing how to cope with the delays that can happen between the AudioStreamPlayers and process frames. But it helps to know exactly what you are working on to give you better details about the problem you are having. Using a single AudioStreamPlayer with stems and transitions works accordingly for me, since the player syncs them in 4.3 but it's a bit different than I'd intuit it, and it's a very lackluster and barebones transitions editor ATM, especially for musicians who aren't programmers, who are probably used to working with DAWs. It should still work but isn't as intuitive as it could be. I'm not sure if there is a tutorial or documentation on how to use it properly but I feel like the whole editor for transition could use an overhaul... just sayin'. |
|
Is it possible to have AudioStreamInteractive inside of another stream? I assumed it would be, because of the examples about nesting AudioStreamSynchronized inside of AudioStreamInteractive, etc. For context I was trying to do something like this: |
This PR adds 3 types of audio streams used for interactive music support.
These three stream types can be combined to create complex, layered interactive music and transitions between them, similar to software such as WWise, FMOD or Elias.
As an example:
In Godot style, these components are presented stand alone, and can be mixed and matched by the user to achieve the desired function. This supersedes #62896 and #32769.
Screens:
Interactive music, list of clips:
Transitions:
NOTE: Functionality is missing in Tree so it cant be expanded and have the rightmost column always visible, hopfully this will be added/fixed later on.
Playlist:
Synchronizer:
Preview video:
interactive_music_take3.mp4
Sample project:
interactive_music.zip