[WIP] Snapgrid #2917

daschuer · 2020-07-05T13:33:27Z

This is a proof of concept for a new beat detection strategy.

This PR is just an experimental PR on top of #2877

Currently we "move" the detected beats to positions they are considered musical reasonable.
Due to this for instance a loud onset that has nothing to do with the rhythm can trick the detection algorithm, because it is considered as beat.

Here we collect all possible onsets and experiential show it in beat grid.
The user can now try with looping these onsets what are actual beats.

Instead of shifting loud onsets it will snap to another possible onset.
This avoids notable double beat issues when syncing, because we are not off-beat, but just snapped to a less loud beat.

We still need to allow shifting in a small region, because of quantitation noise and jitter due to a live drummer or different characteristics of the drums. But this can now limited to a small unnoteabe region of +- 25 ms.

This algorithm is a good foundation for omitted, skipped or added beat detection.

I have tested this with
"Kahn - Helter Skelter"
https://www.youtube.com/watch?v=-byQwJ2Uoh0

At 0:31.80 the rhythm is shifted. The original qm algorithm has hard times to detect that. It is trying to iron out this region, which is musical wrong. Instead the beat grid should be snapped to the new rhythm.

Known Issues:
In a "Boom / Tshack" Rhythm the spectral difference is a lot more sensitive for Tshack than for Boom. I have used thresshold_smoothing to adjust the sensitivity. For detecting all Booms I had to increase the sensitivity to 0.05. This creates some false positives in the first region of the track. Here a value of 0.02 works perfect.

In other tracks with a deeper Boom, we need more sensitivity to detect it. Detecting the Booms is very important, because thy are likely the downbeats.

I think we should use this snapGrid when manually edit the beat grid. I am not sure if we should actually show it, because you can already predict the snap positions from the waveform.

@hacksdump @crisclacerda

What do you think?

Most of the implementation is copied and slighly cleaned for consistency from beatutils. The rest comes from mixxxdj#2847

Change needed to do multi feature detection without redoing the STFTs

The change in qm-dsp broke queen mary beats

…nge constructors

… into rhythm-detector

daschuer · 2020-07-05T13:35:19Z

Be-ing · 2020-07-05T17:07:01Z

This is an interesting idea. On the community call the other day we briefly discussed an idea to store the analysis results separately from the user adjusted beats so the analysis results could be used to help the infer right/left features. Repsys seems to do something like this from the video demo, but I have yet to look at the repsys code in detail. This "snapgrid" may serve that purpose if we store it as an immutable part of the protobuf.

As for whether it would be helpful to show this on the waveform sometimes, we can defer that decision until @hacksdump's project is further along.

daschuer · 2020-07-06T01:08:04Z

Blue + first peak : Boom
Red + second peak: Tshack

This happens because Tshack is not that high but it has many more bins in the spectral difference.
The bin size is ~50 Hz

Be-ing · 2020-07-12T00:18:17Z

@satchelspencer we got this idea from Repsys. We're working on building a new beatgrid format and analyzer in Mixxx capable of working with variable tempos. We'd be curious to get your input on this work.

Be-ing · 2020-07-12T06:22:42Z

I gave this a test with some jazz tracks with variable tempos and rhythms. I think this is a really promising idea. I think it could be useful for more than adjusting the beatgrid. It may be useful in some cases as an alternative grid to quantize to, particularly for setting off beat cue points and adjusting manual loops.

Be-ing · 2020-07-12T06:44:04Z

I propose we call this the "sound grid" to contrast with the beat grid which marks musical time.

@hacksdump give this a try. I think we should show this unobtrusively on the scrolling waveforms, for example small lines at the top & bottom of the waveforms, when it is relevant. This could be when moving the downbeats, cue points, and loop boundaries. We may consider turning quantize into a 3 state switch: off, beat quantize, sound quantize. Or maybe we could have beatloops and sync quantize to the beatgrid whereas cues and manual loops quantize to the sound grid.

daschuer · 2020-07-12T12:14:26Z

I think this is only useful for editing the beatgrid.
Quantize during performance should snap to the beatgrid and probably to integer fractions of it.

The issue is that the snap grid is the raw onset detection output with a moving threshold. It is not ironed. Different instruments have a different form of unsets and we have the quantize jitter.

An alternative name would be "Onsets". Or "Raw Onsets" or such.

Be-ing · 2020-07-12T12:17:13Z

"Onsets" sounds good to me.

Swiftb0y · 2020-07-12T12:47:34Z

What about "transients"? At least it seems like that's what you mean with "onsets".

daschuer · 2020-07-12T14:01:46Z

Onsets is the special term for exactly this:
https://en.wikipedia.org/wiki/Onset_%28audio%29?wprov=sfla1

Be-ing · 2020-07-20T17:38:43Z

This algorithm is a good foundation for omitted, skipped or added beat detection.

Can you elaborate on this? In my testing of #2877, the biggest problem was the analyzer detecting a very incorrect tempo when there are no loud rhythmic elements but the tempo is actually constant. Could this help with that somehow?

daschuer · 2020-07-21T12:18:56Z

Yes.

This is in contrast to the queen Mary algorithm, which has no expectation of skipped or added beats build in.
It warps the beat to a smooth tempo changing grid. That reaches as many beats as possible and works perfect to follow tracks like the fold4, warp5 track, but fails for added and skipped brats.

The new approach is to not move beats out of the onset region, to not produce double beats when beatmatching.

Be-ing · 2020-07-21T18:47:18Z

Great, if we can overcome the problem of inaccurately detecting tempo changes when there are not prominent beats, I think users will not need to manually tell Mixxx to assume a constant tempo.

* Adjust the Boom vs Tshak compensation curve with more test tracks * Use a smmoothing filter for the floating threshold after detecting a beat * Look also to the falling edge of a beat when finding SD peaks

src/analyzer/analyzerrhythm.cpp

github-actions · 2021-01-22T00:51:21Z

This PR is marked as stale because it has been open 90 days with no activity.

Holzhaus

There are merge conflicts and a bunch of compiler warnings now.

Holzhaus · 2021-03-05T10:30:09Z

src/preferences/dialog/dlgprefrhythm.ui

+    </widget>
+   </item>
+   <item row="6" column="0">
+    <widget class="QLabel" name="label_21">


AutoUic: /home/jan/Projects/mixxx/src/preferences/dialog/dlgprefrhythm.ui: Warning: The name 'label_21' (QLabel) is already in use, defaulting to 'label_211'.

Holzhaus · 2021-03-05T10:31:20Z

src/analyzer/analyzerrhythmstats.cpp

+            max = tempos.value();
+        }
+    }
+    return mode;


If tempos.hasNext() is false, then mode is returned uninitialized.

Holzhaus · 2021-03-05T10:32:14Z

src/analyzer/analyzerrhythm.cpp

+
+} // namespace
+
+AnalyzerRhythm::AnalyzerRhythm(UserSettingsPointer pConfig)


/home/jan/Projects/mixxx/src/analyzer/analyzerrhythm.cpp: In constructor ‘AnalyzerRhythm::AnalyzerRhythm(UserSettingsPointer)’: /home/jan/Projects/mixxx/src/analyzer/analyzerrhythm.cpp:52:52: error: unused parameter ‘pConfig’ [-Werror=unused-parameter] 52 | AnalyzerRhythm::AnalyzerRhythm(UserSettingsPointer pConfig) | ~~~~~~~~~~~~~~~~~~~~^~~~~~~

Holzhaus · 2021-03-05T10:32:33Z

src/analyzer/analyzerrhythm.cpp

+std::vector<double> AnalyzerRhythm::computeSnapGrid() {
+    int size = m_detectionResults.size();
+
+    int dfType = 3; // ComplexSD


unused variable

Holzhaus · 2021-03-05T10:33:17Z

src/analyzer/analyzerrhythmbpm.cpp

+        // here we detect if the segment has a constant tempo or not
+        if (partLenght > m_beatsPerBar * 2) {
+            int middle = partLenght / 2;            
+            auto beatsAtLeft = QVector<double>::fromStdVector(std::vector<double>(


/home/jan/Projects/mixxx/src/analyzer/analyzerrhythmbpm.cpp: In member function ‘std::tuple<QVector<double>, QMap<int, double> > AnalyzerRhythm::FixBeatsPositions()’: /home/jan/Projects/mixxx/src/analyzer/analyzerrhythmbpm.cpp:158:49: error: ‘static QVector<T> QVector<T>::fromStdVector(const std::vector<T>&) [with T = double]’ is deprecated: Use QVector<T>(vector.begin(), vector.end()) instead. [-Werror=deprecated-declarations] 158 | auto beatsAtLeft = QVector<double>::fromStdVector(std::vector<double>( | ^~~~~~~~~~~~~

poelzi · 2021-03-05T11:06:15Z

Interesting, I will give this a try soon.
I think if we are able to not only detect onset, but transient as well, we could find the proper alignment of phrases based on that. Most music as far as I can see, always start a phrase with a transient. But I guess this is more something when #2961 lands.

github-actions · 2021-06-04T01:14:57Z

This PR is marked as stale because it has been open 90 days with no activity.

crisclacerda and others added 19 commits June 23, 2020 21:27

add rhythm analyzer

36ada6f

rhythm detector preferences

5013059

rhythm detector output downbeats to beapmap

2487e6d

rhythm analyzer fix downbeat positions

8ee1fe4

rhythm analyzer add cleanups

a54340c

rhythm analyzer style fixes

4469fb7

rhythm analyzer add tempo detection

485cca9

Most of the implementation is copied and slighly cleaned for consistency from beatutils. The rest comes from mixxxdj#2847

qm-dsp - allow for multi feature beat detection.

1ab8f62

Change needed to do multi feature detection without redoing the STFTs

rhythm analyzer - dirty hack to keep mix building.

9ff3f9f

The change in qm-dsp broke queen mary beats

rhythm analyzer - implement multi feature beat detection

d643861

rhythm analyzer - use std::vector constructors rather the newer QT ra…

5ba7497

…nge constructors

rhythm analyzer - add new files to CMakeLists

1fa23ab

rhythm analyzer - don't re-analyze with there are beats

ce94ecb

rhythm analyzer - do not use even vector size for computing median tempo

6b54980

rhythm analyzer - improved agreement calculation

07e4d79

rhythm analyzer - improved fixedBeatsPositions and bpm calculation

1fcf369

rhythm analyzer - remove old comment

9cb681d

Merge branch 'rhythm-detector' of https://github.com/crisclacerda/mixxx…

13ac5f8

… into rhythm-detector

SnapGrid proof of concept

cc876a1

daschuer mentioned this pull request Jul 6, 2020

[WIP] Remove beatmap and beatgrid #2861

Closed

7 tasks

print fftbins

56d8320

Holzhaus marked this pull request as draft July 24, 2020 08:31

daschuer added 4 commits August 7, 2020 13:20

Merge remote-tracking branch 'upstream/master' into snapgrid

bc0fe48

Use a magice curve to be more sesitive in the Boom region

a3ae5c0

Improve Onset detection

8bcba01

* Adjust the Boom vs Tshak compensation curve with more test tracks * Use a smmoothing filter for the floating threshold after detecting a beat * Look also to the falling edge of a beat when finding SD peaks

tweak thresshold smoothing

f15dc52

Holzhaus reviewed Aug 15, 2020

View reviewed changes

src/analyzer/analyzerrhythm.cpp Outdated Show resolved Hide resolved

Fix typo

b0fc100

Be-ing changed the base branch from master to main October 23, 2020 23:16

github-actions bot added the stale Stale issues that haven't been updated for a long time. label Jan 22, 2021

daschuer mentioned this pull request Mar 5, 2021

Ironing beats #3626

Merged

Holzhaus reviewed Mar 5, 2021

View reviewed changes

github-actions bot removed the stale Stale issues that haven't been updated for a long time. label Mar 6, 2021

github-actions bot added the stale Stale issues that haven't been updated for a long time. label Jun 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Snapgrid #2917

[WIP] Snapgrid #2917

daschuer commented Jul 5, 2020

daschuer commented Jul 5, 2020

Be-ing commented Jul 5, 2020 •

edited

Loading

daschuer commented Jul 6, 2020

Be-ing commented Jul 12, 2020

Be-ing commented Jul 12, 2020

Be-ing commented Jul 12, 2020 •

edited

Loading

daschuer commented Jul 12, 2020

Be-ing commented Jul 12, 2020

Swiftb0y commented Jul 12, 2020 •

edited

Loading

daschuer commented Jul 12, 2020

Be-ing commented Jul 20, 2020

daschuer commented Jul 21, 2020

Be-ing commented Jul 21, 2020

github-actions bot commented Jan 22, 2021

Holzhaus left a comment

Holzhaus Mar 5, 2021

Holzhaus Mar 5, 2021

Holzhaus Mar 5, 2021

Holzhaus Mar 5, 2021

Holzhaus Mar 5, 2021

poelzi commented Mar 5, 2021

github-actions bot commented Jun 4, 2021


		} // namespace

		AnalyzerRhythm::AnalyzerRhythm(UserSettingsPointer pConfig)

[WIP] Snapgrid #2917

Are you sure you want to change the base?

[WIP] Snapgrid #2917

Conversation

daschuer commented Jul 5, 2020

daschuer commented Jul 5, 2020

Be-ing commented Jul 5, 2020 • edited Loading

daschuer commented Jul 6, 2020

Be-ing commented Jul 12, 2020

Be-ing commented Jul 12, 2020

Be-ing commented Jul 12, 2020 • edited Loading

daschuer commented Jul 12, 2020

Be-ing commented Jul 12, 2020

Swiftb0y commented Jul 12, 2020 • edited Loading

daschuer commented Jul 12, 2020

Be-ing commented Jul 20, 2020

daschuer commented Jul 21, 2020

Be-ing commented Jul 21, 2020

github-actions bot commented Jan 22, 2021

Holzhaus left a comment

Choose a reason for hiding this comment

Holzhaus Mar 5, 2021

Choose a reason for hiding this comment

Holzhaus Mar 5, 2021

Choose a reason for hiding this comment

Holzhaus Mar 5, 2021

Choose a reason for hiding this comment

Holzhaus Mar 5, 2021

Choose a reason for hiding this comment

Holzhaus Mar 5, 2021

Choose a reason for hiding this comment

poelzi commented Mar 5, 2021

github-actions bot commented Jun 4, 2021

Be-ing commented Jul 5, 2020 •

edited

Loading

Be-ing commented Jul 12, 2020 •

edited

Loading

Swiftb0y commented Jul 12, 2020 •

edited

Loading