chore: update rekordbox kaitai definitions #13293

Swiftb0y · 2024-05-26T18:30:09Z

Should also fix #13291.

Lets see if this builds without updating any additional changes.

Swiftb0y · 2024-05-26T18:47:36Z

Apparently not, seems like we need an updated kaitaistruct runtime. It also doesn't seem appropriate to vendor that anymore. Would ExternalProject_Add instead be an option? Can somebody help me with the required CMake legwork for that?

JoergAtGithub · 2024-05-26T18:57:44Z

We have it in VCPKG

Swiftb0y · 2024-05-26T19:04:27Z

yeah, but thats about it, its not packaged on any linux distro. https://repology.org/project/kaitai-struct-cpp-stl-runtime/versions

daschuer · 2024-05-26T20:09:24Z

We cannot download external sources on Launchpad. So let's just update our vendored version.

Swiftb0y · 2024-05-28T20:25:05Z

lib/kaitai/kaitai/kaitaistream.h

+    // workaround for https://github.com/kaitai-io/kaitai_struct_cpp_stl_runtime/issues/67
+    static std::string bytes_to_str(const std::string src, const std::string& enc) {
+      return bytes_to_str(src, enc.c_str());                       
+    }


only changed I made to the kaitai files except copy-pasting in the newest commit and moving them to match the includes the kaitai-compiler generates.

Swiftb0y · 2024-05-28T20:28:00Z

src/library/rekordbox/kaitaistructs/rekordbox_anlz.ksy

+        # NOTE: unmasking is disabled because the C++ backend of kaitai_struct
+        # doesn't seem to support typecasting as of kaitai version 0.10?
+        # process:  'xor(is_masked ? mask : [0])'
+    instances:
+      c:
+        value: len_entries
+      # mask:
+      #   value: |
+      #     [
+      #       (0xCB+c).as<u1>, (0xE1+c).as<u1>, (0xEE+c).as<u1>, (0xFA+c).as<s1>, (0xE5+c).as<s1>, (0xEE+c).as<s1>, (0xAD+c).as<s1>, (0xEE+c).as<s1>,
+      #       (0xE9+c).as<u1>, (0xD2+c).as<u1>, (0xE9+c).as<u1>, (0xEB+c).as<s1>, (0xE1+c).as<s1>, (0xE9+c).as<s1>, (0xF3+c).as<s1>, (0xE8+c).as<s1>,
+      #       (0xE9+c).as<u1>, (0xF4+c).as<u1>, (0xE1+c).as<u1>
+      #     ].as<bytes>
+      raw_mood:


see comment

Swiftb0y · 2024-05-28T20:29:06Z

src/library/rekordbox/rekordboxfeature.cpp

-        for (std::vector<rekordbox_pdb_t::table_t*>::iterator table =
-                        reckordboxDB.tables()->begin();
-                table != reckordboxDB.tables()->end();
-                ++table) {
-            if ((*table)->type() == tableOrder[tableOrderIndex]) {
-                uint16_t lastIndex = (*table)->last_page()->index();
-                rekordbox_pdb_t::page_ref_t* currentRef = (*table)->first_page();
+        for (const auto& table : *rekordboxDB.tables()) {
+            if (table->type() == tableOrder[tableOrderIndex]) {
+                uint16_t lastIndex = table->last_page()->index();
+                rekordbox_pdb_t::page_ref_t* currentRef = table->first_page();


this didn't build anymore because the table_t* changed to a unique_ptr<table_t>. I took the liberty and modernized the loop in the same step (same below).

Swiftb0y · 2024-05-30T10:04:41Z

@0xsuk can you test this PR to see whether this doesn't break anything and fixes your string-encoding issue? Thank you.

0xsuk · 2024-05-30T13:23:36Z

Everything looks good after I changed UTF-16BE to UTF-16LE in src/library/rekordbox/rekordboxfeature.cpp line 259.

0xsuk · 2024-05-30T13:27:49Z

Thank you :)

This makes it easier to ignore warnings in the autogenerated code. It also makes more sense since the `*.ksy` files are effectively vendored and the generated C++ files are autogenerated. Both something that doesn't belong in `src/`.

Swiftb0y · 2024-05-30T13:30:11Z

whoops. good catch, I assumed kaitai would take care of the conversion automatically. Fixed

Swiftb0y · 2024-05-30T13:30:28Z

Thanks for testing.

Swiftb0y · 2024-05-30T13:31:59Z

Ready for review. I suggest to go through on a commit by commit basis. The PR looks bigger as it is because most of the changes are either auto-generated or artifacts because I moved files around.

Swiftb0y · 2024-06-07T10:55:21Z

friendly ping, would be nice if this made it into 2.4.2. I've highlighted the significant changes above.

Swiftb0y · 2024-06-08T11:03:29Z

CMakeLists.txt

+add_library(rekordbox_metadata STATIC EXCLUDE_FROM_ALL
+  lib/rekordbox-metadata/rekordbox_pdb.cpp
+  lib/rekordbox-metadata/rekordbox_anlz.cpp
+)
+target_include_directories(rekordbox_metadata SYSTEM PUBLIC lib/rekordbox-metadata)
+target_link_libraries(mixxx-lib PRIVATE rekordbox_metadata)
+
 # Kaitai for reading Rekordbox libraries
 add_library(Kaitai STATIC EXCLUDE_FROM_ALL
-  lib/kaitai/kaitaistream.cpp
+  lib/kaitai/kaitai/kaitaistream.cpp
 )


I got a lot of code quality complaints on the output of the kaitai compiler, so I moved the generated code to lib/ (first and foremost because the linters already ignore that and second because it also makes sense to treat autogenerated code that way).

daschuer

The code looks good and I rely on the tests done by @0xsuk
Thank you.

daschuer · 2024-06-09T14:34:25Z

How important is the fix? Does this rectify a release?

Swiftb0y · 2024-06-09T16:21:41Z

Thank you.

How important is the fix? Does this rectify a release?

Well this PR contains two fixes. First the string decoding issue reported in #13291 and the second one concerns the history playlists created by rekordbox. I don't even think we ingest those.

I don't think an extra release is warranted as only the string decoding issue is affected, judging by the delay with which this was reported, I wouldn't say its critical enough to justify the effort.

brunchboy · 2024-06-09T16:44:27Z

It’s not just history playlists. Any table which has had deletions over time will likely be missing a few rows at the end without the structure fix.

brunchboy · 2024-06-09T16:46:47Z

And not just deletions, rows are sometimes not present in row groups for inscrutable reasons. The fact that the problem was noticed in the history playlist table shows it could happen anywhere. We need to scan the entire last row group for rows marked as present, not assume we can stop scanning based on the value of num_rows.

brunchboy · 2024-06-09T16:47:39Z

Which is not to say that it necessarily merits an urgent release, I have no idea how many people rely on scanning rekordbox USBs in the first place. 😄

Swiftb0y · 2024-06-09T16:47:57Z

Ah, thank you. In that case it would probably justify a release. Considering there is also #13309 #11373 and #10923 which all could be caused by that issue.

brunchboy · 2024-06-09T16:49:23Z

Hmm, those all could be impacted but they sound worse than this! Please feel free to ping me if this doesn’t fix those other issues and perhaps I can help investigate the problematic export files.

Swiftb0y · 2024-06-09T16:51:20Z

That would be wonderful. We can close these issues once we have a release with the fix and ask the respective issue authors to check again. In case it still happens, I would refer them to you (where exactly? crate-digger repo / deep-symmetry zulip) if thats alright for you.

brunchboy · 2024-06-09T16:52:13Z

Let’s start on the Zulip instance and then I can create issues as appropriate.

github-actions bot added library rekordbox labels May 26, 2024

Swiftb0y force-pushed the chore/update-rekordbox-kaitai-definitions branch from ab24091 to 4ab1375 Compare May 28, 2024 20:21

github-actions bot added the build label May 28, 2024

Swiftb0y force-pushed the chore/update-rekordbox-kaitai-definitions branch from 4ab1375 to 277b5c1 Compare May 28, 2024 20:21

Swiftb0y commented May 28, 2024

View reviewed changes

update kaitai cpp stl runtime to v0.10.1

3731915

Swiftb0y force-pushed the chore/update-rekordbox-kaitai-definitions branch from 277b5c1 to 7df39ab Compare May 28, 2024 20:31

Swiftb0y marked this pull request as ready for review May 28, 2024 20:32

Swiftb0y force-pushed the chore/update-rekordbox-kaitai-definitions branch 2 times, most recently from 9a05287 to c9ca71f Compare May 30, 2024 13:00

Swiftb0y added 2 commits May 30, 2024 15:29

chore: update rekordbox kaitai definitions

26ef892

make kaitai rekordbox code a library

157feda

This makes it easier to ignore warnings in the autogenerated code. It also makes more sense since the `*.ksy` files are effectively vendored and the generated C++ files are autogenerated. Both something that doesn't belong in `src/`.

Swiftb0y force-pushed the chore/update-rekordbox-kaitai-definitions branch from c9ca71f to 157feda Compare May 30, 2024 13:30

Swiftb0y mentioned this pull request May 31, 2024

Rekordbox USB to also read usb historys #12306

Open

Swiftb0y commented Jun 8, 2024

View reviewed changes

Swiftb0y requested a review from daschuer June 8, 2024 11:07

Swiftb0y added this to the 2.4.2 milestone Jun 9, 2024

rekordbox: incorporate Deep-Symmetry/crate-digger#32 fix

85d95df

daschuer approved these changes Jun 9, 2024

View reviewed changes

daschuer merged commit bc17387 into mixxxdj:2.4 Jun 9, 2024
14 checks passed

Swiftb0y deleted the chore/update-rekordbox-kaitai-definitions branch June 9, 2024 16:17

acolombier mentioned this pull request Jun 10, 2024

Missing Rekordbox Playlists on USB Import #13309

Closed

ronso0 mentioned this pull request Aug 9, 2024

Unicode DeviceSQL strings in PDB are UTF-16-BE, not UTF-16-LE #13291

Closed

Swiftb0y mentioned this pull request Sep 18, 2024

Rekordbox Library Tracks Disappear #11373

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: update rekordbox kaitai definitions #13293

chore: update rekordbox kaitai definitions #13293

Swiftb0y commented May 26, 2024

Swiftb0y commented May 26, 2024

JoergAtGithub commented May 26, 2024

Swiftb0y commented May 26, 2024

daschuer commented May 26, 2024

Swiftb0y May 28, 2024

Swiftb0y May 28, 2024

Swiftb0y May 28, 2024 •

edited

Loading

Swiftb0y commented May 30, 2024

0xsuk commented May 30, 2024

0xsuk commented May 30, 2024

Swiftb0y commented May 30, 2024

Swiftb0y commented May 30, 2024

Swiftb0y commented May 30, 2024

Swiftb0y commented Jun 7, 2024

Swiftb0y Jun 8, 2024

daschuer left a comment

daschuer commented Jun 9, 2024

Swiftb0y commented Jun 9, 2024 •

edited

Loading

brunchboy commented Jun 9, 2024

brunchboy commented Jun 9, 2024 •

edited

Loading

brunchboy commented Jun 9, 2024

Swiftb0y commented Jun 9, 2024

brunchboy commented Jun 9, 2024

Swiftb0y commented Jun 9, 2024

brunchboy commented Jun 9, 2024

chore: update rekordbox kaitai definitions #13293

chore: update rekordbox kaitai definitions #13293

Conversation

Swiftb0y commented May 26, 2024

Swiftb0y commented May 26, 2024

JoergAtGithub commented May 26, 2024

Swiftb0y commented May 26, 2024

daschuer commented May 26, 2024

Swiftb0y May 28, 2024

Choose a reason for hiding this comment

Swiftb0y May 28, 2024

Choose a reason for hiding this comment

Swiftb0y May 28, 2024 • edited Loading

Choose a reason for hiding this comment

Swiftb0y commented May 30, 2024

0xsuk commented May 30, 2024

0xsuk commented May 30, 2024

Swiftb0y commented May 30, 2024

Swiftb0y commented May 30, 2024

Swiftb0y commented May 30, 2024

Swiftb0y commented Jun 7, 2024

Swiftb0y Jun 8, 2024

Choose a reason for hiding this comment

daschuer left a comment

Choose a reason for hiding this comment

daschuer commented Jun 9, 2024

Swiftb0y commented Jun 9, 2024 • edited Loading

brunchboy commented Jun 9, 2024

brunchboy commented Jun 9, 2024 • edited Loading

brunchboy commented Jun 9, 2024

Swiftb0y commented Jun 9, 2024

brunchboy commented Jun 9, 2024

Swiftb0y commented Jun 9, 2024

brunchboy commented Jun 9, 2024

Swiftb0y May 28, 2024 •

edited

Loading

Swiftb0y commented Jun 9, 2024 •

edited

Loading

brunchboy commented Jun 9, 2024 •

edited

Loading