Chunked loading of training data

[LukasBeiske]

This will fix #2413.

• ctapipe-train-energy-regressor
• ctapipe-train-particle-classifier
• ctapipe-train-disp-reconstructor

[Tobychev]

Is there any test that checks that the _read_tabel functions are correct?

[maxnoe]

This looks extremely similar between the three tools. Can we refactor the code into a single common function?

[LukasBeiske]

This looks extremely similar between the three tools. Can we refactor the code into a single common function?

For the disp tool, the calculation of true_disp needs columns which get dropped right after this calculation is done. But we could try to keep these needed columns if all of them are loaded, which only happens in the disp tool, and calculate true_disp outside of _read_table.

For the other two tools, I see no problem.

[LukasBeiske]

This looks extremely similar between the three tools. Can we refactor the code into a single common function?

I did this now, but maybe there are better ways to do this. I also don't have a strong opinion on whether this is worth it or if we should leave it as it was before.

Is there any test that checks that the _read_tabel functions are correct?

I haven't come up with anything useful for this yet, since most of it is already covered by the existing tests of the QualityQuery and the ChunkIterator. If you have a good idea, please let me know.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (7751fc1) 60.60% compared to head (bb01357) 60.60%.
Report is 4 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2423   +/-   ##
=======================================
  Coverage   60.60%   60.60%           
=======================================
  Files           3        3           
  Lines          33       33           
=======================================
  Hits           20       20           
  Misses         13       13           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.