flow assetusage:update does not work for large resource tables

[daniellienert]

The command ended up in a memory usage of 2GB of ram after proccessing ~400 records

[daniellienert]

The problem, is that the complete result is fetched and converted to an array in https://github.com/Flowpack/Flowpack.Neos.AssetUsage/blob/main/Classes/Command/AssetUsageCommandController.php#L104.

With the super generic architecture we have here, it seems not possible to easily switch to a doctrine iterator to do stream processing on large data sets.

[bwaidelich]

I just stumbled upon the same issue. Even the initial run of this command fails after a few thousand entries and there is no way to specify an offset

[bwaidelich]

When increasing the memory_limit to 8GB, the command runs through but the progress bar jumps from 5% to 100% and only a fraction of the ~100k assets are added to the usage table

[bwaidelich]

the progress bar jumps from 5% to 100%

Ah, that's because there are lots of continue-calls that skip the $this->output->progressAdvance()