Extreme memory usage when applying catalog rules

[ioweb-gr]

Preconditions (*)

1. OpenMage 20.0.10

Steps to reproduce (*)

1. Create 11 catalog rules which apply discounts to products where category is one of 1,2,....N and at least 10 clauses where manufacturer is 1..N to apply a discount percentage
2. Use a medium sized catalog of about 40K products
3. Try to apply all rules

e.g.

Expected result (*)

1. Rules are applied

Actual result (*)

1. Memory exhausted. In my case it required 16GB of RAM to actually execute the \Mage_Adminhtml_Promo_CatalogController::applyRulesAction

e.g. to simulate in a CLI script

<?php
require_once('app/Mage.php');
umask(0);
Mage::app('admin');

ini_set('memory_limit', '1G');
Mage::getModel('catalogrule/rule')->applyAll();

Will yield

PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 75497472 bytes) in /var/www/vhosts/example.com/httpdocs/app/code/community/Varien/Profiler.php on line 169

And it will crash until I put 16G in the limit.

[ioweb-gr]

One additional problem in these conditions, if I added more than 15 manufacturers in the same clause ( the image has 5), the rule is not processed entirely.

[ADDISON74]

Could you check this issue with different OpenMage versions? I would start first with Magento 1.9.4.5.

[ioweb-gr]

I can tell you we've been carrying this problem since Magento 1.9.4.1 but it was manageable because it required approximately 2G-3G of memory. However as our needs have increased the past years we've reached the point where we reach 16G of memory and I felt it's at a point we need to debug this.

We also had some workarounds of external scripts running the rules one by one and this suited us until now as well.

I could try later versions probably if that would help though

[elidrissidev]

@ioweb-gr Could you test #3140 and see if it fixes your issue?

[rvelhote]

@ioweb-gr We don't have this problem and we have a catalog about the same size as yours, lots of promotions going on all the time and more complex rules than the ones you showed.

How many store views do you have? We found that Ctb_MktCatalogrules_Model_Resource_Rule when using flat tables and multiple store views is very slow. We made a small change that improved the performance massively.

Try changing line #216 in Mage_CatalogRule_Model_Resource_Rule to ignore the condition and force it use the non-flat table version of the rule indexing in the else statement.

Also, without a profiler (and a stack trace) we can just guess. Run that script with a profiler so we can see where the bottleneck is.

[ioweb-gr]

@elidrissidev it didn't help unfortunately.

@rvelhote We have only 1 store view. We have turned flat tables off quite some time ago. I will give it a shot and test your suggestion for the else condition though.

Regarding the profiler I tried using blackfire but unfortunately the dump is so big that blackfire rejects it :(

[rvelhote]

@ioweb-gr If you have flat tables turned off then it won't make a difference. In our case, we have flat tables active (because it's better for performance) with 103 store views and that code change made a huge difference when applying rules.

Regarding the profiling, I suggest Xhprof if Blackfire doesn't accept the traces. Or you can use Ubench and litter your code with benchmark tags and try to narrow it down. Probably even better because it will minimize the profiler overhead.

[fballiano]

what if you do "manufacturer is one of (....)" instead of the way the rule is created now?

[ioweb-gr]

@fballiano it's not possible it only allows is and is not

[fballiano]

auch sorry

[ioweb-gr]

This is a different feature request alltogether hehe :)