perf: BOM Update Tool

[marination]

First Draft

get_boms_in_bottom_up_order

• Create child-parent map once and fetch value from child key to get parents
• Get parents recursively for a leaf node (get all ancestors)
• Approx. 44 secs for 4lakh 70k BOMS (used to go on for over 5-10 mins earlier, maybe more idk)

update_cost

• Remove double RM cost calculation
• Update child items in db as well (optionally)
• Don't regenerate exploded items for each BOM while updating cost (only in update cost job/bom function)🤦🏻‍♀️
• Approx 5 minutes for 20k BOMs. [WIP]

There were correctness issues that could all be solved with a newer AlGoRiThM, that took preference. So first correctness >> then perf

New BOM Cost Updation Flow

Dependants = Immediate Parents, Dependencies = Children (parent-child is not accurate really)

Consider the following BOMs and their dependencies:

• While updating all BOMs, we update the Raw Material BOMs or leaf nodes first and work our way up
• An important thing here: We must only update a BOM after all its Dependencies are resolved/updated
• That means: α (alpha) cannot be updated until A,B,1,2,3,4 are updated
• For this we update BOMs one level at a time, and within a level we process BOMs parallelly as they are independent.
• BOMs within each level are split into batches of 20k (takes 5 mins) and queued for updation
• Once a level is done a Cron Job will prep and start processing the next level

Working:

• Given the above example, on submitting the BOM Update Log, Raw Material BOMs will be processed first (batch size = 20k). The status will be In Progress

  

• These Jobs will keep updating their row in the table in the Log and keep checking if all other jobs of this level are done (via bom_batches table)

• The Current Level is also maintained in the Progress Section, which will increment as we progress up the levels

• Once a level is complete and the cron (every 5 mins) determines a level is complete (all rows of current level have status Pending -> Completed), it will prepare next level parent boms (dependants) and kickstart the next level

  

• How do we determine "Parent BOMs" ? : By checking immediate parents with no Dependencies. Here 1 has immediate parents A and α (alpha). A can be considered since it has no unresolved/non-updated Dependencies, but α (alpha) has A and B as unresolved dependencies. So we proceed with A as parent of 1 in the next level.

• A Cron Job will run every 5 minutes and check for In Progress jobs and start processing the next level if all jobs of current level are completed

• The Log will be set as Complete when there are no more Dependants/ Parent BOMs to resolve/update

---

Important change/fix to test

• Rate updation as per Raw Material Cost as per used to not work, which now does
• Rates are reset rightfully on save and fetched as per Valuation/Last Purchase/Price List

---

To Test:

• Create BOM with 3 levels (RM BOM, Subassembly BOM that uses RM BOM, FG BOM that uses Subassemble BOM)
• Set RM cost as per Valuation Rate > Change valuation rate of Item in RM BOM > Update all costs via tool > check
• Set RM cost as per Valuation Rate with different BOM currency (USD)> Change valuation rate of Item in RM BOM > Update all costs via tool > check
• Set RM cost as per Price List > Change Price List rate of Item in RM BOM > Update all costs via tool > check
• Set RM cost as per Price List in different Price List currency > Change Price List rate of Item in RM BOM > Update all costs via tool > check

Todo:

• Clearing BOM Update Logs periodically (or atleast the table in it)

[codecov]

Codecov Report

Merging #31072 (3fa0a46) into develop (5c69378) will decrease coverage by 0.04%.
The diff coverage is 86.23%.

@@             Coverage Diff             @@
##           develop   #31072      +/-   ##
===========================================
- Coverage    63.50%   63.46%   -0.05%     
===========================================
  Files          986      988       +2     
  Lines        67457    67574     +117     
===========================================
+ Hits         42838    42885      +47     
- Misses       24619    24689      +70     

Impacted Files	Coverage Δ
...t/accounts/report/sales_register/sales_register.py	73.68% <ø> (-8.19%)	⬇️
erpnext/e_commerce/redisearch_utils.py	36.42% <ø> (ø)
erpnext/hooks.py	100.00% <ø> (ø)
...cturing/doctype/bom_update_tool/bom_update_tool.py	79.31% <0.00%> (-1.94%)	⬇️
erpnext/loan_management/doctype/loan/loan.py	67.78% <44.44%> (ø)
...ext/stock/doctype/stock_entry/stock_entry_utils.py	93.33% <66.66%> (ø)
...facturing/doctype/bom_update_log/bom_update_log.py	89.89% <87.67%> (+5.37%)	⬆️
...uring/doctype/bom_update_log/bom_updation_utils.py	88.11% <88.11%> (ø)
erpnext/manufacturing/doctype/bom/bom.py	87.63% <93.47%> (-0.14%)	⬇️
...ounts/doctype/purchase_invoice/purchase_invoice.py	83.48% <100.00%> (ø)
... and 36 more

[marination]

Update cost tests are breaking. Will fix while writing tests

[marination]

Was an issue with concurrent jobs and incorrect locking sequence causing lost connections to db server.
Fixed and works as expected.
4 jobs parallelly take ~ 12-14 minutes (4*20k = 80,000 BOMs)

[ankush]

Tested with deeply nested BOM with different costs for each RM and some ops cost.

RM cost

• Started with everything valued @ 0
• Update all bottom-most item (RM) to some valuation rate and did BOM update (scheduled job trigger manually)

Exploded item cost == individual subassy costs.

---

Ops cost

• Used same BOM, revised and replaced a revision with ops.
• Ran update tool

Increased levels (by moving this full subassy on another FG)

Partitioning / update logic LGTM 👍

---

unrelated feature request (I added these columns in BOM explorer (v. hacky implementation) but this might help in actually exploring costs 😬 )