perf: use `LEFT JOIN` instead of `NOT EXISTS` #36221

s-aga-r · 2023-07-20T13:44:51Z

Source / Ref: ISS-23-24-02079

Issue: The BOM Update Log gets stuck in Queued (timeout) if there are a large number of BOMs.

Before:

171906 function calls (171733 primitive calls) in 83.689 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     24/1    0.000    0.000   83.689   83.689 {built-in method builtins.exec}
        1    0.000    0.000   83.689   83.689 <string>:1(<module>)
        1    0.000    0.000   83.689   83.689 bom_updation_utils.py:157(get_leaf_boms)
        1    0.000    0.000   83.689   83.689 database.py:286(sql_list)
        1    0.000    0.000   83.689   83.689 database.py:111(sql)
        1    0.000    0.000   83.678   83.678 cursors.py:129(execute)
        1    0.000    0.000   83.678   83.678 cursors.py:306(_query)
        1    0.000    0.000   83.678   83.678 connections.py:542(query)
        1    0.000    0.000   83.678   83.678 connections.py:763(_read_query_result)
        1    0.000    0.000   83.678   83.678 connections.py:1154(read)
     7117    0.016    0.000   83.646    0.012 connections.py:683(_read_packet)
    14234    0.010    0.000   83.626    0.006 connections.py:728(_read_bytes)
    14234    0.005    0.000   83.607    0.006 {method 'read' of '_io.BufferedReader' objects}
       38    0.000    0.000   83.602    2.200 socket.py:575(readinto)
       38   83.602    2.200   83.602    2.200 {method 'recv_into' of '_socket.socket' objects}
        1    0.000    0.000   73.674   73.674 connections.py:1233(_read_result_packet)
        1    0.007    0.007   73.674   73.674 connections.py:1266(_read_rowdata_packet)
     7109    0.007    0.000    0.021    0.000 connections.py:1279(_read_row_from_packet)
     7115    0.003    0.000    0.012    0.000 protocol.py:165(read_length_coded_string)
        1    0.000    0.000    0.010    0.010 database.py:82(connect)

After

179055 function calls (178948 primitive calls) in 17.308 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000   17.308   17.308 {built-in method builtins.exec}
        1    0.000    0.000   17.308   17.308 <string>:1(<module>)
        1    0.000    0.000   17.308   17.308 <ipython-input-5-ab8ea441ed2a>:1(get_leaf_boms)
        1    0.000    0.000   17.307   17.307 utils.py:64(execute_query)
        1    0.000    0.000   17.306   17.306 database.py:111(sql)
        1    0.000    0.000   17.303   17.303 cursors.py:129(execute)
        1    0.000    0.000   17.303   17.303 cursors.py:306(_query)
        1    0.000    0.000   17.303   17.303 connections.py:542(query)
        1    0.000    0.000   17.302   17.302 connections.py:763(_read_query_result)
        1    0.000    0.000   17.302   17.302 connections.py:1154(read)
     7113    0.015    0.000   17.271    0.002 connections.py:683(_read_packet)
    14226    0.010    0.000   17.252    0.001 connections.py:728(_read_bytes)
    14226    0.005    0.000   17.234    0.001 {method 'read' of '_io.BufferedReader' objects}

erpnext/manufacturing/doctype/bom_update_log/bom_update_log.py

erpnext/manufacturing/doctype/bom_update_log/bom_updation_utils.py

codecov · 2023-07-28T12:00:28Z

Codecov Report

Merging #36221 (148d466) into develop (dac9fd6) will decrease coverage by 0.01%.
Report is 5 commits behind head on develop.
The diff coverage is 20.00%.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop   #36221      +/-   ##
===========================================
- Coverage    64.25%   64.25%   -0.01%     
===========================================
  Files          785      785              
  Lines        60965    60968       +3     
===========================================
+ Hits         39173    39174       +1     
- Misses       21792    21794       +2

Files Changed	Coverage Δ
erpnext/hooks.py	`100.00% <ø> (ø)`
...facturing/doctype/bom_update_log/bom_update_log.py	`43.36% <ø> (ø)`
...uring/doctype/bom_update_log/bom_updation_utils.py	`44.23% <0.00%> (-1.32%)`	⬇️
erpnext/accounts/utils.py	`73.44% <100.00%> (ø)`

... and 1 file with indirect coverage changes

* perf: use `LEFT JOIN` instead of `NOT EXISTS` (cherry picked from commit 58d8675) # Conflicts: # erpnext/manufacturing/doctype/bom_update_log/bom_updation_utils.py * fix: long queue `process_boms_cost_level_wise` (cherry picked from commit 148d466) * chore: `conflicts` --------- Co-authored-by: s-aga-r <sagarsharma.s312@gmail.com>

* perf: use `LEFT JOIN` instead of `NOT EXISTS` (cherry picked from commit 58d8675) # Conflicts: # erpnext/manufacturing/doctype/bom_update_log/bom_updation_utils.py * fix: long queue `process_boms_cost_level_wise` (cherry picked from commit 148d466) * chore: `conflicts` * fix: failing test case --------- Co-authored-by: s-aga-r <sagarsharma.s312@gmail.com>

## [13.52.8](v13.52.7...v13.52.8) (2023-08-01) ### Bug Fixes * added validation for unique serial numbers in pos invoice ([#36302](#36302)) ([a165b37](a165b37)) * only publish repost progress to doc subscriber (backport [#36400](#36400)) ([#36403](#36403)) ([e9df064](e9df064)) * removed "fetch_from" (backport [#36365](#36365)) ([#36387](#36387)) ([c574494](c574494)) ### Performance Improvements * use `LEFT JOIN` instead of `NOT EXISTS` (backport [#36221](#36221)) ([#36384](#36384)) ([cdc86bd](cdc86bd))

# [14.33.0](v14.32.1...v14.33.0) (2023-08-01) ### Bug Fixes * allow fully depreciated existing assets ([#36378](#36378)) ([43b85c5](43b85c5)) * change fieldtype from Currency to Float for the valuation rate in the stock report ([93bd4c7](93bd4c7)) * cost center filter for fetching payments ([c9daa85](c9daa85)) * Defined "Open" Status as default (backport [#36421](#36421)) ([#36424](#36424)) ([33a9477](33a9477)) * filter by cost center in trial balance ([02428b4](02428b4)) * german translations ([9c8e2a3](9c8e2a3)) * GL Entries should not be splitted based on cost center allocation in PCV ([ade13e6](ade13e6)) * group item reorder by (warehouse, material_request_type) (backport [#35818](#35818)) ([#36425](#36425)) ([516191b](516191b)) * Ignore account closing balance for financial statement ([800417e](800417e)) * in payment_entry 'Unallocated Amount' cal is broken ([#36369](#36369)) ([cd3b85c](cd3b85c)) * incorrect `idx` on JE's after reconciliation ([80eb875](80eb875)) * incorrect qty set in the serial no picker ([57b19a5](57b19a5)) * incorrect usage `get_cached_value` on single doctypes ([8f72a68](8f72a68)) * **Item Group:** allow root deletion ([1a6be5e](1a6be5e)) * job card suggest holiday as start date (backport [#35958](#35958)) ([#36423](#36423)) ([29ddd26](29ddd26)) * Job Card validation fixed when displaying total completed quantity ([7b3bcd3](7b3bcd3)) * multiple issues related to Production Plan (backport [#36377](#36377)) ([#36381](#36381)) ([a799e1b](a799e1b)) * not able to make material request (backport [#36416](#36416)) ([#36426](#36426)) ([99e7406](99e7406)) * only publish repost progress to doc subscriber (backport [#36400](#36400)) ([#36402](#36402)) ([32bdb7c](32bdb7c)) * overallocation validation misfire on normal invoices ([#36349](#36349)) ([09af485](09af485)) * paid_amount when the group is mode of payment ([50ef358](50ef358)) * process_owner is not link User (backport [#36420](#36420)) ([#36422](#36422)) ([289dc36](289dc36)) * removed "fetch_from" (backport [#36365](#36365)) ([#36386](#36386)) ([6d051f5](6d051f5)) * root type in account map for balance sheet ([#36303](#36303)) ([5b56296](5b56296)) * show invoices name instead of object address ([e802f0c](e802f0c)) * show tds & tcs separately ([619b0fe](619b0fe)) * test with None conditon in PE ([479cab0](479cab0)) * typo in loyalty program throw message ([#36432](#36432)) ([782a4d1](782a4d1)) ### Features * add party type filter ([32fea64](32fea64)) ### Performance Improvements * avoid full table scan in sle count check ([#36428](#36428)) ([04f9915](04f9915)) * move project status reminder to hourly (backport [#36372](#36372)) ([#36373](#36373)) ([4ac60cd](4ac60cd)) * use `LEFT JOIN` instead of `NOT EXISTS` (backport [#36221](#36221)) ([#36383](#36383)) ([26a0b3b](26a0b3b))

s-aga-r added backport version-13-hotfix backport version-14-hotfix backport to version 14 labels Jul 20, 2023

github-actions bot added the needs-tests This PR needs automated unit-tests. label Jul 20, 2023

ankush reviewed Jul 20, 2023

View reviewed changes

erpnext/manufacturing/doctype/bom_update_log/bom_update_log.py Show resolved Hide resolved

s-aga-r force-pushed the FIX-ISS-23-24-02079 branch from 5ccd656 to 843190e Compare July 28, 2023 10:24

s-aga-r changed the title ~~fix: long queue process_boms_cost_level_wise~~ perf: use join instead of sub-query Jul 28, 2023

s-aga-r force-pushed the FIX-ISS-23-24-02079 branch from 843190e to c293e21 Compare July 28, 2023 10:40

s-aga-r changed the title ~~perf: use join instead of sub-query~~ perf: use LEFT JOIN instead of NOT EXISTS Jul 28, 2023

s-aga-r force-pushed the FIX-ISS-23-24-02079 branch from c293e21 to 2ed4a11 Compare July 28, 2023 10:48

s-aga-r requested a review from ankush July 28, 2023 10:49

s-aga-r marked this pull request as ready for review July 28, 2023 10:49

s-aga-r requested a review from rohitwaghchaure as a code owner July 28, 2023 10:49

ankush reviewed Jul 28, 2023

View reviewed changes

erpnext/manufacturing/doctype/bom_update_log/bom_updation_utils.py Outdated Show resolved Hide resolved

perf: use LEFT JOIN instead of NOT EXISTS

58d8675

s-aga-r force-pushed the FIX-ISS-23-24-02079 branch from 2ed4a11 to 58d8675 Compare July 28, 2023 11:15

fix: long queue process_boms_cost_level_wise

148d466

s-aga-r merged commit d9ac7f9 into frappe:develop Jul 28, 2023
12 checks passed

s-aga-r removed the needs-tests This PR needs automated unit-tests. label Jul 28, 2023

This was referenced Jul 28, 2023

perf: use LEFT JOIN instead of NOT EXISTS (backport #36221) #36383

Merged

perf: use LEFT JOIN instead of NOT EXISTS (backport #36221) #36384

Merged

ankush mentioned this pull request Oct 20, 2023

Complete version 15 changelog #37601

Closed

github-actions bot locked as resolved and limited conversation to collaborators Feb 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf: use `LEFT JOIN` instead of `NOT EXISTS` #36221

perf: use `LEFT JOIN` instead of `NOT EXISTS` #36221

s-aga-r commented Jul 20, 2023 •

edited

Loading

codecov bot commented Jul 28, 2023

perf: use LEFT JOIN instead of NOT EXISTS #36221

perf: use LEFT JOIN instead of NOT EXISTS #36221

Conversation

s-aga-r commented Jul 20, 2023 • edited Loading

codecov bot commented Jul 28, 2023

Codecov Report

perf: use `LEFT JOIN` instead of `NOT EXISTS` #36221

perf: use `LEFT JOIN` instead of `NOT EXISTS` #36221

s-aga-r commented Jul 20, 2023 •

edited

Loading