feat: Reconcile Payments in background #34596

ruthra-kumar · 2023-03-27T08:43:54Z

For extremely large volumes of Invoices and Payments, Payment Reconciliation tool may timeout. This feature allows for running the same reconciliation tool in the background.

Documentation can be found here

Changes:

2 new doctypes: Process Payment Reconciliation and Process Payment Reconciliation Log have been introduced. These reuse the existing reconciliation logic. Process Payment Reconciliation will have similar filters as Payment Reconciliation. Once submitted, 'Reconcile in Background' button will be availabe. Once triggered, Process Payment Reconciliation Log will be created. Payments are allocated in FIFO order.

Added a checkbox in Accounts Settings to enable/disable this feature
Added few validations in Payment Reconciliation tool to check for running jobs with same filter.

codecov · 2023-03-28T08:42:49Z

Codecov Report

Merging #34596 (f99d479) into develop (fc6486d) will decrease coverage by 0.19%.
The diff coverage is 16.40%.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop   #34596      +/-   ##
===========================================
- Coverage    63.87%   63.68%   -0.19%     
===========================================
  Files          810      813       +3     
  Lines        59750    60010     +260     
===========================================
+ Hits         38166    38220      +54     
- Misses       21584    21790     +206

Impacted Files	Coverage Δ
...iliation_log/process_payment_reconciliation_log.py	`0.00% <0.00%> (ø)`
.../process_payment_reconciliation_log_allocations.py	`0.00% <0.00%> (ø)`
erpnext/hooks.py	`100.00% <ø> (ø)`
...t_reconciliation/process_payment_reconciliation.py	`11.94% <11.94%> (ø)`
...e/payment_reconciliation/payment_reconciliation.py	`94.11% <69.23%> (-1.43%)`	⬇️
erpnext/accounts/utils.py	`75.53% <100.00%> (+0.03%)`	⬆️

... and 9 files with indirect coverage changes

erpnext/accounts/doctype/accounts_settings/accounts_settings.json

erpnext/accounts/doctype/auto_reconcile/auto_reconcile.js

erpnext/accounts/doctype/auto_reconcile/auto_reconcile.py

erpnext/accounts/doctype/auto_reconcile_log/auto_reconcile_log.js

erpnext/accounts/doctype/process_payment_reconciliation/process_payment_reconciliation.py

erpnext/accounts/doctype/process_payment_reconciliation/process_payment_reconciliation_list.js

erpnext/accounts/doctype/process_payment_reconciliation/process_payment_reconciliation.py

1. Split reconcile function into 2 2. While reconciling against payment entry, skip a set_missing_ref_details setter method

pause checkbox has been removed

1. Moved the status section to the bottom in parent doc 2. Using alerts to indicate Job trigger status

* feat: auto reconcile in background * chore: Option to enable auto reconciliation in settings * refactor: validate if feature is enabled in settings * refactor: check for running job while using reconciliation tool * chore: using doc to get filter values * chore: use frappe.db.get_value in validations * chore: cleanup commented out code * chore: replace get_list with get_all * chore: use block scope variable * chore: type information for functions * refactor: flag to ignore job validation check * refactor: update parent doc status if all reconciled * chore: create test_records file * test: create a bunch of vouchers for testing auto reconcile * chore: renamed auto_reconcile to process_payment_reconciliation * chore: another child doctype to hold payments * chore: remove duplicate field * chore: add fetched payments to log * chore: Popup comment message update * chore: replace get_all with get_value * chore: replace label in settings page * chore: remove unit test and records * refactor: status in reconciliation log * refactor: set status in log as well * chore: fix field name * chore: change triggered job name * chore: use status field in list view of log * chore: status while there are no allocations * refactor: split trigger function into two * chore: adding cancelled status * refactor: function trigger queued docs * chore: cron job scheduled * chore: fixing accouts settings json file * chore: typos and variable scope * chore: use 'pluck' in db call * chore: remove redundant whitelist decorator * chore: use single DB call to fetch values * chore: replace get_all with get_value * refactor: use raw db calls to fetch reconciliation log records Using get_doc on `Process Payment Reconciliation Log` is costly when handling large volumes of invoices. Use raw frappe.db.get_all to selectively pull status and reconciled count * chore: update status on successful batch operation * chore: make payment table readonly * chore: ability to pause the background job * chore: remove isolate_each_allocation * chore: more description in progress bar * refactor: partially working state * refactor: update reconcile flag and setting hard limits for fetching * chore: make allocation editable -- NEED TO REVERT * chore: pause button * refactor: skip setter function in Payment Entry for better performan * refactor: split reconcile function and skip a setter function 1. Split reconcile function into 2 2. While reconciling against payment entry, skip a set_missing_ref_details setter method * chore: increase payment limit * refactor: replace frappe.db.get_all with frappe.db.get_value * chore: remove unwanted doctypes * refactor: make allocation table readonly * perf: update ref_details only for newly linked invoices * chore: rename skip flag * refactor(UI): receivable_payable field should auto populate * refactor: no control statements in finally block * chore: cleanup section and rename checkbox * chore: update new fieldname in code * chore: update error msg * refactor: start and pause integrated into status pause checkbox has been removed * refactor: added cancelled status to the log doctype 1. Moved the status section to the bottom in parent doc 2. Using alerts to indicate Job trigger status (cherry picked from commit ed14d1c) # Conflicts: # erpnext/accounts/doctype/accounts_settings/accounts_settings.json

* feat: auto reconcile in background * chore: Option to enable auto reconciliation in settings * refactor: validate if feature is enabled in settings * refactor: check for running job while using reconciliation tool * chore: using doc to get filter values * chore: use frappe.db.get_value in validations * chore: cleanup commented out code * chore: replace get_list with get_all * chore: use block scope variable * chore: type information for functions * refactor: flag to ignore job validation check * refactor: update parent doc status if all reconciled * chore: create test_records file * test: create a bunch of vouchers for testing auto reconcile * chore: renamed auto_reconcile to process_payment_reconciliation * chore: another child doctype to hold payments * chore: remove duplicate field * chore: add fetched payments to log * chore: Popup comment message update * chore: replace get_all with get_value * chore: replace label in settings page * chore: remove unit test and records * refactor: status in reconciliation log * refactor: set status in log as well * chore: fix field name * chore: change triggered job name * chore: use status field in list view of log * chore: status while there are no allocations * refactor: split trigger function into two * chore: adding cancelled status * refactor: function trigger queued docs * chore: cron job scheduled * chore: fixing accouts settings json file * chore: typos and variable scope * chore: use 'pluck' in db call * chore: remove redundant whitelist decorator * chore: use single DB call to fetch values * chore: replace get_all with get_value * refactor: use raw db calls to fetch reconciliation log records Using get_doc on `Process Payment Reconciliation Log` is costly when handling large volumes of invoices. Use raw frappe.db.get_all to selectively pull status and reconciled count * chore: update status on successful batch operation * chore: make payment table readonly * chore: ability to pause the background job * chore: remove isolate_each_allocation * chore: more description in progress bar * refactor: partially working state * refactor: update reconcile flag and setting hard limits for fetching * chore: make allocation editable -- NEED TO REVERT * chore: pause button * refactor: skip setter function in Payment Entry for better performan * refactor: split reconcile function and skip a setter function 1. Split reconcile function into 2 2. While reconciling against payment entry, skip a set_missing_ref_details setter method * chore: increase payment limit * refactor: replace frappe.db.get_all with frappe.db.get_value * chore: remove unwanted doctypes * refactor: make allocation table readonly * perf: update ref_details only for newly linked invoices * chore: rename skip flag * refactor(UI): receivable_payable field should auto populate * refactor: no control statements in finally block * chore: cleanup section and rename checkbox * chore: update new fieldname in code * chore: update error msg * refactor: start and pause integrated into status pause checkbox has been removed * refactor: added cancelled status to the log doctype 1. Moved the status section to the bottom in parent doc 2. Using alerts to indicate Job trigger status (cherry picked from commit ed14d1c)

feat: Reconcile Payments in background (#34596) * feat: auto reconcile in background * chore: Option to enable auto reconciliation in settings * refactor: validate if feature is enabled in settings * refactor: check for running job while using reconciliation tool * chore: using doc to get filter values * chore: use frappe.db.get_value in validations * chore: cleanup commented out code * chore: replace get_list with get_all * chore: use block scope variable * chore: type information for functions * refactor: flag to ignore job validation check * refactor: update parent doc status if all reconciled * chore: create test_records file * test: create a bunch of vouchers for testing auto reconcile * chore: renamed auto_reconcile to process_payment_reconciliation * chore: another child doctype to hold payments * chore: remove duplicate field * chore: add fetched payments to log * chore: Popup comment message update * chore: replace get_all with get_value * chore: replace label in settings page * chore: remove unit test and records * refactor: status in reconciliation log * refactor: set status in log as well * chore: fix field name * chore: change triggered job name * chore: use status field in list view of log * chore: status while there are no allocations * refactor: split trigger function into two * chore: adding cancelled status * refactor: function trigger queued docs * chore: cron job scheduled * chore: fixing accouts settings json file * chore: typos and variable scope * chore: use 'pluck' in db call * chore: remove redundant whitelist decorator * chore: use single DB call to fetch values * chore: replace get_all with get_value * refactor: use raw db calls to fetch reconciliation log records Using get_doc on `Process Payment Reconciliation Log` is costly when handling large volumes of invoices. Use raw frappe.db.get_all to selectively pull status and reconciled count * chore: update status on successful batch operation * chore: make payment table readonly * chore: ability to pause the background job * chore: remove isolate_each_allocation * chore: more description in progress bar * refactor: partially working state * refactor: update reconcile flag and setting hard limits for fetching * chore: make allocation editable -- NEED TO REVERT * chore: pause button * refactor: skip setter function in Payment Entry for better performan * refactor: split reconcile function and skip a setter function 1. Split reconcile function into 2 2. While reconciling against payment entry, skip a set_missing_ref_details setter method * chore: increase payment limit * refactor: replace frappe.db.get_all with frappe.db.get_value * chore: remove unwanted doctypes * refactor: make allocation table readonly * perf: update ref_details only for newly linked invoices * chore: rename skip flag * refactor(UI): receivable_payable field should auto populate * refactor: no control statements in finally block * chore: cleanup section and rename checkbox * chore: update new fieldname in code * chore: update error msg * refactor: start and pause integrated into status pause checkbox has been removed * refactor: added cancelled status to the log doctype 1. Moved the status section to the bottom in parent doc 2. Using alerts to indicate Job trigger status (cherry picked from commit ed14d1c) Co-authored-by: ruthra kumar <ruthra@erpnext.com>

# [14.23.0](v14.22.3...v14.23.0) (2023-04-25) ### Bug Fixes * `PermissionError` in Work Order ([a3568c1](a3568c1)) * Add company field to lower deduction certificate ([#34914](#34914)) ([d732083](d732083)) * add item-code filter for SCR supplied-items batch-no ([7c4f83e](7c4f83e)) * add limit for get_next_stock_reco ([#34937](#34937)) ([7131ff2](7131ff2)) * added validation for extra job card ([a5fde5d](a5fde5d)) * batch qty conversion factor issue fixed in pos transaction ([#34917](#34917)) ([56ef0ba](56ef0ba)) * broken 'set exchange gain/loss' btn in payment entry ([#34940](#34940)) ([00968ba](00968ba)) * Bulk Payment Entry from PO/SO ([#34942](#34942)) ([878d747](878d747)) * click handler should not attempt indexed access of empty array ([#35013](#35013)) ([06f204a](06f204a)) * Common party JV cost center ([#35008](#35008)) ([693007a](693007a)) * duplicate reposting entries of same voucher ([aeac43c](aeac43c)) * FEC report for France accountancy ([#34781](#34781)) ([6aabab2](6aabab2)) * filtering via batch no ([#34951](#34951)) ([3b23fc1](3b23fc1)), closes [#34950](#34950) * Incorrect difference value in Stock and Account Value Comparison report ([e7ca833](e7ca833)) * incorrect OR condition causing timeout error ([c020789](c020789)) * internal Purchase Receipt GL Entries ([f7e436f](f7e436f)) * item not showing in the BOM ([28cd79a](28cd79a)) * pass reference_doctype in link queries (backport [#35038](#35038)) ([#35039](#35039)) ([5cc3978](5cc3978)) * Payment entry with TDS in bank reco statement ([#34961](#34961)) ([f43ea0d](f43ea0d)) * per_billed condition for Payment Entry ([#34969](#34969)) ([f9f42c7](f9f42c7)) * process_loss_percentage in BOM ([4418862](4418862)) * removed depends on for the Employee Detail section ([9d17d3f](9d17d3f)) * respect title_field from doctype to bulk transactions ([#34928](#34928)) ([29aa4a0](29aa4a0)) * set `frappe.flags.company` to call regional code accurately ([789dfd6](789dfd6)) * simplify `erpnext.get_region` ([87595bd](87595bd)) * SLA permissions (backport [#34981](#34981)) ([#34986](#34986)) ([83a1b83](83a1b83)) * stock entry type issue ([33a1608](33a1608)) * **test:** `test_backdated_stock_reco_cancellation_future_negative_stock` ([2ad157b](2ad157b)) * **test:** `test_internal_pr_gl_entries` ([7740ceb](7740ceb)) * Unable to allocate advance against invoice ([#35007](#35007)) ([5045ad6](5045ad6)) * use `functools.wraps` to preserve doc signature ([2ec18eb](2ec18eb)) * use CombineDatetime instead of Timestamp in QB queries ([55da91c](55da91c)) * use filter_by_finance_book instead of only_depreciable_assets in fixed asset register (backport [#35031](#35031)) ([#35035](#35035)) ([5630e81](5630e81)) * Use set instead of db_set as it is called from validate ([#34967](#34967)) ([6650212](6650212)) * v14, Bank Reconcile Tools not cover case JV debit bank ([#35000](#35000)) ([82d8379](82d8379)) * validation for internal transfer entry ([a582354](a582354)) * value of depreciable assets not updating after manual depr entry [v14] ([#35010](#35010)) ([3c75e55](3c75e55)) * wrong qty of remaining work orders to be created when using "Create" > "Work Order" ([#34726](#34726)) ([47df41f](47df41f)) ### Features * Reconcile Payments in background ([#34596](#34596)) ([5923a80](5923a80)) ### Performance Improvements * Journal Entries (backport [#34918](#34918)) ([#35054](#35054)) ([39b5147](39b5147)) ### Reverts * Revert "fix: Rate from LDC in TDS reports (#33699)" ([f44a79f](f44a79f)), closes [#33699](#33699)

ruthra-kumar force-pushed the auto_reconcile_using_fifo branch 2 times, most recently from 8972938 to d7fc1c4 Compare March 28, 2023 06:23

deepeshgarg007 requested changes Mar 28, 2023

View reviewed changes

ruthra-kumar changed the title ~~feat: auto reconcile in background~~ feat: Reconcile Payments in background Apr 3, 2023

ruthra-kumar force-pushed the auto_reconcile_using_fifo branch from 32bb366 to e7cb620 Compare April 4, 2023 12:22

deepeshgarg007 requested changes Apr 9, 2023

View reviewed changes

This was referenced Apr 19, 2023

refactor: refactor set_missing_values and set_missing_ref_details in Payment Entry #34922

Merged

refactor: update ref details on selected references on Payment Entry frappe/hrms#458

Merged

ruthra-kumar added 21 commits April 20, 2023 11:05

feat: auto reconcile in background

9851358

chore: Option to enable auto reconciliation in settings

67dcd44

refactor: validate if feature is enabled in settings

9e7e392

refactor: check for running job while using reconciliation tool

05f25a6

chore: using doc to get filter values

767cda1

chore: use frappe.db.get_value in validations

9649b6b

chore: cleanup commented out code

4e945d4

chore: replace get_list with get_all

5cd97cf

chore: use block scope variable

a6d7eff

chore: type information for functions

e92cc2d

refactor: flag to ignore job validation check

8c7a20a

refactor: update parent doc status if all reconciled

8e2c7bc

chore: create test_records file

01038a4

test: create a bunch of vouchers for testing auto reconcile

a2fb9bd

chore: renamed auto_reconcile to process_payment_reconciliation

c50aa38

chore: another child doctype to hold payments

a4d3d7b

chore: remove duplicate field

b17486d

chore: add fetched payments to log

0c022ca

chore: Popup comment message update

025ab20

chore: replace get_all with get_value

9d23c30

chore: replace label in settings page

3d9733f

ruthra-kumar added 11 commits April 20, 2023 11:11

chore: make allocation editable -- NEED TO REVERT

f23ac7a

chore: pause button

5c2f26d

refactor: skip setter function in Payment Entry for better performan

ddc699d

refactor: split reconcile function and skip a setter function

7250772

1. Split reconcile function into 2 2. While reconciling against payment entry, skip a set_missing_ref_details setter method

chore: increase payment limit

b729714

refactor: replace frappe.db.get_all with frappe.db.get_value

6b8b6f8

chore: remove unwanted doctypes

4047494

refactor: make allocation table readonly

0b8d0bd

perf: update ref_details only for newly linked invoices

170c63b

chore: rename skip flag

0b930e0

refactor(UI): receivable_payable field should auto populate

dfecf49

ruthra-kumar force-pushed the auto_reconcile_using_fifo branch from 556aff1 to dfecf49 Compare April 20, 2023 10:20

ruthra-kumar added 5 commits April 20, 2023 16:18

refactor: no control statements in finally block

0c4cd7c

chore: cleanup section and rename checkbox

81e57ec

chore: update new fieldname in code

1d73e92

chore: update error msg

68f2e0d

refactor: start and pause integrated into status

b46aa3e

pause checkbox has been removed

deepeshgarg007 marked this pull request as ready for review April 21, 2023 12:53

refactor: added cancelled status to the log doctype

f99d479

1. Moved the status section to the bottom in parent doc 2. Using alerts to indicate Job trigger status

ruthra-kumar force-pushed the auto_reconcile_using_fifo branch from 1b50f57 to f99d479 Compare April 21, 2023 13:56

deepeshgarg007 merged commit ed14d1c into frappe:develop Apr 22, 2023

deepeshgarg007 added the backport version-14-hotfix backport to version 14 label Apr 22, 2023

mergify bot mentioned this pull request Apr 22, 2023

feat: Reconcile Payments in background (backport #34596) #34996

Merged

ruthra-kumar deleted the auto_reconcile_using_fifo branch June 14, 2023 16:03

ankush mentioned this pull request Oct 20, 2023

Complete version 15 changelog #37601

Closed

github-actions bot locked as resolved and limited conversation to collaborators Feb 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Reconcile Payments in background #34596

feat: Reconcile Payments in background #34596

ruthra-kumar commented Mar 27, 2023 •

edited

Loading

codecov bot commented Mar 28, 2023 •

edited

Loading

feat: Reconcile Payments in background #34596

feat: Reconcile Payments in background #34596

Conversation

ruthra-kumar commented Mar 27, 2023 • edited Loading

codecov bot commented Mar 28, 2023 • edited Loading

Codecov Report

ruthra-kumar commented Mar 27, 2023 •

edited

Loading

codecov bot commented Mar 28, 2023 •

edited

Loading