Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: migrating data from GL to payment ledger using raw SQL #31182

Conversation

ruthra-kumar
Copy link
Member

@ruthra-kumar ruthra-kumar commented May 30, 2022

Given the relatively large amount of data in GL Entry, the original patch to migrate data from GL Entry to payment ledger may face performance issues. Hence the patch file has been modified to use raw sql to perform insertion directly into Payment Ledger table.

Changes in PR:

  • Patch logic has been modified to use raw SQL for insertion.
  • creation, modified, modified_by doctype fields are also included in migration. Creation field is required by AR/AP report. This also serves in maintaining data integrity.
  • Removed Payment Ledger naming expression

@github-actions github-actions bot added the needs-tests This PR needs automated unit-tests. label May 30, 2022
@ruthra-kumar ruthra-kumar changed the title refactor: migration data from GL to payment ledger using raw SQL refactor: migrating data from GL to payment ledger using raw SQL May 30, 2022
@ruthra-kumar ruthra-kumar requested review from nabinhait and removed request for ankush and marination May 30, 2022 13:47
@codecov
Copy link

codecov bot commented May 30, 2022

Codecov Report

Merging #31182 (e2c5243) into develop (3019f93) will increase coverage by 0.42%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           develop   #31182      +/-   ##
===========================================
+ Coverage    63.04%   63.47%   +0.42%     
===========================================
  Files          986      988       +2     
  Lines        67343    67605     +262     
===========================================
+ Hits         42457    42909     +452     
+ Misses       24886    24696     -190     
Impacted Files Coverage Δ
...unts/report/purchase_register/purchase_register.py 34.21% <0.00%> (-44.08%) ⬇️
...ext/accounts/report/balance_sheet/balance_sheet.py 36.36% <0.00%> (-21.82%) ⬇️
...e_purchase_register/item_wise_purchase_register.py 57.42% <0.00%> (-20.80%) ⬇️
...rpnext/accounts/doctype/shareholder/shareholder.py 80.00% <0.00%> (-10.00%) ⬇️
...urity_shortfall/process_loan_security_shortfall.py 93.75% <0.00%> (-6.25%) ⬇️
erpnext/hr/doctype/job_offer/job_offer.py 82.92% <0.00%> (-4.88%) ⬇️
...it_and_loss_statement/profit_and_loss_statement.py 71.42% <0.00%> (-3.90%) ⬇️
...ext/stock/doctype/stock_entry/stock_entry_utils.py 93.33% <0.00%> (-2.32%) ⬇️
erpnext/stock/doctype/warehouse/warehouse.py 77.62% <0.00%> (-2.10%) ⬇️
...cturing/doctype/bom_update_tool/bom_update_tool.py 79.31% <0.00%> (-1.94%) ⬇️
... and 77 more

@ruthra-kumar ruthra-kumar force-pushed the migrate_gl_to_payment_ledger_using_sql branch 2 times, most recently from 29cdec7 to 8848548 Compare June 6, 2022 07:16
@ruthra-kumar ruthra-kumar marked this pull request as draft June 8, 2022 00:41
@ruthra-kumar ruthra-kumar force-pushed the migrate_gl_to_payment_ledger_using_sql branch from 8848548 to 6c9004b Compare June 8, 2022 12:07
@ruthra-kumar ruthra-kumar marked this pull request as ready for review June 9, 2022 11:12
@ruthra-kumar ruthra-kumar force-pushed the migrate_gl_to_payment_ledger_using_sql branch from 6c9004b to e2c5243 Compare June 10, 2022 07:48
@deepeshgarg007 deepeshgarg007 merged commit 3d76b1a into frappe:develop Jun 10, 2022
# split data into chunks
chunk_size = 1000
try:
for i in range(0, len(gl_entries), chunk_size):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ruthra-kumar ruthra-kumar deleted the migrate_gl_to_payment_ledger_using_sql branch June 14, 2023 16:07
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 25, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
accounts needs-tests This PR needs automated unit-tests.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants