Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add config for implicit transactions to neo4j extractor #2258

Merged
merged 1 commit into from
Aug 14, 2024

Conversation

kristenarmes
Copy link
Contributor

@kristenarmes kristenarmes commented Aug 14, 2024

Description

Motivation and Context

  • Adding the ability to configure the neo4j extractor to use implicit transactions. The default behavior of the extractor will remain the same.
  • Most cases can continue to use the default explicit transactions, but implicit transactions are required for CALL {} IN TRANSACTIONS subqueries. Using this can help improve performance for queries that use too much memory, for example when a Neo.ClientError.General.TransactionOutOfMemoryError exception is thrown this can help break up the query.
  • Also adding default_access_mode=neo4j.READ_ACCESS to the session configuration for all queries for better performance.

How Has This Been Tested?

Ran this in an ETL job that set the config to use implicit transactions for one, and another job that used the default explicit transactions with no config passed and confirmed expected behavior.

Documentation

N/A

CheckList

  • PR title addresses the issue accurately and concisely
  • Updates Documentation and Docstrings
  • Adds tests
  • Adds instrumentation (logs, or UI events)

Signed-off-by: Kristen Armes <karmes@lyft.com>
@boring-cyborg boring-cyborg bot added the area:databuilder From databuilder folder label Aug 14, 2024
@kristenarmes kristenarmes marked this pull request as ready for review August 14, 2024 00:33
Copy link
Contributor

@B-T-D B-T-D left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kristenarmes kristenarmes merged commit 96f642f into main Aug 14, 2024
10 checks passed
@kristenarmes kristenarmes deleted the karmes-neo4j-extractor-implicit-transactions branch August 14, 2024 01:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:databuilder From databuilder folder
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants