Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix misleading messages logged by SpoolDirLineDelimitedSourceConnector #212

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

ashwinpankaj
Copy link

@ashwinpankaj ashwinpankaj commented Jan 8, 2024

#213

Found two misleading logs while testing SpoolDirLineDelimitedSourceConnector

  1. Finished processing 0 record(s) in 0 second(s) is logged even though connector is processing records
  2. Failed to delete input.path sub-directory: /tmp/data/input/fix.json is logged even though that file does not exist in the path as it has been moved after processing

Fixes

  1. Ensure SpoolDirLineDelimitedSourceTask updates recordCount in AbstractSourceTask and calculate processing speed in milliseconds (sometimes files are processed in less than a second, which leaves secondsElapsed set to 0)
  2. getInputPathSubDirsToCleanup() should only return dirs which exist

@ashwinpankaj ashwinpankaj changed the title Fix misleading messages logged by SpoolDirLineDelimitedSourceConnector Fix misleading messages logged by SpoolDirLineDelimitedSourceConnector #213 Jan 8, 2024
@ashwinpankaj ashwinpankaj changed the title Fix misleading messages logged by SpoolDirLineDelimitedSourceConnector #213 Fix misleading messages logged by SpoolDirLineDelimitedSourceConnector Jan 8, 2024
@ashwinpankaj
Copy link
Author

@jcustenborder could you please review this PR ?

1 similar comment
@ashwinpankaj
Copy link
Author

@jcustenborder could you please review this PR ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant