Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix WebHDFS adapter query to retry failed resources #332

Merged
merged 6 commits into from
Jun 19, 2020

Conversation

mdang8
Copy link

@mdang8 mdang8 commented Jun 17, 2020

What does this PR do?

Fixes the query method for the WebHDFS adapter to include failed resources from previous replication runs. The QueryRequest passed to the query method contains a list of failed item IDs from the previous run of replication. These items should be included in the next run to retry creating them in DDF.

Who is reviewing it (please choose AT LEAST two reviewers that need to approve the PR before it can get merged)?

@cjlange
@aaronilovici
@josephthweatt

How should this be tested? (List steps with links to updated documentation)

  1. Build and install replication.
  2. Configure a replication job with the HDFS instance as the source and DDF as the destination.
  3. Upload some files to HDFS to be replicated.
  4. Connect a remote debugger to DDF and set a breakpoint on line 302 in DdfNodeAdapter.
  5. When the breakpoint gets hit, set the value of the response variable to null.
  6. Resume execution in the debugger and verify an error for creating the metadata gets printed in the logs.
  7. On the next replication run, verify a line gets printed in the logs that says "Found failed item IDs: [...]" with the failed ID inside the brackets.
  8. Verify the failed item gets successfully replicated after that run.

Any background context you want to provide?

Integration tests are not included in this pull request.

What are the relevant tickets?

Fixes #331

Screenshots (if appropriate)

Checklist:

  • Documentation Updated
  • Update / Add Unit Tests
  • Update / Add Integration Tests

@mdang8 mdang8 linked an issue Jun 17, 2020 that may be closed by this pull request
@cjlange
Copy link

cjlange commented Jun 18, 2020

build now

Copy link

@aaronilovici aaronilovici left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This LGTM, pending any changes coming from Chris.

@mdang8
Copy link
Author

mdang8 commented Jun 18, 2020

build now please

@mdang8
Copy link
Author

mdang8 commented Jun 19, 2020

build now 🛠️

@josephthweatt
Copy link

Hero-ed successfully on my machine. LGTM!

@mdang8 mdang8 merged commit dcb9d52 into connexta:0.3.x Jun 19, 2020
@mdang8 mdang8 deleted the webhdfs-query-retries branch June 19, 2020 17:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implementation for retries in WebHDFS adapter
4 participants