Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FileSystem.createBulkDelete introduced in Hadoop 3.4.1 #5131

Open
dlmarion opened this issue Dec 3, 2024 · 2 comments
Open

FileSystem.createBulkDelete introduced in Hadoop 3.4.1 #5131

dlmarion opened this issue Dec 3, 2024 · 2 comments
Milestone

Comments

@dlmarion
Copy link
Contributor

dlmarion commented Dec 3, 2024

Could look at bumping Hadoop version from 3.4.0 to 3.4.1 in Accumulo 3.1 branch and changing the GarbageCollector to use this new API.

@dlmarion dlmarion added this to the 3.1.0 milestone Dec 3, 2024
@ctubbsii
Copy link
Member

ctubbsii commented Dec 4, 2024

Currently, we build with 3.4.0 (now that 3.4.1 is out, we'll be bumping to that, since we try to build with the latest). But, we do have a GitHub Actions check to ensure we still are compatible with the oldest one we support. That's currently checking 3.0.3. We can bump the minimum required Hadoop version to 3.4.1 to do this, but it should be well-documented, and once the change is merged forward into the main branch, any cluster testing of the elasticity work would only be able to be done on a 3.4.1 HDFS cluster (which is fine... just something to keep in mind).

@ctubbsii
Copy link
Member

ctubbsii commented Dec 4, 2024

The one big downside to bumping to 3.4.1 as the minimum supported version is that it is currently also the latest version. So, that restricts users to only a single possible HDFS version to run for now. That can be a big problem for users if there's any serious bugs in 3.4.1 that users want to avoid. Right now, they can use any version between 3.0.3 and 3.4.1, so it's easy to avoid versions that have bugs that users want to avoid. That wouldn't be the case after this change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants