DAOS-16445 client: Add function to cycle OIDs non-sequentially

[jolivier23]

We've noticed that with sequential order, object placement is poor.

We get 40% fill for 8GiB files with 25 ranks and 16 targets per rank
with EC_2P1G8. With this patch, we get a much better distribution.

This patch adds the following:

1. A function for cycling oid.hi incrementing by a large prime
2. For DFS, randomize the starting value
3. Modify DFS to cycle OIDs using the new function.

Required-githooks: true

Before requesting gatekeeper:

• Two review approvals and any prior change requests have been resolved.
• Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
• Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
• Commit messages follows the guidelines outlined here.
• Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

• You are the appropriate gatekeeper to be landing the patch.
• The PR has 2 reviews by people familiar with the code, including appropriate owners.
• Githooks were used. If not, request that user install them and check copyright dates.
• Checkpatch issues are resolved. Pay particular attention to ones that will show up on future PRs.
• All builds have passed. Check non-required builds for any new compiler warnings.
• Sufficient testing is done. Check feature pragmas and test tags and that tests skipped for the ticket are run and now pass with the changes.
• If applicable, the PR has addressed any potential version compatibility issues.
• Check the target branch. If it is master branch, should the PR go to a feature branch? If it is a release branch, does it have merge approval in the JIRA ticket.
• Extra checks if forced landing is requested
  • Review comments are sufficiently resolved, particularly by prior reviewers that requested changes.
  • No new NLT or valgrind warnings. Check the classic view.
  • Quick-build or Quick-functional is not used.
• Fix the commit message upon landing. Check the standard here. Edit it to create a single commit. If necessary, ask submitter for a new summary.

[github-actions]

Ticket title is 'Experiment with different oid generation algorithm'
Status is 'Open'
https://daosio.atlassian.net/browse/DAOS-16445

[daosbuild1]

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-14999/2/testReport/

[daosbuild1]

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14999/6/execution/node/1463/log

[jolivier23]

@mchaarawi In our simulations, this does quite a bit better.

[mchaarawi]

I am uncertain as to why this resolves things.
it sounds to me like this issue needs to be fixed at a lower level and not in the POSIX or other middleware layers.

[jolivier23]

I am uncertain as to why this resolves things. it sounds to me like this issue needs to be fixed at a lower level and not in the POSIX or other middleware layers.

I agree to some extent. It sounds like the placement algorithm doesn't work very well with mostly sequential oid allocation (e.g. not even close to balanced placement). I'm not sure how to fix the allocation order in any other way than in the client logic that cycles through them. If we want to fix placement hash such that it works better with sequentially allocated oids, that is probably best but also a layout change.

[jolivier23]

I am uncertain as to why this resolves things. it sounds to me like this issue needs to be fixed at a lower level and not in the POSIX or other middleware layers.

I agree to some extent. It sounds like the placement algorithm doesn't work very well with mostly sequential oid allocation (e.g. not even close to balanced placement). I'm not sure how to fix the allocation order in any other way than in the client logic that cycles through them. If we want to fix placement hash such that it works better with sequentially allocated oids, that is probably best but also a layout change.

It may be good to have this be more configurable so we can play with settings.

I suppose one other thing we could do is to do something similar with oid.lo but I suspect that may still have this issue where sequential allocation doesn't place well.

[mchaarawi]

I am uncertain as to why this resolves things. it sounds to me like this issue needs to be fixed at a lower level and not in the POSIX or other middleware layers.

I agree to some extent. It sounds like the placement algorithm doesn't work very well with mostly sequential oid allocation (e.g. not even close to balanced placement). I'm not sure how to fix the allocation order in any other way than in the client logic that cycles through them. If we want to fix placement hash such that it works better with sequentially allocated oids, that is probably best but also a layout change.

It may be good to have this be more configurable so we can play with settings.

I suppose one other thing we could do is to do something similar with oid.lo but I suspect that may still have this issue where sequential allocation doesn't place well.

if you touch oid.lo then you are introducing a dfs layout change and will have to add compatibility code.
im more worried about non DFS use cases.. it sounds like this is a more generic problem that needs a better solution IMO.

[jolivier23]

I am uncertain as to why this resolves things. it sounds to me like this issue needs to be fixed at a lower level and not in the POSIX or other middleware layers.

I agree to some extent. It sounds like the placement algorithm doesn't work very well with mostly sequential oid allocation (e.g. not even close to balanced placement). I'm not sure how to fix the allocation order in any other way than in the client logic that cycles through them. If we want to fix placement hash such that it works better with sequentially allocated oids, that is probably best but also a layout change.

It may be good to have this be more configurable so we can play with settings.
I suppose one other thing we could do is to do something similar with oid.lo but I suspect that may still have this issue where sequential allocation doesn't place well.

if you touch oid.lo then you are introducing a dfs layout change and will have to add compatibility code. im more worried about non DFS use cases.. it sounds like this is a more generic problem that needs a better solution IMO.

Only option I can think of is making a generic version of oid_gen, something like

daos_cont_oid_gen(coh, &oid);

which basically does what oid_gen does. I can take a crack at it. We may need to cherry-pick this patch in the near term though.

and maybe something like
daos_cont_oid_init(coh, num_reserved) to reserve the first few oids in oid.lo==0

[jolivier23]

We get 80% utilization before ENOSPC with 64GiB files with this patch. Previously, 8GiB files were hitting ENOSPC at around 40% utilization. We are testing 8GiB files next but this looks promising.

[mchaarawi]

it's still surprising to me why the sequential allocation does such a bad job, but this is OK for me and does not introduce layout compatibility issues with existing containers.

[jolivier23]

it's still surprising to me why the sequential allocation does such a bad job, but this is OK for me and does not introduce layout compatibility issues with existing containers.

Agreed, should probably open a ticket on this at some point though maybe not that important with a workaround. I suppose we should call this out in the docs somewhere that this is important. @Michael-Hennecke do you have a place that would fit?

[karthjyojay]

I'm confused by the first sentence of this method. Are oids 32 bit integers which means there are 2^32 possible combinations? Is this function just getting us a unique hi every time it is called? When does it get called?

[jolivier23]

The oid has 2 64-bit integers, lo and hi. The lower 96 bits (all of lo and the lo 32-bits of hi) are supposed to be unique within a container for a given object. DAOS provides a method that allocates a globally unique lo oid upon request for the container. The user, in this case DFS, then has 2^32 unique oids it can allocate to produce a globally unique oid within the container. When it runs out, it will request a new lo (it probably won't run out as that would require the process to create 2^32 + 1 objects).

This function is just a helper function to cycle through the range of available oids.

[jolivier23]

The upper 32-bits of hi are used by DAOS to encode things like the object class