Cache Block Location to save memory #16953
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@maobaolong yup thanks |
jiacheliu3
approved these changes
Mar 2, 2023
core/common/src/main/java/alluxio/util/proto/BlockLocationUtils.java
Outdated
Show resolved
Hide resolved
elega
force-pushed
the
yimin/cache-block-location
branch
from
48086a9
to
a5b6082
Compare
maobaolong
reviewed
Mar 3, 2023
| private static final IndexDefinition<BlockLocation, Long> WORKER_ID_INDEX = | ||
| IndexDefinition.ofNonUnique(BlockLocation::getWorkerId); | ||
|
|
||
| // TODO(maobaolong): Add a metric to monitor the size of mLocationCacheMap |
There was a problem hiding this comment.
As mLocationCacheMap renamed to BLOCK_LOCATION_CACHE , please update the comment together
| private static final IndexedSet<BlockLocation> BLOCK_LOCATION_CACHE = | ||
| new IndexedSet<>(OBJECT_INDEX, WORKER_ID_INDEX); | ||
|
|
||
| private static final Set<String> VALID_MEDIUM_TYPE_VALUES = |
There was a problem hiding this comment.
Athough MEM, SSD, HDD is the default value of MASTER_TIERED_STORE_GLOBAL_MEDIUMTYPE, but how to handle it when MASTER_TIERED_STORE_GLOBAL_MEDIUMTYPE configured to other value?
How about get the set from MASTER_TIERED_STORE_GLOBAL_MEDIUMTYPE?
There was a problem hiding this comment.
i'll add a comment for this one
| Preconditions.checkState(VALID_MEDIUM_TYPE_VALUES.contains(blockLocation.getTier()), | ||
| "TierAlias must be one of {MEM, HDD and SSD} but got %s", | ||
| blockLocation.getTier()); | ||
| Preconditions.checkState(VALID_MEDIUM_TYPE_VALUES.contains(blockLocation.getMediumType()), |
There was a problem hiding this comment.
The supported mediumTypes are read from configuration too.
PropertyKey tierDirMediumConf =
PropertyKey.Template.WORKER_TIERED_STORE_LEVEL_DIRS_MEDIUMTYPE.format(mTierOrdinal);
List<String> dirMedium = Configuration.getList(tierDirMediumConf);
|
alluxio-bot, merge this please |
jiacheliu3
pushed a commit
to jiacheliu3/alluxio
that referenced
this pull request
May 19, 2023
…memory Cache block location object to reduce memory consumption Number of possible Block location value is limited, up to 9 * number of workers. By caching it, memory consumption can be saved. N/A pr-link: Alluxio#16953 change-id: cid-9464ad2e70284b50d4c8a05dd5138f35e78bcb2d
jiacheliu3
pushed a commit
to jiacheliu3/alluxio
that referenced
this pull request
May 19, 2023
Finish the todo task after Alluxio#16953 pr-link: Alluxio#17056 change-id: cid-e6ddc03b08e7a8a18cd53d77fa9a2f0edf4e1f57
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Cache block location object to reduce memory consumption
Why are the changes needed?
Number of possible Block location value is limited, up to 9 * number of workers. By caching it, memory consumption can be saved.
Does this PR introduce any user facing changes?
N/A