[TieredStorage] Make TieredStorage::write_accounts() thread-safe

[yhchiang-sol]

Problem

While accounts-db might not invoke appends_account twice
for the same AccountsFile, TieredStorage::write_accounts()
itself isn't thread-safe, and it depends on the above accounts-db
assumption.

Summary of Changes

This PR makes TieredStorage::write_accounts() thread-safe.
So only the first thread who successfully updates the readonly
flag can proceed and writes the input accounts. All subsequent
calls to write_accounts() will be no-op and return AttemptToUpdateReadOnly
Error.

[codecov]

Codecov Report

Attention: 3 lines in your changes are missing coverage. Please review.

Comparison is base (28a320d) 81.6% compared to head (886182e) 81.6%.
Report is 15 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #35143   +/-   ##
=======================================
  Coverage    81.6%    81.6%           
=======================================
  Files         832      832           
  Lines      224856   224861    +5     
=======================================
+ Hits       183561   183583   +22     
+ Misses      41295    41278   -17     

[brooksprumo]

Overall this runtime checking is what I was hoping to avoid back when I made this comment here #32579 (comment).

Are there any runtime checks on the reader side? I did a quick look, and it appears "no". We should aim to not have any locks/atomics checking for "is ready only" on the reader side. (Or not have more than one when first opening/mapping the file.)

[yhchiang-sol]

Are there any runtime checks on the reader side? I did a quick look, and it appears "no". We should aim to not have any locks/atomics checking for "is ready only" on the reader side. (Or not have more than one when first opening/mapping the file.)

That makes sense. Let me apply to only the writer side and use the reader instance itself for the read.

[brooksprumo]

That makes sense. Let me apply to only the writer side and use the reader instance itself for the read.

Or have debug_assert!()s on the reader side that check is_read_only.

[brooksprumo]

Code looks good, some nits/comments to address

[brooksprumo]