feat: add retry test inject failure midway in gRPC ReadObject

[cojenco]

• Add Retry Test API support for injecting failure midway in gRPC ReadObject

  • Testbench will fail with UNAVAILABLE after a few downloaded data with retry instructions return-broken-stream
  • Testbench will fail with UNAVAILABLE after YKiB of downloaded data with retry instructions return-broken-stream-after-YK

• Update README

• Enable methods that have Retry Test support in gRPC

  •   "storage.buckets.get",
      "storage.buckets.getIamPolicy",
      "storage.buckets.list",
      "storage.hmacKey.get",
      "storage.hmacKey.list",
      "storage.notifications.get",
      "storage.notifications.list",
      "storage.objects.list",
      "storage.objects.get",
      "storage.serviceaccount.get",
    

• Tests pass

• Appropriate changes to README are included in PR

[codecov]

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (f929482) 98.68% compared to head (daa81b6) 98.69%.
Report is 4 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #533   +/-   ##
=======================================
  Coverage   98.68%   98.69%           
=======================================
  Files          50       50           
  Lines        7898     7956   +58     
=======================================
+ Hits         7794     7852   +58     
  Misses        104      104           

Flag	Coverage Δ
unittests	98.69% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
testbench/common.py	96.75% <100.00%> (+0.05%)	⬆️
testbench/database.py	100.00% <100.00%> (ø)
testbench/grpc_server.py	100.00% <100.00%> (ø)
testbench/rest_server.py	97.84% <100.00%> (ø)
tests/test_common.py	99.79% <100.00%> (ø)
tests/test_testbench_retry.py	99.61% <100.00%> (+0.04%)	⬆️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tritone]

One high level question about the approach; let me know what you think.

[cojenco]

As discussed offline, I updated Retry Test API Forced Failures Support description by transport and clarified how retry instructions result in the gRPC context. @tritone PTAL, thank you!

[tritone]

Looks good to me; thanks for the updates.

[cojenco]

@coryan Wanted to check with you for future planning if there is any need to support x-goog-emulator-instructions force failures (e.g. return-503-after-256K) in the grpc server?

This PR adds support to the Retry Test API. Planning to merge today if no issues, thanks!

[coryan]

@coryan Wanted to check with you for future planning if there is any need to support x-goog-emulator-instructions force failures (e.g. return-503-after-256K) in the grpc server?

No, and we can always add the support if it becomes needed.