bug: Common GRPC InactiveRpcError on v0.21 chain

[Unique-Divine]

Background

InactiveRpcError is the only hurdle to all passing tests for v0.21

Context

Line that causes the error in our code base:

   tx_resp: abci_type.TxResponse = self.execute_tx(
     tx, gas_estimate, tx_config=tx_config
   )
   tx_resp: dict[str, Any] = MessageToDict(tx_resp)
   tx_output = self.client.tx_by_hash(tx_hash=tx_resp["txhash"]) 

The simulation succeeds, and the GRPC call of execute_tx also succeeeds, giving us the tx_hash. Note however that the deprecation of broadcast mode BLOCK and migration to SYNC has removed the tx_response.code from the abci_pb2.TxResponse type.

As a result, we manually query the tx using its hash. This is raising an
inactive RPC error in the GRPC call.

Stack trace (example):

• Branch: uv/v0.21
• Commit: 75b72a3

nibiru/tx.py:102: in execute_msgs
    tx_output = self.client.tx_by_hash(tx_hash=tx_resp["txhash"])
nibiru/grpc_client.py:379: in tx_by_hash
    proto_output: tx_service.GetTxResponse = self.stubTx.GetTx(req)
.venv/lib/python3.8/site-packages/grpc/_channel.py:1030: in __call__
    return _end_unary_response_blocking(state, call, False, None)

state = <grpc._channel._RPCState object at 0x7f2de4107220>
call = <grpc._cython.cygrpc.SegregatedCall object at 0x7f2de41c0d00>
with_call = False, deadline = None

Code where the error is raised:

    def _end_unary_response_blocking(
        state: _RPCState, call: cygrpc.SegregatedCall, with_call: bool,
        deadline: Optional[float]
    ) -> Union[ResponseType, Tuple[ResponseType, grpc.Call]]:
        if state.code is grpc.StatusCode.OK:
            if with_call:
                rendezvous = _MultiThreadedRendezvous(state, call, None, deadline)
                return state.response, rendezvous
            else:
                return state.response
        else:
>           raise _InactiveRpcError(state)  # pytype: disable=not-instantiable

E           grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
E           	status = StatusCode.NOT_FOUND
E           	details = "tx not found: 7D1A949BC0DA3B7FA69BA7FF82D2F29170316448C2013D2001B8276BBA5F807F"
E           	debug_error_string = "UNKNOWN:Error received from peer ipv4:127.0.0.1:9090 {grpc_message:"tx not found: 7D1A949BC0DA3B7FA69BA7FF82D2F29170316448C2013D2001B8276BBA5F807F", grpc_status:5, created_time:"2023-07-06T19:35:56.154371464-05:00"}"
E           >

.venv/lib/python3.8/site-packages/grpc/_channel.py:910: _InactiveRpcError