Rework queue and errors in prover proxy

[SantiagoPittella]

What should be done?

We can refactor a couple methods of the HttpProxy implementation in the prover proxy, simplifying the request ID handling and queues.

How should it be done?

• We can replace the sleep in upstream_peer with Notify.
• Simplify queueing and dequeueing.
• Use requestID from request context instead of headers.
• Move dequeuing in case of connection error into fail_to_connect method.

When is this task done?

The task is done when requestID isn't a header anymore and worker's queues are simplified.

Additional context

No response

[SantiagoPittella]

We may consider adding initial configs in the init command for the tx-prover CLI in this issue too:
#955 (comment)

[SantiagoPittella]

As talked with @igamigo , I'm going to implement a single queue to be shared between all workers.
Using Tokio::Notify, when a worker is available to take a job, it will send a notification, and the first request of the queue should be picked up by the worker.

[bobbinth]

As talked with @igamigo , I'm going to implement a single queue to be shared between all workers.
Using Tokio::Notify, when a worker is available to take a job, it will send a notification, and the first request of the queue should be picked up by the worker.

This could work, but also, would the approach similar to the one we have now not work? Basically, keep the status of the worker in the internal data structure and update it when the worker completes the task. Or does this not work for some reason?

[SantiagoPittella]

That works, though in that case, when all workers are occupied, we need to keep the timer that we have in the upstream_peer method to look in this data structure if a worker is available.
With Notify we add a more complexity but get rid of the timer.

Will go for the internal structure first though, and then we can decide if we want to pivot to Notify.

[SantiagoPittella]

May we consider this as done?

Cc @bobbinth

[bobbinth]

Yes - definitely!

Closed by #976.