Postgres pipelining and async blocks

[fiadliel]

Hi!
I've found the pipelining support to work well in simple scenarios, where single queries can be run in parallel, e.g. the result of the query is directly returned to the caller, and the function returns a boxed future, or impl Future. In these cases, multiple queries can be run concurrently with try_join!.

The issue/limitation I have encountered, is that any use of async seems to capture the connection, so that it's impossible to create any further concurrent calls: cannot borrow `*conn` as mutable more than once at a time

A very simple example, just based on the signatures:

fn a(conn: &mut AsyncPgConnection) -> impl Future<Output = QueryResult<Vec<Row>>> {
    todo!()
}

async fn b(conn: &mut AsyncPgConnection) -> QueryResult<Vec<Row>> {
    todo!()
}

async fn a_and_b(mut conn: AsyncPgConnection) {
    let a_fut = a(&mut conn);
    let b_fut = b(&mut conn);

    futures::try_join!(a_fut, b_fut);
}

This will compile as is, because b_fut is evaluated last, and so the connection is only captured once. If the order of evaluation of a_fut and b_buf is reversed, compilation fails.

It's all fine if there is only one async block. But if I want to use two async blocks — say, there are dependent chains of postgres queries a->b and c->d which I'd like to run concurrently — this is impossible.

Has anybody got any workarounds for this? (other than just drop to normal pooling).
And if this is a known and expected ongoing limitation, I can send a PR to update the documentation around pipelining to mention this.

[weiznich]

That's kind of expected behavior as the connection interface is not just postgres specific and other backends need the mutable reference there.

I think this kind of issue can generally be soved by returning a BoxedFuture<> (or even an impl Future<> with explicit captures) from your functions instead of using an async fn. I can see that this is not optimal but I really do not see a better way to provide this kind of interface at all without duplicating essentially the whole connection interface for the postgres connection to allow it using &self instead.

As for the documentation: PR's improving the documentation are always welcome.

[fakeshadow]

I have experiment with usage like this where I impl AsyncConnection trait for &ConnectionType. It kinda works though the downside is also obvious as with how the trait defined it's easy to misuse it with transaction where it shouldn't be. I wonder if said trait can be break up into multiple pieces where &mut &conn can be used for query but not for starting a transaction.

[weiznich]

I just noticed that I never responded here: Yes it might be meaningful to split the trait into two parts for this. I would accept a PR that implements this as long as it contains a demonstration that this actually works.