adbc_driver_snowflake error when writing table to snowflake

[connelld-dpsk12]

What happened?

ERRO[0016]connection.go:275 gosnowflake.(*snowflakeConn).Close context canceled

Ive tries both python 3.8 and 3.10 when using the adbc engine to write_database with polars. The funny thing is, the data does indeed write to the table, however this error still occurs. Looks like other people have had a similar issue.

Stack Trace

ERRO[0016]connection.go:275 gosnowflake.(*snowflakeConn).Close context canceled

How can we reproduce the bug?

df.write_database(connection=conn, table_name=table, if_table_exists='replace', engine='adbc')

Environment/Setup

adbc-driver-snowflake==1.1.0

[lidavidm]

Looks like other people have had a similar issue

Just for reference, where are the other reports about this?

[connelld-dpsk12]

Looks like other people have had a similar issue

Just for reference, where are the other reports about this?

pola-rs/polars#17666 I guess it was on the polars repo

[lidavidm]

CC @joellubi / @zeroshade (are we forgetting to wait for a task somewhere?)

[joellubi]

Hi @connelld-dpsk12 and @lidavidm. I've seen this issue recently as well and it appears to be upstream of us in gosnowflake. I opened an issue with them last week that provides some detail into what's happening: snowflakedb/gosnowflake#1186.

To summarize the bug, it appears that gosnowflake persists the context submitted with a query in the db connection it creates. Go specifically warns against this because the database/sql library automatically caches and pools these connections, so any associated context may be (and in this case is) invalid after it's been returned to the pool.

The actual impact of this to us is unclear. The log emitted comes from connection.go:275, which is the Close() method for a connection. The gosnowflake client attempts to make an API call to Snowflake informing it that the connection is closed, but does so with the invalid context. As far as I can tell, all our client-side code executes as expected and we release relevant resources on our end when we're done. I can only speculate the impact of not updating the SF server-state immediately; I imagine there's a timeout but it might delay warehouse idling (just a guess).

Ideally this gets fixed upstream, but there are a few options for how we can deal with this in the meantime:

• Our primary Execute[Query|Update] methods manually manage a single connection and don't have this issue. The bug arises only when we use the database/sql API which creates a connection pool. The API is used for various auxiliary tasks (such as COUNT(*) to check rows_affected, query information_schema, etc) because it's a bit more convenient. If we use a fixed number of explicit connections for everything and stop using database/sql, we likely won't have this issue any more.
• A slightly simpler variant of the above: We can call db.SetMaxIdleConns(0) after creating the db which seems to prevent pooling of any connections. This would be a smaller code change, but could increase time spent acquiring connections depending on the ADBC action being performed.

[lidavidm]

Ah, thanks for the details! I'd be OK waiting for upstream to fix this properly before we try to hack around it, since they seem to have acknowledged the report

[joellubi]

@lidavidm @connelld-dpsk12 I opened a PR upstream that should fix this: snowflakedb/gosnowflake#1196