MySQL MariaDB ODBC memory allocation error on huge execute

[pickfire]

Please first make sure you have looked at:

• Documentation: https://github.com/mkleehammer/pyodbc/wiki
• Other issues

Environment

To diagnose, we usually need to know the following, including version numbers. On Windows, be
sure to specify 32-bit Python or 64-bit:

• Python: Python 3.8.1
• pyodbc: 4.0.30
• OS: Linux arch 5.5.6-zen1-1-zen unicode parameters on pyodbc -> unixodbc -> freetds #1 ZEN SMP PREEMPT Mon, 24 Feb 2020 12:20:37 +0000 x86_64 GNU/Linux
• DB: MySQL
• driver: MariaDB ODBC 3.1.5 /usr/lib/libmaodbc.so

Issue

Often it is easiest to describe your issue as "expected behavior" and "observed behavior".

When execute in bulk (generating our own bulk upsert statement), after more than 5000 batches being executed consequently, it shows:

pyodbc.OperationalError: ('HY001', '[HY001] [ma-3.1.5][5.7.12]Memory allocation error (0) (SQLPutData)')

executemany does not have this issue but is very, very, very, very slow compared to this method.

[gordthompson]

Are you saying that after you execute 5000 huge upserts you get the error (suggesting a memory leak), or are you saying that a huge upsert of more than 5000 rows produces the error the first time you try it?

[pickfire]

@gordthompson After more than 5000 rows are being upsert with 1000 each round so sixth time (> 5000), or somewhere around there. I even tweak to use 1200 so after the fifth time (> 4800) it also breaks. The total data that is around 6500 rows.

After I try this, even the normal slow executemany breaks too.

[v-chojas]

After I try this, even the normal slow executemany breaks too.

It seems like this may be a bug in the ODBC driver.

[pickfire]

But it used to work though? So maybe I should try the MySQL own ODBC driver?

[keitherskine]

You could perhaps try taking pyodbc out of the picture by running your SQL queries through the isql utility. See if you still get a memory allocation error.

[pickfire]

@keitherskine There is no issue when using it with isql directly with all the batches of data, of course I converted it to string without the ? beforehand.

[pickfire]

Oh, looks like not using the ? works but I am not sure if this is secure though, even though this is used internally.

[gordthompson]

Closed due to inactivity. Feel free to re-open with current information if necessary.