-
Notifications
You must be signed in to change notification settings - Fork 564
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cursor throws an error when trying to insert in SQL Server database when fast_executemany flag is set #371
Comments
The same here: Python: 3.5.5 tried with SQL Server Native Client too. Could not determine, where the problem is, but it seems to have to do with string data. Installed 4.0.22 again and the problems are not there anymore. |
Could you post a example table schema + Python code to reproduce it, and/or ODBC trace log? That will help with the troubleshooting. |
Hey @v-chojas It was happening while trying to insert a pandas dataframe with to_sql. Thats the dataframe, pickled and base64 encoded. Error Message: Function sequence error (0) (SQLParamData) Ahh: It happens only, when fast_executemany is active. Without it (or with the previous release of pyodbc and fast_executemany activated) the inserts won't fail. |
Reproduced with pyodbc 4.0.23 under 64-bit Python 3.6.4 on Windows 7 with SQLAlchemy 1.2.6. Also confirmed that it is specific to 4.0.23 (4.0.22 works) and is related to string parameter values. With pyodbc 4.0.23 this works: import pandas as pd
from sqlalchemy import create_engine, event
cnxn_url = 'mssql+pyodbc://@SQLmyDb'
engine = create_engine(cnxn_url)
@event.listens_for(engine, 'before_cursor_execute')
def receive_before_cursor_execute(conn, cursor, statement, params, context, executemany):
if executemany:
cursor.fast_executemany = True
table_name = 'fast_executemany_test'
df = pd.DataFrame({'col1':[1, 2]})
df.to_sql(table_name, engine, if_exists = 'replace', chunksize = None) ODBC trace: However, if I change the dataframe to contain strings instead of numbers df = pd.DataFrame({'col1':['foo', 'bar']}) then I get
ODBC trace: |
Related issues are being discussed in the Stack Overflow question here. |
Can you create repro with only the pyODBC calls being made? I cannot reproduce it with this:
Values 'foo', 'bar' enter the table correctly. |
@v-chojas - Before posting I tried to reproduce the issue with plain pyodbc under Python 3.6.4 (Windows, 64-bit) but could not. That's why I went with the pandas code and the ODBC logs in case they gave a hint as to what pandas (or perhaps SQLAlchemy) was doing differently. |
Also worth noting that this may have some relation to Python_3. I just tried my repro code under Python 2.7.14 (Windows, 64-bit) and using df = pd.DataFrame({'col1':['foo', 'bar']}) worked without complaint. |
I have the same problem and using Python 2.7.15 didn't fix it for me. I'm using plain pyodbc and getting the issue under Python 2.7.15 and 3.6.5. I tried on both OS X and Windows 10, same result. |
I believe that, at least in my case, the issue is caused by passing an empty string as a param to execute many. I have reproduced it with very simple code (DSN, UID and PWD excluded):
For me, this fails using ODBC Driver 17 for SQL Server on Mac and Windows and on Python 2 and 3. There are a number of ways to make this code work:
I also added a trace as per this issue, but it didn't seem to be particularly revealing:
For now, I can workaround by ensuring that I always use None instead of '', but would appreciate it if someone knows of other workarounds. |
I believe that the issue could be resolved by replacing an empty string with a single space. I tested and it worked on my sample code. I'm not sure if this fix belongs in pyodbc, but it sure would be nice! Please see this SO for more details. In particular, it links to a blog that discusses just this issue. |
We ran into this issue as well. executemany() doesn't seem to handle empty string. Our environment is: python: 2.7.5 |
Ran into this, and replacing empty strings with Running
Full error:
|
Please provide an ODBC trace; if you are using Linux/Mac, upgrade to unixODBC 2.3.7pre (available here on GitHub: https://github.com/lurcher/unixODBC/ ) if you are using ODBC Driver 17 since it fixes a logging bug, and otherwise the trace will not contain any useful information. |
Not sure if this will help anyone else, but I was able to solve the problem by explicitly encoding the string data as ascii. Using the code provided by @v-chojas earlier in the thread, I am able to reproduce the error with this:
Result:
But explicitly encoding the string data as ascii allowed the code to execute correctly. This is the code I used:
Result:
Environment: |
I can also confirm that this affects pyodbc 4.0.23 but not 4.0.22. Explicitly encoding the string value as @veeology mentioned works for me, though I also need to change empty strings to However I've made two further observations:
My guess then is that 4.0.23 is no longer automatically preparing parameter values as binary for I'm personally reverting to 4.0.22 as I have to write a fair bit of extra code to work around this and it incurs a performance penalty of about 25%. However if anybody does use @veeology's workaround for 4.0.23, I think you'll be wanting to encode in My setup: Win10 |
I had been running on my Mac with 4.0.23 and everything was working fine using spaces instead of empty strings, but then I pushed my code to a Windows 2016 server and I'm seeing issues both with the original code I posted above and even if I change the empty string to a space. The good news is that since I'm on Windows I was able to get a decent ODBC trace log. Setup: I hope this helps to track down the issue! I suppose I will also revert to 4.0.22 as per @logicalextreme recommendation for now... |
Thanks for the trace. It appears that pyODBC is calling SQLParamData twice without ever actually sending the data for the parameter with SQLPutData. I will investigate this. |
I have determined that it is caused by small bug in the code, which is not sending 0-length parameters. Try the fix here: #425 |
OK, I had rolled back to 4.0.22 in order to resolve this issue, but it looks like when I run with 4.0.22 something goes screwy when I insert data. When I retrieve the inserted data, it has a bunch of '\x00' characters in between the characters of a bunch of varchars in the database. It's really weird and doesn't happen with version 4.0.23. Any chance you can make another release with the #425 fix? |
@billmccord you're not encoding the strings to |
Not intentionally, but I'm not specifying an encoding anywhere. The database is the default encoding. I ran a test where I insert data and retrieve and compare. Run with 4.0.22 the test has the weird characters. Run with 4.0.23 it doesn't. |
|
Tried pip installing the commit and keep getting errors. I tried: AND
But I'm getting the following error:
Any help or suggestions you can provide would be appreciated. I have a project that is already way overdue because of all the issues encountered. |
Never mind, I see that it is in a different repo, sorry about that. I'll try there. |
Alright, I finally got the pull request that @v-chojas sent to build on Windows. Originally, I had observed that when I ran my test program on my Mac it was working. However, I just tried it on Windows after doing a manual build and I still get the following error when trying to executemany and it doesn't matter if I use an empty string or a single space: It seems the problem is still not fixed on Windows. Please find the ODBC log attached. |
I just noticed that it actually doesn't matter what the string is that is input on Windows. If I change it to 'test' rather than an empty string I still get the same error. If I disable fast_executemany then all of my tests work (empty string or non-empty string). So, it appears that fast_executemany is just broken on Windows. |
Yep, I'm seeing no difference between 4.0.23 and the commit from the pull request on Windows. The
|
Please try PR #467 |
still not working my requirements: python 3.7.0 |
@maki-markie - Did you install from the code on GitHub? Your list says pyodbc 4.0.24, which does not include the fix in #467. |
@gordthompson No, i just did a normal pip install. Is there another instruction how to install it? |
still having some trouble after installing the : pyodbc 4.0.0-unsupported |
@maki-markie What ODBC driver are you using? That message does not look familiar. |
@maki-markie - Are you working with a Caché database? |
@gordthompson hi, sorry for the late reply. yes i am working with cache database. do you have any experience with this? thanks |
@gordthompson #467 seems to relate more to null values than empty string values. I am testing it now. Best workaround so far seems to be to replace empty strings with null or whitespace. |
@logicalextreme re: "I believe that fast_executemany behaviour doesn't occur with a generator/non-sequence" That does seem to be the case. Details in issue #500 |
Did that answer the question? Should we close this and pursue #500? |
Yes, I'd say that this can be closed. #467 addressed the main issue; the mention of #500 was just a tangent resulting from an earlier comment. |
I've been watching this thread for several months and still not able to use the fast execute. Name: pyodbc
EDIT: Nevermind. I found the issue with the ODBC driver, was using an obsolete one and downloaded the latest to fix the issue. Thanks!~ |
@maki-markie Check your sql driver once you get the latest pyodbc. I would bet that is where the issue lies. |
@eddyizm he is not using SQL Server, but I would not be surprised if the ODBC driver does not support parameter arrays; fast_executemany was originally designed for and works best with ODBC Driver 17 for SQL Server, although it may also be usable with other drivers. |
I am Trying to insert the data from CSV file to SQL server dynamically using python below is the code But i am getting the pyodbc.Error: ('07002', '[07002] [Microsoft][SQL Server Native Client 11.0]COUNT field incorrect or syntax error (0) (SQLExecDirectW)') error can any one please help this? |
@Puneeth2 - Your question does not appear to be related to the behaviour of pyodbc itself, so you might have better luck getting an answer on Stack Overflow. |
sql server cannot find ? using python from Linux OS? SQL Server ---in Windows OS How to solve ?sql server cannot find |
@selvakarna - Your question does not appear to be related to this particular issue. Try asking for help on Stack Overflow. |
I came across this bug recently. Python: 3.7.3 |
I'm having the same issue as well. I've installed the pyodbc version in #467. My pyodbc version is 4.0.18b89. |
@nrhead - Try upgrading pyodbc to version 4.0.27. |
Upgraded to version 4.0.27 and explicitly encoded the
|
Same issue with 4.0.27 |
@ethan-deng - Please open a new issue and include an MCVE that clearly illustrates the specific problem you are currently seeing. |
The solution of #617 and install the bug fix helps. pip install git+https://github.com/v-makouz/pyodbc.git@bugfix_unicode_size --user |
Closed due to inactivity. |
Solution is to go to the ODBC Driver created, change character format from utf-8 to ASCII. Keep changing and testing. There is server miscommunication due to character set. |
Please first make sure you have looked at:
Environment
To diagnose, we usually need to know the following, including version numbers. On Windows, be
sure to specify 32-bit Python or 64-bit:
Issue
Trying to use fast execute many option to insert data to the database throws the error shown below. It works if the flag is not set.
Expected behavior:
Cursor should be able to write to database without throwing an error.
Observed behavior:
The text was updated successfully, but these errors were encountered: