TST: enable MySQL tests #6613

jorisvandenbossche · 2014-03-12T09:23:14Z

PR to fix and enable the MySQL tests for the new sql functionality.

floats (decimal.Decimal) were not converted due to a small bug in the coerce_float arg. This is now OK.
MySQL has no real BOOL type (it is just an alias for tiny int). So I changed the tests for that (so a bool column is converted to int64 (or float if there are NA's)

We should maybe add somewhere in the sql docs an overview of the limitations of the type conversion.

jorisvandenbossche · 2014-03-12T09:37:09Z

@jreback Is there a way to see if the tests are not skipped anymore on Travis? (because before they were) I just see 'Ran 4980 tests in 266.519s OK (SKIP=107)' Can I see which are the 107 skipped?

jreback · 2014-03-12T09:38:57Z

go to the build
at the very bottom are little arrows to the skipped tests

jorisvandenbossche · 2014-03-12T09:42:55Z

ah yes, I see, the print_skipped. Thanks! And they are not skipped anymore, and passing!

jreback · 2014-03-12T09:44:46Z

looks good

jreback · 2014-03-12T13:22:33Z

also enabled on the windows builds.....

merge when ready

jorisvandenbossche · 2014-03-12T13:39:35Z

Nice!
For me this is ready. @mangecoeur OK for you?

mangecoeur · 2014-03-12T13:50:04Z

@jorisvandenbossche all good with me

TST: enable MySQL tests

jreback · 2014-03-12T15:31:45Z

I got this to work on windows (and all tests pass).

Oracle makes this EXTRA hard to download setup the server!

postgres so much easier!

jreback · 2014-03-12T15:48:49Z

I think you should catch these tests and skip
e.g. pymysql is installed but no server is running (or can't login)

secondarily these tests take a LONG time on windows! 150s!
no tests seems that slow, maybe a server setting (its a stock install)

======================================================================
ERROR: test_to_sql_replace (pandas.io.tests.test_sql.TestPostgreSQLAlchemy)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/home/jreback/pandas/pandas/io/tests/test_sql.py", line 778, in setUp
    self.pandasSQL = sql.PandasSQLAlchemy(self.conn)
  File "/mnt/home/jreback/pandas/pandas/io/sql.py", line 615, in __init__
    meta.reflect(self.engine)
  File "/usr/local/lib/python2.7/site-packages/SQLAlchemy-0.7.7-py2.7-linux-x86_64.egg/sqlalchemy/schema.py", line 2474, in reflect
    conn = bind.contextual_connect()
  File "/usr/local/lib/python2.7/site-packages/SQLAlchemy-0.7.7-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2489, in contextual_connect
    self.pool.connect(),
  File "/usr/local/lib/python2.7/site-packages/SQLAlchemy-0.7.7-py2.7-linux-x86_64.egg/sqlalchemy/pool.py", line 224, in connect
    return _ConnectionFairy(self).checkout()
  File "/usr/local/lib/python2.7/site-packages/SQLAlchemy-0.7.7-py2.7-linux-x86_64.egg/sqlalchemy/pool.py", line 387, in __init__
    rec = self._connection_record = pool._do_get()
  File "/usr/local/lib/python2.7/site-packages/SQLAlchemy-0.7.7-py2.7-linux-x86_64.egg/sqlalchemy/pool.py", line 739, in _do_get
    con = self._create_connection()
  File "/usr/local/lib/python2.7/site-packages/SQLAlchemy-0.7.7-py2.7-linux-x86_64.egg/sqlalchemy/pool.py", line 188, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python2.7/site-packages/SQLAlchemy-0.7.7-py2.7-linux-x86_64.egg/sqlalchemy/pool.py", line 270, in __init__
    self.connection = self.__connect()
  File "/usr/local/lib/python2.7/site-packages/SQLAlchemy-0.7.7-py2.7-linux-x86_64.egg/sqlalchemy/pool.py", line 330, in __connect
    connection = self.__pool._creator()
  File "/usr/local/lib/python2.7/site-packages/SQLAlchemy-0.7.7-py2.7-linux-x86_64.egg/sqlalchemy/engine/strategies.py", line 80, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python2.7/site-packages/SQLAlchemy-0.7.7-py2.7-linux-x86_64.egg/sqlalchemy/engine/default.py", line 281, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/local/lib/python2.7/site-packages/psycopg2/__init__.py", line 179, in connect
    connection_factory=connection_factory, async=async)
OperationalError: (OperationalError) could not connect to server: Connection refused
        Is the server running on host "localhost" and accepting
        TCP/IP connections on port 5432?
 None None

----------------------------------------------------------------------
Ran 80 tests in 0.837s

FAILED (SKIP=26, errors=16)

jorisvandenbossche · 2014-03-12T15:55:51Z

Yes, catching the errors is on the to do list in #6292.

For the time, I experienced it as very varying, but the last time I ran them on Windows (all sql tests), it was 26 s for 80 tests.

I think one way to speed up the tests is maybe just making the connection and test tables once for one test class, and not for every test seperately (using setup_class instead of setup), but then should look that no test collide with table names.

jorisvandenbossche · 2014-03-12T15:57:21Z

Are there examples in the code base how the OperationalError can be catched in the tests?

jreback · 2014-03-12T16:02:13Z

look in utils/testing.py at the @network decorator. needs something like this for sql (or could just do it in the connect method (where you try except around it and skip if appropraite)

jreback · 2014-03-12T16:10:58Z

Something funny going on...

I though its only the MySQLAlchemy tests that take an exceptional amount of time, the other (and postgres) are pretty fast.

I though tit was 0.8.3 that was not compiled with c-extensions (as I had originally did pip install), but I just installed the binaries of 0.9.3

INSTALLED VERSIONS
------------------
commit: 6899ed662bdec20aaa3b9f9563236a0c4385395e
python: 2.7.5.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 42 Stepping 7, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.12.0
Cython: 0.19.2
numpy: 1.8.0
scipy: 0.13.0
statsmodels: 0.5.0
IPython: 1.1.0
sphinx: None
patsy: 0.2.1
scikits.timeseries: None
dateutil: 2.2
pytz: 2013.8
bottleneck: 0.7.0
tables: 3.0.0
numexpr: 2.2.2
matplotlib: 1.3.1
openpyxl: 1.7.0
xlrd: 0.9.2
xlwt: 0.7.5
xlsxwriter: 0.5.0
lxml: None
bs4: 4.3.2
html5lib: 1.0b3
bq: None
apiclient: None
rpy2: None
sqlalchemy: 0.9.3
pymysql: 0.6.1.None
psycopg2: 2.5.2 (dt dec pq3 ext)

C:\Users\Jeff Reback\Documents\GitHub\pandas>c:\python27-64\Scripts\nosetests.exe build\lib.win-amd64-2.7\pandas\io\tests\test_sql.py:TestMySQL -v
test_create_and_drop_table (pandas.io.tests.test_sql.TestMySQL) ... ok
test_execute_sql (pandas.io.tests.test_sql.TestMySQL) ... ok
test_invalid_flavor (pandas.io.tests.test_sql.TestMySQL) ... ok
test_read_sql (pandas.io.tests.test_sql.TestMySQL) ... ok
test_roundtrip (pandas.io.tests.test_sql.TestMySQL) ... ok
test_to_sql (pandas.io.tests.test_sql.TestMySQL) ... ok
test_to_sql_append (pandas.io.tests.test_sql.TestMySQL) ... ok
test_to_sql_fail (pandas.io.tests.test_sql.TestMySQL) ... ok
test_to_sql_replace (pandas.io.tests.test_sql.TestMySQL) ... ok
test_tquery (pandas.io.tests.test_sql.TestMySQL) ... ok
----------------------------------------------------------------------
Ran 10 tests in 11.836s


C:\Users\Jeff Reback\Documents\GitHub\pandas>c:\python27-64\Scripts\nosetests.exe build\lib.win-amd64-2.7\pandas\io\tests\test_sql.py:TestSQLApi
............
----------------------------------------------------------------------
Ran 12 tests in 0.306s

C:\Users\Jeff Reback\Documents\GitHub\pandas>c:\python27-64\Scripts\nosetests.exe build\lib.win-amd64-2.7\pandas\io\tests\test_sql.py:TestMySQLAlchemy -v
test_create_table (pandas.io.tests.test_sql.TestMySQLAlchemy) ... ok
test_date_parsing (pandas.io.tests.test_sql.TestMySQLAlchemy) ... ok
test_default_date_load (pandas.io.tests.test_sql.TestMySQLAlchemy) ... ok
test_default_type_conversion (pandas.io.tests.test_sql.TestMySQLAlchemy) ... ok
test_drop_table (pandas.io.tests.test_sql.TestMySQLAlchemy) ... ok
test_execute_sql (pandas.io.tests.test_sql.TestMySQLAlchemy) ... ok
test_mixed_dtype_insert (pandas.io.tests.test_sql.TestMySQLAlchemy) ... ok
test_read_sql (pandas.io.tests.test_sql.TestMySQLAlchemy) ... ok
test_read_table (pandas.io.tests.test_sql.TestMySQLAlchemy) ... ok
test_read_table_absent (pandas.io.tests.test_sql.TestMySQLAlchemy) ... ok
test_read_table_columns (pandas.io.tests.test_sql.TestMySQLAlchemy) ... ok
test_roundtrip (pandas.io.tests.test_sql.TestMySQLAlchemy) ... ok
test_to_sql (pandas.io.tests.test_sql.TestMySQLAlchemy) ... ok
test_to_sql_append (pandas.io.tests.test_sql.TestMySQLAlchemy) ... ok
test_to_sql_fail (pandas.io.tests.test_sql.TestMySQLAlchemy) ... ok
test_to_sql_replace (pandas.io.tests.test_sql.TestMySQLAlchemy) ... ok

----------------------------------------------------------------------
Ran 16 tests in 134.062s

jorisvandenbossche · 2014-03-13T10:33:50Z

@jreback What if you only run the PostgreSQLAlchemy tests? Are these faster than the MySQLAlchemy?

jorisvandenbossche · 2014-03-13T10:39:06Z

It seems to be going faster after some runs of the tests, but now (on Windows) I get 25s for both PostgreSQL and MySQL tests, and 8-9 s for without MySQL.

jreback · 2014-03-13T10:41:14Z

Postgres is fine about 10s on my machine

that's why I am thinking its a setting on the MySQL server but nothing obvious

it's just a stock setup

jorisvandenbossche · 2014-03-13T10:47:06Z

Is it a problem for the test suite? (for testing on travis, that it takes to long)?

For speeding up the tests, possible things to explore:

not make the connection for every test, but just once for the test class (although I don't know if this is the time consuming step, should profile it, as in all other actions (reading/writing tables, etc) also always connections are made)
Maybe not all tests that are now tested in the flavor test classes should be tested for all flavors? Eg if to_sql if_exists works is maybe not flavor dependent? And only leave the real flavor issues (type conversions etc) in those test classes?

jreback · 2014-03-13T10:54:01Z

it's quite fast on Travis and since Postgres is fast I am thinking maybe it's some setting specific to MySQL

seems really odd

your suggestions are good but prob don't need to optimize the testing as a high priority

TST: enable MySQL tests

3ac2bdf

jorisvandenbossche mentioned this pull request Mar 12, 2014

ENH: sql support with SQLAlchemy - follow-up #6292

Closed

17 tasks

jreback added SQL labels Mar 12, 2014

jreback added this to the 0.14.0 milestone Mar 12, 2014

jorisvandenbossche added a commit that referenced this pull request Mar 12, 2014

Merge pull request #6613 from jorisvandenbossche/sql-mysql2

66611ee

TST: enable MySQL tests

jorisvandenbossche merged commit 66611ee into pandas-dev:master Mar 12, 2014

jorisvandenbossche deleted the sql-mysql2 branch March 12, 2014 13:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TST: enable MySQL tests #6613

TST: enable MySQL tests #6613

jorisvandenbossche commented Mar 12, 2014

jorisvandenbossche commented Mar 12, 2014

jreback commented Mar 12, 2014

jorisvandenbossche commented Mar 12, 2014

jreback commented Mar 12, 2014

jreback commented Mar 12, 2014

jorisvandenbossche commented Mar 12, 2014

mangecoeur commented Mar 12, 2014

jreback commented Mar 12, 2014

jreback commented Mar 12, 2014

jorisvandenbossche commented Mar 12, 2014

jorisvandenbossche commented Mar 12, 2014

jreback commented Mar 12, 2014

jreback commented Mar 12, 2014

jorisvandenbossche commented Mar 13, 2014

jorisvandenbossche commented Mar 13, 2014

jreback commented Mar 13, 2014

jorisvandenbossche commented Mar 13, 2014

jreback commented Mar 13, 2014

TST: enable MySQL tests #6613

TST: enable MySQL tests #6613

Conversation

jorisvandenbossche commented Mar 12, 2014

jorisvandenbossche commented Mar 12, 2014

jreback commented Mar 12, 2014

jorisvandenbossche commented Mar 12, 2014

jreback commented Mar 12, 2014

jreback commented Mar 12, 2014

jorisvandenbossche commented Mar 12, 2014

mangecoeur commented Mar 12, 2014

jreback commented Mar 12, 2014

jreback commented Mar 12, 2014

jorisvandenbossche commented Mar 12, 2014

jorisvandenbossche commented Mar 12, 2014

jreback commented Mar 12, 2014

jreback commented Mar 12, 2014

jorisvandenbossche commented Mar 13, 2014

jorisvandenbossche commented Mar 13, 2014

jreback commented Mar 13, 2014

jorisvandenbossche commented Mar 13, 2014

jreback commented Mar 13, 2014