Dev Environment Breaks with MySQL 8.3 During Installation

[tsisodia10]

Description

When setting up the development environment with MySQL version 8.3, pip install -e ".[dev]" fails.

Expected Behavior

Running pip install -e ".[dev]" should successfully install all necessary development dependencies without errors, even with MySQL 8.3.

Current Behavior

The installation process breaks when attempting pip install -e ".[dev]". The following errors are encountered:

src/MySQLdb/_mysql.c:527:9: error: call to undeclared function 'mysql_ssl_set'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
              mysql_ssl_set(&(self->connection), key, cert, ca, capath, cipher);
              ^
      src/MySQLdb/_mysql.c:527:9: note: did you mean 'mysql_close'?
      /usr/local/Cellar/mysql/8.3.0/include/mysql/mysql.h:797:14: note: 'mysql_close' declared here
      void STDCALL mysql_close(MYSQL *sock);
                   ^
      src/MySQLdb/_mysql.c:1795:9: error: call to undeclared function 'mysql_kill'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
          r = mysql_kill(&(self->connection), pid);
              ^
      src/MySQLdb/_mysql.c:1795:9: note: did you mean 'mysql_ping'?
      /usr/local/Cellar/mysql/8.3.0/include/mysql/mysql.h:525:13: note: 'mysql_ping' declared here
      int STDCALL mysql_ping(MYSQL *mysql);
                  ^
      src/MySQLdb/_mysql.c:2011:9: error: call to undeclared function 'mysql_shutdown'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
          r = mysql_shutdown(&(self->connection), SHUTDOWN_DEFAULT);
              ^
      3 errors generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for mysqlclient
Successfully built feast
Failed to build mysqlclient
ERROR: Could not build wheels for mysqlclient, which is required to install pyproject.toml-based projects

Steps to reproduce

1. Create and activate a new Conda environment for feast:

conda create --name feast python=3.9
conda activate feast

2. Install dependencies:

pip install pip-tools
brew install mysql@8.3
brew install xz protobuf openssl zlib
pip install cryptography -U
conda install protobuf
conda install pymssql
pip install -e ".[dev]"

[redhatHameed]

Note: MySQL 8.3 release last month(14 Dec 2023) . @tsisodia10 you tried with older version <8.3 which is working fine right ?

[tsisodia10]

Note: MySQL 8.3 release last month(14 Dec 2023) . @tsisodia10 you tried with older version <8.3 which is working fine right ?

@redhatHameed That's right, I tried with the older version(8.0 to be precise).

[rohanahata]

same for me. 8.3 just falls apart

[tokoko]

I looked at how mysqlclient is used in the project and found that we are actually barely using it at all. mysql online store implementation uses pymysql, the only place mysqlclient is used right now is a single test for sql registry and even that's done through sqlalchemy, therefore it will be pretty straightforward to change that test to use pymysql as well. I'll open a PR to remove mysqlclient from feast requirements altogether.

[sudohainguyen]

I just got a workaround, in case you're working with Linux, try:
apt-get install default-libmysqlclient-dev pkg-config
hope this unblocks you guys

[expediamatt]

@tokoko 👏 👏 👏 👏 👏 thank you. This removes a huge build headache that has bitten at least three devs on my team, and multiple times in the last year at that.

[tokoko]

@expediamatt wasted countless hours on that myself before 😆 Figuring out what version of mariadb/mysql you need for different linux distros is no fun.

[TomMonkeyMan]

same for mysql Ver 8.3.0 for macos13.6 on x86_64 (Homebrew)

I got an error exactly the same when installing mysqlclient (2.2.0) via pip.

It works for me after downgrading mysql version to:

mysql Ver 8.0.36 for macos13.6 on x86_64 (Homebrew)

[eduardogpg]

@TomMonkeyMan same. I had to downgrade to Mysql for it to work.
For Mac :
brew uninstall mysql
brew install mysql@8 # Very important use 8

[tokoko]

FYI, looks like the underlying issue was also fixed 5 days ago with PyMySQL/mysqlclient#690. mysqlclient==2.2.3 should be working fine.