[Snowflake] dbt compile throws memory exception for ffi.callback() on M1 Macbook Air
#19
Comments
|
@jplynch77 Thanks for re-opening. This is frustrating, and I'm not sure if there's anything we can do about it until snowflakedb/snowflake-connector-python#799 is patched. Am I thinking about that right? Over in dbt-labs/dbt-core#3162, we talked about potentially figuring out a way to undo the monkey-patch of At the very least, we have relaxed our dependency on |
jtcohen6
changed the title
dbt compile throws memory exception for ffi.callback() on M1 Macbook Air dbt compile throws memory exception for ffi.callback() on M1 Macbook Air
|
Subscribed - we've also had a user reporting on this with Edit: SUCCESS: |
|
@jtcohen6 Yeah as far as I can tell patching the underlying problem in connector is the only real path forward that's not a hack. I just don't know if anyone's even trying to do that though. Based on this comment it seems like it's might be quite difficult to fix the underlying issue and it doesn't seem like there's been any progress. So if there is some patch you all can figure out, or if the Snowflake people can figure something else out for the connector, I think that's probably going to the way to get this working in the short/medium term. I did get it installed via Rosetta as @poudrouxj points out, it just makes me sad that I have to resort to that. 😢 |
|
@jplynch77 Just left a comment over there: snowflakedb/snowflake-connector-python#799 (comment). If you could give that a read, and let me know if it makes sense to you, I'd really appreciate it. I'd love to figure out a way for us to just call |
|
I struggled getting the VSCode DBT Power User extension working along the brew-installed DBT by @poudrouxj's method above. That is, getting hyperlink table names wasn't working. Instead, I installed the rosetta version of one of miniconda/miniforge/[mambaforge]https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-MacOSX-x86_64.sh (I've linked direct links. If downloading on your own, just specify the x86_64 or Intel version) and install as normal, placing it at e.g. I then created an alias in In that base environment I create a dbt environment and install it as usual. # install x86 conda at ~/miniconda_rosetta
# add `alias rosetta='eval "$(/Users/thomas/miniconda_rosetta/bin/conda shell.zsh hook)"'` to profile, restart terminal
rosetta
conda create -n dbt python=3.9
conda activate dbt
# Check that `conda info` includes `platform : osx-64` and not `platform : osx-arm64`, or you installed normal conda, not x86
pip install dbt # executable will be installed in ~/miniconda_rosetta/envs/dbt/bin/dbtNow, in VSCode, I can specify the dbt conda environment, and poweruser works as expected. |
|
Instead of the workaround above (installing Homebrew via Rosetta), I customized a docker image based on this image https://hub.docker.com/r/xemuliam/dbt. Docker Desktop now natively supports M1 architectures, and this image can build arm64 Linux containers that do not exhibit the FFI issue. |
|
This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days. |
Describe the bug
This is basically dbt-labs/dbt-core#3162, but I'm still running into the problem with
send_anonymous_usage_stats: Falseas well as trying the0.21.0-b1prerelease version where thisanonymous_usageissue is patched.dbt compileanddbt runboth throwMemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks.Steps To Reproduce
On a 2020 Macbook Air running Big Sur 11.5. Installed
dbtas followsdbt installs fine with either version, and
dbt -versionworks fine as well. However when I dodbt compileordbt runI run into theffi.callback() error. I got this error with both0.20.0and0.21.0-b1.I know there was an attempted fix as described in this comment but I think that only addresses the case where this is affecting anonymous tracking and the problem here seems to be with the primary
snowflake-connector-pythonconnection to Snowflake (see trace below).There is also an open issue on
snowflake-connector-pythonrepo but no progress towards fixing that yet.Expected behavior
dbt compileanddbt runrun without error.Screenshots and log output
Full trace.
System information
Which database are you using dbt with?
The output of
dbt --version:The operating system you're using:
MacOS Big Sur 11.5
The output of
python --version:Python 3.9.6
The text was updated successfully, but these errors were encountered: