-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DBT cli throw memory exception for ffi.callback() on Mac with Apple Sillicon #3162
Comments
Hey @alyiwang, thanks for the issue! We've been hearing about plenty of installation errors with M1 macs, though this is the first I'm seeing of this specific error. Unfortunately, I don't have access to the hardware to reproduce and debug this, so I'm operating on a mix of intuition and what I can find online. It looks like this error is coming from In the meantime, you could try:
|
Hey @alyiwang, I was having similar issues and I was able to work through it using the following:
|
Hi @jtcohen6 and @mikelanzago , thanks for your responses! I spent quite some time on this issue, and as you mentioned, it is a dependency issue with snowflake-connector-python, more specifically, it's some of it's dependencies, like pyOpenSSL and pyarrow, that's having compatibility issue with M1 chip. I was able to install latest pyOpenssl via pip, but the installation of snowflake-connector-python failed by pyarrow error. After many try and fail, I got To make the installation work, a couple changes will need to be made from dbt to:
Meanwhile, I got the docker image version of DBT to work locally, even though I don't prefer this way of installation, this is the only thing working for me to unblock my work. I will definitely try out what Mike detailed later today. Thanks! |
@alyiwang This is absolutely our plan for the next minor version of dbt (v0.20.0). |
Just ran into the issue as well today on my new work computer. If there is anything I can do to help out I'd be happy to chip in. The |
We've just run headfirst into this issue with a colleague using a new macbook 😂 ; any rough timelines for a v0.20.0 pre-release @jtcohen6? |
Same issue here guys, looking forward to this new release to fix my local setup. |
Workaround installing though pip following this thread: psycopg/psycopg2#1200 |
I found this link to be helpful: https://nhancv.medium.com/macbook-pro-m1-homebrew-install-error-7170e4816894 |
TL;DR: if you experience the error during config:
send_anonymous_usage_stats: False Details: Here's the stack trace (
|
This is a weird one. The error is in But the reason Since dbt only supports python3, I'd like to figure out a way to tell @vklimontovich I'm glad you were able to get this working in the meantime by disabling anonymous usage tracking. Of course, I'd like to avoid it being an either/or choice! cc @kwigley in case you find this interesting :) |
@jtcohen6 wouldn't wrapping tracking code into Something like this one:
|
@vklimontovich Have you been able to confirm that the addition of that try/except results in error-free runs? (i.e. by cloning That's fine as a temporary solution—if you can confirm it works, I'd welcome a PR for it. In addition, I'd like to find a way to work around the |
@jtcohen6 Is this issue supposed to be resolved with |
@brittianwarner We didn't include the proposed fix above in v0.20.0. You're welcome to try turning off anonymous tracking, and see if that fixes it. Over the past few weeks, there have been a few reports of successful installs on M1s by other means: #3239 (comment) |
@jtcohen6 do you have any insight into when the patch to resolve this issue will be released so we do not need to find these work arounds? |
@kwigley and I just took a look at this, trying to figure out if there's a way we can undo the urllib3/pyopenssl monkey patch that If we can't figure that out—or even if we can—let's add We'll do our best to sneak in this fix for v0.20.1. |
Hi - I’m having this issue- this is the first time trying to use dbt so a little frustrating - what‘s the latest and what should a newbie do here to get the connection working please? |
@Nathancalon If you're using an M1 Mac, we highly recommend you install dbt via Homebrew + Rosetta, as outlined here: https://hackernoon.com/apple-m1-chip-how-to-install-homebrew-using-rosetta-su12331b That's what I've been doing on my M1 for months now. While it's a few extra steps to set up, it's been working without a hitch. |
Hey guys, I fixed this ffi problem by reinstalling the libffi using homebrew, and install cffi wheel: https://files.pythonhosted.org/packages/3e/9b/660d6da900af1976a8b4efea713a7ce9e514bf4659eff9b17f90f00be1cf/cffi-1.15.0-cp39-cp39-macosx_11_0_arm64.whl |
I've tried installing dbt@0.18.0 with Homebrew using Rosetta and I get an error: Building wheels for collected packages: cryptography × Building wheel for cryptography (pyproject.toml) did not run successfully. any help would be much appreciated! |
@tpark464 I would recommend using the latest version of dbt and the associated adapter ( |
for anyone still having issues, this works!!! |
I have same issue with M1 mac, we use Snowflake with dbt.
|
Describe the bug
dbt run
ordbt --version
throw errorCannot allocate write+execute memory for ffi.callback().
Steps To Reproduce
On Macbook pro with M1 chip
brew tap fishtown-analytics/dbt
brew install dbt
add default profile.yml with connection to snowflake
create new project
dbt init
dbt run
with the two sample models, or simplydbt --version
Expected behavior
Should complete successfully, instead of throwing exception. The process works fine on Intel-based Macbook pro.
Screenshots and log output
System information
Which database are you using dbt with?
The output of
dbt --version
:The operating system you're using:
macOS Big Sur 11.2.2
The output of
python --version
:3.8.7
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: