[ADAP-1087] [Regression] Loosen redshift-connector pin

[WillAyd]

Is this a regression in a recent version of dbt-redshift?

• I believe this is a regression in dbt-redshift functionality
• I have searched the existing issues, and I could not find an existing issue for this regression

Current Behavior

The redshift-connector dependency is hard pinned to a particular version. The comments in setup.py indicate that this is because redshift-connector does not follow SemVer and can release breaking changes at any time. However, this has the consequence of making this library much more difficult to install, particularly with the shift in python packaging to enforce constraints

A concrete example of this is Airflow 2.7, which uses the following default constraints:
https://raw.githubusercontent.com/apache/airflow/constraints-2.7.2/constraints-3.11.txt

You'll note the version of redshift-connector is 2.0.914 but this package jumped from 2.0.913 directly to 2.0.915, so it is essentially uninstallable by default in Airflow

Expected/Previous Behavior

pip install with constraint should still have a valid installation candidate

Steps To Reproduce

pip install dbt-redshift --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.7.2/constraints-3.11.txt

Relevant log output

ERROR: Cannot install dbt-redshift==0.13.0, dbt-redshift==0.13.1, dbt-redshift==0.14.0, dbt-redshift==0.14.1, dbt-redshift==0.14.2, dbt-redshift==0.14.3, dbt-redshift==0.14.4, dbt-redshift==0.15.0, dbt-redshift==0.15.1, dbt-redshift==0.15.2, dbt-redshift==0.15.3, dbt-redshift==0.16.0, dbt-redshift==0.16.1, dbt-redshift==0.17.0, dbt-redshift==0.17.1, dbt-redshift==0.17.2, dbt-redshift==0.18.0, dbt-redshift==0.18.1, dbt-redshift==0.18.2, dbt-redshift==0.19.0, dbt-redshift==0.19.1, dbt-redshift==0.19.2, dbt-redshift==1.4.0, dbt-redshift==1.4.1, dbt-redshift==1.5.0, dbt-redshift==1.5.1, dbt-redshift==1.5.10, dbt-redshift==1.5.11, dbt-redshift==1.5.2, dbt-redshift==1.5.3, dbt-redshift==1.5.4, dbt-redshift==1.5.5, dbt-redshift==1.5.6, dbt-redshift==1.5.7, dbt-redshift==1.5.8, dbt-redshift==1.5.9, dbt-redshift==1.6.0, dbt-redshift==1.6.1, dbt-redshift==1.6.2, dbt-redshift==1.6.3, dbt-redshift==1.6.4, dbt-redshift==1.6.5, dbt-redshift==1.7.0 and dbt-redshift==1.7.1 because these package versions have conflicting dependencies.

The conflict is caused by:
    dbt-redshift 1.7.1 depends on redshift-connector==2.0.915
    dbt-redshift 1.7.0 depends on boto3~=1.26.157
    dbt-redshift 1.6.5 depends on redshift-connector==2.0.915
    dbt-redshift 1.6.4 depends on boto3~=1.26.157
    dbt-redshift 1.6.3 depends on boto3~=1.26.157
    dbt-redshift 1.6.2 depends on boto3~=1.26.157
    dbt-redshift 1.6.1 depends on boto3~=1.26.157
    dbt-redshift 1.6.0 depends on boto3~=1.26.157
    dbt-redshift 1.5.11 depends on boto3~=1.26.157
    dbt-redshift 1.5.10 depends on boto3~=1.26.157
    dbt-redshift 1.5.9 depends on boto3~=1.26.157
    dbt-redshift 1.5.8 depends on boto3~=1.26.157
    dbt-redshift 1.5.7 depends on boto3~=1.26.26
    dbt-redshift 1.5.6 depends on boto3~=1.26.26
    dbt-redshift 1.5.5 depends on boto3~=1.26.26
    dbt-redshift 1.5.4 depends on boto3~=1.26.26
    dbt-redshift 1.5.3 depends on boto3~=1.26.26
    dbt-redshift 1.5.2 depends on boto3~=1.26.26
    dbt-redshift 1.5.1 depends on boto3~=1.26.26
    dbt-redshift 1.5.0 depends on boto3~=1.26.26
    dbt-redshift 1.4.1 depends on boto3~=1.26.26
    dbt-redshift 1.4.0 depends on boto3~=1.26.26
    dbt-redshift 0.19.2 depends on boto3<1.16 and >=1.4.4
    dbt-redshift 0.19.1 depends on boto3<1.16 and >=1.4.4
    dbt-redshift 0.19.0 depends on boto3<1.16 and >=1.4.4
    dbt-redshift 0.18.2 depends on boto3<1.12 and >=1.4.4
    dbt-redshift 0.18.1 depends on boto3<1.12 and >=1.4.4
    dbt-redshift 0.18.0 depends on boto3<1.12 and >=1.4.4
    dbt-redshift 0.17.2 depends on boto3<1.12 and >=1.4.4
    dbt-redshift 0.17.1 depends on boto3<1.12 and >=1.4.4
    dbt-redshift 0.17.0 depends on boto3<1.12 and >=1.4.4
    dbt-redshift 0.16.1 depends on boto3<1.12 and >=1.4.4
    dbt-redshift 0.16.0 depends on boto3<1.12 and >=1.4.4
    dbt-redshift 0.15.3 depends on boto3<1.10.0 and >=1.6.23
    dbt-redshift 0.15.2 depends on boto3<1.10.0 and >=1.6.23
    dbt-redshift 0.15.1 depends on boto3<1.10.0 and >=1.6.23
    dbt-redshift 0.15.0 depends on boto3<1.10.0 and >=1.6.23
    dbt-redshift 0.14.4 depends on boto3<1.11.0 and >=1.4.4
    dbt-redshift 0.14.3 depends on boto3<1.10.0 and >=1.6.23
    dbt-redshift 0.14.2 depends on boto3<1.10.0 and >=1.6.23
    dbt-redshift 0.14.1 depends on boto3<1.10.0 and >=1.6.23
    dbt-redshift 0.14.0 depends on boto3<1.10.0 and >=1.6.23
    dbt-redshift 0.13.1 depends on boto3<1.10.0 and >=1.6.23
    dbt-redshift 0.13.0 depends on boto3<1.10.0 and >=1.6.23
    The user requested (constraint) redshift-connector==2.0.914
    The user requested (constraint) boto3==1.28.17

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

Environment

- OS: pop_OS!
- Python: 3.11
- dbt-core (working version): n/a
- dbt-redshift (working version): n/a
- dbt-core (regression version): n/a
- dbt-redshift (regression version): n/a

Additional Context

No response

[EsauOlivaLeaoHelix]

I have the same problem

[colin-rogers-dbt]

We can loosen this up somewhat, unfortunately 2.0.914 is not compatible with dbt-redshift as it will break our logging framework: #601 (comment)

[mikealfare]

This was resolved by #704 in version 1.7.2.

[EsauOlivaLeaoHelix]

Now we can use a different dbt-connector but Airflow constrain file is tied to redshift-connector==2.014 so the problem is the same:

ERROR: Cannot install -r /usr/local/airflow/requirements/requirements.txt (line 26) because these package versions have conflicting dependencies.

The conflict is caused by:
dbt-redshift 1.7.2 depends on redshift-connector!=2.0.914, <=2.0.918 and >=2.0.913
The user requested (constraint) redshift-connector==2.0.914

[WillAyd]

@EsauOlivaLeaoHelix I don't think there is much this project can do about that. It looks like airflow 2.8 pins to redshift-connector 2.0.918 so that should work, but if you need airflow 2.7 I think you will have to provide your own constraints file for this (at your own risk of course)