Contribution guide should include dependencies

[m8ram]

Hi,

Installing the Python requirements on my new system failed due to some missing dependencies.

I have been able to determine the following:

• gcc
• python3-devel
• libffi-devel

So far I have not been able to determine what I need to resolve:

      gcc -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC -Iast27/Include -I/home/user/.virtualenv/crhc-cli/include -I/usr/include/python3.11 -c ast27/Custom/typed_ast.c -o build/temp.linux-x86_64-cpython-311/ast27/Custom/typed_ast.o
      In file included from ast27/Custom/typed_ast.c:3:
      ast27/Custom/../Include/compile.h:5:10: fatal error: code.h: No such file or directory
          5 | #include "code.h"
            |          ^~~~~~~~
      compilation terminated.
      error: command '/usr/bin/gcc' failed with exit code 1

For what it's worth: Version 1.5.5 of typed-ast can be installed without problem.

[waldirio]

Hello @m8ram

Thank you for the heads up. Could you please share the baseOS that you are using? also, the version? For example, rhel8.4 or CentOS 8.1? Something like that.

With that, I'll create a fresh install, minimal installation, will collect all the requirements and will update our page.

You can use the code below

# dnf provides */code.h

This will tell you which package contains the file in question

# dnf provides */code.h
Updating Subscription Management repositories.
Last metadata expiration check: 1:34:09 ago on Tue 17 Oct 2023 05:48:46 PM UTC.
platform-python-debug-3.6.8-1.el8.i686 : Debug version of the Python runtime
Repo        : rhel-8-for-x86_64-appstream-rpms
Matched from:
Filename    : /usr/include/python3.6dm/code.h

platform-python-debug-3.6.8-1.el8.x86_64 : Debug version of the Python runtime
Repo        : rhel-8-for-x86_64-appstream-rpms
Matched from:
Filename    : /usr/include/python3.6dm/code.h
...
python39-devel-3.9.16-1.module+el8.8.0+17625+b531f198.x86_64 : Libraries and header files needed for Python development
Repo        : rhel-8-for-x86_64-appstream-rpms
Matched from:
Filename    : /usr/include/python3.9/code.h
Filename    : /usr/include/python3.9/cpython/code.h

python39-devel-3.9.16-1.module+el8.8.0+18968+3d7b19f0.1.x86_64 : Libraries and header files needed for Python development
Repo        : rhel-8-for-x86_64-appstream-rpms
Matched from:
Filename    : /usr/include/python3.9/code.h
Filename    : /usr/include/python3.9/cpython/code.

I suspect that this is python39-devel, but once you share the information above, I'll be able to tell you for sure which one should be around.

Thank you again for the heads up!
Waldirio

[m8ram]

Hi Waldirio,
This is on my fedora 38 (CSB) system. Since the gcc command line included -I/usr/include/python3.11 I checked python3-devel and I have python3-devel-3.11.6-1.fc38.x86_64 installed.
Regards
Bram

[waldirio]

Hello,

Thank you! I can see the error, I believe typed-ast got deprecated and it's causing the issue, once it was supported up to python3.7. It still working on my dev env once I have multiple versions as well.

Allow me some time to review and fix it.

Thank you!
Waldirio

[waldirio]

Hello,

Please, proceed as below

Install the packages below

git
gcc
python3-devel
libffi-devel

and remove the line below from requirements.txt

typed-ast==1.4.3

After that, proceed with the installation, in your virtual environment

(crhc-cli) [root@server crhc-cli]# pip install -r requirements.txt 

The installation should finish with no issues.

Please, let me know if it works for you. I'll do a double check on the dependencies for this package, run all the tests and if not necessary anymore, I'll remove it.

Thank you again!
Waldirio

[m8ram]

Hi Waldirio,

Confirmed, I have deleted the virtual environment, removed the typed-ast from the requirements.tst and with the listed dependencies the installation of the python requirements now completes without errors.

The python command works but running pyinstaller to generate the binary does throw a traceback:

$ pyinstaller crhc.py 
Traceback (most recent call last):
  File "/home/bmertens/.virtualenv/crhc-cli/bin/pyinstaller", line 8, in <module>
    sys.exit(run())
             ^^^^^
  File "/home/bmertens/.virtualenv/crhc-cli/lib64/python3.11/site-packages/PyInstaller/__main__.py", line 107, in run
    parser = generate_parser()
             ^^^^^^^^^^^^^^^^^
  File "/home/bmertens/.virtualenv/crhc-cli/lib64/python3.11/site-packages/PyInstaller/__main__.py", line 78, in generate_parser
    import PyInstaller.building.build_main
  File "/home/bmertens/.virtualenv/crhc-cli/lib64/python3.11/site-packages/PyInstaller/building/build_main.py", line 35, in <module>
    from PyInstaller.depend import bindepend
  File "/home/bmertens/.virtualenv/crhc-cli/lib64/python3.11/site-packages/PyInstaller/depend/bindepend.py", line 26, in <module>
    from PyInstaller.depend import dylib, utils
  File "/home/bmertens/.virtualenv/crhc-cli/lib64/python3.11/site-packages/PyInstaller/depend/utils.py", line 33, in <module>
    from PyInstaller.depend import bytecode
  File "/home/bmertens/.virtualenv/crhc-cli/lib64/python3.11/site-packages/PyInstaller/depend/bytecode.py", line 95, in <module>
    _call_function_bytecode = bytecode_regex(rb"""
                              ^^^^^^^^^^^^^^^^^^^^
  File "/home/bmertens/.virtualenv/crhc-cli/lib64/python3.11/site-packages/PyInstaller/depend/bytecode.py", line 60, in bytecode_regex
    pattern = re.sub(
              ^^^^^^^
  File "/usr/lib64/python3.11/re/__init__.py", line 185, in sub
    return _compile(pattern, flags).sub(repl, string, count)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bmertens/.virtualenv/crhc-cli/lib64/python3.11/site-packages/PyInstaller/depend/bytecode.py", line 62, in <lambda>
    lambda m: _instruction_to_regex(m[1].decode()),
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bmertens/.virtualenv/crhc-cli/lib64/python3.11/site-packages/PyInstaller/depend/bytecode.py", line 40, in _instruction_to_regex
    return re.escape(bytes([dis.opmap[x]]))
                            ~~~~~~~~~^^^
KeyError: 'CALL_FUNCTION'

After upgrading pyinstaller to 6.1.0 and pyinstaller-hooks-contrib to 2023.10 using pip install -U pyinstaller the issue is resolved.

[waldirio]

Hello,

Thank you @m8ram for the heads up.

I'm doing some follow up to update all the modules used in the project. pip install --upgrade pip and pip install --upgrade -r requirements.txt should do the trick, but I'm seeing that the second one it's not working as expected, bumping all the packages to the latest version, even when using --upgrade-strategy=eager

Doing the steps below, I can see that I got the latest version of each module

# for b in $(pip freeze | cut -d"=" -f1); do echo - $b;pip install --upgrade $b; done

Note. It was necessary to run a couple of times, once pip it was not pulling correctly the dependencies. After that, I was able to see only the issues as presented below

- docutils
Requirement already satisfied: docutils in /root/.virtualenv/crhc-cli/lib64/python3.11/site-packages (0.18.1)
Collecting docutils
  Using cached docutils-0.20.1-py3-none-any.whl.metadata (2.8 kB)
Using cached docutils-0.20.1-py3-none-any.whl (572 kB)
Installing collected packages: docutils
  Attempting uninstall: docutils
    Found existing installation: docutils 0.18.1
    Uninstalling docutils-0.18.1:
      Successfully uninstalled docutils-0.18.1
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
sphinx-rtd-theme 1.3.0 requires docutils<0.19, but you have docutils 0.20.1 which is incompatible.
Successfully installed docutils-0.20.1

- pydantic_core
Requirement already satisfied: pydantic_core in /root/.virtualenv/crhc-cli/lib64/python3.11/site-packages (2.10.1)
Collecting pydantic_core
  Using cached pydantic_core-2.11.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.5 kB)
Requirement already satisfied: typing-extensions!=4.7.0,>=4.6.0 in /root/.virtualenv/crhc-cli/lib64/python3.11/site-packages (from pydantic_core) (4.8.0)
Using cached pydantic_core-2.11.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.9 MB)
Installing collected packages: pydantic_core
  Attempting uninstall: pydantic_core
    Found existing installation: pydantic_core 2.10.1
    Uninstalling pydantic_core-2.10.1:
      Successfully uninstalled pydantic_core-2.10.1
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
pydantic 2.4.2 requires pydantic-core==2.10.1, but you have pydantic-core 2.11.0 which is incompatible.
Successfully installed pydantic_core-2.11.0

I'll run all the tests with the latest version of the packages, in linux and windows environment and will confirm if it's working properly.

Thank you again!
Waldirio

[waldirio]

Hello @m8ram

The requirements.txt was updated to the latest version of python modules available. I tested and it's working properly.

Thank you!
Waldirio