Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available. #93

Closed
utapyngo opened this issue May 15, 2020 · 49 comments
Assignees
Labels
bug Something isn't working

Comments

@utapyngo
Copy link

utapyngo commented May 15, 2020

Trying to use Python 3.6 on Ubuntu 16.04:

Execute installation script
##[debug]Check if Python hostedtoolcache folder exist...
##[debug]Deleting Python 3.6.10
##[debug]Create Python 3.6.10 folder
##[debug]Copy Python binaries to hostedtoolcache folder
##[debug]Create additional symlinks (Required for UsePythonVersion VSTS task)
##[debug]Upgrading PIP...
##[debug]Looking in links: /tmp/tmp5xjr9lqz
##[debug]Requirement already satisfied: setuptools in /opt/hostedtoolcache/Python/3.6.10/x64/lib/python3.6/site-packages (40.6.2)
##[debug]Requirement already satisfied: pip in /opt/hostedtoolcache/Python/3.6.10/x64/lib/python3.6/site-packages (18.1)
##[debug]Collecting pip
##[debug]Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.",)) - skipping
##[debug]Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.",)) - skipping
##[error]The process '/bin/bash' failed with exit code 1
##[debug]Node Action run completed with exit code 1
##[debug]Finishing: Set up Python
@konradpabjan
Copy link
Collaborator

Could you provide a minimal repo? Not sure if this has been fixed or maybe there is some extra configurations or perhaps you're using a self-hosted runner?

I tried the following and it everything seems to be working.

    runs-on: ubuntu-16.04
    steps:
    - name: Setup-Python Test
      uses: actions/setup-python@v2
      with:
        python-version: 3.6.10 (also tried with 3.6 as that defaults to 3.6.10)

@utapyngo
Copy link
Author

We are using a self-hosted runner.
I have kind of fixed this by compiling openssl1.1 and specifying the path in an environment variable. Now the problem is that I must do it manually for every Python version because the Github Actions runner does not work with openssl1.1.

@konradpabjan
Copy link
Collaborator

Yup, it's failing towards the end of the the setup script, ssl related: https://github.com/actions/python-versions/blob/10f5e8e4f5b3118624f86839a5c64c6e095094f7/installers/nix-setup-template.sh#L54

If you try to do ./python -m pip install --ignore-installed pip locally, it spits out the same message. After the extensive testing with other machines, I got this to show up once. I used a combination of solutions outlined here: https://stackoverflow.com/questions/45954528/pip-is-configured-with-locations-that-require-tls-ssl-however-the-ssl-module-in/57240184 (did you use something similar?) Seems like this is highly dependent on the self-hosted machine and what is installed already.

I don't know of a good solution at this time... we could update the README with some instructions, but ideally I would like it to just work. Another option is to maybe update the setup-templates which I think should do the trick. During compile time, we have all the SSL modules installed beforehand, so I there isn't anything else we can do: https://github.com/actions/python-versions/blob/10f5e8e4f5b3118624f86839a5c64c6e095094f7/builders/ubuntu-python-builder.psm1#L68

@jakemwood
Copy link

We are also experiencing problems with this on a self-hosted runner. Has this been solved yet?

Upgrading PIP...
Looking in links: /tmp/tmpopvlgd1t
Requirement already satisfied: setuptools in /opt/hostedtoolcache/Python/3.7.8/x64/lib/python3.7/site-packages (47.1.0)
Requirement already satisfied: pip in /opt/hostedtoolcache/Python/3.7.8/x64/lib/python3.7/site-packages (20.1.1)
##[error]WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
##[error]WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
##[error]WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
##[error]WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
##[error]WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
##[error]WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
ERROR: Could not find a version that satisfies the requirement pip (from versions: none)
ERROR: No matching distribution found for pip
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
##[error]WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
##[error]The process '/bin/bash' failed with exit code 1

@debtha
Copy link

debtha commented Aug 4, 2020

Has anyone found a resolution for this issue? I just upgraded from Python 2.7.5 to Python 3.8.1 and upgraded Pip and having this same issue.

@EvanPHamilton
Copy link

https://help.dreamhost.com/hc/en-us/articles/360001435926-Installing-OpenSSL-locally-under-your-username seems like it might help

@utapyngo
Copy link
Author

@EvanPHamilton, that's what I did, and it helped. The problem is that the environment variables must be set individually for every Python version because the Github Actions runner does not work with openssl1.1.

@BjornWF
Copy link

BjornWF commented Aug 14, 2020

@utapyngo , i'm trying to install tesseract-ocr on Windows 10
I need to install the "tesserocr-2.4 .0-cp37-cp37m-win-amd64.whl" wheel and then get the error

image

I set my environment variable for path for tesseract-ocr. Any suggestions? I would really appreciate any assistance, Regard bjorn

@BjornWF
Copy link

BjornWF commented Aug 15, 2020

Has anyone found a resolution for this issue? I just upgraded from Python 2.7.5 to Python 3.8.1 and upgraded Pip and having this same issue.

@debtha , did you come right here?

@mathangpeddi
Copy link

Screenshot (582)

I am trying to install flask in Windows and it shows a similar error. Any idea how to resolve this error? Thanks in advance!

@chasays
Copy link

chasays commented Oct 26, 2020

resolved.

  1. first install openssl, please refer to this page
  2. install python and ./configure --with-openssl=/home/username/openssl
  3. at last, run python3 -m ssl and nothing outputs, it's ok.

@l1771250640
Copy link

resolved.

  1. first install openssl, please refer to this page
  2. install python and ./configure --with-openssl=/home/username/openssl
  3. at last, run python3 -m ssl and nothing outputs, it's ok.

thanks, resolved.

@kvamsi7
Copy link

kvamsi7 commented Nov 15, 2020

resolved.

  1. first install openssl, please refer to this page
  2. install python and ./configure --with-openssl=/home/username/openssl
  3. at last, run python3 -m ssl and nothing outputs, it's ok.

could you please elaborate on the second step, please ?

@l1771250640
Copy link

resolved.

  1. first install openssl, please refer to this page
  2. install python and ./configure --with-openssl=/home/username/openssl
  3. at last, run python3 -m ssl and nothing outputs, it's ok.

could you please elaborate on the second step, please ?

download python source from https://www.python.org/downloads/source/ such as Python-x.x.x.tgz, then tar -xvf Python-x.x.x.tgz, and cd Python-x.x.x, do /configure --with-openssl=/home/username/openssl.

@zimventures
Copy link

resolved.

  1. first install openssl, please refer to this page
  2. install python and ./configure --with-openssl=/home/username/openssl
  3. at last, run python3 -m ssl and nothing outputs, it's ok.

I was able to get a version of python built to use a openssl in a home directory, however the setup-python doesn't seem to ever want to use that version. Even setting up the python and python3 symlink in /usr/bin still result it in trying to install its own version and failing. Any advice on how to force setup-python to use a specific python binary?

@git8951
Copy link

git8951 commented Dec 30, 2020

resolved.

1. first install openssl, please refer to [this page](https://help.dreamhost.com/hc/en-us/articles/360001435926-Installing-OpenSSL-locally-under-your-username)

2. install python and `./configure --with-openssl=/home/username/openssl`

3. at last, run `python3 -m ssl` and nothing outputs, it's ok.

I install ssl and use same command , last command not working "python3 -m ssl" , same problem not resolved

@ghost
Copy link

ghost commented Jan 30, 2021

I have the problem to not be able to find the bash_profile

I am using Kali Linux on VMWare, and it copied the vmware iso (not the installer, i don't know if there is a difference)

I did all others steps but I am guessing the change in the bash_profile is mandatory, unfortunately i cant do it

Can someone help ?

@rlf89
Copy link

rlf89 commented Feb 1, 2021

I have the problem to not be able to find the bash_profile

I am using Kali Linux on VMWare, and it copied the vmware iso (not the installer, i don't know if there is a difference)

I did all others steps but I am guessing the change in the bash_profile is mandatory, unfortunately i cant do it

Can someone help ?

try . ~/.profile instead

or editing bashrc

https://askubuntu.com/questions/510709/i-cannot-find-bash-profile-in-ubuntu

@rlf89
Copy link

rlf89 commented Feb 1, 2021

I managed to include SSL library with

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/home/username/openssl/lib/

before step
./configure --with-openssl=/home/username/openssl

now
python3 -m ssl
give no errors

@awerchniak
Copy link

I was encountering this problem while attempting to RUN pip install numpy in a docker image that extends one of the Amazon Sagemaker Tensorflow Serving images (reference), which runs Ubuntu 16.04. In case anyone encounters this in the future, the reason that this was failing for me is that my custom script was modifying the LD_LIBRARY_PATH environment variable. After moving the RUN pip install numpy call to before that change, it's working!

@stmckeni
Copy link

stmckeni commented Jun 16, 2021

#93 (comment)

resolved.

  1. first install openssl, please refer to this page
  2. install python and ./configure --with-openssl=/home/username/openssl
  3. at last, run python3 -m ssl and nothing outputs, it's ok.

#93 (comment)

I managed to include SSL library with

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/home/username/openssl/lib/

before step
./configure --with-openssl=/home/username/openssl

now
python3 -m ssl
give no errors

this worked with the openssl 1.1.1k version for python 3.9.5 on Linux lite ( or a custom kernel based on ubuntu 20.04) .

I had installed openssl source built it following the instructions from that link and then configured from python readme.rst with ./configure --with-openssl=/home/username/openssl --enable-optimizations followed the python make and build instructions and after python3 -m ssl I got no errors and pip worked. I even finally got to update to pip 21.1.2. These posted have helped me the most on a barely mainstream custom linux kernel thank you again everyone.

@DiKorsch
Copy link

The solution used here (https://bugs.python.org/issue34028#msg339455) helped me:

  1. I installed openssl with the default prefix (to /usr/local)
  2. Then I needed to set LD_LIBRARY_PATH:
    export LD_LIBRARY_PATH=$LD_LIBARY_PATH:/usr/local/lib64:/usr/local/lib
  3. Afterwards, so that python configuration succeeds, I needed to set LD_FLAGS:
    export LDFLAGS="-L/usr/local/lib -L/usr/local/lib64 -Wl,-rpath,/usr/local/lib -Wl,-rpath,/usr/local/lib64"
  4. Finally, after configuring with ./configure --with-openssl=/usr/local/ and installing with make && make install python3 -m ssl worked

@arpanbaruah
Copy link

If any body still facing issue
Follow these steps
install openssl with /home/username

set
export PATH=$HOME/openssl/bin:$PATH
export LD_LIBRARY_PATH=/home/username/openssl/lib
export LC_ALL="en_US.UTF-8"
export LDFLAGS="-L/home/username/openssl/lib -Wl,-rpath,/home/username/openssl/lib"
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/home/username/openssl/lib/

execute in Python folder
./configure -prefix=/home/username/Python38/ --with-openssl=/home/username/openssl
make
make install
python3 -m ssl

@rtadros125
Copy link

Nothing worked for me until I yum installed openssl-devel and bzip2-devel before building python on CentOS. Neither LD manipulation nor setting --with-openssl while configuring the build was necessary. The names of those packages for apt would be libssl-dev and libbz2-dev, but I have not tried it.

@nikita-bykov nikita-bykov added the bug Something isn't working label Dec 6, 2021
@Huyassin
Copy link

Huyassin commented Feb 27, 2022

In Ubuntu 20.04 I fixed the similar error, trying to install Django by using super user sudo -s
And # pip3 install django

@leeco-22
Copy link

resolved.

  1. first install openssl, please refer to this page
  2. install python and ./configure --with-openssl=/home/username/openssl
  3. at last, run python3 -m ssl and nothing outputs, it's ok.

This is great resolved, thank bro

@leeco-22
Copy link

The solution used here (https://bugs.python.org/issue34028#msg339455) helped me:

  1. I installed openssl with the default prefix (to /usr/local)
  2. Then I needed to set LD_LIBRARY_PATH:
    export LD_LIBRARY_PATH=$LD_LIBARY_PATH:/usr/local/lib64:/usr/local/lib
  3. Afterwards, so that python configuration succeeds, I needed to set LD_FLAGS:
    export LDFLAGS="-L/usr/local/lib -L/usr/local/lib64 -Wl,-rpath,/usr/local/lib -Wl,-rpath,/usr/local/lib64"
  4. Finally, after configuring with ./configure --with-openssl=/usr/local/ and installing with make && make install python3 -m ssl worked

I followed your guide but still got some error bro, "/usr/local/bin/python3.10: No module named openssl"

@Lazik
Copy link

Lazik commented Mar 29, 2022

Worked for me. I had to install openssl and then it worked like a charm.

@qudos-tech
Copy link

3. python3 -m ssl

working but still getting this error: WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available. Please advise...

@IvanZosimov
Copy link
Contributor

Hi, @qudos-tech 👋 Could you provide us with some more details about your case in order to be able to reproduce it. You can use this template as example.

@gezabohus
Copy link

This is on CentOS 7.

Openssl-devel and bzip2-devel mentioned here are up to date.

I have installed Python 3.10 as root so everyone can use it.
Got the failures mentioned in the subject when trying to do

python3 -m pip install requests

Installed openssl for myself as suggested above and I still get the same errors when trying to install python packages.

python3 -m ssl

results in

Traceback (most recent call last):
File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.10/ssl.py", line 98, in
import _ssl # if we can't import it, let the error propagate
ModuleNotFoundError: No module named '_ssl'

@panticmilos
Copy link
Contributor

panticmilos commented May 30, 2022

Hi @gezabohus,

Your case is a bit specific because you are using CentOS compared to other self-hosted runners. The issue here is that prebuilt Python binaries are prebuilt for Ubuntu so we cannot guarantee this Python build would work on other distributions. If you would like to build Python on your own for CentOS there is no need to use the setup-python action, since this action is using prebuilt binaries as explained above. Your issue is not related to setup-python action, probably you can open a topic on the python bug tracker: https://bugs.python.org or https://github.com/python/cpython/issues.

Cheers

@gezabohus
Copy link

Hi @panticmilos,

Thanks for the info.

I wasn't completely clear, sorry. When I wrote "installed" I meant going through the get tar - configure - make cycle. I suppose that should build from source. Right?

@panticmilos
Copy link
Contributor

Hi @gezabohus,

You are welcome. Yes, that is correct, these are building phases. And since you are building it on your own, that's why I think raising the issue on one of these two bug trackers could help you. If I can help you additionally feel free to continue the discussion. :)

@gezabohus
Copy link

Hi @panticmilos,

Got it. Thanks for the info.

@gezabohus
Copy link

Hi all,

A bit of fudging with this answer solved my problem. (Plus using virtual env.)

@shuoyinn
Copy link

Thanks a lot for all these answers. For me, I also found that it also worked even after I removed the openssl installed in my home directory and cancel export LD_LIBRARY_PATH for it's lib. So I just tried many other times for digging into it. And I summarize as follows:

Firstly. I installed openssl in my home directory and set its related LD_LIBRARY_PATH before I installed python3.9 from source code with ./configure --with-openssl={openssl path I just installed manually}. And ./pip3 install worked without any SSL error.

Afterwards, I removed the openssl and unset the environment variable for it, but still no SSL error when I use ./pip3 to install packages.

Another time, I remove all the things mentioned above and install openssl again. This time I didn't set its LD_LIBRARY_PATH and only set --with-openssl={openssl path I just installed manually} and then install python3.9 before I removed openssl again. So there is no manually installed openssl any more. Still, it worked.

In conclusion, I guess the procedure of 'configure' python3.9 needs an openssl directory and only needs for 'configure', for some reason it cannot find one (I mean it cannot find the default openssl the system already has). So just install and set one explicitly. After configuring, making and installing python, it is of no use for normally running python and pip.

I encountered this problem on one of my lab servers (labeled as 1), but everything went well on another (labeled as 0). I cannot find where is different between them since both default openssl are on /usr (but for 1 is on /usr/local/bin and for 0 on /usr/bin).

@dsame dsame self-assigned this Aug 1, 2022
@dsame
Copy link
Contributor

dsame commented Aug 1, 2022

hello @utapyngo

With the fact that self-hosted environment must have openssl libraries installed and with the solution provided @chasays i believe the issue can be closed. But please feel free to reopen this issue or create another one in case if the problem still exists.

@dsame dsame closed this as completed Aug 1, 2022
@clh15683
Copy link

On Ubuntu (tested on 22.04) it is sufficient to install libssl-dev and then rerun configure. There is no need to install a local openssl.

@bozimmerman
Copy link

On Ubuntu (tested on 22.04) it is sufficient to install libssl-dev and then rerun configure. There is no need to install a local openssl.

If re-running configure on a machine that already has the required libs is all that's needed, can that be added to setup-python? I feel like almost none of the comments are focusing on the actual Action we are supposedly trying to use.

@BlackFuffey
Copy link

resolved.

1. first install openssl, please refer to [this page](https://help.dreamhost.com/hc/en-us/articles/360001435926-Installing-OpenSSL-locally-under-your-username)

2. install python and `./configure --with-openssl=/home/username/openssl`

3. at last, run `python3 -m ssl` and nothing outputs, it's ok.

Thank you very much this pretty much saved my life :D

@asanc115
Copy link

asanc115 commented Mar 2, 2023

Got this working for installing Python 3.11.2 on Amazon Linux 2. (An OS Similar to CentOS 7/Rhel 7) I had to install a different openssl version, which Amazon Repo's provide. You can still leave the regular openssl rpm installed, but you must REMOVE openssl-devel if installed. The three packages you must install AFTER REMOVING openssl-devel are:

  • openssl11-1.1.1g
  • openssl11-libs-1.1.1g
  • openssl11-devel-1.1.1g

Once installed just run ./configure --enable-optimizations (no need for the --with-openssl as the configure script already has a defined locations to look for your openssl, you only need to provide it if your openssl is not within those standard directories, you can find the locations in the script)

Do not let the g fool you, amazon updates their RPM's without changing the letter at the end, so yes it's patched for vulnerabilities.

Reference: https://alas.aws.amazon.com/AL2/ALAS-2023-1934.html

Hope this helps!

Also some more info if interested:

Everytime you run ./configure it creates a config.log. I saw it was finding my original openssl in the config.log, however, it was not happy with the version I was using, and the capabilities of that version based on some tests it would do. Which is what led me to find out there was a different openssl Amazon provided in which python3.11.2 was happy with.

@miaogen123
Copy link

I managed to include SSL library with

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/home/username/openssl/lib/

before step ./configure --with-openssl=/home/username/openssl

now python3 -m ssl give no errors

this solution works for me too!

@DDvO
Copy link

DDvO commented Oct 19, 2023

The reason can also be that the OpenSSL installation (e.g, in /usr/local/lib) is messed up.
Invoking python and in the interactive shell import ssl reveals where it tries to find libssl.
See also https://stackoverflow.com/questions/58280484/ssl-module-in-python-is-not-available-on-osx

@enahwe
Copy link

enahwe commented Jan 31, 2024

Those commands will fix the issue:

$ sudo yum install yum-utils
$ sudo yum-builddep python3

or

$ sudo apt build-dep python3
$ sudo apt install pkg-config

or

$ sudo dnf install dnf-plugins-core
$ sudo dnf builddep python3

See: https://devguide.python.org/getting-started/setup-building/index.html#linux

@zackees
Copy link

zackees commented Mar 16, 2024

In case anyone is running into this issue with the amazonlinux:2 docker image, here is a Dockerfile that allows python 3.11.8 to be compiled and installed, by first building openssl from source and setting the proper link flags for the python build.

Thanks to everyone that's contributed to working through this issue. Collectively I was able to piece the proper magic settings to get it working. And right when I was almost about to give up!

# Use an official Amazon Linux 2 base image
FROM amazonlinux:2
# Notes: amazonlinux:2023 uses php 8.1, and does not support
# php 7.2-7.4.


# Install Apache, PHP, and other necessary packages
RUN yum update -y \
    && amazon-linux-extras enable php7.2 \
    && yum install -y \
    # Install commands.
    httpd \
    openssh-server \
    cronie \
    sudo \
    dos2unix \
    which \
    # Optimnizes yum install
    deltarpm \
    procps \
    nano \
    os-prober \
    mariadb-server \
    bash-completion \
    ppp \
    postfix \
    # Install nohup which needs the coreutils
    coreutils nohup \
    # Install php
    php \
    php-cli \
    php-fpm \
    php-bcmath \
    php-bz2 \
    php-calendar \
    php-ctype \
    php-curl \
    php-date \
    php-dom \
    php-exif \
    php-fileinfo \
    php-filter \
    php-ftp \
    php-gd \
    php-gettext \
    php-hash \
    php-iconv \
    php-intl \
    php-json \
    php-libxml \
    php-mbstring \
    php-mysqli \
    php-mysqlnd \
    php-openssl \
    php-pcntl \
    php-pcre \
    php-pdo \
    php-pdo_mysql \
    php-pdo_pgsql \
    php-pdo_sqlite \
    php-pgsql \
    php-phar \
    php-posix \
    php-readline \
    php-reflection \
    php-session \
    php-shmop \
    php-simplexml \
    php-sockets \
    php-spl \
    php-sqlite3 \
    php-standard \
    php-sysvmsg \
    php-sysvsem \
    php-sysvshm \
    php-tokenizer \
    php-xml \
    php-xmlreader \
    php-xmlwriter \
    php-xsl \
    php-zip \
    php-zlib \
    mod_ssl \
    # Don't leave garbage behind, we are a container so we won't need it.
    && yum clean all

# More php packages, todo: merge above with below.
RUN yum install -y php-{pear,cgi,common,curl,mbstring,gd,mysqlnd,gettext,bcmath,json,xml,fpm,intl,zip,imap} && yum clean all

# Install C++ toolchain for building native add-ons for Node.js/python. Note
# that this must be done here while python is python 2.7. After python 3 is install
# this command will fail to work.
RUN yum install -y gcc44 gcc-c++ libgcc44 cmake wget tar gzip make && yum clean all
RUN yum install -y @development zlib-devel bzip2 bzip2-devel readline-devel sqlite sqlite-devel xz xz-devel libffi-devel openssl-devel && yum clean all
# RUN yum install -y compat-openssl10-devel
RUN yum install -y openssl-devel yum-utils ncurses-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel expat-devel && yum clean all
RUN yum-builddep python3 -y

# Install local openssl

# wget https://www.openssl.org/source/openssl-1.1.1g.tar.gz
RUN wget https://www.openssl.org/source/openssl-1.1.1g.tar.gz
RUN tar -xvzf openssl-1.1.1g.tar.gz
RUN cd openssl-1.1.1g && ./config --prefix=/root/openssl --openssldir=/root/openssl no-ssl2 && make && make install
RUN yum groupinstall "Development Tools"

# Install python 3.9.6
# As of right now we cannot build any python version higher than 3.9.6
ARG PYTHON_MAJOR_VERSION=3
ARG PYTHON_MINOR_VERSION=11
ARG PYTHON_PATCH_VERSION=8
ARG PYTHON_VERSION=${PYTHON_MAJOR_VERSION}.${PYTHON_MINOR_VERSION}.${PYTHON_PATCH_VERSION}
ARG PYTHON_SHORT_VERSION=${PYTHON_MAJOR_VERSION}.${PYTHON_MINOR_VERSION}
WORKDIR /opt
RUN wget https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tgz
RUN tar xzf Python-${PYTHON_VERSION}.tgz
WORKDIR /opt/Python-${PYTHON_VERSION}

ENV LD_LIBRARY_PATH="/root/openssl/lib:/usr/local/lib64:/usr/local/lib"
ENV LDFLAGS="-L/root/openssl/lib -L/usr/local/lib -L/usr/local/lib64 -Wl,-rpath,/root/openssl/lib -Wl,-rpath,/usr/local/lib -Wl,-rpath,/usr/local/lib64"
ENV CPPFLAGS="-I/usr/local/include"
RUN ./configure --enable-optimizations --with-openssl=/root/openssl
# RUN ./configure --enable-optimizations
RUN make altinstall
RUN rm -f /opt/Python-${PYTHON_VERSION}.tgz
RUN rm /usr/bin/python3 || true
RUN ln -s /opt/Python-${PYTHON_VERSION}/python /usr/bin/python3
RUN rm /usr/bin/pip3 || true
RUN ln -s /usr/local/bin/pip${PYTHON_SHORT_VERSION} /usr/bin/pip3
WORKDIR /

# Configure python environment
RUN pip3 install --upgrade pip
RUN pip3 install setuptools wheel
RUN pip3 install cryptography==41.0.5
RUN pip3 install wormhole-tx

# Ensure /usr/bin and /usr/sbin (where yum is located) are in the PATH
ENV PATH="/usr/bin:/usr/sbin:${PATH}"

# INSTALL NODE.js AND NPM
# Note that the highest version of Node we can install is 17, because of our version of
# glibc. Fixing this requires a different base image (not amazon linux 2).
ARG NODE_VERSION=17
RUN curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.38.0/install.sh | bash \
    && . ~/.nvm/nvm.sh \
    && nvm install $NODE_VERSION \
    && nvm alias default $NODE_VERSION \
    && nvm use default \
    && echo 'export NVM_DIR="$HOME/.nvm"' >> /root/.bashrc \
    && echo '[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"' >> /root/.bashrc \
    && echo '[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"' >> /root/.bashrc \
    && echo 'export PATH="$NVM_DIR/versions/node/$(nvm version)/bin:$PATH"' >> /root/.bashrc \
    && source /root/.bashrc

# Install npm packages
RUN . ~/.bashrc && npm install --location=global pm2 http-server

# Install ripgrep
RUN curl -LO https://github.com/BurntSushi/ripgrep/releases/download/13.0.0/ripgrep-13.0.0-x86_64-unknown-linux-musl.tar.gz \
    && tar xvf ripgrep-13.0.0-x86_64-unknown-linux-musl.tar.gz \
    && mv ripgrep-13.0.0-x86_64-unknown-linux-musl/rg /usr/local/bin/ \
    && rm -rf ripgrep-13.0.0-x86_64-unknown-linux-musl.tar.gz ripgrep-13.0.0-x86_64-unknown-linux-musl



# TODO Move up to the top of the file.
RUN if ! id apache > /dev/null 2>&1; then echo "Apache user does not exist" && exit 1; fi
RUN mkdir -p /var/lib/php/sessions && chown apache:apache /var/lib/php/sessions


# Install glibc and AWS CLI v2
RUN yum install -y glibc
WORKDIR /tmp
RUN if [ "$(uname -m)" = "x86_64" ]; then \
    curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"; \
    elif [ "$(uname -m)" = "aarch64" ]; then \
    curl "https://awscli.amazonaws.com/awscli-exe-linux-aarch64.zip" -o "awscliv2.zip"; \
    else \
    echo "Unsupported architecture"; \
    exit 1; \
    fi \
    && unzip awscliv2.zip \
    && sudo ./aws/install \
    && rm -rf /tmp/awscliv2.zip /tmp/aws

WORKDIR /root

# Ensure the AWS CLI binary is in the PATH
ENV PATH="/usr/local/aws-cli/aws/v2/current/bin:${PATH}"

# Copy AWS config file
COPY image_src/home/ec2-user/.aws /root/.aws

# Set the default value of BUILD_ENV to "test"
ARG BUILD_ENV=test
ENV BUILD_ENV=${BUILD_ENV}

# Setup SSH.
RUN mkdir -p /etc/ssh/host_keys/
# Copy SSH host keys
COPY ssh/test1.pem /etc/ssh/ssh_host_rsa_key
RUN chmod 600 /etc/ssh/ssh_host_rsa_key
COPY ssh/test1.pem /etc/ssh/host_keys/

ARG SSH_PASS=tssx2hJr229Xo8WpOyKmR0cZBqvl7n
ARG USE_SSHD=
# if USE_SSHD is set, then set it as an environment variable
ENV USE_SSHD=${USE_SSHD}


# Set the URL
ARG URL=
# if URL is set, then set it as an environment variable
ENV URL=${URL}
RUN \
    if [ -z "$URL" ]; \
    then echo "Error: URL environment variable is not set. Please set it to the current URL." && exit 1; \
    else echo "export URL=$URL" >> /root/.bashrc; \
    fi

# Set root password
RUN echo "root:${SSH_PASS}" | chpasswd

# COPY ENTIRE SOURCE DIRECTORY ONTO IMAGE.
COPY image_src/ /
# Apply dos2unix recursively to the copied files
RUN find /home/ -type f -name "*.sh" -exec dos2unix {} \;

# As far as I know, docker COPY commands are unconditional. So we copy the
# directory unconditionally and if the URL is localhost then we don't symlink
# but instead mount the twdb directory on the host machine to /var/www/twdb.
COPY --chown=apache:apache twdb/ /twdb/
# Note that linking is not workign and I don't know why, but directory moving
# seems to work just fine.
#RUN if [ "$URL" != "localhost" ]; then ln -s /twdb /var/www/twdb; fi
# move folder instaad
ARG USE_VOLUME=0
RUN if [ "$USE_VOLUME" != "1" ]; \
    # Move the copy of the twdb directory to /var/www/twdb \
    then mv /twdb /var/www/twdb; \
    # Otherwise remove the twdb directory, since the real twdb directory will be mounted \
    else rm -rf /twdb; \
    fi


COPY entrypoint /home/ec2-user/entrypoint
RUN find /home/ -name "entrypoint" -exec dos2unix {} \;
COPY jobs.config.js /home/ec2-user/jobs.config.js
COPY fmin_db_metrics_cron /home/ec2-user/fmin_db_metrics_cron
# COPY ./image_src/etc/cron.d /etc/cron.d
COPY docker /home/ec2-user/docker

# install twdb/python/requirments.txt
COPY twdb/python /var/www/twdb/python
RUN pip3 install /var/www/twdb/python


RUN /home/ec2-user/docker/final_install.sh

# Open necessary ports
EXPOSE 80 443 22
ENV PORT 80

# Start Apache and cron
CMD ["sh", "-c", "/home/ec2-user/entrypoint"]

@enahwe
Copy link

enahwe commented Jun 27, 2024

I found a really good and complete explanation bellow.

  • Title:
    How to install a shared Python for multiple accounts without SSL issues

https://medium.com/@enahwe/how-to-06bc8a042345

@adarshan-intel
Copy link

The solution used here (https://bugs.python.org/issue34028#msg339455) helped me:

  1. I installed openssl with the default prefix (to /usr/local)
  2. Then I needed to set LD_LIBRARY_PATH:
    export LD_LIBRARY_PATH=$LD_LIBARY_PATH:/usr/local/lib64:/usr/local/lib
  3. Afterwards, so that python configuration succeeds, I needed to set LD_FLAGS:
    export LDFLAGS="-L/usr/local/lib -L/usr/local/lib64 -Wl,-rpath,/usr/local/lib -Wl,-rpath,/usr/local/lib64"
  4. Finally, after configuring with ./configure --with-openssl=/usr/local/ and installing with make && make install python3 -m ssl worked

This is the only comment that worked for me, Thanks a ton!!!!!!!

@dronov-dmitry
Copy link

dronov-dmitry commented Nov 22, 2024

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests