Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build python3.6 deb packages and install them into docker-snmp-sv2 #245

Merged
merged 5 commits into from
Feb 9, 2017
Merged

Build python3.6 deb packages and install them into docker-snmp-sv2 #245

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
cc9aa47
Build python3.6 and add it into docker-snmp-sv2
qiluo-msft Jan 29, 2017
80d2923
(reformat)
qiluo-msft Jan 29, 2017
c77e64b
Download python3 source from azure storage mirror
qiluo-msft Feb 4, 2017
785a242
Remove unused dependencies
qiluo-msft Feb 8, 2017
257f091
(comment)
qiluo-msft Feb 8, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 28 additions & 33 deletions dockers/docker-snmp-sv2/Dockerfile.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,45 +1,40 @@
FROM docker-config-engine

COPY \
COPY [ \
{% for deb in docker_snmp_sv2_debs.split(' ') -%}
debs/{{ deb }}{{' '}}
{%- endfor -%}
debs/
"debs/{{ deb }}",
{%- endfor %} \
"/debs/"]

COPY python-wheels /python-wheels
COPY python-wheels/sswsdk-*-py3-*.whl /python-wheels/
COPY python-wheels/asyncsnmp-*-py3-*.whl /python-wheels/

# enable -O for all Python calls
ENV PYTHONOPTIMIZE 1

## Pre-install the fundamental packages
## Install Python SSWSDK (SNMP subagent dependency)
## Install SNMP subagent
## Clean up
RUN apt-get update && apt-get install -y libmysqlclient-dev libmysqld-dev libperl-dev libpci-dev libpci3 libsensors4 libsensors4-dev libwrap0-dev

RUN dpkg -i \
{% for deb in docker_snmp_sv2_debs.split(' ') -%}
debs/{{ deb }}{{' '}}
{%- endfor %}

RUN rm -rf /debs

# install supervisor
# install libsnmp30 dependencies
# install dependencies for sonic-cfggen
Copy link
Collaborator

@lguohan lguohan Feb 8, 2017 •
edited by qiluo-msft
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this #Resolved

# install libpython3.6-dev and pip dependencies
# TODO: remove libpython3.6-dev, its and pip's dependencies if we can get pip3 directly
Copy link
Contributor

@stcheng stcheng Feb 4, 2017 •
edited by qiluo-msft
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is 'its'? #Resolved

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

libpython3.6-dev's (dependencies)

In reply to: 99458291 [](ancestors = 99458291)

# install subagent
RUN apt-get -y install build-essential wget libssl-dev openssl supervisor && \
rm -rf /var/lib/apt/lists/* && \
wget https://www.python.org/ftp/python/3.5.2/Python-3.5.2.tgz && \
tar xvf Python-3.5.2.tgz && cd Python-3.5.2 && \
./configure --without-doc-strings --prefix=/usr --without-pymalloc --enable-shared && \
make && make install && \
ldconfig && \
cd .. && rm -rf Python-3.5.2 && rm Python-3.5.2.tgz && \
pip3 install --no-cache-dir /python-wheels/*py3*.whl hiredis && \
# clean up
Copy link
Contributor

@stcheng stcheng Feb 4, 2017 •
edited by qiluo-msft
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this comment shall be moved lower? #WontFix

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All these comments are related to below single RUN statement. It's difficult to comment near the exact line, any idea?

In reply to: 99458299 [](ancestors = 99458299)

Copy link
Contributor

@jleveque jleveque Feb 6, 2017 •
edited by qiluo-msft
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One option: you could split the RUN statement into three RUN statements, update, install and clean up. #Resolved

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right. Both options have the same effect. The merging option is kind of manual 'docker squash', which remove the unnecessary intermediate layer storage from docker image's point of view. In future when 'docker squash' becomes mature, we can prevent this situation.

In reply to: 99679265 [](ancestors = 99679265)

Copy link
Collaborator

@lguohan lguohan Feb 7, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we are using docker --squash now, we can seperate into different RUNs now. #Resolved

RUN apt-get update && apt-get install -y supervisor \
libperl5.20 libpci3 libwrap0 \
libexpat1-dev \
python-lxml python-jinja2 python-netaddr python-ipaddr python-yaml \
Copy link
Collaborator

@lguohan lguohan Feb 8, 2017 •
edited by qiluo-msft
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

python-lxml python-jinja2 python-netaddr python-ipaddr python-yaml [](start = 12, length = 67)

we should remove these now, included in the sonic-config-engine #Resolved

curl gcc && \
Copy link
Collaborator

@lguohan lguohan Feb 8, 2017 •
edited by qiluo-msft
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc [](start = 18, length = 2)

do we need gcc? #Resolved

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is needed to install pip by 'get_pip.py'

In reply to: 99996933 [](ancestors = 99996933)

dpkg -i \
{% for deb in docker_snmp_sv2_debs.split(' ') -%}
debs/{{ deb }}{{' '}}
{%- endfor %} && \
rm -rf /debs && \
curl https://bootstrap.pypa.io/get-pip.py | python3.6 && \
python3.6 -m pip install --no-cache-dir /python-wheels/*py3*.whl hiredis && \
rm -rf /python-wheels && \
python3 -m sonic_ax_impl install && \
python3 -m pip uninstall -y pip setuptools && \
/bin/bash -c "rm -rf /usr/lib/python3.5/{unittest,lib2to3,tkinter,idlelib,email,test}" && \
apt-get -y purge build-essential libssl-dev openssl && \
apt-get clean -y && apt-get autoclean -y && apt-get autoremove -y && \
python3.6 -m sonic_ax_impl install && \
apt-get -y purge libpython3.6-dev libexpat1-dev curl gcc && \
apt-get clean -y && apt-get autoclean -y && apt-get autoremove -y --purge && \
find / | grep -E "__pycache__" | xargs rm -rf && \
rm -rf ~/.cache

Expand All @@ -51,4 +46,4 @@ COPY ["config.sh", "/usr/bin/"]
## Although exposing ports is not need for host net mode, keep it for possible bridge mode
EXPOSE 161/udp 162/udp

ENTRYPOINT /usr/bin/config.sh && /usr/bin/supervisord
ENTRYPOINT /usr/bin/config.sh && /usr/bin/supervisord
2 changes: 1 addition & 1 deletion dockers/docker-snmp-sv2/supervisord.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ command=/usr/sbin/snmpd -f -LS4d -u Debian-snmp -g Debian-snmp -I -smux,mteTrigg
priority=100

[program:snmp-subagent]
command=/usr/bin/env python3 -m sonic_ax_impl
command=/usr/bin/env python3.6 -m sonic_ax_impl
priority=200

[program:rsyslogd]
Expand Down
3 changes: 2 additions & 1 deletion rules/docker-snmp-sv2.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@

DOCKER_SNMP_SV2 = docker-snmp-sv2.gz
$(DOCKER_SNMP_SV2)_PATH = $(DOCKERS_PATH)/docker-snmp-sv2
$(DOCKER_SNMP_SV2)_DEPENDS += $(SNMP) $(SNMPD)
## TODO: remove LIBPY3_DEV if we can get pip3 directly
$(DOCKER_SNMP_SV2)_DEPENDS += $(SNMP) $(SNMPD) $(PY3) $(LIBPY3_DEV)
$(DOCKER_SNMP_SV2)_PYTHON_WHEELS += $(ASYNCSNMP_PY3)
$(DOCKER_SNMP_SV2)_LOAD_DOCKERS += $(DOCKER_CONFIG_ENGINE)
SONIC_DOCKER_IMAGES += $(DOCKER_SNMP_SV2)
Expand Down
38 changes: 38 additions & 0 deletions rules/python3.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# python3.5.2+
Copy link
Contributor

@stcheng stcheng Feb 4, 2017 •
edited by qiluo-msft
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3.6.0-1? #Resolved

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'+' indicates any later version

In reply to: 99458337 [](ancestors = 99458337)

Copy link
Contributor

@stcheng stcheng Feb 6, 2017 •
edited by qiluo-msft
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I know, but why not using a fixed version? #Resolved

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

snmpagent depends on python3.5.2+. Currently 3.6.0-1 was the only backport-able version for Debian Jessie fulfill our requirement. So I keep the filename and the comment as accurate as possible, in order not to mislead that 3.6.0-1 is the lowest require version.

In reply to: 99676879 [](ancestors = 99676879)

Copy link
Collaborator

@lguohan lguohan Feb 7, 2017 •
edited by qiluo-msft
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with shuotian, should remove 3.5.2+. we specify 3.6.0-1 that what we have validated. For example, we might use something in this package that is only available for 3.6.0-1, then 3.5.2 is no longer supported. But the questions is since we have no plan to validate/test 3.5.2 in the future, when are we gong to know it will 3.5.2. #Resolved

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed.

In reply to: 99899345 [](ancestors = 99899345)


PYTHON_VER=3.6.0-1
PYTHON_PNAME=python3.6

export PYTHON_VER
export PYTHON_PNAME

LIBPY3_MIN = lib$(PYTHON_PNAME)-minimal_$(PYTHON_VER)_amd64.deb
$(LIBPY3_MIN)_SRC_PATH = $(SRC_PATH)/python3
$(LIBPY3_MIN)_DEPENDS +=
$(LIBPY3_MIN)_RDEPENDS +=
SONIC_MAKE_DEBS += $(LIBPY3_MIN)

LIBPY3_STD = lib$(PYTHON_PNAME)-stdlib_$(PYTHON_VER)_amd64.deb
$(eval $(call add_derived_package,$(LIBPY3_MIN),$(LIBPY3_STD)))
$(LIBPY3_STD)_DEPENDS += $(LIBMPDECIMAL)
$(LIBPY3_STD)_RDEPENDS += $(LIBPY3_MIN) $(LIBMPDECIMAL)

LIBPY3 = lib$(PYTHON_PNAME)_$(PYTHON_VER)_amd64.deb
$(eval $(call add_derived_package,$(LIBPY3_MIN),$(LIBPY3)))
$(LIBPY3)_DEPENDS += $(LIBPY3_STD)
$(LIBPY3)_RDEPENDS += $(LIBPY3_MIN) $(LIBPY3_STD)

PY3_MIN = $(PYTHON_PNAME)-minimal_$(PYTHON_VER)_amd64.deb
$(eval $(call add_derived_package,$(LIBPY3_MIN),$(PY3_MIN)))
$(PY3_MIN)_RDEPENDS += $(LIBPY3_MIN)

PY3 = $(PYTHON_PNAME)_$(PYTHON_VER)_amd64.deb
$(eval $(call add_derived_package,$(LIBPY3_MIN),$(PY3)))
$(PY3)_DEPENDS += $(PY3_MIN) $(LIBPY3)
$(PY3)_RDEPENDS += $(PY3_MIN) $(LIBPY3) $(LIBPY3_MIN)

LIBPY3_DEV = lib$(PYTHON_PNAME)-dev_$(PYTHON_VER)_amd64.deb
$(eval $(call add_derived_package,$(LIBPY3_MIN),$(LIBPY3_DEV)))
$(LIBPY3_DEV)_DEPENDS += $(LIBPY3) $($(LIBPY3)_DEPENDS)
$(LIBPY3_DEV)_RDEPENDS += $(LIBPY3) $($(LIBPY3)_RDEPENDS)

18 changes: 0 additions & 18 deletions src/python3.5/build.sh
View file
Open in desktop

This file was deleted.

34 changes: 34 additions & 0 deletions src/python3/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
.ONESHELL:
SHELL = /bin/bash
.SHELLFLAGS += -e

PYTHON_VER=3.6.0
PYTHON_DEB_VER=1
PYTHON_PNAME=python3.6

MAIN_TARGET = lib$(PYTHON_PNAME)-minimal_$(PYTHON_VER)-$(PYTHON_DEB_VER)_amd64.deb
DERIVED_TARGETS = lib$(PYTHON_PNAME)-stdlib_$(PYTHON_VER)-$(PYTHON_DEB_VER)_amd64.deb \
lib$(PYTHON_PNAME)_$(PYTHON_VER)-$(PYTHON_DEB_VER)_amd64.deb \
$(PYTHON_PNAME)-minimal_$(PYTHON_VER)-$(PYTHON_DEB_VER)_amd64.deb \
$(PYTHON_PNAME)_$(PYTHON_VER)-$(PYTHON_DEB_VER)_amd64.deb \
lib$(PYTHON_PNAME)-dev_$(PYTHON_VER)-$(PYTHON_DEB_VER)_amd64.deb
#$(PYTHON_PNAME)-dev_$(PYTHON_VER)-$(PYTHON_DEB_VER)_amd64.deb

$(addprefix $(DEST)/, $(MAIN_TARGET)): $(DEST)/% :
## Obtaining the python3
wget 'https://sonicstorage.blob.core.windows.net/packages/$(PYTHON_PNAME)_$(PYTHON_VER).orig.tar.xz?sv=2015-04-05&sr=b&sig=d42Wh1CA9NZvlskhW4fpWcHVgc7N3IKhdFzyeO2zbRA%3D&se=2027-02-02T01%3A00%3A57Z&sp=r' -O $(PYTHON_PNAME)_$(PYTHON_VER).orig.tar.xz
wget 'https://sonicstorage.blob.core.windows.net/packages/$(PYTHON_PNAME)_$(PYTHON_VER)-$(PYTHON_DEB_VER).debian.tar.xz?sv=2015-04-05&sr=b&sig=KLX9pMJ3zpQvGBo6ZjzoZXgooMJRUUwMx8ZaTJtywK0%3D&se=2027-02-02T00%3A59%3A34Z&sp=r' -O $(PYTHON_PNAME)_$(PYTHON_VER)-$(PYTHON_DEB_VER).debian.tar.xz
wget 'https://sonicstorage.blob.core.windows.net/packages/$(PYTHON_PNAME)_$(PYTHON_VER)-$(PYTHON_DEB_VER).dsc?sv=2015-04-05&sr=b&sig=95s%2FC4vKY6bRKtkUTz%2BmHLqOllBOYbfP3zV5ayAuzSM%3D&se=2027-02-02T01%3A00%3A26Z&sp=r' -O $(PYTHON_PNAME)_$(PYTHON_VER)-$(PYTHON_DEB_VER).dsc

## Build
dpkg-source -x $(PYTHON_PNAME)_$(PYTHON_VER)-$(PYTHON_DEB_VER).dsc
pushd $(PYTHON_PNAME)-$(PYTHON_VER)
sudo apt-get install devscripts
mk-build-deps
sudo dpkg -i $(PYTHON_PNAME)-build-deps_$(PYTHON_VER)-$(PYTHON_DEB_VER)_amd64.deb || sudo apt-get install -f
dpkg-buildpackage -us -uc -b
popd

cp $(DERIVED_TARGETS) $* $(DEST)/

$(addprefix $(DEST)/, $(DERIVED_TARGETS)): $(DEST)/% : $(DEST)/$(MAIN_TARGET)