Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial commit for introducing the SONiC Platform Development Environ… #3778

Closed
wants to merge 10 commits into from
21 changes: 20 additions & 1 deletion Makefile.work
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
# * BUILD_NUMBER: Desired version-number to pass to the building-system.
# * ENABLE_DHCP_GRAPH_SERVICE: Enables get-graph service to fetch minigraph files
# through http.
# * ENABLE_PDE: Enables platform development environment.
# * ENABLE_ZTP: Enables zero touch provisioning.
# * SHUTDOWN_BGP_ON_START: Sets admin-down state for all bgp peerings after restart.
# * INCLUDE_KUBERNETES: Allows including Kubernetes
Expand Down Expand Up @@ -187,6 +188,7 @@ SONIC_BUILD_INSTRUCTION := make \
BUILD_NUMBER=$(BUILD_NUMBER) \
BUILD_TIMESTAMP=$(BUILD_TIMESTAMP) \
ENABLE_DHCP_GRAPH_SERVICE=$(ENABLE_DHCP_GRAPH_SERVICE) \
ENABLE_PDE=$(ENABLE_PDE) \
ENABLE_ZTP=$(ENABLE_ZTP) \
SHUTDOWN_BGP_ON_START=$(SHUTDOWN_BGP_ON_START) \
INCLUDE_KUBERNETES=$(INCLUDE_KUBERNETES) \
Expand Down Expand Up @@ -216,7 +218,18 @@ SONIC_BUILD_INSTRUCTION := make \
ENABLE_SYNCHRONOUS_MODE=$(ENABLE_SYNCHRONOUS_MODE) \
$(SONIC_OVERRIDE_BUILD_VARS)

.PHONY: sonic-slave-build sonic-slave-bash init reset
PDESUBMODULES = src/sonic-linux-kernel \
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i do not think this is needed.

src/sonic-utilities \
src/sonic-platform-common \
src/sonic-swss-common \
src/sonic-py-swsssdk \
src/redis-dump-load \
src/sonic-sairedis \
src/sonic-platform-daemons \
src/sonic-platform-pde \
src/sonic-ztp

.PHONY: sonic-slave-build sonic-slave-bash init initpde reset
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get rid of initpde

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to be able to build a minimal subset of SONiC with the PDE docker enabled. We can either handle this as part of make init (or in this case make initpde), or we would have to modify slave.mk to ignore NON-PDE required build targets. In this case, we think its easier to maintain a list of essential submodules for the PDE and make it part of the "make initpde" step for doing a build. Can we leave this change as it is or do yo have an alternative suggestion?

We automate the PDE build internally by doing the following.

make initpde
make configure PLATFORM=broadcom
make target/sonic-broadcom-pde.bin

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lguohan, @jleveque - can you please address these responses from Bill (following comments also)? We have come up with a way of doing a minimal build for PDE (reduced build time etc), but it seems that you didn't like this. Do you have an alternative in mind? Thanks.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lguohan, @jleveque - can you please address these responses from Bill (following comments also)? We have come up with a way of doing a minimal build for PDE (reduced build time etc), but it seems that you didn't like this. Do you have an alternative in mind? Thanks.

@lguohan, @jleveque - I never heard back. Should I setup a call to discuss?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to run this "make initpde" to differentiate the Full SONIC image build. Actually, ODM will not run this step, since the step is done by BRCM internally to generate the light weight PDE repo for ODM. Another reason for this is to protect the BRCM SONIC changes with just platform related common change exposing to ODM when working with ODM on ODM HWQ projects.

For PDE, we also have another option which is to run PDE as a docker in Full SONIC image. In the FULL sonic image runtime, we can launch PDE by pde.sh script. The script will disable most dockers in SONIC image and just reserve the necessary dockers with a lightweight environment for ODM platform testing. By default, the PDE docker will not be triggered in Full SONIC image.

Geans

make initpde

make configure PLATFORM=broadcom

make target/sonic-broadcom-pde.bin

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lguohan, @jleveque - no movement on this topic - please offer some morning slots and I will try to setup a call - thx.


.DEFAULT_GOAL := all

Expand Down Expand Up @@ -277,6 +290,12 @@ showtag:
@echo $(SLAVE_IMAGE):$(SLAVE_TAG)
@echo $(SLAVE_BASE_IMAGE):$(SLAVE_BASE_TAG)

initpde:
@for submodule in $(PDESUBMODULES) ; do \
git submodule update --init --recursive $$submodule ; \
done
@git submodule foreach --recursive '[ -f .git ] && echo "gitdir: $$(realpath --relative-to=. $$(cut -d" " -f2 .git))" > .git'

init :
@git submodule update --init --recursive
@git submodule foreach --recursive '[ -f .git ] && echo "gitdir: $$(realpath --relative-to=. $$(cut -d" " -f2 .git))" > .git'
Expand Down
71 changes: 71 additions & 0 deletions dockers/docker-pde/Dockerfile.j2
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
{% from "dockers/dockerfile-macros.j2" import install_debian_packages, install_python_wheels, copy_files %}
FROM docker-config-engine-stretch

ARG docker_container_name
ENV PYTHONPATH=/usr/share/sonic/platform

RUN [ -f /etc/rsyslog.conf ] && sed -ri "s/%syslogtag%/$docker_container_name#%syslogtag%/;" /etc/rsyslog.conf

# Make apt-get non-interactive
ENV DEBIAN_FRONTEND=noninteractive

# Update apt's cache of available packages
RUN apt-get update && apt-get install -y ipmitool telnet busybox kmod

RUN pip install pytest pexpect

{% if docker_pde_debs.strip() -%}
# Copy locally-built Debian package dependencies
COPY \
{% for deb in docker_pde_debs.split(' ') -%}
debs/{{ deb }}{{' '}}
{%- endfor -%}
/debs/

# Install locally-built Debian packages and implicitly install their dependencies
RUN dpkg_apt() { [ -f $1 ] && { dpkg -i $1 || apt-get -y install -f; } || return 1; }{%- for deb in docker_pde_debs.split(' ') %}; \
dpkg_apt /debs/{{ deb }}
{%- endfor %}
{%- endif %}

{% if docker_pde_pydebs.strip() -%}
# Copy locally-built Debian package dependencies
COPY \
{% for deb in docker_pde_pydebs.split(' ') -%}
python-debs/{{ deb }}{{' '}}
{%- endfor -%}
/debs/

# Install locally-built Debian packages and implicitly install their dependencies
{%- for deb in docker_pde_pydebs.split(' ') %}
RUN dpkg_apt() { [ -f $1 ] && { dpkg -i $1 || apt-get -y install -f; } || return 1; }; dpkg_apt /debs/{{ deb }}
{%- endfor %}
{%- endif %}

{% if docker_pde_whls.strip() -%}
# Copy locally-built Python wheel dependencies
COPY \
{% for whl in docker_pde_whls.split(' ') -%}
python-wheels/{{ whl }}{{' '}}
{%- endfor -%}
/python-wheels/

# Install locally-built Python wheel dependencies
RUN pip install \
{% for whl in docker_pde_whls.split(' ') -%}
/python-wheels/{{ whl }}{{' '}}
{%- endfor %}
{% endif %}


# Clean up
RUN apt-get clean -y; \
apt-get autoclean -y; \
apt-get autoremove -y
RUN rm -rf /debs ~/.cache

RUN mkdir -p /home/pde

COPY ["supervisord.conf", "/etc/supervisor/conf.d/"]
COPY ["docker_init.sh", "/usr/bin/"]
ENTRYPOINT ["/usr/bin/docker_init.sh"]
63 changes: 63 additions & 0 deletions dockers/docker-pde/base_image_files/pde-test-harness
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
#!/bin/bash

usage()
{
echo "usage: $0 <help/list/commands>"
exit 1
}

if [ $# -lt 1 ]
then
usage
fi

case $1 in
-h | help)
usage
;;
-l | ls | list)
docker exec -it -w /usr/local/sonic_pde_tests pde ls
exit 0
;;
*)
;;
esac

# Make sure only root can run our script
if [ "$(id -u)" != "0" ]; then
echo "This script must be run as root" 1>&2
exit 1
fi

# Stop swss/syncd service to ensure there is only 1 SAI instance at a time
systemctl stop swss

if [ $(grep -c 'platform:ro' $(which pde.sh)) -gt 0 ]; then
systemctl stop pde
sleep 1
docker rename pde pde.orig
sed -i 's#platform:ro#platform#g' $(which pde.sh)
sed -i 's#hwsku:ro#hwsku#g' $(which pde.sh)
systemctl start pde
fi

# Allow 5 seconds for the docker-pde to be online
err=1
for t in $(seq 5); do
if [ $(docker ps | grep -c pde) -gt 0 ]; then
err=0
break
fi
sleep 1
done
if [ ${err} -ne 0 ]; then
echo "docker-pde is not alive, existing ..."
exit 1
fi

# Now launch the PDE pytest
docker exec -it -w /usr/local/sonic_pde_tests pde pytest $*

# Restart the swss service
systemctl start swss

31 changes: 31 additions & 0 deletions dockers/docker-pde/docker_init.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
#!/usr/bin/env bash

set -x

start_bcm()
{
[ -e /dev/linux-bcm-knet ] || mknod /dev/linux-bcm-knet c 122 0
[ -e /dev/linux-user-bde ] || mknod /dev/linux-user-bde c 126 0
[ -e /dev/linux-kernel-bde ] || mknod /dev/linux-kernel-bde c 127 0
}

PLATFORM_DIR=/usr/share/sonic/platform
HWSKU_DIR=/usr/share/sonic/hwsku

mkdir -p /etc/sai.d/

if [ -f $HWSKU_DIR/sai.profile ]; then
cp $HWSKU_DIR/sai.profile /etc/sai.d/sai.profile

SAI_CONFIG=`cat /etc/sai.d/sai.profile | rev | cut -d "/" -f1 | rev`

ln -s $HWSKU_DIR/$SAI_CONFIG /tmp/brcm_sai_config.bcm
ln -s $HWSKU_DIR/port_config.ini /tmp/port_config.ini

parse_port_config.py > /tmp/test_config.ini
fi

start_bcm

exec /usr/bin/supervisord

13 changes: 13 additions & 0 deletions dockers/docker-pde/supervisord.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
[supervisord]
logfile_maxbytes=1MB
logfile_backups=2
nodaemon=true

[program:rsyslogd]
command=/usr/sbin/rsyslogd -n -iNONE"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo

priority=1
autostart=true
autorestart=false
stdout_logfile=syslog
stderr_logfile=syslog
dependent_startup=true
5 changes: 4 additions & 1 deletion files/build_templates/docker_image_ctl.j2
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,10 @@ start() {
source $ASIC_CONF
fi

{%- if docker_container_name == "database" %}
{%- if docker_container_name == "pde" %}
# PDE does not have access to database
HWSKU=`cat /usr/share/sonic/device/$PLATFORM/default_sku | cut -d ' ' -f1`
{%- elif docker_container_name == "database" %}
# Don't mount HWSKU in {{docker_container_name}} container.
HWSKU=""
{%- else %}
Expand Down
22 changes: 22 additions & 0 deletions files/build_templates/pde.service.j2
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
[Unit]
Description=PDE container
{% if sonic_asic_platform == 'broadcom' %}
Requires=opennsl-modules.service
ConditionPathExists=!/usr/bin/swss.sh
{% endif %}

After=updategraph.service
{% if sonic_asic_platform == 'broadcom' %}
After=opennsl-modules.service
{% endif %}

Before=

[Service]
User={{ sonicadmin_user }}
ExecStartPre=/usr/bin/{{ docker_container_name }}.sh start
ExecStart=/usr/bin/{{ docker_container_name }}.sh wait
ExecStop=/usr/bin/{{ docker_container_name }}.sh stop

[Install]
WantedBy=multi-user.target
51 changes: 51 additions & 0 deletions platform/broadcom/docker-pde.mk
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# Docker image for SONiC Platform Development Environment (PDE)


ifeq ($(ENABLE_PDE), y)

DOCKER_PDE_STEM = docker-pde
DOCKER_PDE = $(DOCKER_PDE_STEM).gz
DOCKER_PDE_DBG = $(DOCKER_PDE_STEM)-$(DBG_IMAGE_MARK).gz

$(DOCKER_PDE)_PATH = $(DOCKERS_PATH)/$(DOCKER_PDE_STEM)

$(DOCKER_PDE)_DEPENDS += $(PYTHON_NETIFACES)
$(DOCKER_PDE)_DEPENDS += $(SONIC_PLATFORM_PDE) $(BRCM_SAI)

$(DOCKER_PDE_RDEPENDS += $(PYTHON_NETIFACES)

$(DOCKER_PDE)_PYTHON_DEBS += $(SONIC_UTILS)
$(DOCKER_PDE)_PYTHON_WHEELS += $(SONIC_PLATFORM_COMMON_PY2)
ifeq ($(PDDF_SUPPORT), y)
$(DOCKER_PDE)_PYTHON_WHEELS += $(PDDF_PLATFORM_API_BASE_PY2)
endif
$(DOCKER_PDE)_PYTHON_WHEELS += $(SONIC_DAEMON_BASE_PY2)
$(DOCKER_PDE)_DBG_DEPENDS = $($(DOCKER_CONFIG_ENGINE_STRETCH)_DBG_DEPENDS)
$(DOCKER_PDE)_DBG_IMAGE_PACKAGES = $($(DOCKER_CONFIG_ENGINE_STRETCH)_DBG_IMAGE_PACKAGES)

$(DOCKER_PDE)_LOAD_DOCKERS = $(DOCKER_CONFIG_ENGINE_STRETCH)

SONIC_DOCKER_IMAGES += $(DOCKER_PDE)
SONIC_STRETCH_DOCKERS += $(DOCKER_PDE)
SONIC_INSTALL_DOCKER_IMAGES += $(DOCKER_PDE)

SONIC_STRETCH_DBG_DOCKERS += $(DOCKER_PDE_DBG)
SONIC_DOCKER_DBG_IMAGES += $(DOCKER_PDE_DBG)
SONIC_INSTALL_DOCKER_DBG_IMAGES += $(DOCKER_PDE_DBG)

$(DOCKER_PDE)_CONTAINER_NAME = pde
$(DOCKER_PDE)_RUN_OPT += --net=host --privileged -t
$(DOCKER_PDE)_RUN_OPT += -v /etc/sonic:/etc/sonic:ro
$(DOCKER_PDE)_RUN_OPT += -v /host/machine.conf:/host/machine.conf:ro
$(DOCKER_PDE)_RUN_OPT += -v /usr/lib/python2.7/dist-packages:/usr/share/sonic/classes:ro
$(DOCKER_PDE)_RUN_OPT += -v /var/log/syslog:/var/log/syslog:ro
$(DOCKER_PDE)_RUN_OPT += -v /lib/modules:/lib/modules:ro
$(DOCKER_PDE)_RUN_OPT += -v /boot:/boot:ro

$(DOCKER_PDE)_RUN_OPT += -v /usr/share/sonic/device/pddf:/usr/share/sonic/device/pddf:ro
$(DOCKER_PDE)_BASE_IMAGE_FILES += pde-test:/usr/local/bin/pde-test
$(DOCKER_PDE)_BASE_IMAGE_FILES += pde-bench:/usr/local/bin/pde-bench
$(DOCKER_PDE)_BASE_IMAGE_FILES += pde-stress:/usr/local/bin/pde-stress
$(DOCKER_PDE)_BASE_IMAGE_FILES += pde-bench-knet:/usr/local/bin/pde-bench-knet

endif
11 changes: 11 additions & 0 deletions platform/broadcom/one-pde-image.mk
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# sonic broadcom one image installer

SONIC_ONE_PDE_IMAGE = sonic-broadcom-pde.bin
$(SONIC_ONE_PDE_IMAGE)_MACHINE = broadcom
$(SONIC_ONE_PDE_IMAGE)_IMAGE_TYPE = onie
$(SONIC_ONE_PDE_IMAGE)_INSTALLS += $(BRCM_OPENNSL_KERNEL)
$(SONIC_ONE_PDE_IMAGE)_INSTALLS += $(PDDF_PLATFORM_MODULE)
$(SONIC_ONE_PDE_IMAGE)_LAZY_INSTALLS += $(SONIC_ONE_IMAGE_LAZY_INSTALLS)

$(SONIC_ONE_PDE_IMAGE)_DOCKERS += $(DOCKER_PDE) $(DOCKER_PLATFORM_MONITOR) $(DOCKER_DATABASE)
SONIC_INSTALLERS += $(SONIC_ONE_PDE_IMAGE)
7 changes: 7 additions & 0 deletions platform/broadcom/sonic-pde-tests.mk
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# sonic pde package

SONIC_PLATFORM_PDE = sonic-platform-pde_1.0_amd64.deb
$(SONIC_PLATFORM_PDE)_SRC_PATH = $(SRC_PATH)/sonic-platform-pde
$(SONIC_PLATFORM_PDE)_DEPENDS += $(BRCM_SAI) $(BRCM_SAI_DEV) $(SWIG)

SONIC_DPKG_DEBS += $(SONIC_PLATFORM_PDE)
5 changes: 5 additions & 0 deletions rules/config
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,11 @@ DEFAULT_PASSWORD = YourPaSsWoRd
# If not set (default behavior) the default minigraph built into the image will be used.
# ENABLE_DHCP_GRAPH_SERVICE = y

# ENABLE_PDE - Enable platform development environment
# If set to y the PDE docker and associated tools and applications will be built as part of the
# SONiC image and present on the system after installation.
# ENABLE_PDE = y

# ENABLE_ZTP - installs Zero Touch Provisioning support.
# ENABLE_ZTP = y

Expand Down
5 changes: 5 additions & 0 deletions slave.mk
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,7 @@ $(info "ENABLE_SYNCD_RPC" : "$(ENABLE_SYNCD_RPC)")
$(info "ENABLE_ORGANIZATION_EXTENSIONS" : "$(ENABLE_ORGANIZATION_EXTENSIONS)")
$(info "HTTP_PROXY" : "$(HTTP_PROXY)")
$(info "HTTPS_PROXY" : "$(HTTPS_PROXY)")
$(info "ENABLE_PDE" : "$(ENABLE_PDE)")
$(info "ENABLE_ZTP" : "$(ENABLE_ZTP)")
$(info "SONIC_DEBUGGING_ON" : "$(SONIC_DEBUGGING_ON)")
$(info "SONIC_PROFILING_ON" : "$(SONIC_PROFILING_ON)")
Expand Down Expand Up @@ -825,6 +826,7 @@ $(addprefix $(TARGET_PATH)/, $(SONIC_INSTALLERS)) : $(TARGET_PATH)/% : \
export sonic_asic_platform="$(patsubst %-$(CONFIGURED_ARCH),%,$(CONFIGURED_PLATFORM))"
export enable_organization_extensions="$(ENABLE_ORGANIZATION_EXTENSIONS)"
export enable_dhcp_graph_service="$(ENABLE_DHCP_GRAPH_SERVICE)"
export enable_pde="$(ENABLE_PDE)"
export enable_ztp="$(ENABLE_ZTP)"
export include_system_telemetry="$(INCLUDE_SYSTEM_TELEMETRY)"
export include_restapi="$(INCLUDE_RESTAPI)"
Expand Down Expand Up @@ -915,17 +917,20 @@ $(addprefix $(TARGET_PATH)/, $(SONIC_INSTALLERS)) : $(TARGET_PATH)/% : \
DEBUG_SRC_ARCHIVE_FILE="$(DBG_SRC_ARCHIVE_FILE)" \
scripts/dbg_files.sh


DEBUG_IMG="$(INSTALL_DEBUG_TOOLS)" \
DEBUG_SRC_ARCHIVE_FILE="$(DBG_SRC_ARCHIVE_FILE)" \
USERNAME="$(USERNAME)" \
PASSWORD="$(PASSWORD)" \
BUILD_TARGET="$@" \
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not this BUILD_TARGET used anyway. is there something else missing?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We pass the BUILD_TARGET to the docker build slave so we can support the build target sonic-broadcom-pde.bin to differenciate it from the normal build. Please let me know if there is a better way to handle this.

IMAGE_TYPE=$($*_IMAGE_TYPE) \
./build_debian.sh $(LOG)

USERNAME="$(USERNAME)" \
PASSWORD="$(PASSWORD)" \
TARGET_MACHINE=$($*_MACHINE) \
IMAGE_TYPE=$($*_IMAGE_TYPE) \
BUILD_TARGET="$@" \
SONIC_ENABLE_IMAGE_SIGNATURE="$(SONIC_ENABLE_IMAGE_SIGNATURE)" \
SIGNING_KEY="$(SIGNING_KEY)" \
SIGNING_CERT="$(SIGNING_CERT)" \
Expand Down