Add protobuf and dashapi to sonic-mgmt #15743

Pterosaur · 2023-07-07T15:34:07Z

Why I did it

The testcases in sonic-mgmt need the packages of protobuf and dashapi

Work item tracking

Microsoft ADO (number only):

How I did it

Because the docker of sonic-mgmt is based on ubuntu20.04, it cannot directly install the packages compiled by slave due to dependency issues. Download related packaged directly from Azp.

How to verify it

Check azp stats.

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

Signed-off-by: Ze Gan <ganze718@gmail.com>

lguohan · 2023-07-08T06:15:04Z

i think the software should be installed in docker-ptf not docker-mgmt

lguohan · 2023-07-08T06:19:03Z

where do we plan to run the sonic-mgmt pytest? is it running from sonic-mgmt or is it from docker-ptf?

lguohan · 2023-07-08T19:03:25Z

can you use docker_sonic_mgmt_debs, so that people if they want to build locally, they can still build it without depending on azp?

Pterosaur · 2023-07-09T02:47:39Z

where do we plan to run the sonic-mgmt pytest? is it running from sonic-mgmt or is it from docker-ptf?

All protobuf and dash payloads are for control plane which is loaded in sonic-mgmt, docker-ptf is only used for packet-gen of dataplane. So, To insteall protobuf and dash in sonic-mgmt is reasonable.

Pterosaur · 2023-07-09T02:57:42Z

can you use docker_sonic_mgmt_debs, so that people if they want to build locally, they can still build it without depending on azp?

I tried it before. But the docker of sonic-mgmt is based on ubuntu20.04 instead of debian bullseye, There are some differences on dependencies.
For example, the python3.9 is for bullseye but python3.8 is for ubuntu20.04. All docker_sonic_mgmt_debs are compiled under bullseye, so when I want to install the necessary package, python3-protobbuf.deb, it will be broken due to 3.8 < 3.9.
If I forcefully upgrade the python3.8 to python3.9, there are many existing dependencies are broken again,
Python version is one but not ONLY one dependency issue. So, the easiest way is to download them from Azp.

But in the future, when the sonic-mgmt is upgraded to ubuntu22.04, I believe we can directly install them from docker_sonic_mgmt_debs.

lguohan · 2023-07-09T07:25:26Z

Let's just build and install in sonic-mgmt container?

lguohan · 2023-07-09T07:25:41Z

Let's just build and install in sonic-mgmt container?

Pterosaur · 2023-07-12T07:40:26Z

/azp run

azure-pipelines · 2023-07-12T07:40:31Z

No commit pushedDate could be found for PR 15743 in repo sonic-net/sonic-buildimage

Signed-off-by: Ze Gan <ganze718@gmail.com>

This reverts commit 0d415b6.

Signed-off-by: Ze Gan <ganze718@gmail.com>

qiluo-msft · 2023-07-13T18:49:00Z

dockers/docker-sonic-mgmt/Dockerfile.j2

+# Install protobuf 3.21.12
+RUN mkdir -p /tmp/protobuf \
+    cd /tmp/protobuf \
+    && wget https://sonicstorage.blob.core.windows.net/public/ubuntu/20.04/libprotobuf-dev_3.21.12-3_amd64.deb \


20.04

It is also used in FROM in the beginning of the file. Will dpkg fail if we upgrade the FROM line to new version? #Pending

These packets are compiled in ubuntu20.04. Actually, a better method is to install these packets from bullseye slave by sonic_mgmt_debs variable, but due to bullseye and ubuntu dependencies issues, like python 3.8 and python 3.9, we have to install a pre-compiled version.
If we upgrade this docker to a new version, like ubuntu22.04, we can follow the typical method to build this docker.

Please consider 20.04 as a named constant.

qiluo-msft · 2023-07-13T18:52:34Z

dockers/docker-sonic-mgmt/Dockerfile.j2

+# Install dash-api
+RUN cd /tmp \
+    && mkdir -p /usr/lib/python3/dist-packages/dash_api \
+    && wget https://raw.githubusercontent.com/sonic-net/sonic-buildimage/master/src/sonic-dash-api/pypkg/__init__.py -O /usr/lib/python3/dist-packages/dash_api/__init__.py \


init.py

This file is inside the same repo, why wget? #Pending

qiluo-msft · 2023-07-13T18:53:19Z

dockers/docker-sonic-mgmt/Dockerfile.j2

+RUN cd /tmp \
+    && mkdir -p /usr/lib/python3/dist-packages/dash_api \
+    && wget https://raw.githubusercontent.com/sonic-net/sonic-buildimage/master/src/sonic-dash-api/pypkg/__init__.py -O /usr/lib/python3/dist-packages/dash_api/__init__.py \
+    && git clone https://github.com/sonic-net/sonic-dash-api.git \


sonic-dash-api.git

This repo is a submodule, it should be available in the working directory. #Pending

this is within the docker, not available.

Do we need to worry if this code on a release branch, but it actually wget/git clone some master files? For example, when branch out 202311, this dockerfile will pull master code.

If we want to access the files in src, we have to mount the src folder to Docker build workspace. But the files in src may be too many(More than 30GB) for Docker daemon which will dramatically downgrade the performance of Docker daemon.

After the sonic-mgmt upgrade to 20.04, I believe we can follow the typical method, docker_mgmt_debs, to install them.

Please explore # 1: you do not need to mount src folder, just copy what you need to Docker build workspace.

Under the current framework, I didn't find a script to do that.

I created a issue #15848. And ask for solving in future PR.

dockers/docker-sonic-mgmt/Dockerfile.j2

Signed-off-by: Ze Gan <ganze718@gmail.com>

#### Why I did it The testcases in sonic-mgmt need the packages of protobuf and dashapi ##### Work item tracking - Microsoft ADO **(number only)**: #### How I did it Because the docker of sonic-mgmt is based on ubuntu20.04, it cannot directly install the packages compiled by slave due to dependency issues. Download related packaged directly from Azp. #### How to verify it Check azp stats.

Add protobuf and dashapi to sonic-mgmt

bcf8e31

Signed-off-by: Ze Gan <ganze718@gmail.com>

Pterosaur force-pushed the dash_for_mgmt branch from 2c2ef50 to bcf8e31 Compare July 7, 2023 15:39

Pterosaur marked this pull request as ready for review July 8, 2023 01:42

Pterosaur requested review from qiluo-msft, xumia and lguohan as code owners July 8, 2023 01:42

Pterosaur force-pushed the dash_for_mgmt branch 6 times, most recently from 78eeecc to 18770b1 Compare July 11, 2023 09:56

Pterosaur force-pushed the dash_for_mgmt branch from 18770b1 to edd980c Compare July 12, 2023 07:43

Fix comment

0d415b6

Signed-off-by: Ze Gan <ganze718@gmail.com>

Pterosaur force-pushed the dash_for_mgmt branch from edd980c to 0d415b6 Compare July 12, 2023 09:34

Revert "Fix comment"

f6c2d3b

This reverts commit 0d415b6.

Pterosaur force-pushed the dash_for_mgmt branch from a87a582 to 140fac2 Compare July 13, 2023 06:56

Fix comments

aea6980

Signed-off-by: Ze Gan <ganze718@gmail.com>

Pterosaur force-pushed the dash_for_mgmt branch from 140fac2 to aea6980 Compare July 13, 2023 12:51

lguohan previously approved these changes Jul 13, 2023

View reviewed changes

qiluo-msft reviewed Jul 13, 2023

View reviewed changes

Pterosaur requested a review from qiluo-msft July 13, 2023 23:45

xumia reviewed Jul 14, 2023

View reviewed changes

dockers/docker-sonic-mgmt/Dockerfile.j2 Outdated Show resolved Hide resolved

Remove unzip

fe31591

Signed-off-by: Ze Gan <ganze718@gmail.com>

Pterosaur dismissed lguohan’s stale review via fe31591 July 14, 2023 04:37

Pterosaur requested a review from xumia July 14, 2023 06:33

lguohan approved these changes Jul 14, 2023

View reviewed changes

qiluo-msft merged commit a248459 into sonic-net:master Jul 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add protobuf and dashapi to sonic-mgmt #15743

Add protobuf and dashapi to sonic-mgmt #15743

Pterosaur commented Jul 7, 2023

lguohan commented Jul 8, 2023

lguohan commented Jul 8, 2023

lguohan commented Jul 8, 2023

Pterosaur commented Jul 9, 2023 •

edited

Loading

Pterosaur commented Jul 9, 2023

lguohan commented Jul 9, 2023

lguohan commented Jul 9, 2023

Pterosaur commented Jul 12, 2023

azure-pipelines bot commented Jul 12, 2023

qiluo-msft Jul 13, 2023 •

edited

Loading

Pterosaur Jul 13, 2023

qiluo-msft Jul 14, 2023

qiluo-msft Jul 13, 2023 •

edited

Loading

qiluo-msft Jul 13, 2023 •

edited

Loading

lguohan Jul 13, 2023

qiluo-msft Jul 13, 2023 •

edited

Loading

Pterosaur Jul 13, 2023 •

edited

Loading

qiluo-msft Jul 14, 2023

Pterosaur Jul 14, 2023

qiluo-msft Jul 14, 2023

Add protobuf and dashapi to sonic-mgmt #15743

Add protobuf and dashapi to sonic-mgmt #15743

Conversation

Pterosaur commented Jul 7, 2023

Why I did it

Work item tracking

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

lguohan commented Jul 8, 2023

lguohan commented Jul 8, 2023

lguohan commented Jul 8, 2023

Pterosaur commented Jul 9, 2023 • edited Loading

Pterosaur commented Jul 9, 2023

lguohan commented Jul 9, 2023

lguohan commented Jul 9, 2023

Pterosaur commented Jul 12, 2023

azure-pipelines bot commented Jul 12, 2023

qiluo-msft Jul 13, 2023 • edited Loading

Choose a reason for hiding this comment

Pterosaur Jul 13, 2023

Choose a reason for hiding this comment

qiluo-msft Jul 14, 2023

Choose a reason for hiding this comment

qiluo-msft Jul 13, 2023 • edited Loading

Choose a reason for hiding this comment

qiluo-msft Jul 13, 2023 • edited Loading

Choose a reason for hiding this comment

lguohan Jul 13, 2023

Choose a reason for hiding this comment

qiluo-msft Jul 13, 2023 • edited Loading

Choose a reason for hiding this comment

Pterosaur Jul 13, 2023 • edited Loading

Choose a reason for hiding this comment

qiluo-msft Jul 14, 2023

Choose a reason for hiding this comment

Pterosaur Jul 14, 2023

Choose a reason for hiding this comment

qiluo-msft Jul 14, 2023

Choose a reason for hiding this comment

Pterosaur commented Jul 9, 2023 •

edited

Loading

qiluo-msft Jul 13, 2023 •

edited

Loading

qiluo-msft Jul 13, 2023 •

edited

Loading

qiluo-msft Jul 13, 2023 •

edited

Loading

qiluo-msft Jul 13, 2023 •

edited

Loading

Pterosaur Jul 13, 2023 •

edited

Loading