Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added initial version of HLD "SONiC reproduceable build" web server population script #976

Merged
merged 2 commits into from
Apr 25, 2022
Merged

Added initial version of HLD "SONiC reproduceable build" web server population script #976

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b505a38
web server population script HLD
orenreiss Mar 22, 2022
7ed5fe4
pull request feedbacks fixes
orenreiss Apr 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 129 additions & 0 deletions doc/sonic-build-system/web_file_server_population_script.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
# Web file server population script
 
 
## _Revision History_

Date | Description
----------- | -------------
3/9/22 | First draft


## _Table of Contents_

1. [Overview](#Overview)
1. [In scope](#subparagraph1)
2. [Download web packages from external file storage](#paragraph1)
1. [Work Items](#subparagraph1)
3. [Upload web packages to trusted file storage](#paragraph2)
1. [Work Items](#subparagraph1)
4. [User Input](#paragraph3)
5. [Script outputs](#paragraph4)
6. [Build flow integration](#paragraph5)
7. [Source tree location](#paragraph6)
8. [Q&A](#paragraph7)



## Overview
The SONiC reproducible build framework provides a way to lock the versions of external packages. It will lock the versions for all the docker images including the slave docker images, host images, and docker slave containers which are used to build the SONiC targets, such as deb files, python wheels, etc.
SONiC reproducible build allows to identify specific version of a web package retrieved by wget/curl and download it from an external file storage.
The web site may be not stable, and the web packages may be replaced to another one unexpectedly, so we do not want to depend on the web packages at a remote web site.
Before a web package is used in the SONiC repository, the package should be uploaded to a trusted file storage. Currently SONiC offers to do that manually or automatically via Jenkins pipeline.
The file Server population script is a complementary utility for “SONiC reproducible build” and suppose to ease the process of downloading the web packages from an external file storage and uploading them to trusted file storage.
The script will be triggered manually or part of the build flow (part of “make freeze”) whenever the developer intend to populate the web server which packages associated to specific tag.

### In scope:
- Download web packages from external file storage
- Rename and upload web packages to trusted file storage
- Script output

## Download web packages from external file storage
The script will identify all relevant web packages, collect their hash value and url and then download them from the remote site to a local cache.
The script search over the source tree path recursively and look for ‘version-web’ files.
Those file are being parsed in order to retrieve the path to the package and its hash value. The script is expecting for such format:
[package url]==[hash value]

```sh
For example:
https://storage.googleapis.com/golang/go1.14.2.linux-amd64.tar.gz==ebef065e4d35572af5b03e2be957a9c6c5063b38
```

### Work Items:
- Identify relevant versions-web files
- Parse versions-web files and identify relevant web packages
- Download web packages from their external file storage


## Upload web packages to trusted file storage

Although there is no version for a package downloaded by wget/curl, the hash value of the package file can be used as the version value.
The script would rename the Web package and then upload it from cache to a trusted file server. The file name format in the file storage is as below:
[original file name]-[hash value]
```sh
For an example:
go1.14.2.linux-amd64.tar.gz-ebef065e4d35572af5b03e2be957a9c6c5063b38
```
For the same web package, it supports to have multiple versions of the file in the file storage
for example:
```sh
For an example:
go1.14.2.linux-amd64.tar.gz-ebef065e4d35572af5b03e2be957a9c6c5063b38
go1.14.2.linux-amd64.tar.gz-b73d6366c0bce441d20138c150e2ddcdf406e654
```
Same file name is used with different hash

### Work Items:
- Rename web package
- Upload web package to trusted file storage
- Generate output version file (optional)

## User Input:
The user would provide the script the following parameters:

| Name | Description | Tag | Type | Default value |
|------------- |------------------------------------------------------------------------------------------------------------ |-------------- |--------- |------------------------------------ |
| source | Search path for source tree where to look for<br> compilation artifacts (version files) | -s, --source | String | CWD<br>(current working directory) |
| cache | Path for temporary storage where to download Web Packages | -c, --cache | String | CWD/tmp |
| output | Output file name to hold the list of packages.<br> The file will be created in the format of version files | -o, --output | String | none |
| print | Print level verbosity | -p, --print | integer | 1 (info level) |
| user | User for trusted server authentication | -u, --user | String | none |
| key | key server authentication | -k, --key | String | none |
| Destination | URL for destination web file server | -d, --dest | String | none |


## Script outputs:

The script would print out to STDOUT its current download progress (progress bar) and error messages in case of errors.
The script can produce a list of packages that were uploaded in a format of version file.
The script would return 0 in case of success and an errors code (see table below) in case of failure.
### Error codes:

Code | Description | Comments
------------ | -------------------------------------| -------------
0 | Success |
-1 | Cannot write to cache |
-2 | Cannot access trusted server |
-3 | Cannot authenticate trusted server |


## Build flow integration:
Script will be integrated into SONiC build environment as part of “make freeze” where invocation of the script is optional. User may choose to call script by defining following environment variables:

```sh
export SONIC_REPRODUCABLE_BUILD_UPLAOD_ENABLE=1
export SONIC_REPRODUCABLE_BUILD_UPLAOD_USER=<User for trusted server authentication>
export SONIC_REPRODUCABLE_BUILD_UPLAOD_KEY=<API key server authentication>
export SONIC_REPRODUCABLE_BUILD_UPLAOD_DEST=< URL for destination web file server>
```
orenreiss marked this conversation as resolved.
Show resolved Hide resolved

Another approach is to call the script manually and provide as an input the pat where the script would search for version files (version files which were generated as part of "make freeze" invocation or by previous invocation of the script).

## Source tree location:

The script will be located under
sonic-buildimage/scripts/populate_file_web_server.py

## Q&A
Who should run this script?
When should user run this script?