diff --git a/docs/2024/ci-scanner/updates/2024-06-20.md b/docs/2024/ci-scanner/updates/2024-06-20.md new file mode 100644 index 000000000..f53f6387b --- /dev/null +++ b/docs/2024/ci-scanner/updates/2024-06-20.md @@ -0,0 +1,51 @@ +--- +title: Week 3 +author: Rajul Jha +--- + + +# Week 3 +*(June 14, 2024 - June 20, 2024)* + +## Meeting 1 +*(June 19, 2024)* + +## Attendees +* [Rajul Jha](https://github.com/rajuljha) +* [Kaushlendra](https://github.com/Kaushl2208) +* [Shaheem Azmal](https://github.com/shaheemazmalmmd) +* Katharina + + +## Discussions + +* Worked on adding nomos json output. But not able to access the theMatches object's licenseAndMatchPositions object. Stuck on this for nomos task. +* Bumped up spdx_tools library to latest version. Tested with both platforms (GH Actions and Gitlab). +* Discussed an issue in the github workflow, not supporting multiple architectures. + * The GH Actions Runner does not support multi-architecture images. + * We discussed potential solutions for the same including trying to build the image in the GH Action itself or utilizing an emulator like qemu. + +## Work Done + +* Upgraded the spdx_tools library [(#PR2762)](https://github.com/fossology/fossology/pull/2762) and did performance analysis for both versions. + * With Version `0.0.0a2`: + ![Screenshot](/img/ci/Spdx_tools_Version0.8.0a2.png) + + * With Version `0.8.2`: + ![Screenshot](/img/ci/Spdx_tools_Version_0.8.2.png) + +* Worked on providing custom `keyword.conf` file during CI pipeline. It works as follows: + * User creates a custom `keyword.conf` file following [this](https://github.com/fossology/fossology/blob/master/src/copyright/agent/keyword.conf) pattern. + * They set an environment variable called `KEYWORD_CONF_FILE_PATH` in CI providers settings and set its value to the path of the `keyword.conf` file. + * Then, when the pipeline is triggered, say on `push` to `main`, then the keyword scanner reads the custom file and scans for the keywords specified by the user. + * Keep in mind that the current keyword file is overwritten by the script, as discussed with the mentors. + +## Planning for next week + +* Test the keyword functionality with GH Actions and Gitlab CI and send out a PR for the same. +* Work on providing multi architecture support for GH Actions. +* Study about how to implement differential scans. \ No newline at end of file diff --git a/docs/2024/ci-scanner/updates/2024-06-27.md b/docs/2024/ci-scanner/updates/2024-06-27.md new file mode 100644 index 000000000..f92a681cc --- /dev/null +++ b/docs/2024/ci-scanner/updates/2024-06-27.md @@ -0,0 +1,61 @@ +--- +title: Week 4 +author: Rajul Jha +--- + + +# Week 4 +*(June 21, 2024 - June 27, 2024)* + +## Meeting 1 +*(June 26, 2024)* + +## Attendees +* [Rajul Jha](https://github.com/rajuljha) +* [Kaushlendra](https://github.com/Kaushl2208) +* [Shaheem Azmal](https://github.com/shaheemazmalmmd) +* [Avinal Kumar](https://github.com/avinal) +* Katharina + +## Discussions + +* Talked about how the differential scans need to work in detail, like: + * Should they be triggered on `release` only? As it makes sense to scan between two different versions when a new release it being published. + * Or should the user be able to scan between two different versions via any sort of trigger, like on `push` or `commit` as well? +* Talked about the current implementation of custom keyword functionality, and how it can be + extrapolated similarly for different scanners also. +* We talked about fleshing out the design and implementation details first, then work on improving speed. + +## Work Done + +* Completed custom keyword functionality. [(#PR2769)](https://github.com/fossology/fossology/pull/2769) The details are as follows: + * User can now setup a GH Action or Gitlab CI pipeline and pass environment variables to scan for custom keywords however they want. This gives them immense control over what they wan't to scan. + * Added some validity checks for the `keyword.conf` file as during testing, I found out that the scanner breaks if an invalid file is passed to it. + * Set up a test docker image [(rjknightmare/fo-ci-test:latest)](https://hub.docker.com/layers/rjknightmare/fo-ci-test/latest/images/sha256-ac72768b8c4383a4aca05a50cd53ae427db9645b42619ad82b53e38b3b876b38?context=repo) for testing. +Here are the test results for the same: + * GH Actions: + + ![Screenshot](/img/ci/GH_Actions_1.png) + + * Gitlab CI: + + ![Screenshot](/img/ci/Gitlab_1.png) + +* Set up qemu emulator for the Github Actions environment to provide multiple architecture support, which also reduced the time for action execution significantly. + + ![Screenshot](/img/ci/GH_Actions_2.png) + +* Studied about how we can implement differential (between two different versions) scans and explored these use cases: + * Use Case I: On `release` + * Use Case II: On `push`, `commit` or any other event trigger. + + +## Planning for next week + +* Study further on above use cases and work out a plan for the same. +* Explore the GH Actions and Gitlab API's and use the best endpoints for our use cases. +* Then start working on implementation of differential scans. \ No newline at end of file diff --git a/static/img/ci/GH_Actions_1.png b/static/img/ci/GH_Actions_1.png new file mode 100644 index 000000000..a075fa5be Binary files /dev/null and b/static/img/ci/GH_Actions_1.png differ diff --git a/static/img/ci/GH_Actions_2.png b/static/img/ci/GH_Actions_2.png new file mode 100644 index 000000000..cd76f741e Binary files /dev/null and b/static/img/ci/GH_Actions_2.png differ diff --git a/static/img/ci/Gitlab_1.png b/static/img/ci/Gitlab_1.png new file mode 100644 index 000000000..f1a7f511e Binary files /dev/null and b/static/img/ci/Gitlab_1.png differ diff --git a/static/img/ci/Spdx_tools_Version0.8.0a2.png b/static/img/ci/Spdx_tools_Version0.8.0a2.png new file mode 100644 index 000000000..fac7d3dd2 Binary files /dev/null and b/static/img/ci/Spdx_tools_Version0.8.0a2.png differ diff --git a/static/img/ci/Spdx_tools_Version_0.8.2.png b/static/img/ci/Spdx_tools_Version_0.8.2.png new file mode 100644 index 000000000..e91559494 Binary files /dev/null and b/static/img/ci/Spdx_tools_Version_0.8.2.png differ