Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(report) : Coding period week 3 and week 4 reports #223

Merged
merged 1 commit into from
Jun 27, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions docs/2024/ci-scanner/updates/2024-06-20.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
---
title: Week 3
author: Rajul Jha
---
<!--
SPDX-License-Identifier: CC-BY-SA-4.0

SPDX-FileCopyrightText: 2024 Rajul Jha <rajuljha49gmail.com>
-->

# Week 3
*(June 14, 2024 - June 20, 2024)*

## Meeting 1
*(June 19, 2024)*

## Attendees
* [Rajul Jha](https://github.com/rajuljha)
* [Kaushlendra](https://github.com/Kaushl2208)
* [Shaheem Azmal](https://github.com/shaheemazmalmmd)
* Katharina


## Discussions

* Worked on adding nomos json output. But not able to access the theMatches object's licenseAndMatchPositions object. Stuck on this for nomos task.
* Bumped up spdx_tools library to latest version. Tested with both platforms (GH Actions and Gitlab).
* Discussed an issue in the github workflow, not supporting multiple architectures.
* The GH Actions Runner does not support multi-architecture images.
* We discussed potential solutions for the same including trying to build the image in the GH Action itself or utilizing an emulator like qemu.

## Work Done

* Upgraded the spdx_tools library [(#PR2762)](https://github.com/fossology/fossology/pull/2762) and did performance analysis for both versions.
* With Version `0.0.0a2`:
![Screenshot](/img/ci/Spdx_tools_Version0.8.0a2.png)

* With Version `0.8.2`:
![Screenshot](/img/ci/Spdx_tools_Version_0.8.2.png)

* Worked on providing custom `keyword.conf` file during CI pipeline. It works as follows:
* User creates a custom `keyword.conf` file following [this](https://github.com/fossology/fossology/blob/master/src/copyright/agent/keyword.conf) pattern.
* They set an environment variable called `KEYWORD_CONF_FILE_PATH` in CI providers settings and set its value to the path of the `keyword.conf` file.
* Then, when the pipeline is triggered, say on `push` to `main`, then the keyword scanner reads the custom file and scans for the keywords specified by the user.
* Keep in mind that the current keyword file is overwritten by the script, as discussed with the mentors.

## Planning for next week

* Test the keyword functionality with GH Actions and Gitlab CI and send out a PR for the same.
* Work on providing multi architecture support for GH Actions.
* Study about how to implement differential scans.
61 changes: 61 additions & 0 deletions docs/2024/ci-scanner/updates/2024-06-27.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
---
title: Week 4
author: Rajul Jha
---
<!--
SPDX-License-Identifier: CC-BY-SA-4.0

SPDX-FileCopyrightText: 2024 Rajul Jha <rajuljha49gmail.com>
-->

# Week 4
*(June 21, 2024 - June 27, 2024)*

## Meeting 1
*(June 26, 2024)*

## Attendees
* [Rajul Jha](https://github.com/rajuljha)
* [Kaushlendra](https://github.com/Kaushl2208)
* [Shaheem Azmal](https://github.com/shaheemazmalmmd)
* [Avinal Kumar](https://github.com/avinal)
* Katharina

## Discussions

* Talked about how the differential scans need to work in detail, like:
* Should they be triggered on `release` only? As it makes sense to scan between two different versions when a new release it being published.
* Or should the user be able to scan between two different versions via any sort of trigger, like on `push` or `commit` as well?
* Talked about the current implementation of custom keyword functionality, and how it can be
extrapolated similarly for different scanners also.
* We talked about fleshing out the design and implementation details first, then work on improving speed.

## Work Done

* Completed custom keyword functionality. [(#PR2769)](https://github.com/fossology/fossology/pull/2769) The details are as follows:
* User can now setup a GH Action or Gitlab CI pipeline and pass environment variables to scan for custom keywords however they want. This gives them immense control over what they wan't to scan.
* Added some validity checks for the `keyword.conf` file as during testing, I found out that the scanner breaks if an invalid file is passed to it.
* Set up a test docker image [(rjknightmare/fo-ci-test:latest)](https://hub.docker.com/layers/rjknightmare/fo-ci-test/latest/images/sha256-ac72768b8c4383a4aca05a50cd53ae427db9645b42619ad82b53e38b3b876b38?context=repo) for testing.
Here are the test results for the same:
* GH Actions:

![Screenshot](/img/ci/GH_Actions_1.png)

* Gitlab CI:

![Screenshot](/img/ci/Gitlab_1.png)

* Set up qemu emulator for the Github Actions environment to provide multiple architecture support, which also reduced the time for action execution significantly.

![Screenshot](/img/ci/GH_Actions_2.png)

* Studied about how we can implement differential (between two different versions) scans and explored these use cases:
* Use Case I: On `release`
* Use Case II: On `push`, `commit` or any other event trigger.


## Planning for next week

* Study further on above use cases and work out a plan for the same.
* Explore the GH Actions and Gitlab API's and use the best endpoints for our use cases.
* Then start working on implementation of differential scans.
Binary file added static/img/ci/GH_Actions_1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added static/img/ci/GH_Actions_2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added static/img/ci/Gitlab_1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added static/img/ci/Spdx_tools_Version0.8.0a2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added static/img/ci/Spdx_tools_Version_0.8.2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.