MF-1830 - Switch to Semaphore 2.0 #1829

WashingtonKK · 2023-06-25T21:57:13Z

What does this do?

Implement CI using Semaphore 2

Which issue(s) does this PR fix/relate to?

Resolves #1830.

List any changes that modify/break current functionality

CI runs tests only on modified modules

Have you included tests for your changes?

N/A

Did you document any new/modified functionality?

Yes

Notes

N/A

codecov · 2023-06-25T22:11:13Z

Codecov Report

Merging #1829 (cbb9b10) into master (106b710) will not change coverage.
Report is 1 commits behind head on master.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1829   +/-   ##
=======================================
  Coverage   64.73%   64.73%           
=======================================
  Files         118      118           
  Lines        9702     9702           
=======================================
  Hits         6281     6281           
  Misses       2753     2753           
  Partials      668      668

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

dborovcanin

@WashingtonKK Please check examples here: https://semaphoreci.com/blog/revving-up-continuous-integration-with-parallel-testing#h-monorepos.

dborovcanin · 2023-06-26T08:29:08Z

.semaphore/semaphore.yml

+  - name: Setup
+    task:
+      env_vars:
+        - name: GO_VERSION
+          value: 1.16.7
+      jobs:
+        - name: Variables
+          commands:
+            - NPROC=$(nproc)
+            - GO_VERSION=1.19.4
+            - PROTOC_VERSION=21.12
+            - PROTOC_GEN_VERSION=v1.28.1
+            - PROTOC_GRPC_VERSION=v1.2.0
+        - name: Update Go
+          commands:
+            - |
+              function version_gt() {
+                test "$(printf ''%s\n'' "$@" | sort -V | head -n 1)" != "$1"
+              }
+            - |
+              CURRENT_GO_VERSION=$(go version | sed 's/[^0-9.]*\([0-9.]*\).*/\1/')
+            - |
+              if version_gt $GO_VERSION $CURRENT_GO_VERSION; then
+                echo "Updating go version from $CURRENT_GO_VERSION to $GO_VERSION ..."
+                sudo rm -rf /usr/bin/go
+                sudo rm -rf /usr/local/go
+                sudo rm -rf /usr/local/bin/go
+                sudo rm -rf /usr/local/golang
+                sudo rm -rf $GOROOT $GOPATH $GOBIN
+                wget https://go.dev/dl/go$GO_VERSION.linux-amd64.tar.gz
+                sudo tar -C /usr/local -xzf go$GO_VERSION.linux-amd64.tar.gz
+                export GOROOT=/usr/local/go
+                export PATH=$PATH:/usr/local/go/bin
+              fi
+            - export GOBIN=$HOME/go/bin
+            - 'export PATH=$PATH:$GOBIN'


We probably do not need to update Go anymore. This was only used as a way of enabling the latest version of Go, but backward compatibility guarantees that it should be just fine if the CI Go version is 1 or 2 (but not more) older than the version in go.mod.

dborovcanin · 2023-06-26T08:37:51Z

.semaphore/semaphore.yml

+            - checkout
+            - echo "Running modified tests...."
+            - echo "" > coverage.txt
+            - |
+              # Get the list of modified Go files
+              MODIFIED_FILES=$(git diff --name-only --diff-filter=AM | grep '\.go$')
+
+              # Extract the unique Go module names from modified files
+              MODIFIED_MODULES=$(go list -e -f '{{.Module.Path}}' $(dirname $MODIFIED_FILES) | sort -u)
+
+              # Run tests for modified modules
+              for module in $MODIFIED_MODULES; do
+                go test -mod=vendor -v -race -tags test -coverprofile=profile.out -covermode=atomic "$module/..."
+                if [ -f profile.out ]; then
+                  cat profile.out >> coverage.txt
+                  rm profile.out
+                fi
+              done     


We have only one module. Also, detecting changes for testing is more complex than checking changed files, since it depends on which files are changed: if we change access control, for example, we need to test messaging, as well.

rodneyosodo

Add test for building binaries

rodneyosodo · 2023-07-04T07:33:38Z

.semaphore/semaphore.yml

+            - go test -v -coverprofile=coverage.out ./...
+            - go tool cover -html=coverage.out -o coverage.html 
+          name: Test Bootstrap
+  - name: Test certs


add tests for cli, coap, consumers, whole http sub package not only /api, and others

Stay consistent and use capital letter for the name of each service (Certs, Things, ...)

rodneyosodo · 2023-07-04T07:35:47Z

.semaphore/semaphore.yml

+            - go test -v -coverprofile=coverage.out ./...
+            - go tool cover -html=coverage.out -o coverage.html 
+          name: Logger test
+  - name: Test all


This doesn't seem to be correct. I'd suggest to test everything probably during merge to main or after a reviewer has commented LGTM

There is probably no need for this test at all, even in cases we merge to master. However, we need to be sure we tun tests when it is relevant (e.g. when proto files change, we need to run almost all the tests).

rodneyosodo

Add auto-cancel on new push

rodneyosodo · 2023-07-04T18:45:57Z

.semaphore/semaphore.yml

-            changeset:
-              comment:
-                matches: "LGTM"
+          when: "branch" = "master" OR "comment" = "LGTM"


It doesn't work for a comment - https://docs.semaphoreci.com/reference/conditions-reference/

drasko

Capitalize service names

rodneyosodo · 2023-07-06T21:05:11Z

.semaphore/semaphore.yml

@@ -0,0 +1,246 @@
+version: v1.0
+name: CI Pipeline


The previous CI run took 23.20m yet the current CI runs on the master take around 24m. Shouldn't we be getting faster times?

Only 3 processes are being run at any instance on the free tier which slows down the whole process. If we had all parallel processes running at the same time concurrently we would have faster times of ~19 min.

SammyOina

remove changes not required in this PR

SammyOina · 2023-07-10T13:31:41Z

cmd/opcua/main.go

@@ -126,6 +126,7 @@ func main() {
 	if err := env.Parse(&httpServerConfig, env.Options{Prefix: envPrefixHttp, AltPrefix: envPrefix}); err != nil {
 		logger.Fatal(fmt.Sprintf("failed to load %s HTTP server configuration : %s", svcName, err))
 	}
+


this change is unnecessary

SammyOina · 2023-07-10T13:31:51Z

docker/.env

@@ -61,6 +61,7 @@ MF_USERS_RESET_PWD_TEMPLATE=users.tmpl
 MF_USERS_PASS_REGEX=^.{8,}$$
 MF_USERS_INSTANCE_ID=

+


SammyOina · 2023-07-10T13:32:00Z

http/api/transport.go

@@ -6,6 +6,7 @@ package api
 import (
 	"context"
 	"encoding/json"
+


probably fixed by rebasing

The change was associated with a merge from master into the branch to keep it up to date.

rodneyosodo · 2023-07-11T16:28:03Z

.semaphore/semaphore.yml

@@ -0,0 +1,430 @@
+version: v1.0


Remove previous CI

dborovcanin

Please add checks for proto file changes, and also connect changes in Things auth with protocol adapters (i.e. adapters must be tested when auth is changed, but artifacts do not need to be published since the code actually did not change). Since all Lint and Test XYZ have the same pattern, check if we can use some kind of matrix and template hee.

dborovcanin · 2023-07-11T21:02:53Z

.semaphore/semaphore.yml

+            - go test -v -coverprofile=coverage.out ./...
+            - go tool cover -html=coverage.out -o coverage.html 
+          name: Logger test
+  - name: Test all


There is probably no need for this test at all, even in cases we merge to master. However, we need to be sure we tun tests when it is relevant (e.g. when proto files change, we need to run almost all the tests).

dborovcanin · 2023-07-11T21:03:52Z

.semaphore/semaphore.yml

+            - docker push mainflux.ws
+          name: Push Ws
+
+  - name: Lint and Test CoAp


Fix typo: CoAP. Check for other typos.

dborovcanin · 2023-07-11T21:04:47Z

.semaphore/semaphore.yml

+    task:
+      jobs:
+        - commands:
+            - make things


We need to build a Docker image, not a binary.

rodneyosodo · 2023-07-12T11:09:39Z

.semaphore/semaphore.yml

+      jobs:
+        - name: Compile For RabbitMQ
+          commands:
+            - MF_BROKER_TYPE=rabbitmq make mqtt


Add empty line

@WashingtonKK this is still not addressed

dborovcanin · 2023-07-13T11:07:39Z

.semaphore/semaphore.yml

+              }
+            - setup_protoc
+            - |
+              check_files() {


dborovcanin · 2023-07-13T11:07:43Z

.semaphore/semaphore.yml

+        - name: Check Generated Protocol Buffer Files
+          commands:
+            - |
+              setup_protoc() {


No need to use functions and call them - you can simply exec this line-by-line.

dborovcanin · 2023-07-13T11:10:52Z

.semaphore/semaphore.yml

+  - name: Things Auth Changes
+    dependencies: []
+    run:
+      when: "change_in('things/policies/auth.proto')"


When changes are in Protobuf files, we need all the tests (i.e. add things/policies/auth.proto to a change_in list for all adapters.

Instead of when: "change_in('things/policies/auth.proto')" we need to check for changes in implementation logic of things - something like when: "change_in(['things/policies/postgres', 'things/policies/service.go'])". Also, no need to run coverage in this block since we did not change the actual code of adapters, so no coverage changes will be present.

dborovcanin · 2023-07-13T12:43:09Z

.semaphore/semaphore.yml

+          commands:
+            - cd things
+            - go test ./... --race


No need for this, it will be triggered by Thing changes.

dborovcanin · 2023-07-13T12:44:31Z

.semaphore/semaphore.yml

+  - name: Push Docker Images
+    dependencies:
+      - Setup
+    run:
+      when: branch = 'master'
+    task:
+      jobs:
+        - name: Push Docker Images
+          commands:
+            - make -j$NPROC latest


No need for this, either. Move publishing to per-service in Build blocks.

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

.semaphore/semaphore.yml

dborovcanin · 2023-07-14T09:28:24Z

.semaphore/semaphore.yml

+        - name: DOCKER_USERNAME                    
+  - name: Lint and Test Things
+    run:
+      when: "change_in(['things', 'cmd/things', 'users/policies/auth.proto'])"


Let's test if changes are in /users/policies/postgres/policies.go and /users/policies/api/grpc/client.go, not user/policies/auth.proto.

.semaphore/semaphore.yml

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

dborovcanin

@WashingtonKK You are missing these:

- echo $DOCKER_TOKEN | docker login --username "$DOCKER_USERNAME" --password-stdin
- docker push mainflux/http:latest

Ofc, this is for HTTP and we need for others as well.

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

drasko

LGTM

* Add pipeline Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Add pipeline Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Add Docker secret Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Remove function Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Remove function Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Modify block code Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Remove coverage on test block Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Fixes Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Update badge Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Remove repeated tests' Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Fix typo Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Fix typo Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Modify docker secrets Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Push docker images Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> --------- Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> Co-authored-by: Drasko DRASKOVIC <drasko.draskovic@gmail.com>

* Add pipeline Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Add pipeline Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Add Docker secret Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Remove function Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Remove function Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Modify block code Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Remove coverage on test block Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Fixes Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Update badge Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Remove repeated tests' Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Fix typo Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Fix typo Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Modify docker secrets Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Push docker images Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> --------- Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> Co-authored-by: Drasko DRASKOVIC <drasko.draskovic@gmail.com> Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* Add pipeline Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Add pipeline Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Add Docker secret Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Remove function Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Remove function Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Modify block code Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Remove coverage on test block Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Fixes Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Update badge Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Remove repeated tests' Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Fix typo Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Fix typo Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Modify docker secrets Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Push docker images Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> --------- Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> Co-authored-by: Drasko DRASKOVIC <drasko.draskovic@gmail.com>

WashingtonKK requested a review from a team as a code owner June 25, 2023 21:57

WashingtonKK force-pushed the setup-semaphore branch from e8cbf9b to 9a01f6d Compare June 25, 2023 21:58

dborovcanin changed the title ~~MF-251 - Change CI~~ MF-1830 - Switch to Semqphore 2.0 Jun 26, 2023

dborovcanin requested changes Jun 26, 2023

View reviewed changes

dborovcanin changed the title ~~MF-1830 - Switch to Semqphore 2.0~~ MF-1830 - Switch to Semaphore 2.0 Jun 26, 2023

WashingtonKK force-pushed the setup-semaphore branch 2 times, most recently from 1684eec to f77cf5e Compare June 27, 2023 09:48

rodneyosodo requested changes Jul 4, 2023

View reviewed changes

drasko requested changes Jul 6, 2023

View reviewed changes

rodneyosodo requested changes Jul 6, 2023

View reviewed changes

dborovcanin force-pushed the setup-semaphore branch from 8bbcfb0 to c65e9e6 Compare July 6, 2023 21:32

WashingtonKK force-pushed the setup-semaphore branch from c65e9e6 to e2a490e Compare July 6, 2023 21:59

dborovcanin force-pushed the setup-semaphore branch from 3c6191a to 77d530e Compare July 7, 2023 14:18

WashingtonKK force-pushed the setup-semaphore branch from 77d530e to e7b00d3 Compare July 7, 2023 14:53

SammyOina requested changes Jul 10, 2023

View reviewed changes

WashingtonKK force-pushed the setup-semaphore branch from e394348 to 6eaf2b2 Compare July 11, 2023 15:38

rodneyosodo requested changes Jul 11, 2023

View reviewed changes

.semaphore/semaphore.yml Outdated

@@ -0,0 +1,430 @@

version: v1.0

Copy link

Member

rodneyosodo Jul 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove previous CI

WashingtonKK reacted with thumbs up emoji

dborovcanin requested changes Jul 11, 2023

View reviewed changes

WashingtonKK force-pushed the setup-semaphore branch from 85837df to f3e4196 Compare July 12, 2023 11:04

rodneyosodo reviewed Jul 12, 2023

View reviewed changes

WashingtonKK force-pushed the setup-semaphore branch from f3e4196 to c8754a1 Compare July 12, 2023 12:20

dborovcanin force-pushed the setup-semaphore branch from c8754a1 to 60f0c7d Compare July 12, 2023 12:20

WashingtonKK force-pushed the setup-semaphore branch 5 times, most recently from bf41b68 to 1ffbae6 Compare July 12, 2023 13:58

dborovcanin requested changes Jul 13, 2023

View reviewed changes

WashingtonKK added 12 commits July 14, 2023 11:14

Add pipeline

61bd6ab

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

Add pipeline

542b9e0

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

Add Docker secret

7af09c6

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

Remove function

151c57f

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

Remove function

a99fee7

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

Modify block code

fb28b26

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

Remove coverage on test block

dac62ba

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

Fixes

95968cd

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

Update badge

103d06d

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

Remove repeated tests'

9b203dd

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

Fix typo

c4bc732

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

Fix typo

005bf72

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

dborovcanin force-pushed the setup-semaphore branch from d2b4160 to c4bc732 Compare July 14, 2023 09:14

dborovcanin requested changes Jul 14, 2023

View reviewed changes

Modify docker secrets

d377752

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

dborovcanin requested changes Jul 14, 2023

View reviewed changes

Push docker images

5c1c37e

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

dborovcanin approved these changes Jul 17, 2023

View reviewed changes

arvindh123 approved these changes Jul 26, 2023

View reviewed changes

Merge branch 'master' into setup-semaphore

cbb9b10

drasko approved these changes Jul 26, 2023

View reviewed changes

drasko merged commit 2b78902 into absmach:master Jul 26, 2023
3 checks passed

WashingtonKK deleted the setup-semaphore branch March 12, 2024 07:37

		@@ -61,6 +61,7 @@ MF_USERS_RESET_PWD_TEMPLATE=users.tmpl
		MF_USERS_PASS_REGEX=^.{8,}$$
		MF_USERS_INSTANCE_ID=

MF-1830 - Switch to Semaphore 2.0 #1829

MF-1830 - Switch to Semaphore 2.0 #1829

Conversation

WashingtonKK commented Jun 25, 2023 • edited by dborovcanin Loading

What does this do?

Which issue(s) does this PR fix/relate to?

List any changes that modify/break current functionality

Have you included tests for your changes?

Did you document any new/modified functionality?

Notes

codecov bot commented Jun 25, 2023 • edited Loading

Codecov Report

dborovcanin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rodneyosodo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rodneyosodo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drasko left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SammyOina left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dborovcanin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dborovcanin left a comment

Choose a reason for hiding this comment

drasko left a comment

Choose a reason for hiding this comment

WashingtonKK commented Jun 25, 2023 •

edited by dborovcanin

Loading

codecov bot commented Jun 25, 2023 •

edited

Loading