NOISSUE - Implementation of gRPC mTLS #1848

arvindh123 · 2023-07-05T15:18:17Z

Pull request title should be MF-XXX - description or NOISSUE - description where XXX is ID of issue that this PR relate to.
Please review the CONTRIBUTING.md file for detailed contributing guidelines.

What does this do?

This PR aims to add MTLS (Mutual Transport Layer Security) support for the gRPC client in the Mainflux . It includes changes to both the gRPC server side and client side. The implementation of MTLS ensures secure communication between the client and server by requiring authentication and encryption for the transmitted data.

Which issue(s) does this PR fix/relate to?

Put here Resolves #XXX to auto-close the issue that your PR fixes (if such)
No

List any changes that modify/break current functionality

No

Have you included tests for your changes?

Did you document any new/modified functionality?

Notes

codecov · 2023-07-05T15:39:07Z

Codecov Report

Merging #1848 (ef8bfc6) into master (9dbe87f) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1848   +/-   ##
=======================================
  Coverage   67.16%   67.16%           
=======================================
  Files         118      118           
  Lines        9076     9076           
=======================================
  Hits         6096     6096           
  Misses       2351     2351           
  Partials      629      629

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

dborovcanin

We need documentation for this, probably with a PR to MF docs.

Makefile

docker/addons/bootstrap/docker-compose.grpc-mtls.yaml

docker/addons/cassandra-reader/docker-compose.grpc-mtls.yaml

docker/ssl/docker-compose.grpc-mtls.yaml

internal/clients/grpc/connect.go

rodneyosodo

I am getting these warning on

make run args "-d"


GNU Make 4.3
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Reading makefiles...
Reading makefile 'Makefile'...
Updating makefiles....
 Considering target file 'Makefile'.
  Looking for an implicit rule for 'Makefile'.
  Trying pattern rule with stem 'Makefile'.
  Trying implicit prerequisite 'Makefile.o'.
  Trying pattern rule with stem 'Makefile'.
  Trying implicit prerequisite 'Makefile.c'.
  Trying pattern rule with stem 'Makefile'.
  Trying implicit prerequisite 'Makefile.cc'.
  Trying pattern rule with stem 'Makefile'.
  Trying implicit prerequisite 'Makefile.C'.
  Trying pattern rule with stem 'Makefile'.
  Trying implicit prerequisite 'Makefile.cpp'.

Makefile

docker/addons/bootstrap/docker-compose.grpc-mtls.yaml

docker/.env

internal/clients/grpc/connect.go

internal/server/grpc/grpc.go

internal/clients/grpc/connect.go

docker/addons/mongodb-reader/docker-compose.grpc-mtls.yaml

internal/clients/grpc/connect.go

rodneyosodo

LGTM

drasko

@dborovcanin to review, please

dborovcanin

Overall, the PR looks good. However, Makefile and the whole process of running services looks a little bit to complex now. I'd like you try to:

Avoid repetition of compose files (especially since ssl/docker-compose.grpc-mtls.yml is just a repetition of 2 sections)
Avoid two compose files for add-ons, as well
Avoid special prefixes for addons
Simplify Makefile (ssl/Makefile looks fine, only the root Makefile needs some love)

author Arvindh <arvindh91@gmail.com> 1688570218 +0530 committer Arvindh <arvindh91@gmail.com> 1689754950 +0530 Rebase with master and squash commits parent 1192325 author Arvindh <arvindh91@gmail.com> 1688570218 +0530 committer Arvindh <arvindh91@gmail.com> 1689174782 +0530 add: rootCA and clientCA in grpc server Signed-off-by: Arvindh <arvindh91@gmail.com> add: rootCA and client certificate in grpc client Signed-off-by: Arvindh <arvindh91@gmail.com> add: docker-compose for grpc-mtls and make target for mtls cert generation Signed-off-by: Arvindh <arvindh91@gmail.com> fix: typo in makefile Signed-off-by: Arvindh <arvindh91@gmail.com> fix: loadCertFile function in internal/clients/grpc/connect.go Signed-off-by: Arvindh <arvindh91@gmail.com> fix: env.parser test Signed-off-by: Arvindh <arvindh91@gmail.com> remove: commented lines Signed-off-by: Arvindh <arvindh91@gmail.com> add: make commands Signed-off-by: Arvindh <arvindh91@gmail.com> update: make commands and grpc clients Signed-off-by: Arvindh <arvindh91@gmail.com> fix: typo in makefile Signed-off-by: Arvindh <arvindh91@gmail.com> fix: loadCertFile function in internal/clients/grpc/connect.go Signed-off-by: Arvindh <arvindh91@gmail.com> remove: commented lines Signed-off-by: Arvindh <arvindh91@gmail.com> update: make commands and grpc clients Signed-off-by: Arvindh <arvindh91@gmail.com> update: make commands and docker-compose Signed-off-by: Arvindh <arvindh91@gmail.com> add: end of line Signed-off-by: Arvindh <arvindh91@gmail.com> fix: typos in makefile Signed-off-by: Arvindh <arvindh91@gmail.com> add: end of line Signed-off-by: Arvindh <arvindh91@gmail.com> fix: typos in makefile Signed-off-by: Arvindh <arvindh91@gmail.com> revert: grafana port in .env Signed-off-by: Arvindh <arvindh91@gmail.com> change: loadCertFile function Signed-off-by: Arvindh <arvindh91@gmail.com> change: certficate logic Signed-off-by: Arvindh <arvindh91@gmail.com> change: env name and update in compose file Signed-off-by: Arvindh <arvindh91@gmail.com> fix: makefile Signed-off-by: Arvindh <arvindh91@gmail.com> remove: tls env var Signed-off-by: Arvindh <arvindh91@gmail.com> change: ioutil to os for ReadFile Signed-off-by: Arvindh <arvindh91@gmail.com> change loadfile Signed-off-by: Arvindh <arvindh91@gmail.com> remove: test which is no needed Signed-off-by: Arvindh <arvindh91@gmail.com> fix: docker project name Signed-off-by: Arvindh <arvindh91@gmail.com> single docker-compose file Signed-off-by: Arvindh <arvindh91@gmail.com> single docker-compose file Signed-off-by: Arvindh <arvindh91@gmail.com> single docker-compose file Signed-off-by: Arvindh <arvindh91@gmail.com>

Signed-off-by: Arvindh <arvindh91@gmail.com>

arvindh123 · 2023-08-14T16:38:48Z

@mainflux/contributors @mainflux/maintainers
To test this PR and to run gRPC with mTLS , following docs will be helpful

Running Mainflux Core Services

To run all the Mainflux core services, use the following command:

make run

This command will start all the mainflux core services.

If you want to run the Mainflux core services with gRPC mTLS enabled, use the following command:

GRPC_MTLS=true make run

If you want to run the Mainflux core services with gRPC TLS enabled, use the following command:

GRPC_TLS=true make run

This will start all the mainflux core services with secure communication using automatically generated certificates if they are not already available.

Running Specific Addons

If you only want to run specific addons, you can specify them in the command. For example, to run the timescale-reader and timescale-writer addons, use the following command:

make run_addons timescale-reader timescale-writer

This command will start the specified addons.

If you want to run specific addons with gRPC mTLS enabled, use the following command:

GRPC_MTLS=true make run_addons timescale-reader timescale-writer

If you want to run specific addons with gRPC TLS enabled, use the following command:

GRPC_TLS=true make run_addons timescale-reader timescale-writer

This will start the specified addons with secure communication using automatically generated certificates if they are not already available.

Running Addons Services

To run all the addons services, you can use the following command:

make run_addons

This command will start all the addons services.

If you want to run the addons services with gRPC mTLS enabled, you can use the following command:

GRPC_MTLS=true make run_addons

If you want to run the addons services with gRPC TLS enabled, you can use the following command:

GRPC_TLS=true make run_addons

This will start all the addons services with secure communication using automatically generated certificates if they are not already available.

dborovcanin · 2023-08-15T08:57:25Z

docker/addons/certs/docker-compose.yml

+        target: /users-grpc-client${MF_USERS_GRPC_CLIENT_KEY:+.key}
+      - type: bind
+        source: ${MF_ADDONS_CERTS_PATH_PREFIX}${MF_USERS_GRPC_SERVER_CA_CERTS:-./ssl/certs/dummy/server_ca}
+        target: /users-grpc-server-ca${MF_USERS_GRPC_SERVER_CA_CERTS:+.crt}


Add an empty line.

dborovcanin · 2023-08-15T08:57:34Z

docker/addons/influxdb-reader/docker-compose.yml

+        target: /things-grpc-client${MF_THINGS_AUTH_GRPC_CLIENT_KEY:+.key}
+      - type: bind
+        source: ${MF_ADDONS_CERTS_PATH_PREFIX}${MF_THINGS_AUTH_GRPC_SERVER_CA_CERTS:-./ssl/certs/dummy/server_ca}
+        target: /things-grpc-server-ca${MF_THINGS_AUTH_GRPC_SERVER_CA_CERTS:+.crt}


dborovcanin · 2023-08-15T08:58:01Z

docker/addons/mongodb-reader/docker-compose.yml

+        target: /things-grpc-client${MF_THINGS_AUTH_GRPC_CLIENT_KEY:+.key}
+      - type: bind
+        source: ${MF_ADDONS_CERTS_PATH_PREFIX}${MF_THINGS_AUTH_GRPC_SERVER_CA_CERTS:-./ssl/certs/dummy/server_ca}
+        target: /things-grpc-server-ca${MF_THINGS_AUTH_GRPC_SERVER_CA_CERTS:+.crt}


Same. Chack all the other files for this, as well.

docker/docker-compose.yml

Signed-off-by: Arvindh <arvindh91@gmail.com>

drasko · 2023-08-16T14:30:09Z

internal/clients/grpc/connect.go

+type security int
+
+const (
+	WITHOUT_TLS security = iota


Change this to Go idiomatic way of denoting consts.

drasko · 2023-08-16T14:31:47Z

Makefile

+BRANCH ?= $(shell git rev-parse --abbrev-ref HEAD  2>/dev/null || git describe --tags --abbrev=0  2>/dev/null )
+empty:=
+space:= $(empty) $(empty)
+DOCKER_PROJECT ?= $(subst $(space),,$(USER_REPO)_$(BRANCH))


Can you try doing it in one line, but making it readable?

Signed-off-by: Arvindh <arvindh91@gmail.com>

drasko

LGTM

arvindh123 changed the title ~~NOISSUES: Implementation of gRPC mTLS~~ NOISSUE: Implementation of gRPC mTLS Jul 5, 2023

arvindh123 force-pushed the grpc_mtls branch from c666413 to 46ed2f9 Compare July 10, 2023 15:30

arvindh123 marked this pull request as ready for review July 10, 2023 15:31

arvindh123 requested a review from a team as a code owner July 10, 2023 15:31

dborovcanin requested changes Jul 10, 2023

View reviewed changes

dborovcanin force-pushed the grpc_mtls branch 2 times, most recently from 1b32f08 to bb53a21 Compare July 12, 2023 09:29

rodneyosodo requested changes Jul 12, 2023

View reviewed changes

Makefile Outdated Show resolved Hide resolved

arvindh123 force-pushed the grpc_mtls branch 2 times, most recently from 818b45a to e70bfc1 Compare July 12, 2023 15:19

rodneyosodo requested changes Jul 12, 2023

View reviewed changes

docker/addons/bootstrap/docker-compose.grpc-mtls.yaml Outdated Show resolved Hide resolved

rodneyosodo requested changes Jul 13, 2023

View reviewed changes

dborovcanin force-pushed the grpc_mtls branch 2 times, most recently from 277ff34 to 48ebe27 Compare July 17, 2023 14:48

arvindh123 force-pushed the grpc_mtls branch from 4652ee9 to 7e45318 Compare July 19, 2023 08:27

rodneyosodo requested changes Jul 19, 2023

View reviewed changes

internal/clients/grpc/connect.go Outdated Show resolved Hide resolved

internal/clients/grpc/connect.go Show resolved Hide resolved

mteodor reviewed Jul 26, 2023

View reviewed changes

docker/addons/mongodb-reader/docker-compose.grpc-mtls.yaml Outdated Show resolved Hide resolved

mteodor reviewed Jul 26, 2023

View reviewed changes

internal/clients/grpc/connect.go Show resolved Hide resolved

arvindh123 force-pushed the grpc_mtls branch from d21fb81 to 2f9ae6e Compare July 26, 2023 11:34

arvindh123 requested review from dborovcanin and rodneyosodo July 28, 2023 06:48

arvindh123 force-pushed the grpc_mtls branch 2 times, most recently from 735ac0c to 5f0c6e9 Compare August 2, 2023 12:31

rodneyosodo approved these changes Aug 2, 2023

View reviewed changes

drasko reviewed Aug 2, 2023

View reviewed changes

arvindh123 force-pushed the grpc_mtls branch from d6faeb5 to 7defca7 Compare August 7, 2023 11:35

dborovcanin requested changes Aug 8, 2023

View reviewed changes

drasko changed the title ~~NOISSUE: Implementation of gRPC mTLS~~ NOISSUE - Implementation of gRPC mTLS Aug 8, 2023

dborovcanin force-pushed the grpc_mtls branch from c5a521f to e3882d7 Compare August 10, 2023 09:17

arvindh123 force-pushed the grpc_mtls branch from e3882d7 to f512859 Compare August 11, 2023 12:54

arvindh123 added 2 commits August 14, 2023 22:01

single docker-compose file

67faba1

Signed-off-by: Arvindh <arvindh91@gmail.com>

arvindh123 force-pushed the grpc_mtls branch from 31da47e to 67faba1 Compare August 14, 2023 16:32

dborovcanin requested changes Aug 15, 2023

View reviewed changes

arvindh123 added 7 commits August 15, 2023 14:53

fix space and new lines

e1bf33c

Signed-off-by: Arvindh <arvindh91@gmail.com>

fix makefile

873a162

Signed-off-by: Arvindh <arvindh91@gmail.com>

add: GRPC_TLS varaible and imporved logging in gRPC Client

a333234

Signed-off-by: Arvindh <arvindh91@gmail.com>

fix mtls and tls env vars

4e23b75

Signed-off-by: Arvindh <arvindh91@gmail.com>

fix mtls and tls env vars

0fac632

Signed-off-by: Arvindh <arvindh91@gmail.com>

grpc_mtls

40eb4ed

Signed-off-by: Arvindh <arvindh91@gmail.com>

fix docker-compose

8922c4b

Signed-off-by: Arvindh <arvindh91@gmail.com>

drasko requested changes Aug 16, 2023

View reviewed changes

arvindh123 added 2 commits August 16, 2023 21:56

fix makefile

31fb7b3

Signed-off-by: Arvindh <arvindh91@gmail.com>

fix const name to go idomatic

ef8bfc6

Signed-off-by: Arvindh <arvindh91@gmail.com>

drasko approved these changes Aug 16, 2023

View reviewed changes

dborovcanin approved these changes Aug 16, 2023

View reviewed changes

dborovcanin merged commit fde4350 into absmach:master Aug 16, 2023
4 checks passed

dborovcanin deleted the grpc_mtls branch August 16, 2023 17:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NOISSUE - Implementation of gRPC mTLS #1848

NOISSUE - Implementation of gRPC mTLS #1848

arvindh123 commented Jul 5, 2023 •

edited

Loading

codecov bot commented Jul 5, 2023 •

edited

Loading

dborovcanin left a comment

rodneyosodo left a comment

rodneyosodo left a comment

drasko left a comment

dborovcanin left a comment

arvindh123 commented Aug 14, 2023 •

edited

Loading

dborovcanin Aug 15, 2023

dborovcanin Aug 15, 2023

dborovcanin Aug 15, 2023

drasko Aug 16, 2023

drasko Aug 16, 2023

drasko left a comment

NOISSUE - Implementation of gRPC mTLS #1848

NOISSUE - Implementation of gRPC mTLS #1848

Conversation

arvindh123 commented Jul 5, 2023 • edited Loading

What does this do?

Which issue(s) does this PR fix/relate to?

List any changes that modify/break current functionality

Have you included tests for your changes?

Did you document any new/modified functionality?

Notes

codecov bot commented Jul 5, 2023 • edited Loading

Codecov Report

dborovcanin left a comment

Choose a reason for hiding this comment

rodneyosodo left a comment

Choose a reason for hiding this comment

rodneyosodo left a comment

Choose a reason for hiding this comment

drasko left a comment

Choose a reason for hiding this comment

dborovcanin left a comment

Choose a reason for hiding this comment

arvindh123 commented Aug 14, 2023 • edited Loading

Running Mainflux Core Services

Running Specific Addons

Running Addons Services

dborovcanin Aug 15, 2023

Choose a reason for hiding this comment

dborovcanin Aug 15, 2023

Choose a reason for hiding this comment

dborovcanin Aug 15, 2023

Choose a reason for hiding this comment

drasko Aug 16, 2023

Choose a reason for hiding this comment

drasko Aug 16, 2023

Choose a reason for hiding this comment

drasko left a comment

Choose a reason for hiding this comment

arvindh123 commented Jul 5, 2023 •

edited

Loading

codecov bot commented Jul 5, 2023 •

edited

Loading

arvindh123 commented Aug 14, 2023 •

edited

Loading