Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NOISSUE - Sync Env Veriables With Docker Deployment #1841

Merged
merged 11 commits into from
Jul 31, 2023
Merged

NOISSUE - Sync Env Veriables With Docker Deployment #1841

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
c112dc4
Initial Commit: Sync Env Veriables With Docker Deployment
rodneyosodo Jul 3, 2023
93dbfc4
Sync Env Vars With Master
rodneyosodo Jul 10, 2023
1af4b7a
Remove Altprefix
rodneyosodo Jul 20, 2023
59d6fc9
Rename HttpPort to HTTPPort
rodneyosodo Jul 24, 2023
7e586a9
Fix envPrefixDB After Rebase
rodneyosodo Jul 28, 2023
57a6292
Remove Server Parse
rodneyosodo Jul 28, 2023
7707a72
Add Provision For TLS on CoAP
rodneyosodo Jul 28, 2023
ab16831
Fix Exit After Defer
rodneyosodo Jul 28, 2023
3596e78
Remove Unused Function
rodneyosodo Jul 28, 2023
610fce2
Document Undocumentated Env Variables
rodneyosodo Jul 31, 2023
9908c67
Merge branch 'master' into NOISSUE-env
drasko Jul 31, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
109 changes: 57 additions & 52 deletions bootstrap/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

New devices need to be configured properly and connected to the Mainflux. Bootstrap service is used in order to accomplish that. This service provides the following features:

1) Creating new Mainflux Things
2) Providing basic configuration for the newly created Things
3) Enabling/disabling Things
1. Creating new Mainflux Things
2. Providing basic configuration for the newly created Things
3. Enabling/disabling Things

Pre-provisioning a new Thing is as simple as sending Configuration data to the Bootstrap service. Once the Thing is online, it sends a request for initial config to Bootstrap service. Bootstrap service provides an API for enabling and disabling Things. Only enabled Things can exchange messages over Mainflux. Bootstrapping does not implicitly enable Things, it has to be done manually.

Expand All @@ -14,18 +14,18 @@ In order to bootstrap successfully, the Thing needs to send bootstrapping reques

Thing Configuration consists of two logical parts: the custom configuration that can be interpreted by the Thing itself and Mainflux-related configuration. Mainflux config contains:

1) corresponding Mainflux Thing ID
2) corresponding Mainflux Thing key
3) list of the Mainflux channels the Thing is connected to
1. corresponding Mainflux Thing ID
2. corresponding Mainflux Thing key
3. list of the Mainflux channels the Thing is connected to

>Note: list of channels contains IDs of the Mainflux channels. These channels are _pre-provisioned_ on the Mainflux side and, unlike corresponding Mainflux Thing, Bootstrap service is not able to create Mainflux Channels.
> Note: list of channels contains IDs of the Mainflux channels. These channels are _pre-provisioned_ on the Mainflux side and, unlike corresponding Mainflux Thing, Bootstrap service is not able to create Mainflux Channels.

Enabling and disabling Thing (adding Thing to/from whitelist) is as simple as connecting corresponding Mainflux Thing to the given list of Channels. Configuration keeps _state_ of the Thing:

| State | What it means |
|----------|--------------------------------------------------------|
| Inactive | Thing is created, but isn't enabled |
| Active | Thing is able to communicate using Mainflux |
| State | What it means |
| -------- | ------------------------------------------- |
| Inactive | Thing is created, but isn't enabled |
| Active | Thing is able to communicate using Mainflux |

Switching between states `Active` and `Inactive` enables and disables Thing, respectively.

Expand All @@ -35,40 +35,39 @@ Thing configuration also contains the so-called `external ID` and `external key`

The service is configured using the environment variables presented in the following table. Note that any unset variables will be replaced with their default values.

| Variable | Description | Default |
|-------------------------------|-------------------------------------------------------------------------|----------------------------------|
| MF_BOOTSTRAP_LOG_LEVEL | Log level for Bootstrap (debug, info, warn, error) | info |
| MF_BOOTSTRAP_DB_HOST | Database host address | localhost |
| MF_BOOTSTRAP_DB_PORT | Database host port | 5432 |
| MF_BOOTSTRAP_DB_USER | Database user | mainflux |
| MF_BOOTSTRAP_DB_PASS | Database password | mainflux |
| MF_BOOTSTRAP_DB | Name of the database used by the service | bootstrap |
| MF_BOOTSTRAP_DB_SSL_MODE | Database connection SSL mode (disable, require, verify-ca, verify-full) | disable |
| MF_BOOTSTRAP_DB_SSL_CERT | Path to the PEM encoded certificate file | |
| MF_BOOTSTRAP_DB_SSL_KEY | Path to the PEM encoded key file | |
| MF_BOOTSTRAP_DB_SSL_ROOT_CERT | Path to the PEM encoded root certificate file | |
| MF_BOOTSTRAP_ENCRYPT_KEY | Secret key for secure bootstrapping encryption | 12345678910111213141516171819202 |
| MF_BOOTSTRAP_CLIENT_TLS | Flag that indicates if TLS should be turned on | false |
| MF_BOOTSTRAP_CA_CERTS | Path to trusted CAs in PEM format | |
| MF_BOOTSTRAP_PORT | Bootstrap service HTTP port | 9013 |
| MF_BOOTSTRAP_SERVER_CERT | Path to server certificate in pem format | |
| MF_BOOTSTRAP_SERVER_KEY | Path to server key in pem format | |
| MF_THINGS_URL | Base url for Mainflux Things | http://localhost:9000 |
| MF_THINGS_ES_URL | Things service event source URL | localhost:6379 |
| MF_THINGS_ES_PASS | Things service event source password | |
| MF_THINGS_ES_DB | Things service event source database | 0 |
| MF_BOOTSTRAP_ES_URL | Bootstrap service event source URL | localhost:6379 |
| MF_BOOTSTRAP_ES_PASS | Bootstrap service event source password | |
| MF_BOOTSTRAP_ES_DB | Bootstrap service event source database | 0 |
| MF_BOOTSTRAP_EVENT_CONSUMER | Bootstrap service event source consumer name | bootstrap |
| MF_JAEGER_URL | Jaeger server URL | localhost:6831 |
| MF_AUTH_GRPC_URL | Users service gRPC URL | localhost:7001 |
| MF_AUTH_GRPC_TIMEOUT | Users service gRPC request timeout in seconds | 1s |
| MF_SEND_TELEMETRY | Send telemetry to mainflux call home server | true |
| Variable | Description | Default |
| ----------------------------- | ----------------------------------------------------------------------- | -------------------------------------------------- |
| MF_BOOTSTRAP_LOG_LEVEL | Log level for Bootstrap (debug, info, warn, error) | info |
| MF_BOOTSTRAP_DB_HOST | Database host address | localhost |
| MF_BOOTSTRAP_DB_PORT | Database host port | 5432 |
| MF_BOOTSTRAP_DB_USER | Database user | mainflux |
| MF_BOOTSTRAP_DB_PASS | Database password | mainflux |
| MF_BOOTSTRAP_DB_NAME | Name of the database used by the service | bootstrap |
| MF_BOOTSTRAP_DB_SSL_MODE | Database connection SSL mode (disable, require, verify-ca, verify-full) | disable |
| MF_BOOTSTRAP_DB_SSL_CERT | Path to the PEM encoded certificate file | |
| MF_BOOTSTRAP_DB_SSL_KEY | Path to the PEM encoded key file | |
| MF_BOOTSTRAP_DB_SSL_ROOT_CERT | Path to the PEM encoded root certificate file | |
| MF_BOOTSTRAP_ENCRYPT_KEY | Secret key for secure bootstrapping encryption | v7aT0HGxJxt2gULzr3RHwf4WIf6DusPphG5Ftm2bNCWD8mTpyr |
| MF_BOOTSTRAP_HTTP_HOST | Bootstrap service HTTP host | |
| MF_BOOTSTRAP_HTTP_PORT | Bootstrap service HTTP port | 9013 |
| MF_BOOTSTRAP_HTTP_SERVER_CERT | Path to server certificate in pem format | |
| MF_BOOTSTRAP_HTTP_SERVER_KEY | Path to server key in pem format | |
| MF_BOOTSTRAP_EVENT_CONSUMER | Bootstrap service event source consumer name | bootstrap |
| MF_BOOTSTRAP_ES_URL | Bootstrap service event source URL | localhost:6379 |
| MF_BOOTSTRAP_ES_PASS | Bootstrap service event source password | |
| MF_BOOTSTRAP_ES_DB | Bootstrap service event source database | 0 |
| MF_AUTH_GRPC_URL | Users service gRPC URL | localhost:7001 |
| MF_AUTH_GRPC_TIMEOUT | Users service gRPC request timeout in seconds | 1s |
| MF_AUTH_GRPC_CLIENT_TLS | Enable TLS for gRPC client | false |
| MF_AUTH_GRPC_CA_CERTS | CA certificates for gRPC client | |
| MF_THINGS_URL | Base url for Mainflux Things | http://localhost:9000 |
| MF_JAEGER_URL | Jaeger server URL | http://jaeger:14268/api/traces |
| MF_SEND_TELEMETRY | Send telemetry to mainflux call home server | true |
| MF_BOOTSTRAP_INSTANCE_ID | Bootstrap service instance ID | |

## Deployment

The service itself is distributed as Docker container. Check the [`boostrap`](https://github.com/mainflux/mainflux/blob/master/docker/addons/bootstrap/docker-compose.yml#L32-L56) service section in
The service itself is distributed as Docker container. Check the [`boostrap`](https://github.com/mainflux/mainflux/blob/master/docker/addons/bootstrap/docker-compose.yml#L32-L56) service section in
docker-compose to see how service is deployed.

To start the service outside of the container, execute the following shell script:
Expand All @@ -87,26 +86,32 @@ make install

# set the environment variables and run the service
MF_BOOTSTRAP_LOG_LEVEL=[Bootstrap log level] \
MF_BOOTSTRAP_ENCRYPT_KEY=[Hex-encoded encryption key used for secure bootstrap] \
MF_BOOTSTRAP_EVENT_CONSUMER=[Bootstrap service event source consumer name] \
MF_BOOTSTRAP_ES_URL=[Bootstrap service event source URL] \
MF_BOOTSTRAP_ES_PASS=[Bootstrap service event source password] \
MF_BOOTSTRAP_ES_DB=[Bootstrap service event source database] \
MF_BOOTSTRAP_HTTP_HOST=[Bootstrap service HTTP host] \
MF_BOOTSTRAP_HTTP_PORT=[Bootstrap service HTTP port] \
MF_BOOTSTRAP_HTTP_SERVER_CERT=[Path to HTTP server certificate in pem format] \
MF_BOOTSTRAP_HTTP_SERVER_KEY=[Path to HTTP server key in pem format] \
MF_BOOTSTRAP_DB_HOST=[Database host address] \
MF_BOOTSTRAP_DB_PORT=[Database host port] \
MF_BOOTSTRAP_DB_USER=[Database user] \
MF_BOOTSTRAP_DB_PASS=[Database password] \
MF_BOOTSTRAP_DB=[Name of the database used by the service] \
MF_BOOTSTRAP_DB_NAME=[Name of the database used by the service] \
MF_BOOTSTRAP_DB_SSL_MODE=[SSL mode to connect to the database with] \
MF_BOOTSTRAP_DB_SSL_CERT=[Path to the PEM encoded certificate file] \
MF_BOOTSTRAP_DB_SSL_KEY=[Path to the PEM encoded key file] \
MF_BOOTSTRAP_DB_SSL_ROOT_CERT=[Path to the PEM encoded root certificate file] \
MF_BOOTSTRAP_ENCRYPT_KEY=[Hex-encoded encryption key used for secure bootstrap] \
MF_BOOTSTRAP_CLIENT_TLS=[Boolean value to enable/disable client TLS] \
MF_BOOTSTRAP_CA_CERTS=[Path to trusted CAs in PEM format] \
MF_BOOTSTRAP_PORT=[Service HTTP port] \
MF_BOOTSTRAP_SERVER_CERT=[Path to server certificate] \
MF_BOOTSTRAP_SERVER_KEY=[Path to server key] \
MF_SDK_BASE_URL=[Base SDK URL for the Mainflux services] \
MF_SDK_THINGS_PREFIX=[SDK prefix for Things service] \
MF_JAEGER_URL=[Jaeger server URL] \
MF_AUTH_GRPC_URL=[Users service gRPC URL] \
MF_AUTH_GRPC_TIMEOUT=[Users service gRPC request timeout in seconds] \
MF_AUTH_GRPC_CLIENT_TLS=[Boolean value to enable/disable client TLS] \
MF_AUTH_GRPC_CA_CERT=[Path to trusted CAs in PEM format] \
MF_THINGS_URL=[Base url for Mainflux Things] \
MF_JAEGER_URL=[Jaeger server URL] \
MF_SEND_TELEMETRY=[Send telemetry to mainflux call home server] \
MF_BOOTSTRAP_INSTANCE_ID=[Bootstrap instance ID] \
$GOBIN/mainflux-bootstrap
```

Expand Down
4 changes: 4 additions & 0 deletions certs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,17 @@
# Certs Service

Issues certificates for things. `Certs` service can create certificates to be used when `Mainflux` is deployed to support mTLS.
Certificate service can create certificates in two modes:

1. Development mode - to be used when no PKI is deployed, this works similar to the [make thing_cert](../docker/ssl/Makefile)
2. PKI mode - certificates issued by PKI, when you deploy `Vault` as PKI certificate management `cert` service will proxy requests to `Vault` previously checking access rights and saving info on successfully created certificate.

## Development mode

If `MF_CERTS_VAULT_HOST` is empty than Development mode is on.

To issue a certificate:

```bash

TOK=`curl -s --insecure -S -X POST http://localhost/tokens -H 'Content-Type: application/json' -d '{"email":"edge@email.com","password":"12345678"}' | jq -r '.token'`
Expand Down
50 changes: 26 additions & 24 deletions cmd/bootstrap/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,12 @@ import (
"log"
"os"

"github.com/go-redis/redis/v8"
"github.com/jmoiron/sqlx"
chclient "github.com/mainflux/callhome/pkg/client"
"github.com/mainflux/mainflux"
"go.opentelemetry.io/otel/trace"

"github.com/mainflux/mainflux/bootstrap"
"github.com/mainflux/mainflux/bootstrap/api"
bootstrapPg "github.com/mainflux/mainflux/bootstrap/postgres"
rediscons "github.com/mainflux/mainflux/bootstrap/redis/consumer"
redisprod "github.com/mainflux/mainflux/bootstrap/redis/producer"
Expand All @@ -31,21 +33,17 @@ import (
mfsdk "github.com/mainflux/mainflux/pkg/sdk/go"
"github.com/mainflux/mainflux/pkg/uuid"
"github.com/mainflux/mainflux/users/policies"
"go.opentelemetry.io/otel/trace"
"golang.org/x/sync/errgroup"

"github.com/go-redis/redis/v8"
"github.com/jmoiron/sqlx"
"github.com/mainflux/mainflux/bootstrap"
"github.com/mainflux/mainflux/bootstrap/api"
)

const (
svcName = "bootstrap"
envPrefix = "MF_BOOTSTRAP_"
envPrefixDB = "MF_BOOTSTRAP_DB_"
envPrefixES = "MF_BOOTSTRAP_ES_"
envPrefixHttp = "MF_BOOTSTRAP_HTTP_"
envPrefixHTTP = "MF_BOOTSTRAP_HTTP_"
defDB = "bootstrap"
defSvcHttpPort = "9013"
defSvcHTTPPort = "9013"
)

type config struct {
Expand All @@ -72,23 +70,27 @@ func main() {
log.Fatalf("failed to init logger: %s", err)
}

var exitCode int
defer mflog.ExitWithError(&exitCode)

instanceID := cfg.InstanceID
if instanceID == "" {
instanceID, err = uuid.New().ID()
if err != nil {
log.Fatalf("Failed to generate instanceID: %s", err)
logger.Error(fmt.Sprintf("Failed to generate instanceID: %s", err))
exitCode = 1
return
}
}

// Create new postgres client
dbConfig := pgClient.Config{Name: defDB}

db, err := pgClient.SetupWithConfig(envPrefix, *bootstrapPg.Migration(), dbConfig)
db, err := pgClient.SetupWithConfig(envPrefixDB, *bootstrapPg.Migration(), dbConfig)
if err != nil {
logger.Fatal(err.Error())
logger.Error(err.Error())
exitCode = 1
return
}
var exitCode int
defer mflog.ExitWithError(&exitCode)
defer db.Close()

// Create new redis client for bootstrap event store
Expand All @@ -101,7 +103,7 @@ func main() {
defer esClient.Close()

// Create new auth grpc client api
auth, authHandler, err := authClient.Setup(envPrefix, svcName)
auth, authHandler, err := authClient.Setup(svcName)
if err != nil {
logger.Error(err.Error())
exitCode = 1
Expand All @@ -112,10 +114,12 @@ func main() {

tp, err := jaegerClient.NewProvider(svcName, cfg.JaegerURL, instanceID)
if err != nil {
logger.Fatal(fmt.Sprintf("failed to init Jaeger: %s", err))
logger.Error(fmt.Sprintf("failed to init Jaeger: %s", err))
exitCode = 1
return
}
defer func() {
if err := tp.Shutdown(context.Background()); err != nil {
if err := tp.Shutdown(ctx); err != nil {
logger.Error(fmt.Sprintf("error shutting down tracer provider: %v", err))
}
}()
Expand All @@ -125,8 +129,8 @@ func main() {
svc := newService(auth, db, tracer, logger, esClient, cfg)

// Create an new HTTP server
httpServerConfig := server.Config{Port: defSvcHttpPort}
if err := env.Parse(&httpServerConfig, env.Options{Prefix: envPrefixHttp, AltPrefix: envPrefix}); err != nil {
httpServerConfig := server.Config{Port: defSvcHTTPPort}
if err := env.Parse(&httpServerConfig, env.Options{Prefix: envPrefixHTTP}); err != nil {
logger.Error(fmt.Sprintf("failed to load %s HTTP server configuration : %s", svcName, err))
exitCode = 1
return
Expand All @@ -150,9 +154,7 @@ func main() {
// Subscribe to things event store
thingsESClient, err := redisClient.Setup(envPrefixES)
if err != nil {
logger.Error(err.Error())
exitCode = 1
return
logger.Fatal(err.Error())
}
defer thingsESClient.Close()

Expand Down
Loading