MDBF-842 / MDBF-841 - Streaming NGINX logs to standard streams / evaluate .env files values

[RazvanLiviuVarzaru]

Main feature: MDBF-791 - Run Production BuildBot services in docker containers

• MQ_ROUTER_URL for PROD will be the same as DEV, on 8080
• define NGINX_CR_HOST_WG_ADDR . For PROD is hz-bbw5 on 8080. For DEV we should deploy CR on hz-dev-bbm and choose another port (8081) because 8080 is already in use by CrossBar (MQ_ROUTER_URL)
• enable proxy pass to CrossReference in NGINX bb.conf.template and define ENV variable in docker-compose.
• by removing the logs mount, and custom logging in configs, NGINX container will now stream the default access / error logs to the standard out/err streams.
• having logging driver: journald will enable us to collect the logs via journalctl using the container tag bb-nginx.

[RazvanLiviuVarzaru]

@vladbogo I see that .env file as opossed to .env.dev
does not have the BUILDMASTER_WG_IP variable which in the end is given to the Docker Latent Workers as masterFQDN to know where to connect.

Because master machine does not have proper fqdn or the worker machines do not know how to resolve it?
Is my assumption right?

@fauust which will be the WG_IP of hz-bbm2 in the end?

[vladbogo]

@RazvanLiviuVarzaru we probably forgot to add it to .env (this file was never actually used until now). There is a default value that points to the prod config for all env vars. So, if BUILDMASTER_WG_IP is not defined for prod, it will use the default value (for reference

buildbot/utils.py

Line 108 in d2bde09

fqdn = os.getenv("BUILDMASTER_WG_IP", default="100.64.100.1")

)

The idea behind this was the following: prod was up and running before dev and we changed a lot of env vars when setting up the dev env. We decided to use prod defaults because we considered with @fauust that this approach is less prone to forgetting to manually set an env var on prod (or ensuring it will be visible to all systemd services) until prod also uses docker. However, once prod uses docker, the ideas was to change all defaults to dev (or remove them) and cleanup this since having defaults to prod isn't necesssary the best idea.

So, .env and .env.dev should have the same env vars defined. Hope this answers your question.

[RazvanLiviuVarzaru]

@RazvanLiviuVarzaru we probably forgot to add it to .env (this file was never actually used until now). There is a default value that points to the prod config for all env vars. So, if BUILDMASTER_WG_IP is not defined for prod, it will use the default value (for reference

buildbot/utils.py

Line 108 in d2bde09

fqdn = os.getenv("BUILDMASTER_WG_IP", default="100.64.100.1")

)
The idea behind this was the following: prod was up and running before dev and we changed a lot of env vars when setting up the dev env. We decided to use prod defaults because we considered with @fauust that this approach is less prone to forgetting to manually set an env var on prod (or ensuring it will be visible to all systemd services) until prod also uses docker. However, once prod uses docker, the ideas was to change all defaults to dev (or remove them) and cleanup this since having defaults to prod isn't necesssary the best idea.

So, .env and .env.dev should have the same env vars defined. Hope this answers your question.

Thanks @vladbogo , I'm aware of the defaults and how BB picks up these values DEV (containers) vs. PROD. (services).
I was curious about the importance of this variable, my understanding is that Docker Latent Workers cannot connect back to the master if this is not set correctly.

This is the confirmation I want to get from you, because I suppose you configured it in the first place.

[vladbogo]

yes, it's the IP/DNS name of the master used by the latent workers to connect back once a container is started

[cvicentiu]

@RazvanLiviuVarzaru we probably forgot to add it to .env (this file was never actually used until now). There is a default value that points to the prod config for all env vars. So, if BUILDMASTER_WG_IP is not defined for prod, it will use the default value (for reference

buildbot/utils.py

Line 108 in d2bde09

fqdn = os.getenv("BUILDMASTER_WG_IP", default="100.64.100.1")

)
The idea behind this was the following: prod was up and running before dev and we changed a lot of env vars when setting up the dev env. We decided to use prod defaults because we considered with @fauust that this approach is less prone to forgetting to manually set an env var on prod (or ensuring it will be visible to all systemd services) until prod also uses docker. However, once prod uses docker, the ideas was to change all defaults to dev (or remove them) and cleanup this since having defaults to prod isn't necesssary the best idea.
So, .env and .env.dev should have the same env vars defined. Hope this answers your question.

Thanks @vladbogo , I'm aware of the defaults and how BB picks up these values DEV (containers) vs. PROD. (services). I was curious about the importance of this variable, my understanding is that Docker Latent Workers cannot connect back to the master if this is not set correctly.

This is the confirmation I want to get from you, because I suppose you configured it in the first place.

This is one of the reasons why I do not like defaults being set in the code like this. It makes it hard to track what is set and what is not. Having a "fail-fast" approach, where we just won't run unless the .env file is complete is much better. I will add that in my refactoring patches.

[RazvanLiviuVarzaru]

@RazvanLiviuVarzaru we probably forgot to add it to .env (this file was never actually used until now). There is a default value that points to the prod config for all env vars. So, if BUILDMASTER_WG_IP is not defined for prod, it will use the default value (for reference

buildbot/utils.py

Line 108 in d2bde09

fqdn = os.getenv("BUILDMASTER_WG_IP", default="100.64.100.1")

)
The idea behind this was the following: prod was up and running before dev and we changed a lot of env vars when setting up the dev env. We decided to use prod defaults because we considered with @fauust that this approach is less prone to forgetting to manually set an env var on prod (or ensuring it will be visible to all systemd services) until prod also uses docker. However, once prod uses docker, the ideas was to change all defaults to dev (or remove them) and cleanup this since having defaults to prod isn't necesssary the best idea.
So, .env and .env.dev should have the same env vars defined. Hope this answers your question.

Thanks @vladbogo , I'm aware of the defaults and how BB picks up these values DEV (containers) vs. PROD. (services). I was curious about the importance of this variable, my understanding is that Docker Latent Workers cannot connect back to the master if this is not set correctly.
This is the confirmation I want to get from you, because I suppose you configured it in the first place.

This is one of the reasons why I do not like defaults being set in the code like this. It makes it hard to track what is set and what is not. Having a "fail-fast" approach, where we just won't run unless the .env file is complete is much better. I will add that in my refactoring patches.

Agree. I've almost missed this important variable.

[fauust]

@RazvanLiviuVarzaru we probably forgot to add it to .env (this file was never actually used until now). There is a default value that points to the prod config for all env vars. So, if BUILDMASTER_WG_IP is not defined for prod, it will use the default value (for reference

buildbot/utils.py

Line 108 in d2bde09

fqdn = os.getenv("BUILDMASTER_WG_IP", default="100.64.100.1")

)
The idea behind this was the following: prod was up and running before dev and we changed a lot of env vars when setting up the dev env. We decided to use prod defaults because we considered with @fauust that this approach is less prone to forgetting to manually set an env var on prod (or ensuring it will be visible to all systemd services) until prod also uses docker. However, once prod uses docker, the ideas was to change all defaults to dev (or remove them) and cleanup this since having defaults to prod isn't necesssary the best idea.
So, .env and .env.dev should have the same env vars defined. Hope this answers your question.

Thanks @vladbogo , I'm aware of the defaults and how BB picks up these values DEV (containers) vs. PROD. (services). I was curious about the importance of this variable, my understanding is that Docker Latent Workers cannot connect back to the master if this is not set correctly.
This is the confirmation I want to get from you, because I suppose you configured it in the first place.

This is one of the reasons why I do not like defaults being set in the code like this. It makes it hard to track what is set and what is not. Having a "fail-fast" approach, where we just won't run unless the .env file is complete is much better. I will add that in my refactoring patches.

It was in my TODO and never ment to remain, but yep agree, it's worth removing them now:

❯ todo grep mdbf | grep default 
[ ] 304  16-10-2023 17:56 mdbf: remove all the default= in BB configs @wk

[RazvanLiviuVarzaru]

@RazvanLiviuVarzaru we probably forgot to add it to .env (this file was never actually used until now). There is a default value that points to the prod config for all env vars. So, if BUILDMASTER_WG_IP is not defined for prod, it will use the default value (for reference

buildbot/utils.py

Line 108 in d2bde09

fqdn = os.getenv("BUILDMASTER_WG_IP", default="100.64.100.1")

)
The idea behind this was the following: prod was up and running before dev and we changed a lot of env vars when setting up the dev env. We decided to use prod defaults because we considered with @fauust that this approach is less prone to forgetting to manually set an env var on prod (or ensuring it will be visible to all systemd services) until prod also uses docker. However, once prod uses docker, the ideas was to change all defaults to dev (or remove them) and cleanup this since having defaults to prod isn't necesssary the best idea.
So, .env and .env.dev should have the same env vars defined. Hope this answers your question.

Thanks @vladbogo , I'm aware of the defaults and how BB picks up these values DEV (containers) vs. PROD. (services). I was curious about the importance of this variable, my understanding is that Docker Latent Workers cannot connect back to the master if this is not set correctly.
This is the confirmation I want to get from you, because I suppose you configured it in the first place.

This is one of the reasons why I do not like defaults being set in the code like this. It makes it hard to track what is set and what is not. Having a "fail-fast" approach, where we just won't run unless the .env file is complete is much better. I will add that in my refactoring patches.

It was in my TODO and never ment to remain, but yep agree, it's worth removing them now:

❯ todo grep mdbf | grep default 
[ ] 304  16-10-2023 17:56 mdbf: remove all the default= in BB configs @wk

Ok, but this is something to be removed after migration.
I still have a production deployment to do on hz-bbm1 and we don't need to do any changes that can block rollback.