When running multiple environments simultaneously container hostnames sometimes resolve to wrong container

[Elamurugan-Nallathambi]

When I try run Multiple M1, M2 environments DB connectivity inside application not working but from Tableplus it works ok against both environments.

warden env up -d or warden env start shows all containers started and docker ps shows all containers from m1, m2 are running ok.

Do I have to do something different to make both environments able to connect to DB?

[davidalger]

@Elamurugan-Nallathambi Are you getting error messages of any sort? Also how do you have the database connection configured in the M1 and M2 application?

Each individual environment would normally be configured to use db as the database hostname in local.xml or env.php. Docker takes care of routing this to the correct database container using that name on the project's network. For example, you can see in the below, db is resolving to different containers in the below cases:

www-data@site1-php-fpm:11:52 PM:/var/www/html$ ping -c 2 db
PING db (172.20.0.3) 56(84) bytes of data.
64 bytes from site1_db_1.site1_default (172.20.0.3): icmp_seq=1 ttl=64 time=0.074 ms
64 bytes from site1_db_1.site1_default (172.20.0.3): icmp_seq=2 ttl=64 time=0.083 ms

--- db ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1030ms
rtt min/avg/max/mdev = 0.074/0.078/0.083/0.009 ms

www-data@site2-php-fpm:11:51 PM:/var/www/html$ ping -c 2 db
PING db (172.18.0.4) 56(84) bytes of data.
64 bytes from site2_db_1.site2_default (172.18.0.4): icmp_seq=1 ttl=64 time=0.061 ms
64 bytes from site2_db_1.site2_default (172.18.0.4): icmp_seq=2 ttl=64 time=0.101 ms

--- db ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1023ms
rtt min/avg/max/mdev = 0.061/0.081/0.101/0.020 ms

[Elamurugan-Nallathambi]

This is local.xml, env.php config settings

                <host><![CDATA[db]]></host>
                <username><![CDATA[root]]></username>
                <password><![CDATA[magento]]></password>
                <dbname><![CDATA[magento1]]></dbname>

            'host' => 'db',
            'dbname' => 'mvp_dev_dec',
            'username' => 'root',
            'password' => 'magento',

pinging from individual containers shows the db container links from M2 on both php shell prompt [It could be because I started m2 environment early]

~/Sites/warden/m1 (sprint44_changes*!)
$ warden shell
www-data@php-fpm:08:13 AM:/var/www/html$ ping -c 2 db
PING db (172.24.0.8) 56(84) bytes of data.
64 bytes from m2_db_1.warden (172.24.0.8): icmp_seq=1 ttl=64 time=0.092 ms
64 bytes from m2_db_1.warden (172.24.0.8): icmp_seq=2 ttl=64 time=0.140 ms

--- db ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1046ms
rtt min/avg/max/mdev = 0.092/0.116/0.140/0.024 ms

docker ps shows all containers running from both environments along with base warden containers.

Any suggestion on how I can debug to find the issue here? Am running this from Macbook pro with Catalina latest update.

[davidalger]

@Elamurugan-Nallathambi
It definitely looks like the resolver is routing db to the wrong container, and appears to be making the jump via the warden network which both db containers are member of. I've not seen this happen before, but it sort of makes sense. I'm gonna see if there is a way to prevent this from happening in the future, but in the meantime, try using either the network bound address: db.m1_default and db.m2_default; last bit of those addresses corresponds to the network name of each environment.

[davidalger]

What I believe is causing this issue is the fact that db (among other) containers are on the warden network in addition to the default project network docker-compose creates for them, and docker-compose is applying the db alias when attaching it to both networks:

$ docker inspect exampleproject_db_1 -f '{{json .NetworkSettings.Networks}}' | jq
{
  "exampleproject_default": {
    "IPAMConfig": null,
    "Links": null,
    "Aliases": [
      "db",
      "b410aef703c7"
    ],
    "NetworkID": "73bc2f00967fc484a514e27631dc7e724e1705230a11b12c05c41b8d4b9df3ad",
    "EndpointID": "999dc32372ad05ea8c950d7e337b4cb50ed29cc6badc5c9da1f8e0ee82fac535",
    "Gateway": "172.22.0.1",
    "IPAddress": "172.22.0.3",
    "IPPrefixLen": 16,
    "IPv6Gateway": "",
    "GlobalIPv6Address": "",
    "GlobalIPv6PrefixLen": 0,
    "MacAddress": "02:42:ac:16:00:03",
    "DriverOpts": null
  },
  "warden": {
    "IPAMConfig": null,
    "Links": null,
    "Aliases": [
      "db",
      "b410aef703c7"
    ],
    "NetworkID": "4d32e6033f450fa862c948652781cc148710cbdc7c10f1a70f28d0c95f9881a4",
    "EndpointID": "a5d48eaa0bf6b159fc6627b76dda9d0455863942226d698a236e99f12d4a5fab",
    "Gateway": "172.18.0.1",
    "IPAddress": "172.18.0.9",
    "IPPrefixLen": 16,
    "IPv6Gateway": "",
    "GlobalIPv6Address": "",
    "GlobalIPv6PrefixLen": 0,
    "MacAddress": "02:42:ac:12:00:09",
    "DriverOpts": null
  }
}

The end result (although as stated above, I have yet to personally see it…) of this is db can resolve to one of many db containers when multiple envs are started since both php-fpm and db containers are part of warden network on all projects. Some of this is speculation, but it makes sense given the scenario.

So the challenge to permanently resolving this is either getting docker-compose to not apply aliases to both networks or somehow de-coupling service containers to avoid "cross-talk"

[Elamurugan-Nallathambi]

@davidalger thanks for detailed analysis and direction on this. I could able to connect to environment based DB using db.[environment_network_name] or the second alias name from network settings.

It works ok one time while visiting both m1 , m2 frontend from browser and randomly its redirecting to other environment or giving response as File not found.. I am trying to understand where I have issue with network mapping or how it resolves to right container.

It will be really helpful to have multiple environments working since some times need to compare/debug in both environments

[davidalger]

This should be resolve in commit 258213b which will be part of upcoming 0.3.0 release. If installing via git clone, simply checkout develop to start utilizing this code, or use brew unlink warden && brew install warden --HEAD to install via brew from latest develop to take advantage of this (and maybe put it through the paces a bit) ahead of the 0.3.0 release.

Net direction on this is documented in the change log update notes: 8d3ed13. The short version however is this: environment specific containers are no longer part of the warden network avoiding the ambiguity of short DNS names such as db or nginx which could at times arbitrarily resolve to another environments service. This also as a result means that warden up and warden env up now link "peered" global services (currently Traefik and the SSH tunnel) with each environments network so that Traefik can continue to talk to varnish and nginx as well as have SSH tunnels continue to function.

[davidalger]

@Elamurugan-Nallathambi This should be fixed in Warden 0.3.0 for which I've just tagged a release. Environments no longer belong to multiple networks, eliminating the possibility for the non-determinate resolution of service DNS names. :)

[Elamurugan-Nallathambi]

@davidalger just upgraded and tested, it works with both M1 & M2. Thank you so much for keeping this in loop and posting it here.

Just note (its not an issue and am just sharing), with M1

Mutagen configuration does not exist for environment type "magento1"
Mutagen configuration does not exist for environment type "magento1"

[davidalger]

@Elamurugan-Nallathambi Thanks for the note regarding that issue; would be related to the work done on #90 and my not testing it thoroughly enough it seems, as that should not happen on normal environment operations. Easily fixed though, and just pushed up 0.3.1 release to resolve that bug.

[Elamurugan-Nallathambi]

@davidalger wow, that was very fast response and action. Upgraded again and tested, now am no longer seeing this message in M1.