systemd: gitea.service not started after database upgrade

[silverwind]

In the systemd service we recommend a Requires= dependency, but it seems this causes a issue with the service not being started back up when apt upgrades the database dependency. I observed the following sequence with such a configuration during a unattended APT upgrade of mysql.service on Ubuntu 20.04.2 using systemd 245:

May 13 06:34:13 systemd[1]: Stopping Gitea (Git with a cup of tea)...
May 13 06:34:13 systemd[1]: Stopping MySQL Community Server...
May 13 06:34:14 systemd[1]: Stopped MySQL Community Server.
May 13 06:34:14 systemd[1]: Stopped Gitea (Git with a cup of tea).
May 13 06:34:39 systemd[1]: Starting MySQL Community Server...
May 13 06:34:40 systemd[1]: Started MySQL Community Server.

Essentially gitea.service is not started up again after the database upgrade for unknown reason.

I don't fully understand why systemd would not bring up gitea.service in such a case, but I think we may want to consider using Wants= instead of Requires= to indicate a "weak" dependency which may help in such a case, or find out what systemd exactly does here.

https://unix.stackexchange.com/questions/375091 sounds like the same issue but I don't think the solution is acceptable.

[zeripath]

After= might be better?

[silverwind]

We had After= before but is was changed to Requires= in dbb4aee. I do think a dependency like Requires= or Wants= is appropriate but there is something particular that apt/systemd do that prevents the service from starting after the dependency was restarted.

[zeripath]

I think there's nothing stopping us from having a Requires and After for the same value.

I suspect the problem is that without the After the mysqld.service doesn't know that it needs to restart the service. All systemd knew is that it needed to stop the service because a Required dependency no longer existed.

I guess we need to know the semantics of Requires, Wants and After.

My suspicion is that there are subtle implications of these and that Requires won't start a required service and will shutdown if a Requires service is shutdown. Wants will start the wanted service and will shutdown if the wanted service shutsdown. Whereas After will tell the service that it comes after that it should be started after that service is started.

One test of that theory would be to restart the network.target and see if Gitea is restarted after that.

[silverwind]

One particular interesting comment I found on above stackexchange question is:

Amusingly enough, when you do systemctl restart postgresql, systemd remembers to restart its dependent services again. It seems like apt, for some reason, does combination of stop and start instead of restart

This should be testable by creating 2 units with a Require= relation and then stop/start the one that is being required. Systemd docs don't really mention what happens in such a situation in detail.

[noerw]

@silverwind Oh wow thanks for the report, you saved me hours of debugging on my instance, where the same happened the last two days..

[hubertbanas]

I ran into the same issue some time age. This is what resolved it for me

After=syslog.target network.target nginx.service mariadb.service

[zeripath]

OK so it sounds like my suspicion above (#15866 (comment)) is correct and if we change our suggested definition to include After= and Require= we're gonna be fine.

---

In fact it's notable that the lines below regarding socket activation actually suggest this:

###
# If using socket activation for main http/s
###
#
#After=gitea.main.socket
#Requires=gitea.main.socket
#

[lunny]

Reopen per #15881 (comment)

[silverwind]

Doesn't work for me either, simple test:

/etc/systemd/system/unit1.service

[Unit]
Description=unit1
Requires=unit2.service
After=unit2.service

[Service]
ExecStart=ping 127.0.0.1
ExecStop=kill -- $MAINPID
StandardOutput=file:/dev/null

[Install]
WantedBy=multi-user.target

/etc/systemd/system/unit2.service

[Unit]
Description=unit2

[Service]
ExecStart=ping 127.0.0.1
ExecStop=kill -- $MAINPID
StandardOutput=file:/dev/null

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl start unit1 unit2
systemctl stop unit2
systemctl start unit2
systemctl status unit1
# Active: inactive (dead)

Putting Wants= instead of Requires= leads to unit1 never being stopped when unit2 is stopped, better but not ideal.

[silverwind]

I think systemd can not provide a solution and we're better off using Wants= which means gitea will not be stopped when the database stops. Assuming gitea can reconnect to the database, this will work fine.

[hubertbanas]

Here is my working gitea.service for your references. I'm running Debian 10 Buster

[Unit]
Description=Gitea (Git with a cup of tea)
After=syslog.target network.target nginx.service mariadb.service

[Service]
Restart=always
RestartSec=2s
Type=simple
User=git
Group=git
WorkingDirectory=/var/lib/gitea/
ExecStart=/usr/local/bin/gitea web -c /etc/gitea/app.ini
Environment=USER=git HOME=/home/git GITEA_WORK_DIR=/var/lib/gitea

[Install]
WantedBy=multi-user.target

[silverwind]

Yes, it works without Requires= because Requires= is the option that make the service stop but not start back up. I think either After= alone or After= and Wants= are both fine, but we simply can not have Requires= (or BindsTo=) if we want to avoid the service being stopped when the database stops. Quote from Requires= docs:

this unit will be stopped if one of the other units is explicitly stopped

[silverwind]

Related systemd issue: systemd/systemd#19626