Replace spawn-fcgi by systemd socket for wwsympa/sympasoap

[xavierba]

Expected Behavior

On systems using systemd, there is no need to use spawn-fcgi to create the listening socket for wwsympa/sympasoap.

Current Behavior

The wwsympa/sympasoap systemd units are using spawn-fcgi.

Possible Solution

Provide wwsympa.socket/sympasoap.socket and modify wwsympa.service/sympasoap.service to make use of it.

Here are some example units for running wwsympa with httpd on Fedora/CentOS7+.
This is just a quick hack, it needs to be discussed and fine-tuned before a proper PR can be submitted.

wwwsympa.socket:

[Unit]
Description=WWSympa socket for httpd

[Socket]
ListenStream=/var/run/sympa/wwsympa.socket
Accept=false
SocketUser=apache
SocketGroup=apache
SocketMode=0600

[Install]
WantedBy=sockets.target

wwsympa.service:

[Unit]
Description=WWSympa - Web interface for Sympa mailing list manager
After=syslog.target sympa.service

[Service]
Type=simple
ExecStart=/usr/libexec/sympa/wwsympa.fcgi
User=sympa
Group=sympa
StandardInput=socket

[Install]
WantedBy=multi-user.target

Context

spawn-fcgi is currently not available for RHEL/CentOS 8. While looking at adding it to EPEL 8, I found out it is not needed at all.

• https://redmine.lighttpd.net/projects/spawn-fcgi/wiki/Systemd
• https://b.mtjm.eu/fastcgi-systemd-socket-activation.html
• https://www.freedesktop.org/software/systemd/man/systemd.socket.html

[ikedas]

Firstly, I think, you'd be better to prove the new method you propose will work properly. If it was proven, your new method may be added to the administrator manual. Older methods should be left so that compatibility to existing environments will be kept.

May I transfer this issue to sympa-community.github.io?

[xavierba]

The only proof I can provide for now is a sympa 6.2.48 web interface running over httpd on Fedora 30 installed from RPM works after replacing the stock wwsympa service unit file with the above service and socket unit files. Afaik, there is no incompatibility involved in switching from spawn-cgi to systemd socket, the httpd conf file nor anything else beside the unit files did not need to be altered.
The test setup can easily be replicated with the information I'm providing above and I don't expect anyone to just trust my words blindly, quite the contrary and I definitely welcome anyone to see for themselves and give feedback.

About the documentation, while I agree it is important, the systemd units are parts of the code and are generated from templates when building sympa (fcgi and socket path depend on the parameters passed to configure, socket owner/group depends on the webserver used). Imho, there's no reason to write said documentation before the implementation part has been discussed first.

My starting point is spawn-fcgi is not needed when using systemd. Specifically, in the case I'm interest in, it is not available for CentOS 8 and will not be. Also, spawn-fcgi is mostly dead upstream, so sympa better not depend on unmaintained code, especially when there is maintained code that serves the same purpose. Actually, spawn-fcgi is likely doomed to disappear from any systemd based distro sooner or later.

[ikedas]

1. If you have difficulty with specific distro (e.g. RHEL/CentOS 8), you can start with chaniging spec for that distro.
2. Then you (or any others) can add documentation about generic setup instruction (not limited to EL8 users) without swapn-fcgi, keeping instruction on older methods using spawn-fcgi etc.
3. At last, source distro of Sympa may remove spawn-fcgi support, and announce that change to administrators and packagers.

In this way, we will be able to introduce users to new installation/packaging method.

Thus, in the first place, you would be better to implement new method for EL8 RPM (step 1. in above) at first. You will not necessarily have to work for subsequent steps.

---

I'll check your unit files for a while. One of my concern is whether these units support automatic restart: The wwsympa.fcgi itself terminates not only when it crashes but also when the executable files has been updafted.

[xavierba]

Actually, I'm in no hurry for getting sympa running on EL8 for my own purpose. I'm working on this because I fell there is a need for sympa itself to be easily available on RHEL/CentOS 8. And there was already someone that requested it actually, both directly to me and in a sympa issue.

My assessment of the situation is spawn-fcgi is both not needed and dead upstream, and thus must be replaced by systemd socket. Be it for EL8 or any other systemd-based distro. However, before engaging with such a downstream change, I'd rather have upstream at the very least know about what I will be doing and why I will be doing it. At best, I expect upstream to acknowledge the change and help toward making this change as robust and transparent to users as possible, before it even has a chance to reach any end-user.

I'm in no way disagreeing with the steps you are proposing. This definitely is the way I'd like it to be done, but with one difference. I'd rather see upstream sympa host the socket-based unit files for anyone to use if they see fit, than keep them in the Fedora/EPEL garden.
I'm not advocating removing spawn-fcgi based unit file, nor making systemd socket-based unit files the default (at least for now). Other downstream maintainer or user can then choose for themselves if they want to use sysVinit scripts, systemd with spawn-fcgi, systemd with socket activation, or any other method of their choosing. But we are not at this step yet , we're even not at step 1, we're at step 0, discuss, fine-tune and validate/invalidate the proposed change.

About automatic restart, that is a very valid concern. I think the service unit would need to have either Restart=on-failure or possibly Restart=on-abnormal. Another likely needed change is to add PartOf=wwsympa.service to the socket unit.

[ldidry]

Quick question: does systemd socket system allows to have more than one wwsympa.fcgi processus running? Because on heavily used Sympa, you may need more than one processus to handle requests.

[racke]

@ldidry Yes it does. Proof of Concept: sympa-community/sympa-ansible#61

[ldidry]

@racke For what I see, it would create multiple sockets (/run/{{ unit_name }}%i.sock), not just one. We can use multiple sockets in nginx, but it’s not something convenient (as for Apache, it should be doable too, I think, but again with more complexity in the configuration).

[ikedas]

Apache: Using fcgistarter

FYI: With Apache 2.4, fcgistarter may be used as an alternative to spawn-fcgi (However it uses INET socket instead of UNIX socket).

(TBD)

[ikedas]

Systemd: Single socket with multiplexer: Using multiwatch

Multiple worker threads in ArchLinux Wiki says:

If you want to spawn multiple worker threads, it is recommended that you use multiwatch, which will take care of restarting crashed children.

And subsequent example uses multiwatch and spawn-fcgi without considering Systemd socket.

Thus, replacing spawn-fastcgi with Systemd socket, we may provide single socket with multiple workers.

/lib/systemd/system/multiwatch-wwsympa.socket

[Unit]
Description = WWSympa - Web interface for Sympa mailing list manager (socket)

[Socket]
SocketUser = nginx
SocketMode = 0600
ListenStream = /run/sympa/wwsympa.socket

[Install]
WantedBy = sockets.target

/lib/systemd/system/multiwatch-wwsympa.service

[Unit]
Description = WWSympa - Web interface for Sympa mailing list manager (service)
After = syslog.target sympa.service

[Service]
User = sympa
Group = sympa
ExecStart = /usr/bin/multiwatch -f $FCGI_CHILDREN -- /usr/libexec/sympa/wwsympa.fcgi
StandardOutput = null
StandardInput = socket
StandardError = null
Environment="FCGI_CHILDREN=5"
EnvironmentFile=-/etc/sysconfig/sympa
Restart=on-failure

[Install]
WantedBy = multi-user.target

And, for another SocketUser, create this file:

/etc/systemd/system/multiwatch-wwsympa.socket/socket.conf

[Socket]
# With sympa-httpd, uncomment folowing line.
#SocketUser=apache

Pros

• spawn-fcgi will no longer be required.
• Compatible configuration for HTTP servers (nginx & httpd).
• Provided by many distros (ArchLinux, Debian, FreeBSD etc.).

Cons

• Not provided by Fedora/EL/CentOS.

[racke]

Sounds good to me, let me try that in Ansible. Thanks @ikedas

[xavierba]

Pushing multiwatch in Fedora/EPEL shouldn't be an issue and I've made a package for it already. I can push it to sympa-beta COPR if you want to give it a test run.

[ikedas]

@racke thanks!

@xavierba thanks, but for the present I'll do with manually crafted unit files. I'd like to check more other configurations (such as multiple sockets with httpd mod_proxy_balancer) and to post them.

[BenjaminLefoul]

Hi!

@xavierba We have exchanged some emails a few weeks ago regarding this.

What is the status on this issue?
I believe it is perfectly reasonable to start with EPEL8, and in time move the socket file(s?) upstream.

I actually have a bunch of RHEL8 (not centos) machines ready for just this purpose right now.

@ikedas Is the multiwatch solution still viable? And if so, @xavierba can we push it to the fedora project? I am also a packager there (although rather inactive right now) if you need help for a review.

Benjamin

[xavierba]

@BenjaminLefoul if you are happy with what is in my COPR repo [1], that is the conf still using spawn-fcgi, then I can indeed make a real EPEL8 build.
I can adapt later shall sympa move on with systemd sockets or anything else.

[1] https://copr.fedorainfracloud.org/coprs/xavierb/sympa-beta/package/sympa/

[ikedas]

Hi @BenjaminLefoul ,

I drafted an instruction using Systemd socket. See sympa-community/sympa-community.github.io#58. If possible, could you please check if it works as expected?

[BenjaminLefoul]

@xavierba Wouldn't it be easier to start using sockets (and multiwatch?) as soon as possible so we don't have to worry about maintaining spawn-fcgi (and multiwatch) for the rest of rhel8s life?

[BenjaminLefoul]

Hi @BenjaminLefoul ,

I drafted an instruction using Systemd socket. See sympa-community/sympa-community.github.io#58. If possible, could you please check if it works as expected?

And how about a wwsympa@.socket file?
/usr/lib/systemd/system/wwsympa@.socket

[Unit]
Description = WWSympa - Web interface for Sympa mailing list manager (socket)

[Socket]
SocketUser = %i
SocketMode = 0600
ListenStream = /run/sympa/wwsympa.socket

[Install]
WantedBy = sockets.target

And then use systemctl start wwsympa@apache.socket, systemctl start wwsympa@nginx.socket or systemctl start wwsympa@lighttpd.socket?

[BenjaminLefoul]

Okay how about a systemd target file to allow several wwsympa.fcgi processes on a single socket file, without multiwatch?

/usr/lib/systemd/system/wwsympa.target:

[Unit]
Description= WWSympa - Web interface for Sympa mailing list manager
Wants=wwsympa@0.service wwsympa@1.service wwsympa@2.service wwsympa@3.service
 wwsympa@4.service


[Install]
WantedBy=multi-user.target

/usr/lib/systemd/system/wwsympa@.service:

[Unit]
Description=WWSympa - Web interface for Sympa mailing list manager
After=syslog.target sympa.service
PartOf=wwsympa.target

[Service]
Type=simple
ExecStart=/usr/libexec/sympa/wwsympa.fcgi
User=sympa
Group=sympa
StandardInput=socket
Restart=on-failure
Sockets=wwsympa.socket

[Install]
WantedBy=multi-user.target

/usr/lib/systemd/system/wwsympa.socket:

[Unit]
Description=WWSympa socket for httpd

[Socket]
ListenStream=/var/run/sympa/wwsympa.socket
Accept=false
SocketUser=apache
SocketMode=0600

[Install]
WantedBy=sockets.target

And for another webserver:
/etc/systemd/system/wwsympa.socket.d/socket.conf:

[Socket]
SocketUser=nginx

[ikedas]

Hi @BenjaminLefoul ,
I drafted an instruction using Systemd socket. See sympa-community/sympa-community.github.io#58. If possible, could you please check if it works as expected?

And how about a wwsympa@.socket file?
/usr/lib/systemd/system/wwsympa@.socket

If you think such setting may be useful, please draft instruction for that setting and submit it here.

On the other hand, I thought the setting with socket and multiwatch is useful, drafted instruction for that setting and submitted it. Couldn’t you review it?

[Unit]
Description = WWSympa - Web interface for Sympa mailing list manager (socket)

[Socket]
SocketUser = %i
SocketMode = 0600
ListenStream = /run/sympa/wwsympa.socket

[Install]
WantedBy = sockets.target

And then use systemctl start wwsympa@apache.socket, systemctl start wwsympa@nginx.socket or systemctl start wwsympa@lighttpd.socket?

I don’t know if it works with Apache, lighttpd and the others. Please investigate and, if possible, include it to the instruction you will author!

[BenjaminLefoul]

Well it would have been nice not having to push and maintain a new package to the Fedora project. I don't know how far @xavierba went with the multiwatch RPM

But if all we need are 5 parallel wwsympa.fcgi processes which can try to respawn when they crash and use the same socket, then the systemd solution I described in my last post should be enough.

What do you think?

[ikedas]

TMTOWTDI. I submitted what I experienced to share it with the community. Why don’t you submit what you could share with the community?

[xavierba]

@BenjaminLefoul multiwatch rpm is available in my sympa-beta corp repo (for Fedora and EL7/EL8) if you want to give it a try.

[xavierba]

Let me try to recap:

• The basic scenario of one wwsympa process running on one socket is what I drafted in the very first post of this issue.

• Only one wwsympa process might not be enough for all use cases, so one needs to be able to spawn several of them. The number of processes needs to be tunable.
  This can be achieved by either:

  • Multiple instances of both the socket and wwsympa service with the wwsympa@.socket and wwsympa@.service files. The sympa administrator will need to spawn as many instances he needs with systemctl start wwsympa@1; systemctl start wwwsympa@2; ... This supposes to make (non-trivial ?) changes to the webserver(s) configuration.
  • Using a multiplexer. This only needs one wwsympa.socket and wwsympa.service files. The number of instances can be set in the environment file /etc/sysconfig/sympa
    Current candidates are:
    • fcgitstarter (INET socket only, not UNIX socket)
    • multiwatch, which seems the preferred choice.

Do we want to support both the single instance mode and multiple instances mode ?
Or go with only the multiple instances mode ?

Also, the socket(s) need(s) to be owned by the webserver that will be using it/them, so this needs to be properly set out of the box or configurable.
This can be achieved by either:

• Overriding SocketUser in /etc/systemd/system/wwwsympa.socket.d/override.conf.
• Alternatively, a different wwsympa.socket and/or wwwsympa@.socket file with the proper SocketUser can be distributed per web-server sub-package.
• Using wwwsympa@.socket instances with the web-server user as the instance (systemctl start wwsympa@nginx and use SocketUser=%i in the systemd socket file. I think this only adresses the case where there is one socket, not multiple sockets.

Let me know if I have missed anything and what are your preferred choices from the above.

Edit: One cannot use an environment variable and feed it to SocketUser.

[BenjaminLefoul]

Thanks, @xavierba
and thanks for the multiwatch RPM!

I have now tested with apache on RHEL8 with both spawn-fcgi and multiwatch. Both work (I can get the web interface).

For multiwatch I used @ikedas unit files, changing only the multiwatch path from /usr/local[...] to /usr/bin/multiwatch, and only starting the socket with user "apache" (which in turns starts the wwsympa service).

Note that for either case, I had to do a slight SELinux tweak (if you run targeted on the standard red hat policy enforced): /var/run/sympa/wwsympa.socket needs to have target label httpd_var_run_t

semanage fcontext -a -t httpd_var_run_t "/var/run/sympa(/.*)?"
restorecon -r /var/run

I have not tested nginx or lighttpd, but it should be okay when it comes to SELinux (both use the same entrypoint apache does: httpd_exec_t)

I have not tested multiwatch on Fedora, but we could start by pushing to EPEL8?

Edit: ok, multiwatch works on Fedora 31, but it needs another SElinux refinement (connectto). I simply used this:

[root@localhost ~]# cat my-usrsbinhttpd.te 

module my-usrsbinhttpd 1.0;

require {
	type unconfined_service_t;
	type httpd_t;
	class unix_stream_socket connectto;
}

#============= httpd_t ==============
allow httpd_t unconfined_service_t:unix_stream_socket connectto;

[ikedas]

@xavierba wrote:

Do we want to support both the single instance mode and multiple instances mode ?
Or go with only the multiple instances mode ?

IMO, on RPM (Fedora/EPEL), we may support only multiple instances: The greater will serve for the lesser.

Also, the socket(s) need(s) to be owned by the webserver that will be using it/them, so this needs to be properly set out of the box or configurable.
This can be achieved by either:

• Overriding SocketUser in /etc/systemd/system/wwwsympa.socket.d/override.conf.
• Alternatively, a different wwsympa.socket and/or wwwsympa@.socket file with the proper SocketUser can be distributed per web-server sub-package.
• Using wwwsympa@.socket instances with the web-server user as the instance (systemctl start wwsympa@nginx and use SocketUser=%i in the systemd socket file. I think this only adresses the case where there is one socket, not multiple sockets.

Let me know if I have missed anything and what are your preferred choices from the above.

Edit: One cannot use an environment variable and feed it to SocketUser.

My preference is overriding SocketUser by the file like override.conf. The unit with spawn-fcgi used an environment variable to override command line options vary by HTTP servers.

[xavierba]

@ikedas thanks for your answers. Actually, that is what I have implemented in my test specfile.

I have yet to look at lighttpd, which seems to handle the socket on its own and not with spawn-fcgi. I don't know much about lighttpd, that why I kept it for later.

However, the questions were actually more targeted at sympa devs in order to propose a PR.
Do we want to implement it in the same way, that is :

• multiple instances per default with multiwatch
• default socket user nginx, with override to apache/www-user/whatever in a systemd conf override

@racke, @ldidry, it would be great to have your opinion too.

[xavierba]

@racke I have some stuff more or less ready to be used allowing to easily tune the number of instances for both sympa itself and the soap server. I will post it tomorrow when I'm in front of my computer rather than on a phone.

[xavierba]

Here are the relevant commits, which include some new systemd unit files

• https://src.fedoraproject.org/fork/xavierb/rpms/sympa/c/cdc92804c7cfcb348f7a318c1ad8d90cf20d298a?branch=systemd_socket
• https://src.fedoraproject.org/fork/xavierb/rpms/sympa/c/547a959728742ce8b58b4ea11d72d9ddac695099?branch=systemd_socket
• https://src.fedoraproject.org/fork/xavierb/rpms/sympa/c/2a9ee7b8e523b46b8031d9c241474995aadd6294?branch=systemd_socket

One can set the number of wws and soap instances through the WWS_FCGI_CHILDREN and SOAP_FCGI_CHILDREN envvars in /etc/sysconfig/sympa (which would be /etc/default/sympa in Debian), with a default of 5 instances for both.
The default socket user is nginx, but overrides are deployed when one uses the httpd or lighttpd sub-packages.

I welcome any feedback on this and would be happy to help getting this integrated into upstream sympa.

[racke]

Yes, I took a similar approach. However I set the default for SOAP to 2. I will align the variable names with yours and check about other differences. Debian uses www-data as user for Nginx and Apache, so I can hardcode that.

[xavierba]

It was the same FCGI_CHILDREN setting for both WWS and SOAP and set to 5 before, so I used that for both, but I guess it makes sense to reduce the number of SOAP instances to only 2.

[racke]

Yes, in the rare case someone really uses SOAP frequently it can be increased to more than too, but I think 2 is a better default setting.

[xavierba]

Agreed.
https://src.fedoraproject.org/fork/xavierb/rpms/sympa/c/2db634f0f431b4d82e09d1a7eb2a6e92d86221c8?branch=systemd_socket

[xavierba]

@racke: What do you think about this ? Does that match what you've done for Debian ?
sympa-6.2...xavierba:systemd_socket

The default webserver user is set to nginx, as this is what is currently in wwsympa.servicein, but that can be changed, and it doesn't really matter anyway as this is made configurable. The newly introduced configure option can be used in the other unit files in a subsequent commit.

[xavierba]

I've compared the unit files in my tree with what's in Debian 12 package and taken the service/socket descriptions which were better.

The different web server user is already addressed in my tree through a configure option.
There's also the /etc/sysconfig vs /etc/default stuff, which would require yet another configure option.

The only meaningful difference seems to be I have

Restart=on-failure

while Debian has

Restart=always
RestartSec=5

in both the web interface and soap service units.

@racke, why always rather than on-failure ? Also, any particular reason to increase the restart delay from the default 100 ms to 5 seconds ?

[racke]

@xavierba, I actually don't recall that. on-failure seems to be a good choice, so I filed a bug against the Debian package (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1005823).

[xavierba]

@racke what about RestartSec=5 ?

Also, would you have a suggestion for a configure option name to accommodate for /etc/sysconfig vs /etc/default ?

[racke]

RestartSec=5 will be removed as well. Maybe --service-defaults-directory ?

[ikedas]

Sorry for delayed comment. On sympa-6.2...xavierba:systemd_socket

Is --with-web_server_user option for configure script necessary? Of course it is useful, but we'd be better to override the value of SocketUser in *-multiwatch.socket at runtime. Because the packagers need to be able to support different web servers with one wwsympa package.

I suppose (I've not confirmed yet, though) this can be achieved in either of the following ways.

(A) Fill the SocketUser in wwsympa-multiwatch.socket with something meaningless, e.g.:

SocketUser = nobody

and add a file /etc/systemd/system/wwsympa-multiwatch.socket.d/override.conf with the content:

[Socket]
SocketUser = nginx

then start (or reload) the unit.

(B) Rename wwsympa-multiwatch.socket to wwsympa-multiwatch@.socket (and do the same for wwsympa-multiwatch.service), change its content as

SocketUser = %i

and, start this unit by

# systemctl start wwsympa-multiwatch@nginx.socket

thus SocketUser is assigned the value nginx at runtime.

---

I'll confirm these in a few days.

[ikedas]

@xavierba & @racke,

• I guess Restart=always has been chosen to support the behavior of wwsympa.fcgi that trys refreshing itself when it is updated:
  

  sympa/src/cgi/wwsympa.fcgi.in

  Lines 1781 to 1792 in f71f5ce

  	# Exit if wwsympa.fcgi itself has changed.
  	if (defined $birthday) {
  	my $age = [stat $PROGRAM_NAME]->[9];
  	if (defined $age and $birthday != $age) {
  	$log->syslog(
  	'notice',
  	'Exiting because %s has changed since FastCGI server started',
  	$PROGRAM_NAME
  	);
  	exit(0);
  	}
  	}

• Use of EnvironmentFile seems not recommended now: Overriding Environment by drop-in snippet looks recommened: cf. An answer to "How to set environment variable in systemd service?".

[ikedas]

I confirmed.

• (A) using drop-in to specify web server user:
  wwsympa-multiwatch.socket
  wwsympa-multiwatch.service
  Web server user (SocketUser in [Socket] section) should be modified by such as:

  systemctl edit wwsympa-multiwatch.socket

  or directly create/edit the snippet e.g. /etc/systemd/system/wwsympa-multiwatch.socket.d/xxxx.conf.

• (B) using instance name to specify web server user:
  wwsympa@.socket
  wwsympa@.service
  Service should be started as:

  systemctl start wwsympa@webserveruser.socket.

EnvironmentFile directive was removed.

[xavierba]

The idea behind the configure option is to provide a way to generate unit files that are tailored to the target distribution. (eg. Debian will use www-data, while Fedora/EL will use nginx). If need be, this can be overriden with a different user to provide support for other web servers (httpd or lighttpd, like in Fedora/EL package).
The EnvironmentFile directive only serves the purpose to set the number of FCGI children for WWS and SOAP.
As Debian/Fedora uses different location for said environment file, I'd like to also introduce a configure option for that. If/when distributions decide to deprecate /etc/sysconfig/ /etc/default, it could be used to relocate the file easily.

[ikedas]

The idea behind the configure option is to provide a way to generate unit files that are tailored to the target distribution. (eg. Debian will use www-data, while Fedora/EL will use nginx). If need be, this can be overriden with a different user to provide support for other web servers (httpd or lighttpd, like in Fedora/EL package). The EnvironmentFile directive only serves the purpose to set the number of FCGI children for WWS and SOAP.

I understand that --with-web_server_user configure option is not related to Environment/EnvironmentFile directive in systemd unit, but it is relateted to SocketUser directive.

My suggestion is that, instead of new configure option. we'd be better to handle web server users according to the systemd specification: Run systemctl edit xxx.socket and create a dropin snippet including SocketUser directive:

[Socket]
SocketUser=the_user_we_really_want_to_use

[EDIT] The second way (wwsympa@.socket) uses template variable for SocketUser and actual value should be specified as instance name, e.g. systemctl start wwsympa@nginx.socket.

---

As Debian/Fedora uses different location for said environment file, I'd like to also introduce a configure option for that. If/when distributions decide to deprecate /etc/sysconfig/ /etc/default, it could be used to relocate the file easily.

Similar suggestion: Handle FCGI_CHILDREN environment with running systemctl edit xxxx.service and creating a snippet:

[Service]
Environment="FCGI_CHILDREN=number_of_workers_we_want"

n.b.: Since each unit doesn't share the namespace of environment, the same name FCGI_CHILDREN may be used.

---

My suggestions may not be elegant, but they require fewer modifications to our code base.

[EDIT] See also: "Editing provided units".

[ikedas]

More on /etc/{default,sysconfig}/sympa and EnvironmentFile:

• A blog post by a developer of Systemd suggests that distribution-dependent files would be replaced by drop-in snippets

• BTW the manual says that

  ... This option may be specified more than once in which case all specified files are read. ...

  That is, I guess the both lines may be included:

  EnviromentFile=-/etc/default/sympa
  EnviromentFile=-/etc/sysconfig/sympa
  

[ikedas]

If there are no objection to the PR above, it may be merged.

[xavierba]

In the light of your comment, I guess Restart=always is a better fit than Restart=on-failure.
Having both deb/rpm style EnvironmentFile is a smarter and less invasive trick than having a configure option.

However, I find the solution to have a configure option for the webserver user to be more elegant than using override files. Admittedly, that doesn't change much for rpm/deb packages which will still have to provide an override file to be able to ship support for several webservers. However, someone building and installing from source will have an additional configuration step to tackle.
Still in the light of ease of installation, maybe the unit files (both spawnfcgi and multiwatch flavors) can be installed at the proper /usr/lib/systemd/system/ location rather than just be built and left in the tree ?

[ikedas]

However, I find the solution to have a configure option for the webserver user to be more elegant than using override files. Admittedly, that doesn't change much for rpm/deb packages which will still have to provide an override file to be able to ship support for several webservers. However, someone building and installing from source will have an additional configuration step to tackle. Still in the light of ease of installation, maybe the unit files (both spawnfcgi and multiwatch flavors) can be installed at the proper /usr/lib/systemd/system/ location rather than just be built and left in the tree ?

I think in the near future we could rather abandon spawn-fcgi support --- in fact, units for two different methods aren't needed --- and allow multiwatch units to be installed. BTW if that happens, "an additional configuration step to tackle" for people would be to put the override files in place. I feel this would be as simple as specifying an additional configure option at build time.

In any case, there would be no need to implement new options right now.

[ikedas]

If no objection to keeping both methods with systemd for now, I'll merge the PR above.

[xavierba]

Having both the spawn-fcgi and multiwatch based unit files is fine by me. Actually, having both installed as well would be fine too, one can enable either one depending on its own preferences, but that can be done in a different commit/PR. And then dropping spawn-fcgi could be done in yet another commit/PR.

[ikedas]

Having both the spawn-fcgi and multiwatch based unit files is fine by me.

Do you mean the former is required for EPEL7 package you maintain? (EPEL8/9 will adopt the latter.) If you do, you may include a unit file specific to EL7 by your own into the RPM source. In below I'll write why.

Actually, having both installed as well would be fine too, one can enable either one depending on its own preferences,

There are no preferences. Since the method with spawn-fcgi occationally caused problems, so alternative method had been sought. With the method with multiwatch, no problems have been reported so far.
Of course, there might be the other alternatives: @racke suggested the method purely using Systemd sockets. Hownever, his method could only be used with nginx.

After all, in terms of keeping things simple, the Systemd units with multiwatch is the current best practice for Linux with systemd support, I believe.

Reality is not so simple. In reality, old configuration files per distribution (e.g. files in /etc/sysconfig or /etc/default) have to be supported for backward compatibility, for example. Moreover, this is only the case for some variants of many Linux distributions. It is not practical for us to add support for all other distributions --- Gentoo Linux, FreeBSD, macOS, ... --- to the source distribution.

To solve problem, I think that Sympa itself may only provide interfaces to other components. In fact, Sympa provides a pipe interface for MTAs and a FastCGI interface for HTTP servers.

Extremely say, there is no need to include anything else in Sympa source distribution. Configurations for each OS distribution can be provided in separate project(s). I understand that one of the purposes of the assets repository was to achieve this to help such projects.

It could include the actual configuration files for the methods adopted by individual distributions, as well as those that were not adopted.

but that can be done in a different commit/PR. And then dropping spawn-fcgi could be done in yet another commit/PR.

Honestly, as for the Systemd unit included in the source tarball, I believe that only the simplest (without EnvironmentFile) one with multiwatch is needed. That is, I prefer not to include actual unit files specific to EL8/9 (also EL7, Debian, ...).

Sorry for long comment.

[xavierba]

Having both the spawn-fcgi and multiwatch based unit files is fine by me.

Do you mean the former is required for EPEL7 package you maintain? (EPEL8/9 will adopt the latter.) If you do, you may include a unit file specific to EL7 by your own into the RPM source. In below I'll write why.

No, I mean it doesn't hurt to have both. One has to choose which method he wants to use.
About EL7, I've not made my mind yet. I think I very much prefer multiwatch over spawn-fcgi, but I'm not sure switching from spawn-fcgi to multiwatch that late in the EL7 release cycle is a wise move. However, choosing one method and sticking with it for all Fedora/EL releases sounds like the simplest to me.
If Sympa decides to formally drop spawn-fcgi support when using systemd either now or later, that'll help me make a choice ;-)
Nevertheless, I tested updating from an RPM using spawn-fcgi to an RPM using multiwatch without issue a number of times.

Actually, having both installed as well would be fine too, one can enable either one depending on its own preferences,

There are no preferences. Since the method with spawn-fcgi occationally caused problems, so alternative method had been sought. With the method with multiwatch, no problems have been reported so far. Of course, there might be the other alternatives: @racke suggested the method purely using Systemd sockets. Hownever, his method could only be used with nginx.

I suggested to keep both to ease transition, but this should indeed only be temporary. Remember, I was the one who first suggested to add support for systemd sockets rather than relying on spawn-fcgi to create the socket ;-)

After all, in terms of keeping things simple, the Systemd units with multiwatch is the current best practice for Linux with systemd support, I believe.

Agreed.

Reality is not so simple. In reality, old configuration files per distribution (e.g. files in /etc/sysconfig or /etc/default) have to be supported for backward compatibility, for example. Moreover, this is only the case for some variants of many Linux distributions. It is not practical for us to add support for all other distributions --- Gentoo Linux, FreeBSD, macOS, ... --- to the source distribution.

I have no specific knowledge about anything beside SysVInit/systemd or too old to be really useful.
Anyway, I guess if some kind folk would like to add support for other init system, that should certainly be considered.
Yet again, it's better to have stuff upstream rather than let everyone re-invent the wheel.

To solve problem, I think that Sympa itself may only provide interfaces to other components. In fact, Sympa provides a pipe interface for MTAs and a FastCGI interface for HTTP servers.

Extremely say, there is no need to include anything else in Sympa source distribution. Configurations for each OS distribution can be provided in separate project(s). I understand that one of the purposes of the assets repository was to achieve this to help such projects.

It could include the actual configuration files for the methods adopted by individual distributions, as well as those that were not adopted.

but that can be done in a different commit/PR. And then dropping spawn-fcgi could be done in yet another commit/PR.

Honestly, as for the Systemd unit included in the source tarball, I believe that only the simplest (without EnvironmentFile) one with multiwatch is needed. That is, I prefer not to include actual unit files specific to EL8/9 (also EL7, Debian, ...).

One of the point with systemd is to reduce distributions fragmentation. Indeed, that does not solve everything at once, like the /etc/default vs /etc/sysconfig difference, but having both supported in the unit files is not a huge burden to carry in upstream sympa. Writing unit files is not such an easy task and can have some subtleties. See the part of the above discussion about Restart=OnFailure vs Restart=Always. So I strongly think there is value in shipping unit files. And there would also be value in installing the unit files, rather than have people manually copy the files at the proper location. I believe systemd is the most used init system, so let's at least make it easy for a majority of people, while not making it any more difficult for people using anything else.

Sorry for long comment.

No worries :-)

[ikedas]

Reality is not so simple. In reality, old configuration files per distribution (e.g. files in /etc/sysconfig or /etc/default) have to be supported for backward compatibility, for example. Moreover, this is only the case for some variants of many Linux distributions. It is not practical for us to add support for all other distributions --- Gentoo Linux, FreeBSD, macOS, ... --- to the source distribution.

I have no specific knowledge about anything beside SysVInit/systemd or too old to be really useful. Anyway, I guess if some kind folk would like to add support for other init system, that should certainly be considered. Yet again, it's better to have stuff upstream rather than let everyone re-invent the wheel.

I agree to the last sentence. However, "upstream" necessarily does not mean the source distribution of Sympa itself. There is no need to cram everything into one tarball.

To solve problem, I think that Sympa itself may only provide interfaces to other components. In fact, Sympa provides a pipe interface for MTAs and a FastCGI interface for HTTP servers.
Extremely say, there is no need to include anything else in Sympa source distribution. Configurations for each OS distribution can be provided in separate project(s). I understand that one of the purposes of the assets repository was to achieve this to help such projects.
It could include the actual configuration files for the methods adopted by individual distributions, as well as those that were not adopted.

but that can be done in a different commit/PR. And then dropping spawn-fcgi could be done in yet another commit/PR.

Honestly, as for the Systemd unit included in the source tarball, I believe that only the simplest (without EnvironmentFile) one with multiwatch is needed. That is, I prefer not to include actual unit files specific to EL8/9 (also EL7, Debian, ...).

One of the point with systemd is to reduce distributions fragmentation. Indeed, that does not solve everything at once, like the /etc/default vs /etc/sysconfig difference, but having both supported in the unit files is not a huge burden to carry in upstream sympa. Writing unit files is not such an easy task and can have some subtleties. See the part of the above discussion about Restart=OnFailure vs Restart=Always. So I strongly think there is value in shipping unit files. And there would also be value in installing the unit files, rather than have people manually copy the files at the proper location. I believe systemd is the most used init system, so let's at least make it easy for a majority of people, while not making it any more difficult for people using anything else.

The issue with the Restart directive is the same for all distributions that use Systemd, so it should be discussed as the issue about Systemd support in general. However, /etc/sysconfig and /etc/default (and maybe more?) are examples of "fragmentation": The Systemd developers do not recommend them and suggest the method standardised by Systemd (see the article in my comment), and they are just attempts in EL and Debian. Not only that, it is not enough to support sysVinit and Systemd alone.

I think that there is no need for Sympa itself to have a role that encompasses all of such "fragmentations". Such a role should be taken up by separate project. I suggest the use of an assets repository.

[ikedas]

Apparently, my explanation seems not understood.

I will merge the current PR. However at first I have to update the description about Systemd support in the manual.