-
-
Notifications
You must be signed in to change notification settings - Fork 3
qhpsi.8
qmail-queue - queue a mail message for delivery
qmail-queue
qmail-queue reads a mail message from descriptor 0. It then reads envelope information from descriptor 1. It places the message into the outgoing queue for future delivery by qmail-send. The default outgoing queue is /var/indimail/queue. This can be changed by setting environment variable QUEUEDIR to point to a path having a valid queue created by the program queue-fix(8). qmail-queue is setuid to uid of user qmailq.
The deposition and balancing of mail across multiple queues is controlled by the following environment variables.
QUEUE_BASE The base directory having all the indimail queues. If this
is not defined, the control file queue_base is used. This is typically
/var/indimail/queue.
QUEUE_START This is a number which defines the first queue that
should be used.
e.g. QUEUE_START=1 implies the first queue to be /var/indimail/queue/queue1
QUEUE_COUNT This defines the number of queues that should be used for
load balancing. qmail-queue used a random number to select a queue
in a multi-queue setup.
e.g. QUEUE_START=1, QUEUE_COUNT=5 implies 5 queues /var/indimail/queue/queue1, /var/indimail/queue/queue2, ..., /var/indimail/queue/queue5
MIN_FREE
This is the minimum free disk space that should be available in the
filesystem containing the queues. When free space goes below this value,
qmail-multi will return a temporary disk full error "qq write error or
disk full (#4.3.0)". This prevents qmail-send reaching a deadlock in
the case when disk becomes full and there are bounce messages to be
processed. qmail-send in this case keeps on retrying bounce message
which does not succeeed because of insufficient disk space to write the
bounce. This effectively stops processing of local and remote messages
and thus disk space never gets freed up. qmail-multi prevents this
from happening by accepting mails only when disk space greater than
MIN_FREE is present. It uses statfs(2) (statvfs on solaris) to
get the free space on the filesystem.
qmail-queue uses 151 as the directory split. It can be overridden by environment variable CONFSPLIT upto a max of 151.
The envelope information is an envelope sender address followed by a list of envelope recipient addresses. The sender address is preceded by the letter F and terminated by a 0 byte. Each recipient address is preceded by the letter T and terminated by a 0 byte. The list of recipient addresses is terminated by an extra 0 byte. If qmail-queue sees end-of-file before the extra 0 byte, it aborts without placing the message into the queue. If USE_FSYNC USE_FDATASYNC environment variable is defined qmail-queue uses fsync(2) or fdatasync(2) to sync the message file in queue/mess and the envelope in queue/intd file as well. After creating a link to the envelope in queue/todo to queue/intd, qmail-queue syncs the directory using the following two methods.
-
BSD Style Synchronous Directory - This gets enabled if USE_SYNCDIR environment variable is defined. This forces directory syncing.
-
Using FSYNC - qmail-queue opens the file in queue/todo and uses the fsync(2) call to flush data buffers to the disk
-
Using FDATASYNC - qmail-queue opens the file in queue/todo and uses the fdatasync(2) instead of fsync(2) call to flush data buffers to the disk. fdatasync(2) is faster than fsync(2), but does not flush few of the file's metadata.
qmail-queue(8) communicates with todo-proc(8) when it places a message in the outgoing queue using either of the two below methods.
trigger method - This is used when the environment variable DYNAMIC_QUEUE is not set. Every queue in indimail has the path lock/trigger pointing to a named pipe. When a message is queued, qmail-queue writes a null byte to the pipe which causes todo-proc to wake up and scan the todo subdirectory of the queue. This is a disk intensive operation and can overwhelm todo-proc / qmail-send under high injection rate conditions.
message queue method - This is used when the environment variable DYNAMIC_QUEUE is set. Here qmail-queue communicates the inode number and the todo subdirectory name using POSIX message queue /queueN. Here N refers to the queue number (See multi-queue(7). qmail-queue uses mq_open(3) to open the message queue. This method avoids the need for todo-proc / qmail-send to scan the queue subdirectories for new messages, thereby saving on disk IO for better filesystem performance.
If the environment variable ORIGINIPFIELD is set, a "X-Originating-IP" header will be added to every relayed mail.
If the environment variable FASTQUEUE is set, qmail-queue will bypass QHPSI, ORIGINIPFIELD, extraqueue, removeheaders, envheaders, logheaders, mailarchive control files
qmail-queue sets an alarm of 86400 seconds to quit if it doesn't complete. This can be changed by setting DEATH environment variable. This should be less than 1296000 seconds or the value of OSSIFIED environment variable used by qmail-clean, qmail-send, slowq-send, qmta-send. These four programs delete any left over files that have not been accessed for more than OSSIFIED seconds in the intd, mess and todo queue subdirectories. Any file in the intd and mess sub directory that do not have a corresponding file in the info and the todo sub directory are files probably left due to a situation where qmail-queue didn't complete.
qmail-queue use many control files to provide various features.
extraqueue
If this control file exists, each line in the content is added as a
recipient to the recipient list. The environment variable EXTRAQUEUE
also does the same thing. The value of the environment variable
overrides the value in the control file. Also the environment variable
can be just one recipient. There is a potential for loops in case the
recipient is local and the local delivery has any kind of forwarding
rule which results in qmail-queue being invoked. If EXTRAQUEUE is
set but empty, the control will not be read.
quarantine
If this control file exists, recipient list is replaced with its
content. The environment variable QUARANTINE also does the same
thing. The value of the environment variable overrides the value in the
control file. Additionally, comma separated list of original recipient
list is added to the header X-Quarantine-ID. If QUARANTINE is
set but empty, the control will not be read.
envheaders
If this control file exists, qmail-queue passes all headers listed
in the control file to the queue along with the value.
qmail-lspawn(8) and qmail-rspawn(8) will set environment
variables corresponding to the headers with their values. An optional
length, separated by a ':', can be appended to the name of a header. If
a length is specified, the environment variable will be set only if the
actual length of the header value is less than or equal to this value.
This will prevent emails with extra long headers or malicious headers to
exploit memory allocation on the system. The environment variable
ENVHEADERS can be set to use a different name for this control file.
If ENVHEADERS is set but empty, the control file will not be read
and hence disabled. It is a good idea to do this on your SMTP server
that faces the internet.
logheaders
If this control file exists, qmail-queue writes all headers (along
with values) listed in the control file to file descriptor defined by
LOGHEADERFD environment variable. The file descriptor needs to be
open for write. You can open the file descriptor in the startup scripts
for qmail-smtpd, for example. If LOGHEADERS is set but empty,
the control file will not be read.
touch /tmp/header.log
exec 45 >/tmp/header.log
LOGHEADERFD=45
If the file descriptor is not open, logging of header will be silently ignored. Logging of headers can be useful for debugging or logging specific headers in log files. The environment variable LOGHEADERS can be set to use a different name for this control file.
removeheaders
If this control file exists, qmail-queue removes all headers listed
in the control file from the mail message on descriptor 0 before writing
it to the disk. The environment variable REMOVEHEADERS can be set to
use a different name for this control file. If REMOVEHEADERS is set
but empty, the control will not be read.
Every envelope recipient address should contain a username, an @ sign, and a fully qualified domain name.
qmail-queue always adds a Received line to the top of the message. If the environment variable HIDE_HOST is set, qmail-queue does not write hostnames and IP addresses in the received headers.
qmail-queue keeps a copy of the QQEH environment variable (if set), and passes it into the queue. qmail-local and qmail-remote will prepend it to the headers of the email when it is delivered.
mailarchive
if this control file exists, its content can be used for rule based
archival. The environment variable MAILARCHIVE overrides the name of
the mailarchive control file. If MAILARCHIVE is set but empty, the
control file will not be read. A line in this control file can be of the
form
type:regexp:dest_mailbox
where type is F or T. If type is F, rule is appled on the sender. If type is T, the rule is applied on the recipient. type can be omitted to match all recipients (but not senders). regexp is an expression to match the address (sender or recipient as specfied by type). dest_mailbox is a valid email address. regexp can be omitted to match any address. You can set QREGEX environment variable if you want to use actual regex patterns. If QREGEX is not set, filename global pattern will be used for match. regexp can be preceded by a ! sign to negate the expression. A % in dest_mailbox followed by u, d or e gets replaced as below
%u - user component in address without the '@' sign
%d - domain component in address
%e - email address
As stated above, the address used for matching regexp is the sender when type is F. For bounces, you can use <> to match a null sender. e.g.
F:<>:bounce_archive@example.com
When a rule matches a sender or any one of the recipients, the dest_mailbox address after expansion will be added to the existing recipient list. When a rule matches multiple recipients, only one email address will be added to recipient list to avoid duplicates.
The below will make a copy all mails for domain domain to archdomain (for all recipients - practically each and every mail).
::%u@arch%d
The below will make a copy of all mails sent by anyone from your yourdomain address.
F:.*@yourdomain:outgoing@yourdomain
The below will make a copy of all mails sent by anyone not from your yourdomain address. Here we have used ! to negate the result of regexp match.
F:!.*@yourdomain:incoming@yourdomain
NOTE: Invalid regexp are silently discarded.
originipfield
If this control file is set or if the environment variable
ORIGINIPFIELD is set, a "X-Originating-IP" header will be added to
every relayed mail. The environment variable ORIGINIPFIELD overrides
the control file originipfield.
qmail-queue does not inspect the message and does not enforce any restrictions on its contents. However, the recipients probably expect to see a proper header, as described in qmail-header(5) .
If QMAILQUEUE environment variable is set, programs included with qmail which invoke qmail-queue will invoke the the program set by QMAILQUEUE instead.
qmail-queue imposes two constraints on the queue structure: each mess subdirectory must be in the same filesystem as the pid directory; and each todo subdirectory must be in the same filesystem as the intd directory.
A large fraction of today's emails is infected by a virus or a worm. It is necessary to recognize those malicious emails as soon as possible -- already in the DATA phase of the SMTP conversation and to reject them.
Typically, this is facilitated my means of AMaViS, qmail-scanner, or qscanq which facilitate as "wrapper" of the original qmail-queue program invoked by means of QMAILQUEUE environment variable.
However -- in case of virus attacks -- those umbrella programs become part of the problem itself due to their resource requirements.
Today's virus scanner -- in particluar Clam AV -- work in resource efficient client/server mode (clamd/clamdscan) and include the feature to detect virii/worms in the base64 encoded data stream. Thus, there is no necessity to call additional programs (like reformime or ripmime) except for the virus scanner itself.
The QHPSI extension for qmail-queue allows to call an arbitary virus scanner directly to scan the message file in mess subdirectory or it allows plugins to be loaded from the /usr/lib/indimail/plugins directory. This directory can be changed by defining PLUGINDIR environment variable. QHPSI can be advised to pass multiple arguments to the virus scanner for customization. To run external scanner or load scanner plugins, qmail-queue calls qhpsi, a program setuid to qscand. By default, qhpsi looks for the symbol virusscan to invoke the scanner. The symbol can be changed by setting the environment variable QUEUE_PLUGIN_SYMB to the desired symbol.
In order to use the QHPSI, the virus scanner has to have the following qualifications:
a. Correct interpretation of the base64 and perhaps the uudecoded data in
order to detect the virii/worms therein.
b. Results have to be made available on stderr/stdout.
And perhaps:
c. Suppression of 'negative' scan results.
You can set environment variables to set the QPHSI interface as given below.
QHPSI can be set either to the full path of a scanner or to a list of plugins.
is set to full path of the virus scanner, i.e. QHPSI=/usr/bin/clamdscan. If full path is not specified, qmail-queue will use execvp(2) to run the scanner. Else it uses execv(2). Setting QHPSI environment variable turns on the QHPSI interface. The Qmail High Performance Scanner interface QHPSI also allows qmail-queue to read command line arguments taken from the QHPSI environment to be used as a call-interface for an external virus scanner:
QHPSI="clamdscan %s --quiet --no-summary"
Any '%s' in the QHPSI gets replaced with the message filename in the queue's 'mess' directory.
[step]
is set to list of plugins starting with a keyword 'plugin:'.
QHPSI="plugin: clamd.so fsecure.so symantec.so"
Sets qmail-queue to load three plugins clamd, fsecure and symantec from /usr/lib/indimail/plugins directory.
You can also use the plugin generic.so. generic.so plugin can use any scanner defined by the SCANCMD environment variable.
QHPSI="plugin: generic.so"
SCANCMD="clamdscan %s --quiet --no-summary"
Any '%s' in the SCANCMD gets replaced with the message filename in the queue's 'mess' directory.
QHPSIRC
To specify the return code of the virus scanner in case of an infection;
default is 1.
QHPSIRN
To specify the return code of the virus scanner in case of no infection;
default is 0.
REJECTVIRUS
If set to 1, infected mails will be bounced to sender. If set to 2,
infected mails will be blackholed. If set to 0, infected mails will be
accepted. Blackholing does not work if VIRUSFORWARD or QUARANTINE is
defined. Setting to a value > 2 will bounce infected mails to sender
VIRUSFORWARD
Infected mails will be quarantined to the email defined by VIRUSFORWARD.
This will work only if REJECTVIRUS is not equal to 1. Setting
VIRUSFORWARD sets the X-Quarantine-ID containing the list of all
recipients.
QHPSIMINSIZE
The minimal size of the message to invoke the virus scanner; default
is 0. A typical choice would be QHPSIMISIZE=10000 (~10k).
QHPSIMAXSIZE
The maximal size of the message to invoke the virus scanner; default is
unrestricted. A typical choice would be QHPPIMAXSIZE=1000000 (~1M).
When QHPSI is enabled qmail-queue adds the header X-QHPSI, the value of which is either 'infected' or 'clean' (depending on whether the mail as identifed as infected or not).
All of the below programs use the setting of QMAILQUEUE environment variables to execute qmail-queue. They all can either take QMAILQUEUE set to the path of a single qmail-queue frontend or as command line chain of qmail-queue frontend programs. If you specify qmail-queue as one of the programs in QMAILQUEUE, the chain will stop at qmail-queue. Any programs listed after qmail-queue will not get executed. So it is important not to have qmail-queue in the list.
condredirect(1), dot-forward(1), fastforward(1), filterto(1), forward(1), maildirserial(1), mini-smtpd(8), new-inject(1), ofmipd(8), qmail-inject(8), sendmail(8), qmail-local(8), qmail-qmqpd(8), qmail-qmtpd(8), qmail-send(8), qmta-send(8), qnotify(1), qreceipt(1), replier(1), rrforward(1), rrt(1), slowq-send(8), qmail-smtpd(8), srsfilter(1), qmail-spamfilter(8), qmail-multi(8), qmail-qfilter(1), qmail-dkim(8), qscanq-stdin(8)
The below programs also act as a qmail-queue frontend. They can be set in QMAILQUEUE environment variable and will ultimately execute qmail-queue in the end as a default (except for qmail-nullqueue which always exits 0).
qmail-spamfilter(8), qmail-multi(8), qmail-qfilter(1), qmail-dkim(8), qscanq-stdin(8), qmail-nullqueue(8)
qmail-queue does not print diagnostics. It exits with various exit codes detailed below. It exits 0 if it has successfully queued the message. It exits between 1 and 99 if it has failed to queue the message.
All qmail-queue error codes between 11 and 40 indicate permanent errors:
11
Address too long.
31
Mail server permanently refuses to send the message to any recipients.
(Not used by qmail-queue, but can be used by programs offering the
same interface.)
32
Spam or Junk Mail threshold exceeded. This is returned when a spam
filter like bogofilter is used (SPAMFILTER is defined) and the mail
is identified as spam.
33
Message contains a virus.
34
Message contains banned attachments
88
Custom error (=bounce) messages. You have to write the error message to
STDERR and exit 88, in order to use the custom message. Format of the
message:
Dthis is a custom fatal error message
Zthis is a custom temporary failure message
All other qmail-queue error codes indicate temporary errors:
51
Out of memory.
52
Timeout.
53
Write error; e.g., disk full.
54
Unable to read the message or envelope.
55
Unable to read a configuration file.
56
Problem making a network connection from this host. (Not used by
qmail-queue.)
61
Problem with the qmail home directory.
62
Problem with the queue directory.
63
Problem with queue/pid.
64
Problem with queue/mess.
65
Problem with queue/intd.
66
Problem with queue/todo.
67
Problem with uids/gids.
71
Mail server temporarily refuses to send the message to any recipients.
(Not used by qmail-queue.)
72
Connection to mail server timed out. (Not used by qmail-queue.)
73
Connection to mail server rejected. (Not used by qmail-queue.)
74
Connection to mail server succeeded, but communication failed. (Not used
by qmail-queue.)
79
Envelope format error.
81
Internal bug; e.g., segmentation fault.
Samples with clamd/clamdscan:
Here is a typical sample how to customize QHPSI together with Clam AV (clamd/clamdscan) for a tcpserver tcp.smtpd file:
:allow,QHPSI='/usr/bin/clamdscan %s --quiet --no-summary'
Comments:
- The path of 'clamdscan' can be omitted, because it is in
the standard path (/usr/bin).
- In the configuration file clamav.conf, the option
'ScanMail' has to be enabled; clamd has to run as 'root'.
- The argument QHPSI='/usr/bin/clamdscan %s --quiet --no-summary'
tells Clam AV to just provide the return status and print nothing
on the stdout.
- The argument QHPSIRC is not necessary, because
'clamdscan' return with 'RC=1' (the default) in
case a virus infection is recognized.
Sample with McAfee's uvscan:
:allow,QHPSI='/usr/bin/uvscan --secure',QHPSIMAXSIZE='9000000',QHPSIRC='13'
Comments:
- The path of 'uvscan' is '/usr/bin' and can be ommitted.
- 'uvscan' returns with RC=13 in case a virus is found,
therefore, QHPSIRC has to provide this value.
- The virus scanning is omitted, if the size of the message
exceeds 9.000.000 byte, ~ 8.5MB.
You can also set these variables outside tcp.smtp. e.g.
# echo /usr/bin/clamdscan %s --quiet --no-summary \
> /service/qmail.smtpd.25/variables/QHPSI
addresses(5), envelopes(5), qmail-header(5), qmail-inject(8), indimail-control(5), queue-fix(8), qmail-rm(1), qmail-qread(8), qmail-qmqpc(8), qmail-send(8), qmail-smtpd(8), qmail-lspawn(8), qmail-rspawn(8), qmail-internals(8), qmail-multi(8), qmail-spamfilter(8), qmail-nullqueue(8), qmail-direct(8), multi-queue(7), fsync(2) fdatasync(2) mq_open(3) qmail-queue-clients(7)