Exim's bounce text changed - need to update ProcessBounce.pm

[popov-d]

I've noticed that in my installation (Debian-based) there are numerous files in /var/spool/sympa/bounce/bad . Most of them appeared to be bounces from Exim servers (4.84-4.90) in non-DSN mode. The information inside was quite clear, but Sympa could not catch the address of bouncer.

While investigating the problem I found that regular expressions in lib/Sympa/Spindle/ProcessBounce.pm slightly differ from those in actual bounce messages. I changed it to catch information from actual bounces, while trying not to break compatibility with old ones, in case an old Exim is running somewhere.

ProcessBounce.pm-patched-6.2.16-Debian9.txt

ProcessBounce.pm-patched-6.2-current.txt

Version

6.2.16 from Debian Stretch, where problem was found. Exim messages handling is the same in latest 6.2 on GitHub, so problem is still here.

Installation method

Deb from official Debian repository.

Expected behavior

Attached sample message: bounce-sample.txt

This has bounced from a mailbox of a user to a VERP address. It has the right user name in VERP, X-Failed-Recipients and in body text. It should be handled as a bounce from v.tretyakov@krista.ru in sympa-test@krista.ru mailing list. User's bounce count in list subscription should increment and show that message as last bounce.

(I got this behavior with my ProcessBounce.pm)

Actual behavior

Sympa sees none of addresses mentioned above (also code tries to parse all) and puts it into spool/bounce/bad, as an unknown bounce. The following messages show in log:

Oct 17 16:57:03 sympa bounced[2395]: notice Sympa::Spindle::ProcessBounce::_twist() Processing Sympa::Message sympa@krista.ru.1539784620.2679; envelope_sender=; message_id=E1gCmJn-000DNA-Bw@mailx.krista.ru; sender=mailer-daemon@krista.ru
Oct 17 16:57:03 sympa bounced[2395]: notice Sympa::Spindle::ProcessBounce::_twist() VERP in use: bounce Sympa::Message sympa@krista.ru.1539784620.2679 related to v.tretyakov@krista.ru for list Sympa::List sympa-test@krista.ru
Oct 17 16:57:03 sympa bounced[2395]: info Sympa::Spindle::ProcessBounce::_twist() No address found in message Sympa::Message sympa@krista.ru.1539784620.2679

Additional information

Please note that Sympa does see the bouncer's address in VERP, but doen't use it unless body of a message is successfully parsed. I am not sure if there are reasons for such behaviour. My version doesn't change it.

[racke]

Sisimai seems to parse it correctly:

MX: mailx.krista.ru
Domain: krista.ru
Reason: mailboxfull
Advice: Your mailbox v.tretyakov@krista.ru has been reported to be full by your email provider krista.ru.
Subject: [sympa-test] test 8
Recipient: v.tretyakov@krista.ru
Host: krista.ru
Bounce date: 2018-10-17
Reply code: 552
Delivery status: 5.2.2
Diagnostics: LMTP error after end of data: 552 5.2.2 Over quota SESSIONID=

It would be good to update ProcessBounce so it catches it properly.

[ikedas]

Hi @popov-d,
I agree to incorporate your update.

So if possible, could you please submit this change as a PR? It will help accurate works.
If you have difficulty to submit PR, I'll work with the patch you attached.

[ikedas]

Please note that Sympa does see the bouncer's address in VERP, but doen't use it unless body of a message is successfully parsed. I am not sure if there are reasons for such behaviour. My version doesn't change it.

I'm not sure, but Sympa looks parsing message body to get status of delivery failure, not only bouncing addresses.

[popov-d]

Thank you, ikedas. Pull request is in #450.

Is I understand now, «bounce messages» may be not just errors, but also delay warnings, some kinds of automatic replies, etc. Sympa should not treat them all as user errors. It parses message to see if it's surely an error.

What I catch here is an SMTP/LMTP error in EXIM and sometimes with error code inside; I take it also, if available.

[racke]

Correct, Sympa has to sort out all emails coming to the bounce address and qualify them.

[ikedas]

Merged. @popov-d, thanks for improving Sympa!