Skip to content

Mail Analyzing Interface for email bounce: A Perl module to parse RFC5322 bounce mails and generating structured data as JSON from parsed results. Formerly known as bounceHammer 4: an error mail analyzer.

License

Notifications You must be signed in to change notification settings

sisimai/p5-sisimai

Repository files navigation

License Perl CPAN codecov

Important

The default branch of this repository is 5-stable (Sisimai 5) since 2nd February 2024. If you want to clone the old version, see the 4-stable1 branch instead. We have moved away from using both the main and master branches in our development process.

Caution

Sisimai versions 4.25.14p11 and earlier contain a regular expression vulnerability ReDoS: CVE-2022-4891. If you are using one of these versions, please upgrade to v4.25.14p12 or later.

Warning

Sisimai 5 requires Perl 5.26 or later. Check the version of Perl in your system before installing/upgrading by perl -v command.

Note

Sisimai is a Perl module or Ruby Gem, but it can be used in any environment where JSON can be read, such as PHP, Python, Go, and Rust. By obtaining the analysis results, it is very useful for understanding the bounce occurrence status.

What is Sisimai

Sisimai is a library that decodes complex and diverse bounce emails and outputs the results of the delivery failure, such as the reason for the bounce and the recipient email address, in structured data. It is also possible to output in JSON format.

The key features of Sisimai

  • Decode email bounces to structured data
    • Sisimai provides detailed insights into bounce emails by extracting 24 key data points.2
      • Essential information: timestamp, origin
      • Sender information: addresser, senderdomain,
      • Recipient information: recipient, destination, alias
      • Delivery information: action, replycode,action, replycode, deliverystatus
      • Bounce details: reason, diagnosticcode, diagnostictype, feedbacktype, hardbounce
      • Message details: subject, messageid, listid,
      • Additional information: smtpagent, timezoneoffset, lhost, rhost, token, catch
    • Output formats
      • Perl (Hash, Array)
      • JSON (by using JSON module)
      • YAML (YAML module or YAML::Syck module required)
  • Easy to Install, Use.
    • cpan, cpanm, or cpm
    • git clone & make
  • High Precision of Analysis

Command line demo

The following screen shows a demonstration of dump method of Sisimai 5 at the command line using Perl(p5-sisimai) and jq command.

Setting Up Sisimai

System requirements

More details about system requirements are available at Sisimai | Getting Started page.

Install

From CPAN

$ cpanm --sudo Sisimai
--> Working on Sisimai
Fetching http://www.cpan.org/authors/id/A/AK/AKXLIX/Sisimai-4.25.16.tar.gz ... OK
...
1 distribution installed
$ perldoc -l Sisimai
/usr/local/lib/perl5/site_perl/5.30.0/Sisimai.pm

From GitHub

Warning

Sisimai 5 requires Perl 5.26 or later. Check the version of Perl in your system before installing/upgrading by perl -v command.

$ perl -v

This is perl 5, version 30, subversion 0 (v5.30.0) built for darwin-2level

Copyright 1987-2019, Larry Wall
...

$ cd /usr/local/src
$ git clone https://github.com/sisimai/p5-sisimai.git
$ cd ./p5-sisimai

$ make install-from-local
./cpanm --sudo . || ( make cpm && ./cpm install --sudo -v . )
--> Working on .
Configuring Sisimai-v5.1.0 ... OK
Building and testing Sisimai-v5.1.0 ... Password: <sudo password here>
OK
Successfully installed Sisimai-v5.1.0
1 distribution installed

$ perl -MSisimai -lE 'print Sisimai->version'
5.1.0

Usage

Basic usage

Sisimai->rise() method provides the feature for getting decoded data as Perl Hash reference from bounced email messages as the following. Beginning with v4.25.6, new accessor origin which keeps the path to email file as a data source is available.

#! /usr/bin/env perl
use Sisimai;
my $v = Sisimai->rise('/path/to/mbox'); # or path to Maildir/


# In v4.23.0, the rise() and dump() methods of the Sisimai class can now read the entire bounce
# email as a string, in addition to the PATH to the email file or mailbox.
use IO::File;
my $r = '';
my $f = IO::File->new('/path/to/mbox'); # or path to Maildir/
{ local $/ = undef; $r = <$f>; $f->close }
my $v = Sisimai->rise(\$r);

# If you also need analysis results that are "delivered" (successfully delivered), please specify
# the "delivered" option to the rise() method as shown below.
my $v = Sisimai->rise('/path/to/mbox', 'delivered' => 1);

# From v5.0.0, Sisimai no longer returns analysis results with a bounce reason of "vacation" by
# default. If you also need analysis results that show a "vacation" reason, please specify the
# "vacation" option to the rise() method as shown in the following code.
my $v = Sisimai->rise('/path/to/mbox', 'vacation' => 1);

if( defined $v ) {
    for my $e ( @$v ) {
        print ref $e;                   # Sisimai::Fact
        print ref $e->recipient;        # Sisimai::Address
        print ref $e->timestamp;        # Sisimai::Time

        print $e->addresser->address;   # "michitsuna@example.org" # From
        print $e->recipient->address;   # "kijitora@example.jp"    # To
        print $e->recipient->host;      # "example.jp"
        print $e->deliverystatus;       # "5.1.1"
        print $e->replycode;            # "550"
        print $e->reason;               # "userunknown"
        print $e->origin;               # "/var/spool/bounce/new/1740074341.eml"
        print $e->hardbounce;           # 1

        my $h = $e->damn();             # Convert to HASH reference
        my $j = $e->dump('json');       # Convert to JSON string
        print $e->dump('json');         # JSON formatted bounce data
    }
}

Convert to JSON

Sisimai->dump() method provides the feature for getting decoded data as JSON string from bounced email messages like the following code:

#! /usr/bin/env perl
use Sisimai;

# Get JSON string from path of a mailbox or a Maildir/
my $j = Sisimai->dump('/path/to/mbox'); # or path to Maildir/
                                        # dump() is added in v4.1.27
print $j;                               # decoded data as JSON

# dump() method also accepts "delivered" and "vacation" option like the following code:
my $j = Sisimai->dump('/path/to/mbox', 'delivered' => 1, 'vacation' => 1);

Callback feature

c___ (c and three _s, looks like a fishhook) argument of Sisimai->rise and Sisimai->dump is an array reference and is a parameter to receive code references for callback feature. The first element of c___ argument is called at Sisimai::Message->sift for dealing email headers and entire message body. The second element of c___ argument is called at the end of each email file processing. The result generated by the callback method is accessible via Sisimai::Fact->catch.

[0] For email headers and the body

Callback method set in the first element of c___ is called at Sisimai::Message->sift().

#! /usr/bin/env perl
use Sisimai;
my $code = sub {
    my $args = shift;               # (*Hash)
    my $head = $args->{'headers'};  # (*Hash)  Email headers
    my $body = $args->{'message'};  # (String) Message body
    my $adds = { 'x-mailer' => '', 'queue-id' => '' };

    if( $body =~ m/^X-Postfix-Queue-ID:\s*(.+)$/m ) {
        $adds->{'queue-id'} = $1;
    }

    $adds->{'x-mailer'} = $head->{'x-mailer'} || '';
    return $adds;
};
my $data = Sisimai->rise('/path/to/mbox', 'c___' => [$code, undef]);
my $json = Sisimai->dump('/path/to/mbox', 'c___' => [$code, undef]);

print $data->[0]->catch->{'x-mailer'};    # "Apple Mail (2.1283)"
print $data->[0]->catch->{'queue-id'};    # "43f4KX6WR7z1xcMG"

[1] For each email file

Callback method set in the second element of c___ is called at Sisimai->rise() method for dealing each email file.

my $path = '/path/to/maildir';
my $code = sub {
    my $args = shift;           # (*Hash)
    my $kind = $args->{'kind'}; # (String)  Sisimai::Mail->kind
    my $mail = $args->{'mail'}; # (*String) Entire email message
    my $path = $args->{'path'}; # (String)  Sisimai::Mail->path
    my $fact = $args->{'fact'}; # (*Array)  List of Sisimai::Fact

    for my $e ( @$fact ) {
        # Store custom information in the "catch" accessor.
        $e->{'catch'} ||= {};
        $e->{'catch'}->{'size'} = length $$mail;
        $e->{'catch'}->{'kind'} = ucfirst $kind;

        if( $$mail =~ /^Return-Path: (.+)$/m ) {
            # Return-Path: <MAILER-DAEMON>
            $e->{'catch'}->{'return-path'} = $1;
        }

        # Save the original email with an additional "X-Sisimai-Parsed:" header to a different PATH.
        my $a = sprintf("X-Sisimai-Parsed: %d\n", scalar @$fact);
        my $p = sprintf("/path/to/another/directory/sisimai-%s.eml", $e->token);
        my $f = IO::File->new($p, 'w');
        my $v = $$mail; $v =~ s/^(From:.+)$/$a$1/m;
        print $f $v; $f->close;
    }

    # Remove the email file in Maildir/ after decoding
    unlink $path if $kind eq 'maildir';

    # Need to not return a value
};

my $list = Sisimai->rise($path, 'c___' => [undef, $code]);
print $list->[0]->{'catch'}->{'size'};          # 2202
print $list->[0]->{'catch'}->{'kind'};          # "Maildir"
print $list->[0]->{'catch'}->{'return-path'};   # "<MAILER-DAEMON>"

More information about the callback feature is available at Sisimai | How To Parse - Callback Page.

One-Liner

Beginning with Sisimai 4.1.27, Sisimai->dump() method is available and you can get decoded data as JSON using the method.

$ perl -MSisimai -lE 'print Sisimai->dump(shift)' /path/to/mbox

Output example

[
  {
    "destination": "google.example.com",
    "lhost": "gmail-smtp-in.l.google.com",
    "hardbounce": 0,
    "reason": "authfailure",
    "catch": null,
    "addresser": "michitsuna@example.jp",
    "alias": "nekochan@example.co.jp",
    "smtpagent": "Postfix",
    "smtpcommand": "DATA",
    "senderdomain": "example.jp",
    "listid": "",
    "action": "failed",
    "feedbacktype": "",
    "messageid": "hwK7pzjzJtz0RF9Y@relay3.example.com",
    "origin": "./gmail-5.7.26.eml",
    "recipient": "kijitora@google.example.com",
    "rhost": "gmail-smtp-in.l.google.com",
    "subject": "Nyaan",
    "timezoneoffset": "+0900",
    "replycode": 550,
    "token": "84656774898baa90660be3e12fe0526e108d4473",
    "diagnostictype": "SMTP",
    "timestamp": 1650119685,
    "diagnosticcode": "host gmail-smtp-in.l.google.com[64.233.187.27] said: This mail has been blocked because the sender is unauthenticated. Gmail requires all senders to authenticate with either SPF or DKIM. Authentication results: DKIM = did not pass SPF [relay3.example.com] with ip: [192.0.2.22] = did not pass For instructions on setting up authentication, go to https://support.google.com/mail/answer/81126#authentication c2-202200202020202020222222cat.127 - gsmtp (in reply to end of DATA command)",
    "deliverystatus": "5.7.26"
  }
]

Differences between Sisimai 4 and Sisimai 5

The following table show the differences between Sisimai 4.25.16p1 and Sisimai 5. More information about differences are available at Sisimai | Differences page.

Features

Beginning with v5.0.0, Sisimai requires Perl 5.26.0 or later.

Features Sisimai 4 Sisimai 5
System requirements (Perl) 5.10 - 5.38 5.26 - 5.38
Callback feature for the original email file N/A Available3
The number of MTA/ESP modules 68 73
The number of detectable bounce reasons 29 34
Dependencies (Except core modules of Perl) 2 modules 2 modules
Source lines of code 10,800 lines 11,800 lines
The number of tests in t/, xt/ directory 270,000 tests 335,000 tests
The number of bounce emails decoded per second4 541 emails 660 emails
License 2 Clause BSD 2 Caluse BSD
Commercial support Available Available

Decoding Method

Some decoding method names, class names, parameter names have been changed at Sisimai 5. The details of the decoded data are available at LIBSISIMAI.ORG/EN/DATA

Decoding Method Sisimai 4 Sisimai 5
Decoding method name Sisimai->make Sisimai->rise
Dumping method name Sisimai->dump Sisimai->dump
Class name of decoded object Sisimai::Data Sisimai::Fact
Parameter name of the callback hook c___5
Method name for checking the hard/soft bounce softbounce hardbounce
Decode a vacation message by default Yes No
Sisimai::Message returns an object Yes No
MIME decoding class Sisimai::MIME Sisimai::RFC2045
Decoding transcript of SMTP session No Yes6

MTA/ESP Module Names

Three ESP module names have been changed at Sisimai 5. The list of the all MTA/ESP modules is available at LIBSISIMAI.ORG/EN/ENGINE

Sisimai:: Sisimai 4 Sisimai 5
Apple iCloud Mail (added at v5.1.0) None Rhost::Apple
Microsoft Exchange Online Rhost::ExchangeOnline Rhost::Microsoft
Google Workspace Rhost::GoogleApps Rhost::Google
Tencent Rhost::TencentQQ Rhost::Tencent
Yahoo Mail (added at v5.1.0) None Rhost::YahooInc
DragonFly Mail Agent (added at v5.1.0) None Lhost::DragonFly

Bounce Reasons

Five bounce reasons have been added at Sisimai 5. The list of the all bounce reasons sisimai can detect is available at LIBSISIMAI.ORG/EN/REASON

Rejected due to Sisimai 4 Sisimai 5
sender domain authentication(SPF,DKIM,DMARC) SecurityError AuthFailure
low/bad reputation of the sender hostname/IP addr. Blocked BadReputation
missing PTR/having invalid PTR Blocked RequirePTR
non-compliance with RFC7 SecurityError NotCompliantRFC
exceeding a rate limit or sending too fast SecurityError Speeding

Contributing

Bug report

Please use the issue tracker to report any bugs.

Emails could not be decoded

Bounce mails which could not be decoded by Sisimai are saved in the repository set-of-emails/to-be-debugged-because/sisimai-cannot-parse-yet. If you have found any bounce email cannot be decoded using Sisimai, please add the email into the directory and send Pull-Request to this repository.

Other Information

Related sites

See also

Author

@azumakuniyuki

Copyright

Copyright (C) 2014-2024 azumakuniyuki, All Rights Reserved.

License

This software is distributed under The BSD 2-Clause License.

Footnotes

  1. Specify -b 4-stable when you clone Sisimai 4 for example, git clone -b 4-stable https://github.com/sisimai/p5-sisimai.git

  2. The callback function allows you to add your own data under the catch accessor.

  3. The 2nd argument of c___ parameter at Sisimai->rise method

  4. macOS Monterey/1.6GHz Dual-Core Intel Core i5/16GB-RAM/Perl 5.30

  5. c___ looks like a fishhook

  6. Sisimai::SMTP::Transcript->rise Method provides the feature

  7. RFC5322 and related RFCs

About

Mail Analyzing Interface for email bounce: A Perl module to parse RFC5322 bounce mails and generating structured data as JSON from parsed results. Formerly known as bounceHammer 4: an error mail analyzer.

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages