SOAP API output without utf8

[acasadoual]

When I subscribe a user with the SOAP API the email arrives with the wrong accents in ANSI. Example:

Esto es una prueba de acentos á é ñ í ó ú

Version

Sympa 6.2.68, CentOS 7.9, httpd-2.4, php-5.4

Installation method

rpm fedora-epel

Expected behavior

email with utf8 encoding

Actual behavior

email with ansi enconding

Steps to reproduce

subscribe a user with the SOAP API

Additional information

[root@alboran sympa]# grep -i utf8 /usr/libexec/sympa/wwsympa.fcgi
#XXXbinmode STDOUT, ":utf8";
#XXX## Incoming data is utf8-encoded
#XXXbinmode STDIN, ":utf8";
#XXX$in{$p} = Encode::decode('utf8', $in{$p});
return Encode::encode_utf8($s) if Encode::is_utf8($s);
Sympa::Tools::Text::guessed_to_utf8($fn,
Sympa::Tools::Text::guessed_to_utf8($path, @langs);
Sympa::Tools::Text::guessed_to_utf8($fn,
Encode::decode_utf8($in{'subject'}),
$message->reformat_utf8_message([], 'utf-8');
$message->reformat_utf8_message([], $charset);
[root@alboran sympa]# grep -i utf8 /usr/libexec/sympa/sympa_soap_server.fcgi
[root@alboran sympa]#

[ikedas]

@acasadoual , could you please show actual inputs in UTF-8 encoding and actual outputs? "Esto es una prueba de acentos á é ñ í ó ú" (guessed input from your example) doesn't look like an e-mail address.

[acasadoual]

The inputs is a tt2 file.

# cat /var/lib/sympa/list_data/lista4/mail_tt2/welcome.tt2
[%# welcome.tt2 ~%]
From: [% fromlist %]
Subject: [%"Welcome to list %1"|loc(list.name)|qencode%]

Esto es una prueba de acentos á é ñ í ó ú

[%|loc(list.name,domain)%]Welcome to list %1@%2[%END%]
[%|loc(user.email)%]Your subscription email is %1[%END%]

[% TRY %]
[% INSERT "info" %]
[% CATCH %]
[% END %]

[% IF conf.wwsympa_url -%]
[%|loc%]The list homepage:[%END%] [% 'info' | url_abs([list.name]) %]
[%|loc%]General information about mailing lists:[%END%] [% 'help' | url_abs(['introduction.html']) %]
[% END -%]

The received mail is attached image:

[acasadoual]

With sympa 6.2.70 it fails too.

[ikedas]

Could you please be more specific about the steps to reproduce?

• What client did you use? sympa_soap_client.pl or any other?
• Which SOAP function did you use, add or subscribe?
  [EDIT]
• If you delete the info file of the list, does the same problem occur?

[acasadoual]

Hi Ikedas,
I used my own soap client in PHP, but sympa_soap_client.pl fails too.
Occurs with all functions that send an email to the user using a mail_tt2 template
add and subscribe (wellcome.tt2)
del and signoff (bye.tt2)

Thanks you
I attach image for bye.tt2

[ikedas]

Hi @acasadoual ,
Could you please apply this patch and check if the problems will be solved?

The patch is against 6.2.72.

As this patch is a bit complicated, if you feel difficulty to apply it, please let me know.

[acasadoual]

How i can to applied this path in RPM installation?

[ikedas]

To confirm if patch can be applied correctly,

# patch -p3 -d /usr/share/sympa/lib --dry-run < patch_file

(when Makefile.am is not found, hit enter key and skip it.)

If no errors are shown, then apply patch actually:

# patch -p3 -d /usr/share/sympa/lib < patch_file

(Also skip Makefile.am.)

Note that, this patch is under review and it should not be applied to production service: It is strongly recommended to apply it to testing environment.

[acasadoual]

That patch works ok.

[ikedas]

Good news. Thank you @acasadoual !

Does the patch also solve the problem reported on #1682 ?

[acasadoual]

NO, only #1541 (this error)

[acasadoual]

For #1682

Solution:

In line 1528 of /usr/share/sympa/lib/Sympa/WWW/Soap.pm

'- push @ALL, Encode::decode_utf8($k . '=' . $data->{$k});'
'+ push @ALL, $k . '=' . $data->{$k};'

[ikedas]

No, these lines were erased by the patch you applied. Lines around 1528 are now as below:

1524     unless (ref($data) eq 'HASH') {
1525         return undef;
1526     }
1527 
1528     if ($format eq 'as_string') {
1529         return Sympa::WWW::SOAP::Data->type('string')
1530             ->value(join ';',
1531             map { sprintf '%s=%s', $_, $data->{$_} } sort keys %$data);

[acasadoual]

It's true.

Now, all commands of "sympa test soap" are in warning with utf8 character:

# sympa test soap --trusted_application=myapp --trusted_application_password='mypass' --service=info --proxy_vars='USER_EMAIL=myuser@ual.es' --service_parameters='the' https://alboran.ual.es/mysoap            calling authenticateRemoteAppAndRun( myapp, mypass USER_EMAIL=myuser@ual.es,info,the)
0
        _homepage_
                'https://alboran.ual.es/sympa/info/the'
        _info_
                'Lista de distribución del área de Teoría e Historia de la Educación.
Suscripción cerrada.'
        _isOwner_
                '1'
        _listAddress_
                'the@lista.ual.es'
        _subject_
"\x{00c1}" does not map to utf8 at /usr/share/sympa/lib/Sympa/CLI/test/soap.pm line 280.
"\x{00ed}" does not map to utf8 at /usr/share/sympa/lib/Sympa/CLI/test/soap.pm line 280.
"\x{00f3}" does not map to utf8 at /usr/share/sympa/lib/Sympa/CLI/test/soap.pm line 280.
                '\xC1rea Teor\xEDa e Historia de la Educaci\xF3n'

[acasadoual]

only "subject" field, the "info" field is ok

[ikedas]

It's true.

What is true? The patch erased those lines?

Now, all commands of "sympa test soap" are in warning with utf8 character:

What are "all commands"? Could you please show the all command lines you executed, and some examples of the actual results?

[acasadoual]

It's true.

What is true? The patch erased those lines?
Yes, the patch erased those lines.

Now, all commands of "sympa test soap" are in warning with utf8 character:

What are "all commands"? Could you please show the all command lines you executed, and some examples of the actual results?

All commands with subject results:
which
complexWhich
lists
complexLists
info

Example complexWhich:

162
        _homepage_
                'https://alboran.ual.es/sympa/info/alumnos_master_auditoria_cuentas'
        _isEditor_
                '0'
        _isOwner_
                '1'
        _isSubscriber_
                '0'
        _listAddress_
                'alumnos_master_auditoria_cuentas@lista.ual.es'
        _subject_
"\x{00e1}" does not map to utf8 at /usr/share/sympa/lib/Sympa/CLI/test/soap.pm line 280.
"\x{00ed}" does not map to utf8 at /usr/share/sympa/lib/Sympa/CLI/test/soap.pm line 280.
                'M\xE1ster en Auditor\xEDa de Cuentas'

Example which:

162
"\x{00e1}" does not map to utf8 at /usr/share/sympa/lib/Sympa/CLI/test/soap.pm line 280.
"\x{00ed}" does not map to utf8 at /usr/share/sympa/lib/Sympa/CLI/test/soap.pm line 280.
        'homepage=https://alboran.ual.es/sympa/info/alumnos_master_auditoria_cuentas;isEditor=0;isOwner=1;isSubscriber=0;listAddress=alumnos_master_auditoria_cuentas@lista.ual.es;subject=M\xE1ster en Auditor\xEDa de Cuentas'

Example complexLIsts:

139
        _homepage_
                'https://alboran.ual.es/sympa/info/profesores_master_tec_aplicadas_ing_informatica'
        _listAddress_
                'profesores_master_tec_aplicadas_ing_informatica@lista.ual.es'
        _subject_
"\x{00e1}" does not map to utf8 at /usr/share/sympa/lib/Sympa/CLI/test/soap.pm line 280.
"\x{00ed}" does not map to utf8 at /usr/share/sympa/lib/Sympa/CLI/test/soap.pm line 280.
"\x{00ed}" does not map to utf8 at /usr/share/sympa/lib/Sympa/CLI/test/soap.pm line 280.
"\x{00e1}" does not map to utf8 at /usr/share/sympa/lib/Sympa/CLI/test/soap.pm line 280.
                'Profesores con docencia en m\xE1ster tecnolog\xEDas y aplicaciones en ingenier\xEDa inform\xE1tica'

Example lists:

138
"\x{00f3}" does not map to utf8 at /usr/share/sympa/lib/Sympa/CLI/test/soap.pm line 280.
"\x{00f1}" does not map to utf8 at /usr/share/sympa/lib/Sympa/CLI/test/soap.pm line 280.
"\x{00ed}" does not map to utf8 at /usr/share/sympa/lib/Sympa/CLI/test/soap.pm line 280.
        'homepage=https://alboran.ual.es/sympa/info/profesores_master_representacionydiseno;listAddress=profesores_master_representacionydiseno@lista.ual.es;subject=Profesores con docencia en master representaci\xF3n y dise\xF1o en Ingenier\xEDa y Arquitectura'

[ikedas]

@acasadoual ,
Could you please send me the config file of the list where the problem is occurring (alumnos_master_auditoria_cuentas)? I'll confirm using that file.

Note that, please do not copy and paste content of the file, but send it as a file attachment to the email. The email address is found here.

[acasadoual]

I sent you the file.
Thanks.

[ikedas]

It turned out that the problem about "not map to utf8" is the bug in XML-Parser CPAN module:
cpan-authors/XML-Parser#30 .

So we won't try to fix it anymore.