Segmentation errors for gfal-copy's GridFTP plugin on RHEL8

[XMol]

(moved from cern-fts/gfal2-util)

Hello GFAL developers,

we've installed a new VM with RHEL-8 to run our regular transfer tests using the gfal2-tools. They used to run just fine on SL-7, but now our gfal-copy commands randomly fail with Segmentation faults exclusively with the gsiftp transfer protocol.

For some reason that I'm unable to figure out, no core dump files are produced, while that should be possible (I can generate one myself).

$ grep systemd-coredump /var/log/messages | tail
Dec 19 23:16:02 gm-1-kit-e systemd-coredump[3646763]: Process 3646718 (python3) of user 1000 dumped core.
Dec 19 23:16:02 gm-1-kit-e systemd[1]: systemd-coredump@258-3646762-0.service: Succeeded.
Dec 20 01:16:02 gm-1-kit-e systemd-coredump[3658820]: Resource limits disable core dumping for process 3658734 (python3).
Dec 20 01:16:02 gm-1-kit-e systemd-coredump[3658820]: Process 3658734 (python3) of user 1000 dumped core.
Dec 20 01:16:02 gm-1-kit-e systemd[1]: systemd-coredump@259-3658819-0.service: Succeeded.
Dec 20 08:31:02 gm-1-kit-e systemd-coredump[3702089]: Resource limits disable core dumping for process 3702066 (python3).
Dec 20 08:31:02 gm-1-kit-e systemd-coredump[3702089]: Process 3702066 (python3) of user 1000 dumped core.
Dec 20 08:31:02 gm-1-kit-e systemd[1]: systemd-coredump@260-3702087-0.service: Succeeded.
Dec 20 08:39:36 gm-1-kit-e systemd-coredump[3703091]: Process 3703089 (sleep) of user 1000 dumped core.#012#012Stack trace of thread 3703089:#012#0  0x00007fe5fe35d8e8 __nanosleep (libc.so.6)#012#1  0x0000564612d25b47 rpl_nanosleep (sleep)#012#2  0x0000564612d25920 xnanosleep (sleep)#012#3  0x0000564612d22a88 main (sleep)#012#4  0x00007fe5fe29ed85 __libc_start_main (libc.so.6)#012#5  0x0000564612d22b5e _start (sleep)
Dec 20 08:39:36 gm-1-kit-e systemd[1]: systemd-coredump@261-3703090-0.service: Succeeded.

Other information you might find useful...

$ python3 --version
Python 3.6.8

$ rpm -qa gfal2*
gfal2-plugin-file-2.21.5-1.el8.x86_64
gfal2-plugin-gridftp-2.21.5-1.el8.x86_64
gfal2-2.21.5-1.el8.x86_64
gfal2-plugin-http-2.21.5-1.el8.x86_64
gfal2-plugin-srm-2.21.5-1.el8.x86_64
gfal2-plugin-xrootd-2.21.5-1.el8.x86_64
gfal2-util-scripts-1.8.0-1.el8.noarch

$ uname -a
Linux gm-1-kit-e 4.18.0-513.5.1.el8_9.x86_64 #1 SMP Fri Sep 29 05:21:10 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux

Do you have other hints on what we should check to find out more information for you? If you know how to ensure core dumps are actually produced, that would be worth sharing.

Best regards,
Xavier.

[mpatrascoiu]

Hello Xavier,

Repeating my message from cern-fts/gfal2-util#6 , given this is about GridFTP, it's not something we want to deeply investigate. The Gfal2 plan is to abandon GridFTP support.

Maybe I can help you get a stacktrace. Given the gfal2-util clients are shell wrappers, maybe you can try to do the copy via Python3 directly:

import gfal2

ctx = gfal2.creat_context()
params = ctx.transfer_parameters()

ctx.filecopy(params, "src", "dst")

Maybe this way, you'll have better chances of producing a coredump.
Once the coredump is available, we can look whether it's something evident in Gfal2 or coming from the underlying GridFTP library. If it's in Gfal2, I'm interested whether this could affect the other plugins (e.g.: http, xrootd).

Cheers,
Mihai

[XMol]

Thank you for the suggestion, Mihai. I'll give it a try.

[XMol]

This small Python script was running for about 90 minutes without breaking due to a segmentation error...

import gfal2

ctx = gfal2.creat_context()
params = ctx.transfer_parameters()

ctx.filecopy(params, 'file:////bin/bash', 'gsiftp://ppsgridftp-kit.gridka.de:2811//pnfs/gridka.de/dteam/disk-only/force-core-dump.tmp')
while True:
  ctx.filecopy(params, 'gsiftp://ppsgridftp-kit.gridka.de:2811//pnfs/gridka.de/dteam/disk-only/force-core-dump.tmp', 'file:////dev/null')

Anyway, our regular workflow (miraculously) produced three core dump files, of which I've attached the human readable information produced by coredumpctl (gfal2.coredumpctl.txt). If you need the actual core dump files, I can provide them. If you need me to do something specific with the core dump files, please tell me.

Best regards,
Xavier.

[mpatrascoiu]

Hello,

It does look like a double free for the same GSI security context:

                Stack trace of thread 3717519:
                #0  0x00007f19af2bfe31 __libc_free (libc.so.6)
                #1  0x00007f19b05648ec ASN1_OBJECT_free (libcrypto.so.1.1)
                #2  0x00007f19b0575198 asn1_primitive_free.localalias.0 (libcrypto.so.1.1)
                #3  0x00007f19b0575618 asn1_template_free (libcrypto.so.1.1)
                #4  0x00007f19b05752d6 asn1_item_embed_free (libcrypto.so.1.1)
                #5  0x00007f19b0575618 asn1_template_free (libcrypto.so.1.1)
                #6  0x00007f19b05752d6 asn1_item_embed_free (libcrypto.so.1.1)
                #7  0x00007f19b0575539 ASN1_item_free (libcrypto.so.1.1)
                #8  0x00007f199917d697 globus_gsi_proxy_handle_destroy (libglobus_gsi_proxy_core.so.0)
                #9  0x00007f199ac7db8d gss_delete_sec_context (libglobus_gssapi_gsi.so.4)

Could you provide the coredump for gfal-rm -vvv -t 20 gsiftp://atlasgridftp-kit.gridka.de:2811/[..] ?

Cheers,
Mihai

[XMol]

Here you go
core.python3.1000.1b33bd67a03e41faa9fc307a01ea236f.3717501.1703065389000000.lz4.txt. Please remove/ignore the .txt suffix when using the file. I had to append it because GitHub doesn't allow me to upload .lz4.

[XMol]

Hi @mpatrascoiu,

did the coredump help you in any way?

Ciao,
Xavier.

[XMol]

Hi @mpatrascoiu,

the core dumps keep happening for our setup; around 10 times per day, which is less than 1 % of all tests, but still...

Of course, you are free to simply reject the issue on the basis that nobody cares about either gsiftp or RHEL-8 anymore. Simply close this issue and I'll stop bothering you, since we've implemented retries in case of segfault errors in our tests.

Kind regards,
Xavier.