ARM64 macOS variadic arguments not passed properly in ctypes

[Rapptz]

Bug report

Using ctypes with variadic functions on ARM64 macOS machines seems to improperly pass the arguments, leading to truncation.

Minimal repro:

>>> import ctypes
>>> from ctypes import util
>>> libc = ctypes.CDLL(util.find_library("c"))
>>> libc.printf(b"hello %d world\n", 128_000_000)
hello 0 world
14

This happens regardless of casting it to a ctypes.c_int explicitly or not.

>>> libc.printf(b"hello %ld world\n", ctypes.c_int(1))
hello 0 world
14

On my regular machine (in this case an x64 Windows machine) it works as expected:

>>> import ctypes
>>> libc = ctypes.cdll.msvcrt
>>> libc.printf(b"hello %d world\n", 128_000_000)
hello 128000000 world
22

Your environment

I do not personally have a macOS machine, but I got a few others who did have a machine test for me. Their versions were as follows:

Machine 1:
Python 3.10.1, macOS 12.3.1 (21E258)

Machine 2:

Python 3.9.13 (v3.9.13:6de2ca5339, May 17 2022, 11:37:23) 
[Clang 13.0.0 (clang-1300.0.29.30)] on darwin
macOS-12.2.1-arm64-arm-64bit

Machine 3:

~ % python3 --version
Python 3.9.10

~ % sw_vers
ProductName:     macOS
ProductVersion:  12.3.1
BuildVersion:    21E258

• CPython versions tested on: 3.9, 3.10
• Operating system and architecture: ARM64 Apple Silicon macOS

Linked PRs

• GH-92892: Add section about variadic functions to ctypes documentation #99529
• [3.10] GH-92892: Add section about variadic functions to ctypes documentation (GH-99529) #99681
• [3.11] GH-92892: Add section about variadic functions to ctypes documentation (GH-99529) #99682

[ronaldoussoren]

For variadic functions on macOS/arm64 you must explicitly annotate the fixed arguments to the variadic function (for printf the format string argument).

The reason for that is that the macOS/arm64 calling convention is different between plain and variadic functions (for the variadic arguments themselves) and types switches between the to based on the number of arguments passed to the function.

This probably needs some documentation.

P.S. Sorry about the vague language, I'll write something more coherent later.

[ronaldoussoren]

To expand on my previous comment, the following code works as expected:

import ctypes
from ctypes import util
libc = ctypes.CDLL(util.find_library("c"))
libc.printf.argtypes = [ctypes.c_char_p].   # <--- argument type spec
libc.printf(b"hello %d world\n", 128_000_000)

The marked line just before calling printf tells ctypes about the expected types and number of fixed arguments, and that way the ctypes implementation knows that the additional integer argument should be passed using the varargs convention which is different than the argument passing convention for normal functions.

[ronaldoussoren]

I'm not sure how to properly document this requirement, probably a new subsection in the "Calling Functions" section?

[m13253]

Thanks for the explanation. With your information, I fixed the original downstream issue.

I agree that we need to document this requirement. This is both a portability issue and a security issue, because the wrong value is passed into the external function without being noticed or detected.

I would suggest that we “recommend every variadic function to get its arguments specified”, because specifying the arguments does no harm on other platforms.

[m13253]

For anyone coming to this page in the future, here is the Apple official documentation:

https://developer.apple.com/documentation/xcode/writing-arm64-code-for-apple-platforms#Update-Code-that-Passes-Arguments-to-Variadic-Functions

(Read the section named “Update Code that Passes Arguments to Variadic Functions”)

[Rapptz]

Thanks for the explanation and the response, it's greatly appreciated!

I'm not sure how to properly document this requirement, probably a new subsection in the "Calling Functions" section?

Maybe near the bottom of this section https://docs.python.org/3/library/ctypes.html#calling-functions-continued an admonition detailing what needs to be done for ARM64 could be written? Something like...

.. warning::

    When calling variadic functions (such as ``printf``) certain platforms, such as ARM64 for Apple platforms, require specifying ``argtypes`` for the non-variadic parameters. In the case of ``printf`` this would be the ``const char*`` initial parameter.

    .. code-block:: python3

        libc.printf.argtypes = [ctypes.c_char_p].   # <--- argument type spec
    
    Failure to do this will result in undesirable behavior. Since specifying ``argtypes`` doesn't inhibit portability, it's recommended to do so for all variadic functions.

I think this would give it enough attention.

[ronaldoussoren]

Something like that, although I'd prefer a new heading about calling variadic functions instead of inserting a warning about macOS.

[ronaldoussoren]

Note that this is a duplicate of #87046

[ronaldoussoren]

I've created a PR that adds a section about this to the ctypes documentation.