Compilation failure on MSVC when changing the default calling convention

[marzer]

Hello,

I'm trying to use meshoptimizer in a Visual Studio project that overrides the default calling convention with __vectorcall (/Gv), and seeing compilation failures in src/meshoptimizer.h:

1>meshoptimizer.h(732,23): error C2440: 'initializing': cannot convert from 'overloaded-function' to 'void (__vectorcall *)(void *)'
1>meshoptimizer.h(732,23): error C2440: template <typename T> void (*meshopt_Allocator::StorageT<T>::deallocate)(void*) = operator delete;

This appears to be because the MSVC C++ libs explicitly mark operator new and operator delete as __cdecl, rather than implicitly picking up the calling convention from the compiler settings. Since your allocator/deallocate free-function pointers don't specify a calling convention, they get __vectorcall implicitly in this scenario, and thus become incompatible with operator new.

Explicitly changing the calling convention back to __cdecl for the translation unit allows it to compile OK, but then causes linker errors because the other translation units that include meshoptimizer.h are still set to __vectorcall. The only path I can see to using the library in its current form is to change the whole project back to __cdecl, but that comes at a runtime performance cost I'd like to avoid.

(I could also get around it by wrapping the library it in a separately-compiled library and forwarding all the calls through shims with explicit calling conventions, but that's clearly a big headache.)

Would you be amenable to adding a user-configurable #define for explicitly specifying the calling convention of alloc+dealloc function pointers? I'd be happy to prepare a PR for this myself.

[zeux]

Yeah feel free to submit a PR for this. One alternative solution I can see is using extra wrapper functions inside meshopt_Allocator, eg

void* defaultAllocate(size_t size) { return operator new(size); }