Errors compiling on Windows with VS 2022 17.6 LTSC

[mihaicodrean]

Bug description:

I'm getting errors while building-without-installer the default configuration in Windows.

How to reproduce:

Run .\build.ps1.

Expected behavior:

No errors.

Errors and logs:

Here's the first error that I get:

121>tsduck\src\libtsduck\plugins\plugins\tsAbstractDuplicateRemapPlugin.cpp(69,5): error C2672: 'ts::Args::getIntValues': no matching overloaded function found [tsduck\scripts\msvc\tsducklib.vcxproj]
   tsduck\src\libtsduck\base\app\tsArgs.h(936,14): message : could be 'void ts::Args::getIntValues(ts::CompactBitSet<BITS,nullptr> &,const ts::UChar *,bool) const' (compiling source file tsduck\src\libtsduck\plugins\plugins\tsAbstractDuplicateRemapPlugin.cpp) [tsduck\scripts\msvc\tsducklib.vcxproj]
   tsduck\src\libtsduck\plugins\plugins\tsAbstractDuplicateRemapPlugin.cpp(69,5): message : 'void ts::Args::getIntValues(ts::CompactBitSet<BITS,nullptr> &,const ts::UChar *,bool) const': could not deduce templateargument for 'ts::CompactBitSet<BITS,nullptr> &' from 'ts::TSPacketLabelSet' [tsduck\scripts\msvc\tsducklib.vcxproj]
   tsduck\src\libtsduck\base\app\tsArgs.h(921,14): message : or       'void ts::Args::getIntValues(std::bitset<_Bits> &,const ts::UChar *,bool) const' (compiling source file tsduck\src\libtsduck\plugins\plugins\tsAbstractDuplicateRemapPlugin.cpp) [tsduck\scripts\msvc\tsducklib.vcxproj]
   tsduck\src\libtsduck\plugins\plugins\tsAbstractDuplicateRemapPlugin.cpp(69,5): message : 'void ts::Args::getIntValues(std::bitset<_Bits> &,const ts::UChar *,bool) const': could not deduce template argument for'std::bitset<_Bits> &' from 'ts::TSPacketLabelSet' [tsduck\scripts\msvc\tsducklib.vcxproj]
     tsDebugPlugin.cpp
   tsduck\src\libtsduck\base\app\tsArgs.h(906,14): message : or       'void ts::Args::getIntValues(std::set<INT,std::less<_Ty>,std::allocator<_Ty>> &,const ts::UChar *) const' (compiling source file tsduck\src\libtsduck\plugins\plugins\tsAbstractDuplicateRemapPlugin.cpp) [tsduck\scripts\msvc\tsducklib.vcxproj]
   tsduck\src\libtsduck\plugins\plugins\tsAbstractDuplicateRemapPlugin.cpp(69,5): message : 'void ts::Args::getIntValues(std::set<INT,std::less<_Ty>,std::allocator<_Ty>> &,const ts::UChar *) const': could not deduce template argument for 'std::set<INT,std::less<_Ty>,std::allocator<_Ty>> &' from 'ts::TSPacketLabelSet' [tsduck\scripts\msvc\tsducklib.vcxproj]
   tsduck\src\libtsduck\base\app\tsArgs.h(894,14): message : or       'void ts::Args::getIntValues(std::vector<CHARTYPE,std::allocator<_Ty>> &,const ts::UChar *) const' (compiling source file tsduck\src\libtsduck\plugins\plugins\tsAbstractDuplicateRemapPlugin.cpp) [tsduck\scripts\msvc\tsducklib.vcxproj]
     tsDropOutputPlugin.cpp
   tsduck\src\libtsduck\plugins\plugins\tsAbstractDuplicateRemapPlugin.cpp(69,5): message : 'void ts::Args::getIntValues(std::vector<CHARTYPE,std::allocator<_Ty>> &,const ts::UChar *) const': could not deduce template argument for 'std::vector<CHARTYPE,std::allocator<_Ty>> &' from 'ts::TSPacketLabelSet' [tsduck\scripts\msvc\tsducklib.vcxproj]

Environment:

• OS: Windows
• OS version: 10
• TSDuck full version: 3.37-3670 (latest tag)
• Installation type: Rebuilt binary

Additional information:

Visual Studio Professional 2022 (64-bit) - LTSC 17.6.

[lelegard]

On an up-to-date Windows system, TSDuck tag v3.37-3670 compiles correctly. Executing build.ps1 compiles the Win64 and Win32 versions without problem. I just tested it.

The configuration is Windows 11 with Visual Studio 2022 Community Edition, version 17.10.3, containing MSVC 19.40.33811.00.

I don't think that Windows 10 vs. 11 matters. But the MSVC compiler is very unstable. Over the years, I have seen many bugs being created, then fixed. Especially when it comes to cryptic overloading interpretations as seen in your error messages.

When TSDuck version 3.37-3670 was built (official build), it used MSVC 19.39.33522.00 (see tsversion --version=all), which is not much different from the latest one. I don't remember which Visual Studio version it was. The compiler can only identify itself during the compilation, not the Visual Studio environment.

From Visual Studio, it is difficult to get the real compiler version. In your environment, get the value of the macro _MSC_FULL_VER (type it in a source file and hover over it to display the value). For instance, for MSVC 19.40.33811.00, _MSC_FULL_VER is 194033811.

If your version is lower than 193933522, the version which was used to build TSDuck v3.37-3670, you probably have a compiler bug and you need to upgrade to the latest version. As a rule of thumb, I have seen so many bugs in MSVC that I recommend to always upgrade to the latest version. Except in the rare cases where the latest version introduces a new bug. It happens...

Otherwise, the same TSDuck source code compiles quite well with all versions of clang and gcc versions 11 and higher (again, gcc versions 8 to 10 had bugs which prevented the compilation of recent versions of TSDuck).

I remove the "bug" tag since this is not a TSDuck bug.

[mihaicodrean]

For _MSC_FULL_VER, I get 193632544 for LTSC 17.6.11. Upgrading to the latest on LTSC, 17.6.16, yields 193632546 for _MSC_FULL_VER, so not much help.

I have a requirement to stay on LTSC, so what do you suggest in this case?

[lelegard]

I have a requirement to stay on LTSC, so what do you suggest in this case?

If you have the requirement to stay with a buggy compiler instead of one that works, there is nothing I can do for you. I have worked long enough in the industry to understand that such requirement may exist. It is difficult to make management understand that such decision is only counter-productive.

At best, you could install an up-to-date compiler on some other PC, even a personal one, and build an installer (and portable one if you need it). And then install these installers on your dev systems.

[mihaicodrean]

At best, you could install an up-to-date compiler on some other PC, even a personal one, and build an installer (and portable one if you need it). And then install these installers on your dev systems.

That could work, thanks for the idea, although another approach is to understand what exactly is the MS compiler missing and assist it.

BTW, if I use the prebuilt x64 Release library from the installer, then I get this error when linking in a test app:
error LNK2001: unresolved external symbol "private: static class ts::CerrReport * ts::CerrReport::_instance" (?_instance@CerrReport@ts@@0REAV12@EA) fatal error LNK1120: 1 unresolved externals
Could this also be related to the compiler version, or is it something else that I'm missing? I doubt I should define that static myself.

[lelegard]

Having the prebuilt TSDuck 3.37-3670 installed, I can link a sample test application which uses CerrReport. Moreover, the symbol which is in error for you is actually exported by tsduck.dll:

PS C:\> tsversion
3.37-3670
PS C:\> dumpbin /exports "C:\Program Files\TSDuck\bin\tsduck.dll" | Select-String _instance@CerrReport@ts@@0REAV12@EA

       7620 1DC3 011BD340 ?_instance@CerrReport@ts@@0REAV12@EA

So, the symbol is exported. I don't think that the compiler version is to blame since the right symbol is looked for.

[lelegard]

Try the dumpbin command on your system. To check that the installation is correct.

[mihaicodrean]

Yes, it's there in the DLL:

dumpbin /exports "tsduck.dll" | grep _instance@CerrReport@ts@@0REAV12@EA
       7620 1DC3 011BD340 ?_instance@CerrReport@ts@@0REAV12@EA

However, I don't think the liker looks at the DLL, but only at the LIB. And for that, I get:

dumpbin /exports "tsduck.lib" | grep _instance@CerrReport@ts@@0REAV12@EA
                  ?_instance@CerrReport@ts@@0REAV12@EA (private: static class ts::CerrReport * ts::CerrReport::_instance)

I can link a sample test application which uses CerrReport.

I'm actually not using CerrReport directly, but have only added a call to ts::IPInitialize().

[lelegard]

@mihaicodrean

That could work, thanks for the idea, although another approach is to understand what exactly is the MS compiler missing and assist it.

I fully support the idea of understanding first, instead of trying something else. However, your have two distinct issues.

1. You definitely need to understand why you have the missing symbol during the link. This is not normal, even with your compiler.
2. However, the other problem, the inability to find the right overload resolution, is clearly a bug in the compiler C++ front end. This kind of thing cannot be fixed without scrapping the buggy compiler and get a fixed one.

In the second case, you get the error while compiling TSDuck. If you rebuild it elsewhere with a good compiler and install it, you should be safe on the short term. However, the overload failure is on common classes of the TSDuck public API. There is still a risk that you run into a similar problem some day with your applications.

From my experience, the MSVC compiler can be extremely dangerous. Over the years, I ran into multiple compilation bugs. When they are obvious, an error message and a failing compilation, as you get, it is not too dangerous because you see it. However, in at least one occurrence, I got a silent code generation bug. This is the developer's nightmare, your source code is correct, your binary works with other compilers and on other OS, but it fails on Windows. After I reported that bug, it took one year for MS to start working on it, and an additional year to get a fix. This is why I consider MSVC as a time bomb and I absolutely recommend to upgrade to the latest version.

[lelegard]

@mihaicodrean

However, I don't think the liker looks at the DLL, but only at the LIB. And for that, I get:

Good point. But, as you observed, the symbol is there and the linker should find it.

I'm actually not using CerrReport directly, but have only added a call to ts::IPInitialize().*

OK, but ts::IPInitialize() uses an optional parameter which defaults to CERR, a macro which expands to the CerrReport singleton instance. So, your binary code references it.

[mihaicodrean]

I fully support the idea of understanding first, instead of trying something else.

I think the compiler fails to determine the value of the template argument N from the CompactBitSet<N> parameter.

template <std::size_t N>
void ts::Args::getIntValues(CompactBitSet<N>& values, const UChar* name, bool defValue) const

For the call:

getIntValues(_setLabels, u"set-label");

_setLabels is TSPacketLabelSet, which is:

using TSPacketLabelSet = CompactBitSet<32>;

So if I change the call to the following, then at least IntelliSense is happy:

getIntValues<32>(_setLabels, u"set-label");

[lelegard]

Interesting. So, I think we all agree to call that a "compiler bug", right ?

[mihaicodrean]

So, I think we all agree to call that a "compiler bug", right ?

I have a feeling that the CompactBitSet is missing something, because there are no issues with bitset, for example:

using PIDSet = std::bitset<PID_MAX>;
PIDSet             _pids {};
args.getIntValues(_pids, u"pid");

template <std::size_t N>
void ts::Args::getIntValues(std::bitset<N>& values, const UChar* name, bool defValue) const

[mihaicodrean]

Or vice-versa actually, something in CompactBitSet that the compiler doesn't quite understand. For example, if I remove the following from the template definition, then the compiler is pleased:

typename std::enable_if<(BITS > 0)>::type* = nullptr

[lelegard]

I have a feeling that the CompactBitSet is missing something, because there are no issues with bitset, for example

There must be something different but I wouldn't say missing. The idea of TSDuck dev is to test on each push on 3 different OS, 3 compilers, 2 levels of C++ standard (17 and 20), 2 architectures (Intel and Arm), 32000 assertions in unit tests, 2000 high-level tests. So, when everything pass everywhere, we may have some confidence that the code is correct, both on a language and functional perspective. If you add that recent versions of MSVC have changed their behaviour and now correctly resolve the overloading, that clearly means that the component which has something missing is the compiler. Whether it is called "long term support contract" is just the demonstration that LTSC is just BS***

[mihaicodrean]

I tend to agree that this is probably a compiler bug after all, because the underlying cause seems to be the condition in the enable_if construct, (BITS > 0). If I change that to (1 > 0), for example, then the type is found.

[mihaicodrean]

Wow, I cannot believe how many bug reports are there for SFINAE / enable_if for MSVC. Case closed then.

What about the issue with the singleton? Shall I create another issue for it, to discuss it further?

[lelegard]

I cannot believe how many bug reports are there for SFINAE / enable_if for MSVC.

😆😆😆 I told you, MSVC is crap...

What about the issue with the singleton? Shall I create another issue for it, to discuss it further?

Let's continue here.

• Did you recompile your test application after installing the new TSDuck version?
• Is there many other references to TSDuck in the application? What about dumpbin /symbols on the application object file? Especially dumpbin /symbols xxx.obj | Select-String _VERSION_.
• Can you reproduce this on a new empty application referencing TSDuck? Just using IPInitialize for instance.

[mihaicodrean]

Did you recompile your test application after installing the new TSDuck version?

Yes. I even tried with a local build, once I removed the problematic-for-my-compiler std::enable_if.

Is there many other references to TSDuck in the application?

No, I'm just starting to use TSDuck, and stumbled upon that linker error.
BTW, the issue is not with just that singleton, but with any of them, for example ts::UID::Instance() triggers a similar complaint:

error LNK2001: unresolved external symbol "private: static class ts::UID * ts::UID::_instance" (?_instance@UID@ts@@0REAV12@EA)

What about dumpbin /symbols on the application object file? Especially dumpbin /symbols xxx.obj | Select-String _VERSION_.

I don't get _VERSION_ there.

Can you reproduce this on a new empty application referencing TSDuck? Just using IPInitialize for instance.

I'll try that evetually.

[mihaicodrean]

Ah, hold on. It appears that _TSDUCKDLL_USE is not defined, despite having referenced "tsduck.props". Digging further...

[mihaicodrean]

My bad - I haven't had "tsduck.props" referenced properly. With that addressed, the linker error(s) are gone. Thanks for all the help & patience!

[mihaicodrean]

One last thing, hopefully, because maybe this is a known issue: as soon as I bring in "tsUString.h", indirectly, via "tsIPProtocols.h", I get this linker error:

error LNK2001: unresolved external symbol "private: static void __cdecl std::basic_string<char16_t,struct std::char_traits<char16_t>,class std::allocator<char16_t> >::_Start_element_lifetimes(char16_t * const,unsigned __int64)" (?_Start_element_lifetimes@?$basic_string@_SU?$char_traits@_S@std@@V?$allocator@_S@2@@std@@$$FCAXQEA_S_K@Z)

It doesn't appear to be a case of different STL versions, given that the tsduck.dll dynamically links with msvcp140.dll.
Funny enough, I only get a single hit when searching online for "_Start_element_lifetimes". :)

[lelegard]

Did you follow the instructions from here?

Namely, did you reference tsduck.props in your VS project file?

[mihaicodrean]

Yes, of course. Strangely, that error does not occur in Release|x86 with my local build of TSDuck, but only in Release|x64, with the library provided by the public installer.

[lelegard]

I am confused now by the timeline of messages in this thread, as if they were presented in the wrong order. What is the status now? Any remaining error with tsduck.props? What does work and what does not?

[mihaicodrean]

There are three items that we have discussed in this umbrella issue:

1. [SOLVED] The VS 2022 17.6 LTSC compiler bug when using std::enable_if in one use case, that I removed to be able to build the 32-bit TSDuck
2. [SOLVED] The linker issue with the singletons, that was due to my incorrect import of tsduck.props
3. The STL issue when including "tsUString.h", that happens in x64, but not x86.

[lelegard]

The STL issue when including "tsUString.h", that happens in x64, but not x86.

Is this the prebuilt binary for x64 vs. your build for x86? The two were built using different compilers (including a buggy one).

Or are the two from your builds? In which case it is a platform issue, not a compiler issue.

The important point is that the missing symbol of a private method in std::basic_string. So, this not something that can be called by you or TSDuck. It must be called from an inline or template method in the MSVC header files for std::basic_string. And it must be declared in the same header as basic_string.

In my case, this is header xstring in C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\include

I do not find any reference to a private method named _Start_element_lifetimes here.

You could probably look into your headers and you will probably find this private method. But that would mean a major runtime incompatibility between your compiler and subsequent versions.

That would not be surprising (I told you it was crap). This recently emerged as a major breakage of the GitHub CI Windows runners (still not fixed), putting thousands of developers in trouble. The problem was a defective and inconsistent runner image by GitHub folks. But the upper root cause was a major incompatibility between versions of the MSVC runtime.

[lelegard]

My recommendations, in this order:

• Drop so-called LTSC versions of the MSVC compiler and stick to the latest versions. Always. Stay up-to-date.
• Drop the whole thing and move to Linux or macOS. The clang compiler is extremely robust. GCC is fine too, but is ageing.

Probably not realistic in your case, but quite pragmatic.

[lelegard]

About the major GitHub runner breakage because of the mess which was created by incompatible versions of the C++ runtime, you may read the following reports:

actions/runner-images#10004
actions/runner-images#10020
actions/runner-images#10055

Some of them were closed but the problem remains at least on JNI code (affecting TSDuck as well).

You may also find hundredths of reports affecting that many repositories, which reference the issues above. The amount of lost troubleshooting time over hundredths of organizations is above several men x years. And the problem is not over yet.

All because of the MSVC runtime crap...

[mihaicodrean]

Is this the prebuilt binary for x64 vs. your build for x86?

Tried with both the prebuilt x64, and also with my own x64 build - same deal. My x86 build is fine.

Or are the two from your builds? In which case it is a platform issue, not a compiler issue.

No, see above. But to confirm, my builds match the Platform Toolset and the Windows SDK Version for both the TSDuck library & the test application.

In my case, this is header xstring in C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\include

Mine is: c:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.36.32532\include\

I do not find any reference to a private method named _Start_element_lifetimes here.

It looks like this on my end:

private:
    static constexpr void _Start_element_lifetimes(_Elem* const _Ptr, const size_type _Size) {
        // Start element lifetimes to avoid UB. This is a more general mechanism than _String_val::_Activate_SSO_buffer,
        // but likely more impactful to throughput.
#if _HAS_CXX20
        if (_STD is_constant_evaluated()) {
            for (size_type _Idx = 0; _Idx < _Size; ++_Idx) {
                _STD construct_at(_Ptr + _Idx);
            }
        }
#else // ^^^ C++20-or-later / pre-C++20 vvv
        (void) _Ptr;
        (void) _Size;
#endif // _HAS_CXX20
    }

[mihaicodrean]

The test application's obj contains these symbols for the x64 build:

External    | ?_Start_element_lifetimes@?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@$$FCAXQEAD_K@Z (private: static void __cdecl std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >::_Start_element_lifetimes(char * const,unsigned __int64))
External    | __t2m@?_Start_element_lifetimes@?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@CAXQEAD_K@Z ([T2M] private: static void __cdecl std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >::_Start_element_lifetimes(char * const,unsigned __int64))
External    | __m2mep@?_Start_element_lifetimes@?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@$$FCAXQEAD_K@Z ([M2MEP] private: static void __cdecl std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >::_Start_element_lifetimes(char * const,unsigned __int64))
External    | __mep@?_Start_element_lifetimes@?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@$$FCAXQEAD_K@Z ([MEP] private: static void __cdecl std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >::_Start_element_lifetimes(char * const,unsigned __int64))

Notice how they all have char * const as the first parameter, as opposed to char16_t * const from the error message.

[lelegard]

Interesting.

Your version of the compiler and its header files use a static method which is no longer used in subsequent versions. OK, why not. Since this is a template class, with template parameters in the profile of the method, the compiler is supposed to generate the code for that method alongside the application (but shall include only one instance in the link).

The problem is that a header file declares and defines a template method and the corresponding instance is not generated. That is a compiler bug. That's all.

I am not sure if the difference you noted on parameter types matters. The ts::UString class is just an enriched subclass of std::u16string, which uses char16_t* pointers. The symbols which are referenced in your object file (using char* parameters) are not declared as unresolved by the linker. So, this template code must have been generated somehow.

The other problem with other symbols is probably different. When the linker (or the linker-support part of the compiler) generates the template instance, it apparently forget to generate the missing method.

The fact that you don't have the problem with x86 is also interesting. If you look at the gigantic mess of the Windows GitHub runners with the MSVC runtime (see links in my previous post), we see the same symptom: no problem with x86. It seems that there is no longer any change on x86 (quite understandable since this is an obsolete target) and that all changes occur on x64. With a very bad change management policy, since it creates a lot of mess.

I can only have the same recommendation again: get away from that version of the compiler, use the latest version, and stay up-to-date all the time. Not only for TSDuck, for your own applications as well.

[mihaicodrean]

Just for reference, that method was added here: microsoft/STL@72229ff and removed here: microsoft/STL@29fc1c9.

[lelegard]

And there is a bug in the compiler, failing to properly generate the instance code, at least in some cases. So, removing the static template may have have only hidden the compiler bug (unless an independent fix in the compiler was also added in the meantime).