Proposal: Type provider design-time DLLs should be chosen more appropriately

[dsyme]

Recent PRs in the Type Provider SDK repo (e.g. fsprojects/FSharp.TypeProviders.SDK#139) are a big step towards completing our type provider story for .NET Core With this work, properly written type providers using the latest version of the TPSDK can now always produce target code for either .NET Core or .NET Framework – i.e. they properly adjust for the target reference assemblies being used by the compilation. This applies to both generative and erasing type providers.

One final piece of the type provider puzzle is to have the F# compiler load an appropriate design-time DLL depending on whether the compiler is running in .NET Core or .NET Framework.

Currently, the compiler looks for design time DLL alongside the referenced runtime DLLs. (For simple type providers, these DLLs are the same)

We will do something like what is specified her https://docs.microsoft.com/en-us/nuget/schema/analyzers-conventions,

Rough proposal

When a referenced assembly as the usual TypeProviderAssembly attribute, indicating it wants design-time type provider component “MyDesignTime.dll”, then

1. When executing using .NET Core the compiler looks in this order

   ...\typeproviders\fsharpNN\netcoreapp2.0\ARCH\MyDesignTime.dll
   ...\typeproviders\fsharpNN\netcoreapp2.0\MyDesignTime.dll 
   ...\typeproviders\fsharpNN\netstandard2.0\ARCH\MyDesignTime.dll
   ...\typeproviders\fsharpNN\netstandard2.0\MyDesignTime.dll 
   MyDesignTime.dll 

2. When executing using .NET Framework the compiler looks in this order

    ...\typeproviders\fsharpNN\net461\ARCH\MyDesignTime.dll
    ...\typeproviders\fsharpNN\net461\MyDesignTime.dll
    ...\typeproviders\fsharpNN\netstandard2.0\ARCH\MyDesignTime.dll
    ...\typeproviders\fsharpNN\netstandard2.0\MyDesignTime.dll
    MyDesignTime.dll 

relative to the location of the runtime DLL, which is presumed to be in a nuget package.

• When we use ... we mean a recursive upwards directory search looking for a directory names typeproviders, stopping when we find a directory name packages

• WHen we use fsharpNN we mean a successive search backwards for fsharp42, fsharp41 etc. Putting a TPDTC in fsharp41 means the TPDTC is suitable to load into F# 4.1 tooling and later, and has the right to minimally assume FSharp.Core 4.4.1.0

Some examples:

The idea is that a package can contain a type provider design-time for both .NET Core and .NET Framework. This will allow it to load into both compiler, F# Interactive and tools running .NET Framework OR .NET Core.

Example 1 - the ultimate simplest form

Going forward, the very simplest type providers will eventually use a single .NET Standard 2.0 DLL for both the runtime library and the design-time component. Layout:

      lib\netstandard2.0\MyRuntimeAndDesignTime.dll 

As of today however you will also need to ship facades for netstandard.dll, System.Runtime.dll and System.Reflection.dll alongside this component and as such you should either

• adopt one of the layouts below to ensure facade extra files don't end up in the lib directory, OR
• put the facades in a well-known relative location (not under the lib directory in the package) and added AssemblyResolve events to go and find them

Example 2 - FSharp.Data

FSharp.Data would lay out as follows:

    lib\net45\FSharp.Data.dll 
    lib\netstandard2.0\FSharp.Data.dll 
    lib\portable-whatever\FSharp.Data.dll 

    typeproviders\fsharp40\net461\FSharp.Data.DesignTime.dll  
    typeproviders\fsharp40\netcoreapp2.0\FSharp.Data.DesignTime.dll  

Here we are assuming FSharp.Data wants to use different design time DLLs for the two host tooling contexts we care about. Example 3 deals with the case where FSHarp.Data wants to use a single DLL.

Example 3 - FSharp.Data (with .NET Standard design-time DLL)

A layout like this may also be feasible if shipping facades to create a .NET Standard 2.0 TPDTC

    lib\net45\FSharp.Data.dll 
    lib\netstandard2.0\FSharp.Data.dll 
    lib\portable-whatever\FSharp.Data.dll 

    typeproviders\fsharp41\netstandard2.0\FSharp.Data.DesignTime.dll  

See note on facades above.

Example 4 - type providers with 32/64-bit dependencies

A Python type provider may have different dependencies for 32 and 64-bit for both runtime and design-time (the directory names may not be exactly right)

    lib\netstandard2.0\x86\FPython.Runtime.dll 
    lib\netstandard2.0\x64\FPython.Runtime.dll 

    typeproviders\fsharp41\netstandard2.0\x86\FPython.Runtime.dll 
    typeproviders\fsharp41\netstandard2.0\x86\cpython32.dll # some 32-bit DLL needed to run at design-time

    typeproviders\fsharp41\netstandard2.0\x64\FPython.Runtime.dll 
    typeproviders\fsharp41\netstandard2.0\x64\cpython64.dll # some 64-bit DLL needed to run at design-time

plus facades as mentioned above

Non Example 5

Going forward, we should not be happy with type provider packages that look like this - these will be unusable when the compiler executes using .NET Core

    lib\net45\MyTypeProvider.dll 

or even

    lib\net45\MyTypeProvider.dll 
    typeproviders\fsharp40\net45\MyTypeProvider.DesignTime.dll

Again if your TPDTC component is a solitary NET Framework component then it will be unusable when a host tool executes using .NET Core.

Remember, we must be able to load type provider design-time DLLs into many different tooling contexts, notable

• FsAutoComplete.exe .NET Framework running 64-bit
• fsc.exe running on .NET Core 2.0 as either 64-bit or 32-bit
• fsi.exe running on .NET Framework as 32-bit
• fsiAnyCpu.exe running on .NET Framework as either 64-bit or 32-bit
• fsc.exe running on .NET Framework as either 64-bit or 32-bit
• devenv.exe running on .NET Framework 32-bit

and any context that uses FSharp.Compiler.Service.dll as either netstandard 2.0 or a .NET Framework component.

Note the above is effectively a proposed package architecture for any cross-generating compiler plugins. If we ever support other kinds of compiler plugins then we could follow a similar pattern.

[matthid]

In (C#) Roslyn-Analyzer world is it really the compiler figuring out the paths?
To be honest I'm a bit skeptical to add NuGet implementation details to the F#-Compiler...
What about dependencies of the Design-Time assembly?

[dsyme]

In (C#) Roslyn-Analyzer world is it really the compiler figuring out the paths?

No, it's just a related example

To be honest I'm a bit skeptical to add NuGet implementation details to the F#-Compiler...

I think its inevitable in some form. The referenced runtime DLL and the design-time DLL live in the same package and we really have to separate them, but equally be able to locate the latter from the former.

What about dependencies of the Design-Time assembly?

The design-time DLL is loaded using LoadFrom and must carry all its dependencies in the same directory as the DLL (or loadable using standard binding rules after a LoadFrom). It must be a standalone "reflection-loaded adding".

One huge advantage of the above is that each design-time configuration (e.g. x86 and x64) can now carry different dependencies.

Note the situation today is horrendous - the dependencies of the design-time components all end up in the "lib" directory, and mistakenly get added as references. And people hack the search through neighouring packages in terrible ways, e.g. https://github.com/BlueMountainCapital/FSharpRProvider/blob/master/src/RProvider/RProvider.fsx

[dsyme]

To be honest I'm a bit skeptical to add NuGet implementation details to the F#-Compiler...

Note the other alternative would be to have logic in our MSBuild targets file and add a first-class /typeproviderreference:MyDesignTime.dll flag to the compiler.

However I think that's very obscure and won't work for tools like F# Interactive (fsi.exe), which must locate type provider design-time components without resorting to using MSBuild. In particular I don't really want people to have to use

#r "packages\FSharp.Data\lib\net45\FSharp.Data.dll"
#tpr "packages\FSharp.Data\typeproviders\netstandard2.0\FSharp.Data.DesignTime.dll"

in their F# scripts - though admittedly it's being very honest about what's going on. We would end up baking just the same logic in to the compiler in order to give people a good error message telling them where to find the design-time DLL of they forget to add the #tpr reference

[matthid]

Note the other alternative would be to have logic in our MSBuild targets file and add a first-class "/typeproviderreference:MyDesignTime.dll" flag to the compiler. However I think that's very obscure and won't work for tools like F# Interactive (fsi.exe),

To be honest that sounds a lot better and if we add a corresponding #typeproviderr (or however we call it) to F# scripting, we can extend Paket to properly add those into the load scripts including the proper dependencies. This way we leave resolving dependencies to the caller (as it should be).

which must locate type provider design-time components without resorting to using MSBuild.

Yes but a compiler solution introduces a lot of other problems

• what if multiple type providers bundle the same dependencies?
• Framework compatibility and fallback is not too simple to implement correctly, all that needs to be duplicated in the F# compiler (or otherwise we will miss a lot in our compatibility list)
• and it feels wrong

On the other side thinking about supporting design time and runtime dependencies in Paket makes my head burn. (In practice a unified resolution will most likely be fine / we might add special supports in our groups). Also there is a problem with detecting those dependencies: Is a dependency of a type provider package a runtime or a designtime dependency? But at least that will not be a problem of the compiler.

[matthid]

which must locate type provider design-time components without resorting to using MSBuild.

And it will be solved via #2483 ;)

[smoothdeveloper]

@matthid:

To be honest I'm a bit skeptical to add NuGet implementation details to the F#-Compiler...

I see it not as NuGet implementation, but as providing a stable layout relative to the runtime assembly.

To be honest that sounds a lot better and if we add a corresponding #typeproviderr (or however we call it) to F# scripting, we can extend Paket to properly add those into the load scripts including the proper dependencies. This way we leave resolving dependencies to the caller (as it should be).

This could be done leveraging #2483:

#r "typeprovider:path/to/FSharp.Data.dll"

paket in the load script generation would identify assemblies exposing type providers and add the reference in that way rather than simple assembly reference.

The typeprovider resolution handler would be implemented with the conventions @dsyme was sketching, it would help conceal all the implementation details, and keep the manual work from a simple script to get the reference working minimal (unlike @dsyme sample in #3736 (comment))

Yes but a compiler solution introduces a lot of other problems

Lots of valid concerns.

[dsyme]

On the other side thinking about supporting design time and runtime dependencies in Paket makes my head burn. (In practice a unified resolution will most likely be fine / we might add special supports in our groups). Also there is a problem with detecting those dependencies: Is a dependency of a type provider package a runtime or a designtime dependency? But at least that will not be a problem of the compiler.

I don't expect Paket or Nuget will ever seriously get into the business of managing dependencies of compile-time tools and add-ins. At least not in a timeframe that's useful. Indeed I don't think they could - they just don't know enough about host tooling. We could maybe push something bespoke into Paket, but its only use cases would be for F# type-provider add-ins, so there's not much point.

I'm also not terribly keen in hiding resolution logic in MSBuild for similar reasons. On the whole I'd rather have this stuff "close to home" where we have control over it, rather than pushing the problem off to Paket, Nuget etc. If it inspires Nuget to come up with a way of managing compile-time tool dependency chains then that's good and we could eventually switch to using the results, but I think that's unlikely. Even for analyzers mentioned above there is no compile-time dependency management - I think it's assumed each analyzer is independently loadable

what if multiple type providers bundle the same dependencies?

That problem exists in any situation. Type provider design-time components (let's call those TPDTCs) are loaded into host tooling using Assembly.LoadFrom (regardless of how they are specified and found). There is no other choice (unless we either isolate them in a process - AppDomains are no longer an option on .NET Core).

This causes problems - for example, TPDTCs may also not have binding redirects (except ones implemented via adhoc AssemblyResolve events, and ones provided by host tooling, e.g. a redirect to a recent version of FSharp.Core can generally be assumed) .

I don't think Paket and Nuget can help us here - how will Paket know of the binding configuration of Ionide, or VS2019, or FSC, or ... , which I believe it needs to know to do useful dependency management of compile-time addins.

Perhaps in the longer term we can run TPDTCs isolated, e.g. in a separate process (cf. FsAutoComplete.exe). However it is not totally easy to remote the System.Type/Method/... API through a process boundary.

Perhaps there is more we can learn from how Roslyn analyzers are loaded.

all that needs to be duplicated in the F# compiler (or otherwise we will miss a lot in our compatibility list)

The good news is that in practice the design-time component of most TPs can be relatively simple .NET Standard 2.0 components (e.g. this works for FSharp.Data). There are cases where the component has significant non-.NET Standard dependencies (e.g. RProvider) - over time we really just have to hope that these dependencies become available as .NET Standard components.

[dsyme]

which must locate type provider design-time components without resorting to using MSBuild.

And it will be solved via #2483 ;)

I don't think this solves type provider references.

Say that separate TPDTCs are available for .NET Framework 4.5 and .NET Standard 2.0. Which type TPDTC reference is emitted into the load script when running in FSI.EXE (.NET Framework 4.6 32 bit)? What about when type checking in devenv.exe (.NET Framework 32 bit)? When running in the F# Compiler (.NET Core 2.0 64 bit)? The package manager library (Paket.Core.dll) would have to look at the configuration of the tool it is running in (i.e,. FSI.EXE on .NET Core 2.0 x64) and do resolution based on that, but again presumably looking under some equivalent of the typeprovider directories mentioned above (these TPDTC components can't live in lib directories since they are not intended to be referenced in the normal way).

But that's just adding some fairly bespoke resolution rules into Paket. And we'd need another implementation in MSBuild/nuget.

I understand there's some perfect world where the compiler relies on external tooling to tell it about TPDTCs. But relying on external tools causes a lot of procedural problems too, and I'm not sure those tools know that much that's useful, apart from the framework compatibility relationship - which is trivial if all we care about is netstandard2.0, netcoreapp2.0 and net4x.

Note that we made a decision not to rely on external tooling for the resolution of TPDTCs in 2012 - and given the vast churn in nuget/msbuild since then that feels like a wise decision - it feels to me that external package manager tooling actually churns faster than anything else in the dotnettyverse :) - and any bespoke feature we might have put in back then would have been waaaay out of date by now. Better to have control over our destiny and put the rules in the F# compiler.. :)

One reason I'm in favour of the proposal above is that it represents a small variation on where we are today: just 20-50 lines of code in ExtensionTyping.fs, and I think it will get us to a stable place for the next 2-3 years. If we are going to change things in a major way I feel we should be more radical, e.g. look at interpreting type providers, or isolating them, or ... Basically getting rid of type providers as binary components. But that doesn't solve dependencies and so on, unless we use a full IL interpreter. (Also, we could look at what Scala tooling does for macros - the problems are very similar and I know there has been a lot of engineering challenges there).

[matthid]

Perhaps there is more we can learn from how Roslyn analyzers are loaded.

Yes please let's figure out their solution and go down the same/a similar route.
It seems they have to specify all deps by command line: dotnet/roslyn#14866 (comment)

Also it might be that we want to support language agnostic analyzers sooner or later one way or another, but in the mean-time we could even use the unused "fsharp" folder in there:
https://docs.microsoft.com/en-us/nuget/schema/analyzers-conventions

I'll try to respond to the rest of your response as well (as it contains valid critics) but I just wanted to get this part out.

[dsyme]

we could even use the unused "fsharp" folder in there:
https://docs.microsoft.com/en-us/nuget/schema/analyzers-conventions

TPDTCs are not analyzers, so putting them in the analyzers directory seems problematic. Adding "fsharp" to the path makes sense

I should also add this: when executing the F# compiler with .NET Core TPDTCs are loaded using AssemblyLoadContext.Default.LoadFromAssemblyPath - which I assume is the equivalent of Assembly.LoadFrom (untested).

[enricosada]

@dsyme, can we just use a nuget package to ship the design assemblies?
so TP libraries can depend on that package, and that package add the right design.time based on runtime
As a note, that package can also bundle the ProvidedTypes.fs file, so TP libraries just need to add that package and a stuff like

<Compile Include="$(ProvidedTypesSourcePath)" />

the ProvidedTypesSourcePath property is provided by the FSharp.TypeProviders.SDK package, and the source file is inside that.
So for TP lib, is a normal library. and can add deps too (by runtime or native too)

[dsyme]

@dsyme, can we just use a nuget package to ship the design assemblies?

@enricosada No - that would mean the TPDTCs get referenced as normal assemblies (-r) for the compilation, which is wrong. Also, tools like nuget and paket would select the DLLs based on the target runtime, not the compile-time runtime (the runtime and platform being used to execute the F# compiler or other host tool).

[matthid]

I don't expect Paket or Nuget will ever seriously get into the business of managing dependencies of compile-time tools and add-ins. At least not in a timeframe that's useful.

We can be pretty fast in adding and releasing support into Paket. So that point is not valid. Can't speak for NuGet, but if we decide to re-use analyzers infrastructure nobody needs to do anything besides F# SDK and compiler?

Indeed I don't think they could - they just don't know enough about host tooling. We could maybe push something bespoke into Paket, but its only use cases would be for F# type-provider add-ins, so there's not much point.

Paket doesn't have to know anything, because load scripts can add #if conditionals to deal with that if required. For compilation we'd investigate how analyzers work and go down the same route.

I'm also not terribly keen in hiding resolution logic in MSBuild for similar reasons. On the whole I'd rather have this stuff "close to home" where we have control over it, rather than pushing the problem off to Paket, Nuget etc. If it inspires Nuget to come up with a way of managing compile-time tool dependency chains then that's good and we could eventually switch to using the results, but I think that's unlikely. Even for analyzers mentioned above there is no compile-time dependency management - I think it's assumed each analyzer is independently loadable

I don't see the reason why we cannot do the same (ie. say that type providers need to be standalone) or how that is related to adding magic path resolution to the F# compiler.
In reality we could say: Type providers need to be standalone (for packaging and to prevent the design time dependency problem) and all design-time dependencies and the type-provider assemblies are given to the compiler via command line. Those issues are in no way related to each other. I think this statement also addresses all other issues regarding TPDTCs of your first post. What I mean is: The conclusion that we need to handle magic folder resolution in the compiler is in no way related to design time dependencies. This is something package managers can figure out later on their own if they want/can. If it is in the compiler there is never a way to make it work. Therefore it is actually another reason to not add it to the compiler.

And it will be solved via #2483 ;)

I don't think this solves type provider references.

Say that separate TPDTCs are available for .NET Framework 4.5 and .NET Standard 2.0. Which type TPDTC reference is emitted into the load script when running in FSI.EXE (.NET Framework 4.6 32 bit)? What about when type checking in devenv.exe (.NET Framework 32 bit)? When running in the F# Compiler (.NET Core 2.0 64 bit)? The package manager library (Paket.Core.dll) would have to look at the configuration of the tool it is running in (i.e,. FSI.EXE on .NET Core 2.0 x64) and do resolution based on that, but again presumably looking under some equivalent of the typeprovider directories mentioned above (these TPDTC components can't live in lib directories since they are not intended to be referenced in the normal way).

The "Paket"-Plugin in the compiler can figure out the host it is running on and get the correct references. At least that is how I understand @forki's suggested solution. So I don't see the problem. We already have RID resolution support for runtime dependencies in Paket so we could reuse that once we have decided how we want to package TCs with such special requirements.

But that's just adding some fairly bespoke resolution rules into Paket. And we'd need another implementation in MSBuild/nuget.

:/

Note that we made a decision not to rely on external tooling for the resolution of TPDTCs in 2012 - and given the vast churn in nuget/msbuild since then that feels like a wise decision - it feels to me that external package manager tooling actually churns faster than anything else in the dotnettyverse :) - and any bespoke feature we might have put in back then would have been waaaay out of date by now. Better to have control over our destiny and put the rules in the F# compiler.. :)

Or they would have considered the requirement from the start in the new world or abstracted the idea of analyzers into compiler plugins. Nobody knows.

One reason I'm in favour of the proposal above is that it represents a small variation on where we are today: just 20-50 lines of code in ExtensionTyping.fs, and I think it will get us to a stable place for the next 2-3 years.

Maybe, Maybe not.

If we are going to change things in a major way I feel we should be more radical, e.g. look at interpreting type providers, or isolating them, or ... Basically getting rid of type providers as binary components. But that doesn't solve dependencies and so on, unless we use a full IL interpreter. (Also, we could look at what Scala tooling does for macros - the problems are very similar and I know there has been a lot of engineering challenges there).

Yep they need a redesign badly, but as you point out it doesn't really solve this problem in any way (I don't understand how a full IL interpreter would help here?)

TPDTCs are not analyzers, so putting them in the analyzers directory seems problematic. Adding "fsharp" to the path makes sense

To be honest I don't really think it is. In fact it probably should have been considered as compiler plugins from the beginning. We might even consider to use /plugin: instead of /typeprovider and #plugin because then we might be able to support language independent roslyn analyzers in some far-away future with the same infrastructure.
I think it could work pretty well in practice, but yes a bit of testing is needed here as there could be some problems hidden with that approach.

I should also add this: when executing the F# compiler with .NET Core TPDTCs are loaded using AssemblyLoadContext.Default.LoadFromAssemblyPath - which I assume is the equivalent of Assembly.LoadFrom (untested).

Actually they probably should be loaded into their own loadcontext, then a shared TPDTC with different versions for two type providers could technically work. That "might" be a reason to resolve type provider design time dependencies in the compiler (Though I think it can work via command line as well, it is just a bit more difficult to implement correctly). Honestly, I don't think this will be changed at all so it isn't actually a reason in practice.

@enricosada No - that would mean the TPDTCs get referenced as normal assemblies (-r) for the compilation, which is wrong. Also, tools like nuget and paket would select the DLLs based on the target runtime, not the compile-time runtime (the runtime and platform being used to execute the F# compiler or other host tool).

Yes from a quick research it seems like in fact all roslyn-analyzers are self-contained as well (ie. they bundle all dependencies). Threfore, I'd suggest (as a first step) to go down the same route and ignore all issues a design time resolution would introduce.

[dsyme]

Just to note that for F# TPDTCs, it appears to make sense to be able to qualify on the F# version number of the host tooling.

For example

    typeproviders\fsharp41\netstandard2.0\FSharp.Data.DesignTime.dll  
    typeproviders\fsharp43\netstandard2.0\FSharp.Data.DesignTime.dll  

• fsharp41 == "you can use this TPDTC when running in F# 4.1 host tooling of after"
• fsharp42 == "you can use this TPDTC when running in F# 4.2 host tooling or after"
• fsharp43 == "you can use this TPDTC when running in F# 4.3 host tooling or after"

Possibly also other monikers for Fable tooling etc.

For each such version of F# tooling, we would specify a minimum base FSharp.Core that you can assume is minimal for that host tooling, e.g.

• fsharp40 == the TPDTC can assume FSharp.Core 4.4.0.0
• fsharp41 == the TPDTC can assume FSharp.Core 4.4.1.0

and so on. Each TPDTC can be written with respect to the its appropriate FSharp.Core or earlier.

To motivate, FSharp.Core 4.0.0.0 included new contracts for TPDTCs for static parameters on methods which weren't present in 4.3.1.0. But a type provider that wanted to be usable in F# 3.1 tooling couldn't make use of that, and in practice almost no one is using that feature partly for that reason.

Making this mandatory would help allow us to gradually progress the type provider API and to improve the API, extending it to include more capabilities and perhaps even adding a revamped, new API altogether, but still have packages which ship TPDTC components which work with previous-generation tooling.

It's partly because I want F# to have control over this sort of dimension to versioning that I'm not totally happy with relying on external tools to do resolution with respect to runtime assumptions. (I assume Roslyn analyzers will need similar qualification based on roslyn version, but that the solution grid may be different)

I'm a big fan of continuing to extend the TPDTC contracts in interesting directions but in order to do that sanely we will need some scheme like this.

[dsyme]

Tooling RFC is at https://github.com/fsharp/fslang-design/blob/master/tooling/FST-1003-loading-type-provider-design-time-components.md

[cartermp]

Closing this, as the RFC implementation is merged and we're entering the final stages of this work soon

[dmitry-a-morozov]

I'm trying to build a project that consumes Example 3 type provider (.NET Standard design-time DLL) using .NET Core SDK command line.
It fails to find TPDTC dependencies. Where they suppose to be located? Same folder, local nuget cache or somewhere else?
The project targets netcoreapp2.1. I use the latest 2.1.302 sdk.

It builds just fine if I apply
#3303 (comment)

[daniellittledev]

@dsyme regarding the example Example 4 - type providers with 32/64-bit dependencies
Is there any more detail about these paths, you mentioned "the directory names may not be exactly right"

The package I'm looking at has four native assemblies win-x64, win-x86, win-arm and win-64 but the example only shows two x86 and x64.

[dsyme]

Is there any more detail about these paths, you mentioned "the directory names may not be exactly right"

My understanding is that the F# tooling doesn't know anything about these names - rather it does the Assembly.Load from the chosen location and the varsious versions of the .NET Runtime can in some circumstances look in the subdirectories for native components. However I'm not certain of the details and it doesn't appear to be under test.

In general TPDTC with native dependencies should just be avoided to be honest - your TP may need to load into all sorts of F# tooling on different architectures

[daniellittledev]

@dsyme thanks for the information, although this might be bad news for me and anyone making database type providers.

My specific situation is building a type provider for database access using Microsoft.Data.SqlClient. On windows (unix is all managed code) this package has a dependency on Microsoft.Data.SqlClient.SNI.runtime. I haven't seen a NuGet package like this one before, it only contains native binaries, it's structure looks like this.

- runtimes
  - win-arm
    - native
      -  Microsoft.Data.SqlClient.SNI.dll
  - win-arm64
    - native
      -  Microsoft.Data.SqlClient.SNI.dll
  - win-x64
    - native
      -  Microsoft.Data.SqlClient.SNI.dll
  - win-x86
    - native
      -  Microsoft.Data.SqlClient.SNI.dll

I can successfully load Microsoft.Data.SqlClient for the type provider, but it fails to find Microsoft.Data.SqlClient.SNI.dll on windows and there does not seem to be a way to load it, according to my research and your comment.

If I cannot load native a dll, then it is currently not possible to write a TP for Microsoft.Data.SqlClient that runs on windows. This leaves me with two options:

• Not building a type provider for Microsoft.Data.SqlClient
• Alternately, lobbying MS to support a managed-code only option for windows

Do you have any recommendations regarding these options?

[Thorium]

As a third workaround (not sustainable solution), @RonaldSchlenker, what you can do is unload the AssemblyResolve event handler on AppDomain.CurrentDomain before Assembly load call and replace it with your custom function... will involve some hours of frustrating VS debugging.

[SchlenkR]

Thanks @Thorium for the suggested workaround. I was one of the people pinged by @WillEhrendreich in the related issue, so I thought I’ll try to remember the infos I had when I was confronted with that issue (don’t even know where I posted my comments). But it’s currently not an issue that I’m facing.

Anyway, this morning I tried to understand why the SqlClient (the „new“ one) is not able to correctly resolve the native parts that actually match the runtime. In normally compiled apps (net461, net6), it doesn’t seem to be an issue. Perhaps (and maybe the commenters before me already mentioned it), it’s a combination of - host default resolve behavior - TP specialties - MsSqlClient lib resolve behavior - others, but I have no deeper insight on that, and I’m not a TP expert.

To get some more knowledge, I’ve created a stripped-down TP, a very simple one, that logs infos concerning compile time (IDE) and runtime, and does a simple SQL query using MsSqlClient. It’s not ready yet, but might be helpful to understand a bit more and hopefully find a stable way for solving this or related issues.

I’ll post more infos in the related issues.

The repo of the mentioned TP is here:

https://github.com/RonaldSchlenker/DebuggingTypeProvider

[WillEhrendreich]

Too curious not to dig in. Thanks, @RonaldSchlenker , I've forked it already, lol.

[daniellittledev]

To follow up to my previous comment Microsoft.Data.SqlClient does actually have an option to run without native assemblies (use fully managed code instead).

You can avoid loading the native SNI by switching to managed SNI on Windows:

System.AppContext.SetSwitch("Switch.Microsoft.Data.SqlClient.UseManagedNetworkingOnWindows", true)

[TobiasJuderjahn]

Without wanting to be rude, I have to say that creating your own type providers is far too complicated. Unfortunately, the process is completely non-transparent with incomprehensible error messages.
I actually find the idea behind it fascinating, but you have to be an expert to make it work.
I give up. Please don't take this the wrong way, I respect your commitment. I just wanted to give feedback and would explain in more detail if asked.

[vzarytovskii]

Without wanting to be rude, I have to say that creating your own type providers is far too complicated. Unfortunately, the process is completely non-transparent with incomprehensible error messages.

I actually find the idea behind it fascinating, but you have to be an expert to make it work.

I give up. Please don't take this the wrong way, I respect your commitment. I just wanted to give feedback and would explain in more detail if asked.

There are some plans to revamp them at some point and make the sdk part of compiler, but it didn't make it for next release unfortunately.

[TobiasJuderjahn]

That would be great!
At this point, a big thank you to the team in any case. F# is a fantastic language.