-
Notifications
You must be signed in to change notification settings - Fork 528
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Xamarin.Android.Build.Tasks] Better support for netstandard libraries. #1356
Conversation
System.Runtime uses . for both ref and lib assemblies. So I guess that would need to be handled in some way. |
Although System.IO.Compression uses . in its ref and lib directory and you are using that as a test. So maybe it is already handled :) |
Requires dotnet/java-interop#270 |
…forced loading. (#270) Context: dotnet/android#1154 Context: dotnet/android#1356 One of the problems we currently face is that our build system resolves `ref` netstandard libraries. With the current cache system, onces an assembly has been loaded into the Cecil cache it only ever uses that assembly. We might need to replace the current cached version with a new one if we detect a `ref` and need to reload the `lib`.
@@ -48,6 +48,53 @@ | |||
<Reference Include="FSharp.Compiler.CodeDom"> | |||
<HintPath>..\..\packages\FSharp.Compiler.CodeDom.1.0.0.1\lib\net40\FSharp.Compiler.CodeDom.dll</HintPath> | |||
</Reference> | |||
<Reference Include="Newtonsoft.Json"> | |||
<HintPath>..\..\packages\Newtonsoft.Json.11.0.1\lib\net45\Newtonsoft.Json.dll</HintPath> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jonpryor do we have to add all this junk to the Mac installer?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@dellis1972: if we need it at "runtime" -- app build time, packaging time, etc. -- then yes, we need to add this to the macOS installer.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, will need to remember that for when we bump monodroid.
build |
@@ -65,6 +77,11 @@ bool Execute (DirectoryAssemblyResolver resolver) | |||
|
|||
var topAssemblyReferences = new List<AssemblyDefinition> (); | |||
|
|||
LockFile lockFile = null; | |||
if (!string.IsNullOrEmpty (ProjectAssetFile) && File.Exists (ProjectAssetFile)) { | |||
lockFile = LockFileUtilities.GetLockFile (ProjectAssetFile, NullLogger.Instance); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we really want NullLogger.Instance
here? That implies we won't get any log messages from NuGet, or so I assume.
@@ -320,6 +325,7 @@ protected override void OnResume() | |||
IsRelease = true, | |||
UseLatestPlatformSdk = true, | |||
References = { | |||
new BuildItem.Reference ("Java.Interop.Export"), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How'd we manage to not need this before? If we didn't need it before, why's it needed now?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is related to what @pjcollins saw in his tests. Not sure why I needed to add Java.Interop.Export
, bit the test wouldn't build without it.. might have been a change upstream?
Please update |
@@ -120,6 +142,35 @@ bool Execute (DirectoryAssemblyResolver resolver) | |||
readonly List<string> do_not_package_atts = new List<string> (); | |||
int indent = 2; | |||
|
|||
AssemblyDefinition ResolveRuntimeAssemblyForReferenceAssembly (LockFile lockFile, DirectoryAssemblyResolver resolver, AssemblyNameDefinition assemblyNameDefinition) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm assuming this method was cribbed from .NET Core? If that's the case, we should likewise mention that in src/Xamarin.Android.Build.Tasks.tpnitems
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nope, it was written by me. I referenced the Nuget source code to see how the API Nuget.ProjectModel was used but none of this code was copied from .Net Core or Nuget.
@jonpryor @dellis1972 I managed to work around #1401 and test two projects which were affected by this startup issue (e.g.
|
That's ignorable, if unfortunate; we're going to see that once on every process startup. :-( Does Additionally, commercial installs use Release-Configuration assemblies now. Do you see that message with d15-6 or d15-7? If not, then this is because you're using the Debug-configuration
@radekdoulik? From the oss-3a60f76_build_1162.txt build log, I see:
So it is provided to
...it's only |
I think is very hacky as the assumption that ref to lib paths have to match is not always true. I think the better solution would be to use or reuse the already existing logic which does that in msbuild/nuget when you build with netstandard packages and it copies correct files to bin folder (note: I don't know how hard/easy would that be) |
@marek-safar I have been trying to find the code which does the replacement without too much luck so far. So you have any insight please feel free to share it :) This PR does NOT assume So we lookup via the API "System.IO.Packaging" for example, and then the API tells us where to find the "runtime" assembly (if one exists). I don't see that as hacky, I assume that .net core is doing something similar. A hacky solution IMHO would be
which would definitely have problems. |
If I add a |
4bddb52
to
b705f0c
Compare
Debug.WriteLine calls are removed during Compile Time. Since it has the `
[ConditionalAttribute("DEBUG")]` on it.
…On 13 March 2018 at 16:17, Peter Collins ***@***.***> wrote:
@jonpryor <https://github.com/jonpryor>
Does Debug.WriteLine() get written in Release apps? I wouldn't expect it
to; we should verify this.
If I add a Debug.Writeline() message to a new template project and deploy
in Release mode, the message is not present in logcat. Interestingly enough
however, I don't see Debug.WriteLine() output in logcat when deploying in
Debug mode either. I only see those messages in the Debug output pane of
the IDE when deploying in Debug mode. In the case reported above, the
Java.Interop.Export message was printed after deploying in Release mode.
I do not see the same Java.Interop.Export message when using 15.6 however.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1356 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAxeeb2i6gudwPRpBZWByaMpk_hvjkN1ks5td_EygaJpZM4SaQmT>
.
|
To elaborate on @dellis1972' last comment and @pjcollins' observed behavior:
What's happening is this:
In terms of |
|
||
public System.Threading.Tasks.Task LogAsync (LogLevel level, string data) | ||
{ | ||
return System.Threading.Tasks.Task.Run (() => Log (level, data)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this actually safe? See also what AsyncTask
has to go through for async logging...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok probably not...not sure we want ResolveAssemblies
to derive from AsyncTask
though.
<HintPath>..\..\packages\Newtonsoft.Json.11.0.1\lib\net45\Newtonsoft.Json.dll</HintPath> | ||
</Reference> | ||
<Reference Include="NuGet.Frameworks"> | ||
<HintPath>..\..\packages\NuGet.Frameworks.4.6.0\lib\net46\NuGet.Frameworks.dll</HintPath> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These packages are what need to be added to src/Xamarin.Android.Build.Tasks.tpnitems
. If they need to be redistributed, they need to be in a .tpnitems
file, with corresponding license information.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done. Don't think we need to add System.Runtime
or System.Runtime.InteropServices
since I don't think we need to ship those as they should be resolved as part of the .net runtime.. in theory.
16579a3
to
77b5dc4
Compare
We need some feedback on this from the NuGet team. Please @rrelyea and @rohit21agrawal could you advise on how to best resolve this? |
@rrelyea @rohit21agrawal this is related to what we were talking on-site: if the @dellis1972 maybe we could even send them a PR that adds ^? Possibly in https://github.com/NuGet/NuGet.BuildTasks/blob/dev/src/Microsoft.NuGet.Build.Tasks/ResolveNuGetPackageAssets.cs? |
2986d3a
to
323c3d2
Compare
Fixes dotnet#1154, dotnet#1162 Netstandard packages sometimes ship with both reference and implementation assemblies. The Nuget build task `ResolveNuGetPackageAssets` only resolves the `ref` version of the assemblies. There does not seem to be away way to FORCE Nuget to resolve the lib one. How .net Core manages to do this is still a mistery. That said the Nuget `ResolveNuGetPackageAssets` does give us a hint as to how to use the `project.assets.json` file to figure out what `lib` version of the package we should be including. This commit reworks `ResolveAssemblies` to attempt to map the `ref` to a `lib` if we find a Referenece Assembly. Historically we just issue a warning (which will probably be ignored), but now we will use the `project.assets.json` file to find the implementation version of the `ref` assembly. We need to be using `Nuget.ProjectModel` since it an API for querying the `project.assets.json`. We make use of the Nuget build properties `$(ProjectLockFile)` for the location of the `project.assets.json` , `$(NuGetPackageRoot)` for the root folder of the Nuget packages and `$(NuGetTargetMoniker)` for resolving which `TargetFrameworks` we are looking for. All of these properties should be set by Nuget. If they are not we should fallback to the default behaviour and just issue the warning. { "version": 3, "targets": { "MonoAndroid,Version=v8.1": { "System.IO.Packaging/4.4.0": { "type": "package", "compile": { "ref/netstandard1.3/System.IO.Packaging.dll": {} }, "runtime": { "lib/netstandard1.3/System.IO.Packaging.dll": {} } }, } } } The code above is a cut down sample of the `project.assets.json`. So our code will first resolve the `targets`. We use `$(NuGetTargetMoniker)` to do this. For an android project this should have a value of `MonoAndroid,Version=v8.1`. Note we do NOT need to worry about the version here. When Nuget restores the packages it creates the file so it will use the correct version. Next we try to find the `System.IO.Packaging`. We need to look at the `lockFile.Libraries` to get information about the install path in the Nuget folder, and then `target.Libraries` to pick out the `runtime` item. Once we have resolved the path we need to then combine that with the `$(NuGetPackageRoot)` to get the full path to the new library. If at any point during all of this code we don't get what we expect (i.e a null) we should abort and just issue the warning. The only real concern with this is if the format of the `project.assets.json` file changes. It is not ment to be edited by a human so there is the possibiltity that the Nuget team will decide to either change the schema or even migrate to a new file format. Hopefully we can just update the `Nuget` nuggets if that happens.
25dead6
to
b8ee9c3
Compare
Thanks alot for this PR 👍 |
…s. (#1356) Fixes: #1154 Fixes: #1162 NetStandard packages sometimes ship with both reference and implementation assemblies. The NuGet build task `<ResolveNuGetPackageAssets/>` only resolves the `ref` version of the assemblies. There does not seem to be away way to *force* NuGet to resolve the lib one. This commit reworks the `<ResolveAssemblies/>` task to attempt to map the `ref` to a `lib` if we find a Referenece Assembly. Historically we just issue a warning (which will probably be ignored), but now we will use the `project.assets.json` file to find the implementation version of the `ref` assembly, by using the `NuGet.ProjectModel` package to find the library which is associated with a reference assembly. We thus "ignore" reference assemblies, using the "referenced"/"real" assemblies instead, to ensure that our existing build process continues to generate usable packages. *Note*: An alternative approach would be to "embrace" reference assemblies, allowing them to be used in the build. This doesn't currently work with our build system, as we assume assemblies will contain native libraries, Android resources, and other artifacts which must be extracted at build time, and reference assemblies will not contain these artifacts. We would like to explore the "embrace" strategy, but this will require far more effort to support.
LogWarning ($"Could not resolve target for '{TargetMoniker}'"); | ||
return null; | ||
} | ||
var libraryPath = lockFile.Libraries.FirstOrDefault (x => x.Name == assemblyNameDefinition.Name); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Library's name property is the package id - is it guaranteed that AssemblyNameDefninition.Name will have a matching package id?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@rohit21agrawal that is a good point. I might have to rework this a bit then :(
cc @jonpryor should we revert?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about emitting AssemblyMetadataAttribute with the PackageId from msbuild, using something like https://mobile.twitter.com/kzu/status/977257467917819905
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I reworked the code in #1459 :)
…forced loading. (#270) Context: dotnet/android#1154 Context: dotnet/android#1356 One of the problems we currently face is that our build system resolves `ref` netstandard libraries. With the current cache system, onces an assembly has been loaded into the Cecil cache it only ever uses that assembly. We might need to replace the current cached version with a new one if we detect a `ref` and need to reload the `lib`.
Context: #1154 Context: #1356 Commit c7b9a50 [broke the build][0], because we neglected to cherry-pick a dependency from Java.Interop (doh!): [0]: https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android-d15-7/13/ Tasks/ResolveAssemblies.cs(183,32): error CS1739: The best overload for 'Load' does not have a parameter named 'forceLoad' Bump to Java.Interop/d15-7/4e1965d5, which adds the `DirectoryAssemblyResolver.Load(string fileName, bool forceLoad)` overload, which should fix the above error.
Is this already published on nuget? |
Xamarin.android is not installed using nuget. I think lt will be published through a Visual Studio update. However there is still an unmerged PR |
@@ -3,4 +3,17 @@ | |||
<package id="FSharp.Compiler.CodeDom" version="1.0.0.1" targetFramework="net45" /> | |||
<package id="FSharp.Core" version="3.1.2.5" targetFramework="net451" /> | |||
<package id="Irony" version="0.9.1" targetFramework="net45" /> | |||
<package id="Newtonsoft.Json" version="11.0.1" targetFramework="net451" /> | |||
<package id="NuGet.Common" version="4.6.0" targetFramework="net462" /> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You need to switch to PackageReference, this would have been two entries or so, with no changes in the csproj ;)
Fixes #1154, #1162
Netstandard packages sometimes ship with both reference and
implementation assemblies. The Nuget build task
ResolveNuGetPackageAssets
only resolves the
ref
version of the assemblies. There doesnot seem to be away way to FORCE Nuget to resolve the lib one.
How .net Core manages to do this is still a mistery. That said
the Nuget
ResolveNuGetPackageAssets
does give us a hint as tohow to use the
project.assets.json
file to figure out whatlib
version of the package we should be including.This commit reworks
ResolveAssemblies
to attempt to map theref
to alib
if we find a Referenece Assembly. Historicallywe just issue a warning (which will probably be ignored), but
now we will use the
project.assets.json
file to find theimplementation version of the
ref
assembly.We need to be using
Nuget.ProjectModel
since it an API forquerying the
project.assets.json
. We make use of the Nuget build properties$(ProjectLockFile)
for the location of theproject.assets.json
,
$(NuGetPackageRoot)
for the root folder of the Nuget packagesand
$(NuGetTargetMoniker)
for resolving whichTargetFrameworks
we are looking for. All of these properties should be set by Nuget.
If they are not we should fallback to the default behaviour and just issue the warning.
The code above is a cut down sample of the
project.assets.json
. So ourcode will first resolve the
targets
. We use$(NuGetTargetMoniker)
todo this. For an android project this should have a value of
MonoAndroid,Version=v8.1
. Note we do NOT need to worry about the versionhere. When Nuget restores the packages it creates the file so it will
use the correct version.
Next we try to find the
System.IO.Packaging
. We need to look at thelockFile.Libraries
to get information about the install path in theNuget folder, and then
target.Libraries
to pick out theruntime
item.
Once we have resolved the path we need to then combine that with the
$(NuGetPackageRoot)
to get the full path to the new library. If at anypoint during all of this code we don't get what we expect (i.e a null) we
should abort and just issue the warning.
The only real concern with this is if the format of the
project.assets.json
file changes. It is not ment to be edited by a human so there is the
possibiltity that the Nuget team will decide to either change the schema or
even migrate to a new file format. Hopefully we can just update the
Nuget
nuggets if that happens.