Unlisted Packages been found in Paket Outdated

[madhon]

Paket is picking packages which have been unlisted from Nuget
e.g. RX-PlatformServices

Paket.Dependencies has
nuget Rx-PlatformServices 2.2.5

when running paket outdated it finds v2.3 which is noted as been unlisted at the nuget site
https://www.nuget.org/packages/Rx-PlatformServices/2.3.0

[forki]

mhm how can we detect this?

/cc @maartenba

[maartenba]

Don't laugh... Packages are considered unlisted if their Published property matches a magic date:

    // The magic unpublished date is 1900-01-01T00:00:00
    private static readonly DateTime _unpublished = new DateTimeOffset(1900, 1, 1, 0, 0, 0, TimeSpan.FromHours(-8)).DateTime;

[agross]

@forki @maartenba We should make a list of magic values used by NuGet.

[forki]

Maarten you are a true hero for us.

[agross]

@maartenba 👍

[maartenba]

@forki @agross We have had all these pains building MyGet so happy to help :-)

[forki]

Such a nice feature.

[forki]

@maartenba

<?xml version="1.0" encoding="utf-8"?>
<entry xml:base="https://www.nuget.org/api/v2/" xmlns="http://www.w3.org/2005/Atom" xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
  <id>https://www.nuget.org/api/v2/Packages(Id='Rx-PlatformServices',Version='2.3')</id>
  <category term="NuGetGallery.V2FeedPackage" scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" />
  <link rel="edit" title="V2FeedPackage" href="Packages(Id='Rx-PlatformServices',Version='2.3')" />
  <title type="text">Rx-PlatformServices</title>
  <summary type="text"></summary>
  <updated>2014-08-21T19:17:53Z</updated>
  <author>
    <name>Microsoft</name>
  </author>
  <link rel="edit-media" title="V2FeedPackage" href="Packages(Id='Rx-PlatformServices',Version='2.3')/$value" />
  <content type="application/zip" src="https://www.nuget.org/api/v2/package/Rx-PlatformServices/2.3.0" />
  <m:properties>
    <d:Version>2.3</d:Version>
    <d:NormalizedVersion>2.3.0</d:NormalizedVersion>
    <d:Copyright>Copyright (C) Microsoft Corporation</d:Copyright>
    <d:Created m:type="Edm.DateTime">2014-01-27T19:28:43.397</d:Created>
    <d:Dependencies>Rx-Interfaces:2.2:|Rx-Core:2.2:</d:Dependencies>
    <d:Description>Reactive Extensions Platform Services Library used to access platform-specific functionality and enlightenment services.</d:Description>
    <d:DownloadCount m:type="Edm.Int32">262653</d:DownloadCount>
    <d:GalleryDetailsUrl>https://www.nuget.org/packages/Rx-PlatformServices/2.3.0</d:GalleryDetailsUrl>
    <d:IconUrl>http://go.microsoft.com/fwlink/?LinkId=261274</d:IconUrl>
    <d:IsLatestVersion m:type="Edm.Boolean">false</d:IsLatestVersion>
    <d:IsAbsoluteLatestVersion m:type="Edm.Boolean">false</d:IsAbsoluteLatestVersion>
    <d:IsPrerelease m:type="Edm.Boolean">false</d:IsPrerelease>
    <d:Language>en-US</d:Language>
    <d:Published m:type="Edm.DateTime">2014-01-27T19:28:43.397</d:Published>
    <d:PackageHash>H3aiFtRAsdG2mN5O6p/tvOiDbRRISYEQAnTDJGheX9Kmal0YS3txJbqlOuOtJB6XbkVA7/vtvdsWDswSKItFyA==</d:PackageHash>
    <d:PackageHashAlgorithm>SHA512</d:PackageHashAlgorithm>
    <d:PackageSize m:type="Edm.Int64">196416</d:PackageSize>
    <d:ProjectUrl>http://go.microsoft.com/fwlink/?LinkId=261273</d:ProjectUrl>
    <d:ReportAbuseUrl>https://www.nuget.org/package/ReportAbuse/Rx-PlatformServices/2.3.0</d:ReportAbuseUrl>
    <d:ReleaseNotes m:null="true" />
    <d:RequireLicenseAcceptance m:type="Edm.Boolean">true</d:RequireLicenseAcceptance>
    <d:Summary m:null="true" />
    <d:Tags>Rx Reactive Extensions Observable LINQ Events</d:Tags>
    <d:Title>Reactive Extensions - Platform Services Library</d:Title>
    <d:VersionDownloadCount m:type="Edm.Int32">362</d:VersionDownloadCount>
    <d:MinClientVersion m:null="true" />
    <d:LastEdited m:type="Edm.DateTime">2014-01-27T19:32:41.41</d:LastEdited>
    <d:LicenseUrl>http://go.microsoft.com/fwlink/?LinkID=261272</d:LicenseUrl>
    <d:LicenseNames>MS-EULA</d:LicenseNames>
    <d:LicenseReportUrl></d:LicenseReportUrl>
  </m:properties>
</entry>

and https://www.nuget.org/packages/Rx-PlatformServices/2.3.0 shows "unlisted". but I don't find the magic date.

[maartenba]

It does however on https://www.nuget.org/api/v2/Packages?$filter=Id eq 'Rx-PlatformServices' and Version eq '2.3'

[forki]

So they give different results. Da fuq?
On Nov 6, 2014 6:45 PM, "Maarten Balliauw" notifications@github.com wrote:

It does however on https://www.nuget.org/api/v2/Packages?$filter=Id eq
'Rx-PlatformServices' and Version eq '2.3'

—
Reply to this email directly or view it on GitHub
#327 (comment).

[maartenba]

[forki]

I can confirm now. The two different APIs give different Publishing dates.

[forki]

thanks to @maartenba's excellent help it's now fixed in 0.11.16.

You might have to use paket update -fsince the OData result is usually cached.

[madhon]

Excellent news, thanks to all

[JonCanning]

Can I include unlisted packages if needed? RavenDB.Client requires the unlisted Microsoft.CompilerServices.AsyncTargetingPack

[forki]

That's a joke right?

Which ravendb package?
On Nov 12, 2014 2:36 PM, "Jon Canning" notifications@github.com wrote:

Can I include unlisted packages if needed? RavenDB.Client requires the
unlisted Microsoft.CompilerServices.AsyncTargetingPack

—
Reply to this email directly or view it on GitHub
#327 (comment).

[sergey-tihon]

https://www.nuget.org/packages/RavenDB.Client/ ;)

[isaacabraham]

Wonderful :-)

[forki]

please try 0.13.1 - it should work with RavenDB.Client

[mexx]

It's not good, but it's absolutely valid scenario.
Imaging you as package author state a dependency to a package and the maintainer of this one decides afterwards to unlist it, but the one published by you is still listed.
So what should the library author do in this case?
Sure she should provide an update asap and unlist own package versions depending on unlisted one.

But even than, if a Paket user have specified a dependency and it have dependency on unlisted package and no other version present, Paket should warn the user, but not fail.

@forki is awesome, fix arrives while I'm writing the comment ;)

[JonCanning]

You sir, are a scholar and a gentleman

[forki]

This is a very special edge case since there is not a single unlisted package anymore. Otherwise it would have worked in 0.13.0

[forki]

@maartenba @jeffhandley I think the API just broke: see https://www.nuget.org/api/v2/Packages?$filter=Id%20eq%20%27Rx-PlatformServices%27%20and%20Version%20eq%20%272.3.0%27

gives me:

<?xml version="1.0" encoding="utf-8"?>
<feed xml:base="https://www.nuget.org/api/v2/" xmlns="http://www.w3.org/2005/Atom" xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
  <id>https://www.nuget.org/api/v2/Packages</id>
  <title type="text">Packages</title>
  <updated>2014-11-13T10:12:51Z</updated>
  <link rel="self" title="Packages" href="Packages" />
  <author>
    <name />
  </author>
</feed>

This worked 2 days ago!

[yannisgu]

NuGet v3! Maybe...

[forki]

I'm calling v2 here. It just gives different result. This breaks Paket and maybe even nuget.exe

[maartenba]

@jeffhandley @forki Seems the API indeed had a breaking change...

Version seems to now hold 2.3
NormalizedVersion now seems to hold 2.3.0

https://www.nuget.org/api/v2/Packages?$filter=Id eq 'Rx-PlatformServices' and Version eq '2.3.0' --> null
https://www.nuget.org/api/v2/Packages?$filter=Id eq 'Rx-PlatformServices' and Version eq '2.3' --> result
https://www.nuget.org/api/v2/Packages?$filter=Id eq 'Rx-PlatformServices' and NormalizedVersion eq '2.3.0' -> result

This is a breaking change: TeamCity, ProGet, MyGet, Artifactory, Nexus etc. all support the first URL from this list, and internally use the raw Packages collection as well (e.g. MyGet's proxy feeds, ProGet's feed connectors). Newer clients will not break as they tend to use the Search API, older clients will barf on this I think.

See later in thread.

[forki]

Thanks Maarten I thought I'm stupid.

just to complete the list:

https://www.nuget.org/api/v2/Packages?$filter=Id%20eq%20%27Rx-PlatformServices%27%20and%20NormalizedVersion%20eq%20%272.3%27 --> null

[jeffhandley]

We'll take a look, but I don't think we made any changes here. We haven't really been touching the v2 API while preparing to release v3 Preview, so I'm not sure what would have happened to cause this behavior to change.

I know that Version vs. NormalizedVersion has been a distinction for a while, where Version exposes the raw version specified in the package (and this package says "2.3" not "2.3.0") whereas NormalizedVersion is normalized to a SemVer format.

You say this worked 2 days ago. With this exact package id/version pair, looking for it with "2.3"? Or is it possible that this edge case of this package having "2.3" specified in the nuspec was not detected before?

[jeffhandley]

BTW Both of the following work:

1. nuget.exe install Rx-PlatformServices -Version 2.3.0
2. nuget.exe install Rx-PlatformServices -Version 2.3

[forki]

Thanks for the info. Maybe something else returned 2.3 instead of 2.3.0
On Nov 13, 2014 11:39 AM, "Jeff Handley" notifications@github.com wrote:

We'll take a look, but I don't think we made any changes here. We haven't
really been touching the v2 API while preparing to release v3 Preview, so
I'm not sure what would have happened to cause this behavior to change.

I know that Version vs. NormalizedVersion has been a distinction for a
while, where Version exposes the raw version specified in the package (and
this package says "2.3" not "2.3.0") whereas NormalizedVersion is
normalized to a SemVer format.

You say this worked 2 days ago. With this exact package id/version pair,
looking for it with "2.3"? Or is it possible that this edge case of this
package having "2.3" specified in the nuspec was not detected before?

—
Reply to this email directly or view it on GitHub
#327 (comment).

[maartenba]

Seems to be related to this specific package. Tried a couple of others and there the Version is still intact.
Any chance there is an activity log for this specific package to see what changed behaviour on it?

[jeffhandley]

Yeah, you'd likely see 2.3 show up in some places, including from the nuspec itself. If you download the nupkg for the package and crack it open, you'll see "2.3" for the version. We normalize those versions before we bake them in anywhere in the NuGet VS client. And then we normalize versions while asking for package restore and whatnot too. But we have to expose Version verbatim in some places for backwards compat from before we normalized versions.

[forki]

Ok I think I found the issue. Thanks everyone for helping.

The new api v3 gives the normalized version. The old v2 gives 2.3.

The good news is: it's not a breaking change. Just something we have to
consider during nuget v3 vs v2 usage.

Thanks
On Nov 13, 2014 11:44 AM, "Jeff Handley" notifications@github.com wrote:

Yeah, you'd likely see 2.3 show up in some places, including from the
nuspec itself. If you download the nupkg for the package and crack it open,
you'll see "2.3" for the version. We normalize those versions before we
bake them in anywhere in the NuGet VS client. And then we normalize
versions while asking for package restore and whatnot too. But we have to
expose Version verbatim in some places for backwards compat from before we
normalized versions.

—
Reply to this email directly or view it on GitHub
#327 (comment).

[jeffhandley]

Cool. Yeah, API v3 is going to try to hide a lot of the old quirks like this, normalizing versions through and through. If you always use NormalizedVersion against API v2, you should be good.

[forki]

Yes. Very cool. Thanks
On Nov 13, 2014 11:49 AM, "Jeff Handley" notifications@github.com wrote:

Cool. Yeah, API v3 is going to try to hide a lot of the old quirks like
this, normalizing versions through and through. If you always use
NormalizedVersion against API v2, you should be good.

—
Reply to this email directly or view it on GitHub
#327 (comment).

[forki]

The only problem is the fallback. Not every feed supports v3 so we have to
normalize before we ask v2 for package details, right?
On Nov 13, 2014 11:53 AM, "Steffen Forkmann" sforkmann@gmail.com wrote:

Yes. Very cool. Thanks
On Nov 13, 2014 11:49 AM, "Jeff Handley" notifications@github.com wrote:

Cool. Yeah, API v3 is going to try to hide a lot of the old quirks like
this, normalizing versions through and through. If you always use
NormalizedVersion against API v2, you should be good.

—
Reply to this email directly or view it on GitHub
#327 (comment).

[jeffhandley]

Right. I'm pretty sure our clients have been normalizing on all requests for a long time.

[forki]

Ok thanks. That's something we can easily do.
On Nov 13, 2014 11:57 AM, "Jeff Handley" notifications@github.com wrote:

Right. I'm pretty sure our clients have been normalizing on all requests
for a long time.

—
Reply to this email directly or view it on GitHub
#327 (comment).

[forki]

It actually worked and cut our resolver time in half.

[jeffhandley]

When asking for normalized versions, we will hijack the request and serve it from the search service instead of the db. So that makes sense.

[forki]

no that part isn't affected since we cache this data already on disk.
It's just the "give me all version no.s of a given package" that improves things.