Slow when multiple repos

[vchekan]

When I have many repositories, paket works quite slow because it seems to me, paket does evaluation of every source with all supported protocols until one response is successful.

The worst case I experienced, was converting from nuget of a large project, which use our internal nuget repo mirror of nuget.org and another one, based on CI server. In process of conversion, paket ignored disable section of nuget.config, and added back nuget.org for both v2 and v3. Thus, import ended up with 4 servers, nuget v2,v3 and 2 of our internal ones. Repeating every protocol for every repository for every package in large project took >1hr :(

I'm thinking, would it be possible to have lazily evaluated member in repository object, which would probe for supported version upon first call and than use only the successful protocol for given repository? This way it will eliminate a lot of failed http requests and save time.

[forki]

For the convert-from-nuget issue I created #1098

Regarding feed queries: we query all configured feeds that are written above a nuget package line in the dependencies file. An easy workaround is the following:

source http://fastfeed

nuget FastPackage1 // looks only into http://fastfeed
nuget FastPackage2 // looks only into http://fastfeed

source http://slowfeed
nuget SlowPackage1 // loks into http://slowfeed and http://fastfeed

"which would probe for supported version upon first call and than use only the successful protocol for given repository"

yes that is some I have planned for long time. Hope to implement that soon

[forki]

first results are coming in. 2.5.0-alpha002 is 4.2x faster on such a scenario with multiple feeds. try it out

[vchekan]

I've tried e317a91. Marginally faster. Still 55min.
For some reason it keeps trying to interpret v3 json as xml.

{
  "@id": "https://www.nuget.org/packages/{id}/{version}/ReportAbuse",
  "@type": "ReportAbuseUriTemplate/3.0.0-beta",
  "comment": "URI template used by NuGet Client to construct Report Abuse URL for packages"
},
{
  "@id": "https://api.nuget.org/v3/stats0/totals.json",
  "@type": "TotalStats/3.0.0-rc",
  "comment": "Endpoint to get stats totals to display in nuget.org home page"
}

],
"@context": {
"@vocab": "http://schema.nuget.org/services#",
"comment": "http://www.w3.org/2000/01/rdf-schema#comment"
}
}
Source 'https://api.nuget.org/v3/index.json' exception: System.Xml.XmlException: Data at the root level is invalid. Line 1, positi
on 1.
at System.Xml.XmlTextReaderImpl.Throw(Exception e)
at System.Xml.XmlTextReaderImpl.ParseRootLevelWhitespace()
at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
at System.Xml.XmlLoader.Load(XmlDocument doc, XmlReader reader, Boolean preserveWhitespace)
at System.Xml.XmlDocument.Load(XmlReader reader)
at System.Xml.XmlDocument.LoadXml(String xml)
at Paket.NuGetV2.parseODataDetails[a](String nugetURL, PackageName packageName, a version, String raw) in C:\funprojects\Paket
src\Paket.Core\NuGetV2.fs:line 183
at Paket.NuGetV2.getDetailsFromNuGetViaOData@284-9.Invoke(Unit unitVar) in C:\funprojects\Paket\src\Paket.Core\NuGetV2.fs:line
284
at Microsoft.FSharp.Control.AsyncBuilderImpl.callA@851.Invoke(AsyncParams1 args) --- End of stack trace from previous location where exception was thrown --- at Microsoft.FSharp.Control.AsyncBuilderImpl.commit[a](Result1 res)
at Microsoft.FSharp.Control.CancellationTokenOps.RunSynchronously[a](CancellationToken token, FSharpAsync1 computation, FSharp Option1 timeout)
at Microsoft.FSharp.Control.FSharpAsync.RunSynchronously[T](FSharpAsync1 computation, FSharpOption1 timeout, FSharpOption1 c ancellationToken) at Paket.NuGetV2.tryNext@622(String root, Boolean force, FSharpList1 sources, PackageName packageName, SemVerInfo version, FSh
arpList`1 xs) in C:\funprojects\Paket\src\Paket.Core\NuGetV2.fs:line 629

Trying source 'https://www.nuget.org/api/v2'

[forki]

could you please mail me your dependencies and lock files? sforkmann @ gmail

1h update is very very unusal. Maybe I can find the issue.

[forki]

latest version runs all calls against different NuGet protocols in parallel and takes the fastest response. But please still send the dependencies and lock files.

[vchekan]

It is convert-from-nuget command.
Here is initial nuget.config:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <config>
       <add key="repositorypath" value="$(Solutiondir)\..\packages" />
    </config>

    <disabledPackageSources>
         <add key="nuget.org" value="true" />
         <add key="mirror" value="true" />
         <add key="NTent" value="true" />
    </disabledPackageSources>

    <packageSources>
      <add key="nuget-proxy" value="http://localhost:8081/nexus/service/local/nuget/ntent/" />
    </packageSources>
</configuration>

As you see, all sources except nexus is disabled
When I run import, it generate paket.dependencies:

source http://ci-server.cb.ntent.com/guestAuth/app/nuget/v1/FeedService.svc/
source https://api.nuget.org/v3/index.json
source https://www.nuget.org/api/v2/
source http://localhost:8081/nexus/service/local/nuget/ntent/

nuget angularjs 1.4.3 framework: >= net45
nuget AngularTemplates.Compile 1.0.0 framework: >= net45
nuget Antlr 3.4.1.9004 framework: >= net45
nuget Autofac 3.5.0 framework: >= net45
nuget Autofac.Owin 3.1.0 framework: >= net45
nuget Autofac.WebApi 3.1.0 framework: >= net45
nuget Autofac.WebApi2 3.4.0 framework: >= net45
nuget Autofac.WebApi2.Owin 3.2.0 framework: >= net45
...
93 lines, 15 of which is private packages on local repo

4 "source" is more than I'd like to have, and because it is created during the import, there is no opportunity to edit them out until import is complete :(

I am going to use Sonatype's Nexus server, which is more known for Maven support, but it has very good nuget server implementation, in particular, so called "group server", which is a proxy which will do requests to multiple sources and cache result. So it would be extremely benefitial to avoid requests to disabled servers.

BTW, I'm at today's build, 8e4a0e3

Is there any way to redirect output to a text file? Import from nuget will ask for some confirmations on console which makes it impossible to redirect output :(

[forki]

4 "source" is more than I'd like to have, and because it is created during the import, there is no opportunity to edit them out until import is complete :

ah so you are complaining about the convert-from-nuget performance? fortunately that is a step that you only do once. After that you want to edit your paket.dependencies file and remove unneeded sources. you also might want to run simplify and relax some of the version restrictions.
If you do that, then what is the runtime for paket update?

(For the convert-from-nuget issue I created #1098 and @theimowski will take a look)

[theimowski]

@vchekan do I understand correctly that those feeds listed in disabled section are defined in different, higher-level nuget.config file and are identified by the key attribute?

[vchekan]

@theimowski yes, that's correct.

[theimowski]

Ok I think I know how it could be fixed. Gonna try send a fix this weekend

[forki]

@vchekan Paket 2.7.2 has fixed convert-from-nuget. Thanks to @theimowski

[vchekan]

Measured my benchmark project, it takes now 2 minutes and 8 seconds to import a project which took 1hr before. Great job!