Paket mixed responses and downloads

[SavaNDragos]

I found a pretty ugly bug on the new servers (not sure if it matters but the servers are windows server 2012 with Jenkins). We also are using our internal feed.
The bug is like this:
When paket asks for details related to a package and you have more packages for which it needs details, it will end up mixing the reponses and also the downloads:
From what I can tell there isn't a secure way to receive the callback from the nuget feed and they look to be processed randonly.

So from the log:
...
Response from http://nuget.mycompany.net/NugetWebFeed/nuget/Packages?$filter=Id eq 'Graphite.NET' and Version eq '1.1':

Graphite.NET
<feed xml:base="http://nuget.mycompany.net/NugetWebFeed/nuget/"....

<title type="text">Packages</title> ... http://nuget.mycompany.net/NugetWebFeed/nuget/Packages(Id='Mycompany.Assambly',Version='0.0.075.8635') <title type="text">Mycompany.Assambly</title>

In from of the response I added the package name (for debug purpose: Graphite.NET <?xml version="1.0).
Now what is more funny is because everything is parallel you can end up getting as a response a package:

Response from http://nuget.mycompany.net/NugetWebFeed/nuget/Packages?$filter=Id eq 'Microsoft.Web.Services3' and NormalizedVersion eq '3.0.0':

Microsoft.Web.Services3 PK��

5�Gզ�� �� � � _

The mix happens when the downloads take place also. So even if you received a good response you can get a wrong package.

At the end the builds fail because in packages\apackage you will have binaries belonging to anothe package.

This is happening on all our new servers.

At the moment I tried to limit the calls to the internal feed [singleton when making calls] and the response get back ok, but there is no parallel anymore.
We want to use the official release so any suggestions?

[SavaNDragos]

I should not this is happening with both restore and update (stuff i tested)

[forki]

That sounds really strange. The parallel calls are not matching the results? So that's a bug in the async parallel calls?

/cc @dsyme ever heard of something like that in other F# code?

[forki]

Is there a way to reproduce this behavior? What did you change exactly to make it work?

[SavaNDragos]

I used Monitor.Enter Paket.Constants.InUse in the download packages method but this is not ok because it is not intended to be used in async

[forki]

Could you also please test with version 1.21.1?

[SavaNDragos]

Have no idea how to reproduce it, on my machine it works fine but on the new build servers it happens.

[forki]

Any details about the installed .NET framework versions? And does the
1.21.1 work?
On Aug 7, 2015 11:29 AM, "SavaNDragos" notifications@github.com wrote:

Have no idea how to reproduce it, on my machine it works fine but on the
new build servers it happens.

—
Reply to this email directly or view it on GitHub
#966 (comment).

[SavaNDragos]

Tested it with 1.21.1 and is still happening.
the machine have .net framework 4.5.2 and VS 2013.

[forki]

Ok good, so at least this is not an error in the latest FSharp.Core.
I'm currently in vacation, but I'd love to investigate this on Monday. I
hope we can find the issue.
On Aug 7, 2015 11:45 AM, "SavaNDragos" notifications@github.com wrote:

Tested it with 1.21.1 and is still happening.
the machine have .net framework 4.5.2 and VS 2013.

—
Reply to this email directly or view it on GitHub
#966 (comment).

[SavaNDragos]

I rerun the same tests i have done on windows 8 and everything works fine. Also the old servers are fine (Windows server 2008 with .net framework 4.5.2 and VS 2010, 2012, 2013).
Conclusion is only affected from what I have seen Windows Server 2012 R2 (maybe the R2 matters).

[SavaNDragos]

Ok so played a little bit more: so if we make some async operations to runSynchroniosly paket works fine (just strange, was expected to fail after a couple of tries but it looks fine).
My changes where:
1.In let getFromUrl (auth:Auth option, url : string) ... return client.DownloadString(Uri(url))
2.in let getXmlFromUrl (auth:Auth option, url : string) ... return client.DownloadString(Uri(url))
3.in let safeGetFromUrl (auth:Auth option, url : string) ... let raw = client.DownloadString(Uri(url))
In let getDetailsFromNuGetViaODataFast\let getDetailsFromNuGetViaOData:
4.5.let raw = getFromUrl(auth,url) |> Async.RunSynchronously
6.let raw =
match response with
| Some(r) -> async { return r } |> Async.RunSynchronously
| None ->
let url = sprintf "%s/odata/Packages(Id='%s',Version='%s')" nugetURL package (version.ToString())
getXmlFromUrl(auth,url) |> Async.RunSynchronously

[forki]

Could you please send a pull request for easier diff?

Btw: thanks for being so patient and helping with finding the root cause.
On Aug 7, 2015 16:18, "SavaNDragos" notifications@github.com wrote:

Ok so played a little bit more: so if we make some async operations to
runSynchroniosly paket works fine (just strange, was expected to fail after
a couple of tries but it looks fine).
My changes where:
1.In let getFromUrl (auth:Auth option, url : string) ... return
client.DownloadString(Uri(url))
2.in let getXmlFromUrl (auth:Auth option, url : string) ... return
client.DownloadString(Uri(url))
3.in let safeGetFromUrl (auth:Auth option, url : string) ... let raw =
client.DownloadString(Uri(url))
In let getDetailsFromNuGetViaODataFast\let getDetailsFromNuGetViaOData:
4.5.let raw = getFromUrl(auth,url) |> Async.RunSynchronously
6.let raw =
match response with
| Some(r) -> async { return r } |> Async.RunSynchronously
| None ->
let url = sprintf "%s/odata/Packages(Id='%s',Version='%s')" nugetURL
package (version.ToString())
getXmlFromUrl(auth,url) |> Async.RunSynchronously

—
Reply to this email directly or view it on GitHub
#966 (comment).

[SavaNDragos]

I have created the pull #969

[SavaNDragos]

Have you managed to look over the pull request?

[forki]

Yes, but I try to reproduce this on our own 2012 R2 servers. No luck. Does
paket's own build work for you?
On Aug 10, 2015 2:31 PM, "SavaNDragos" notifications@github.com wrote:

Have you managed to look over the pull request?

—
Reply to this email directly or view it on GitHub
#966 (comment).

[SavaNDragos]

the build with the modifications i made works perfect
for replication you need the old build (whitout the proposed changes)

[forki]

yes that's what I meant. if you clone the current Paket repo on your 2012
R2 servers and just run build.cmd - does that repro the failure for you?

2015-08-10 14:48 GMT+02:00 SavaNDragos notifications@github.com:

the build with the modifications i made works perfect
for replication you need the old build (whitout the proposed changes)

—
Reply to this email directly or view it on GitHub
#966 (comment).

[SavaNDragos]

Everything executed ok so all tests pass.

[forki]

yep same here I tried to run this like 10 times. I can't reproduce your error.
can you create a small project were it reproduces? I really want to fix that guy

[forki]

/cc @part-timeDev

[SavaNDragos]

We are using a lot of internal libraries from our internal feed, tried to replicate with only libraries from nuget.org but couldn't. Also I downgraded a paket to nuget and nuget has no issue with getting the packages.
No idee why the issue happens but I know that is in the async code.

[forki]

yes I understand that's related to the async code (maybe there is a compiler/jit bug). but I need to debug the real issue. If I just disable the parallel stuff then we have slow restore and don't really know why.

[forki]

you replaced AsyncDownloadString (from FSharp.Core) with DownloadString. How about trying a task based async version and converting to F#'s async!?

[forki]

look at #971 I wonder if that would work better.

[SavaNDragos]

going to test it in the next 30 min

[forki]

added also DownloadFileTaskAsync

[SavaNDragos]

I tested it several times (the changes you made) and it works. Can you create a new version?

[forki]

@dsyme @latkin @mexx @tpetricek this suggestes that the bug is somewhere in https://github.com/fsharp/fsharp/blob/0313094fa5d19fd871c83e7e45c35d1fad65daf9/src/fsharp/FSharp.Core/control.fs#L2156

@SavaNDragos yes will do.

[forki]

@SavaNDragos done.

[SavaNDragos]

thanks, I will start now to deploy it on the servers :d

[SavaNDragos]

Run it on the build server with a clean cache and error is still there (less packages failling).

[SavaNDragos]

This is absurd, the changes you made works but the executable I downloaded seem to have an issue, so i downloaded the code again compiled it obtained the lose files and deployed them and succes. Will wait till tomorrow to see what happens. [one server with run with the downloalded executable also]

[SavaNDragos]

"I downloaded seem to have an issue" .. the same issue

[forki]

Maybe I messed up the release. Will check later
On Aug 10, 2015 7:20 PM, "SavaNDragos" notifications@github.com wrote:

"I downloaded seem to have an issue" .. the same issue

—
Reply to this email directly or view it on GitHub
#966 (comment).

[SavaNDragos]

I think the late part was an issue from my side, I installed latest version with choco and the error persisted so I just downloaded the executable and deployed it manually on a build server.
All builds on that server looks great no paket issue dedected.
I will clear the nuget cache on 2 other build servers and deploy paket 1.26 to confirm that everything is perfect.

[forki]

so it's fixed in 1.26.0 and we can close this case?

if yes, then please open an issue at https://gitter.im/Microsoft/visualfsharp - it seems we have identified an issue with AsyncDownloadString in FSharp.Core.

[forki]

@SavaNDragos since you are the only one that can reproduce that error, are you up for further rounds of testing? If the issue is really outside of Paket (whereever that might be) then we still want to fix that.

[SavaNDragos]

sounds good, I have allready cloned a build server

[SavaNDragos]

I think it should be marked as resolved