Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chocolatey doesn't always decompress downloads appropriately (support automatic decompression) #1056

Closed
pascalberger opened this issue Nov 17, 2016 · 31 comments

Comments

@pascalberger
Copy link
Contributor

What You Are Seeing?

Vsix file downloaded using Install-ChocolateyVsixPackage is zipped, which results in checksum validation error.

What is Expected?

File should be downloaded as Vsix, without being packed in a Zip file.

How Did You Get This To Happen? (Steps to Reproduce)

Try installing this package.
Source code here

@ferventcoder
Copy link
Member

I wonder if when it downloads it, it comes down as a zip ?

@ferventcoder
Copy link
Member

Since this skips the verifier - can you provide a full output log (debug/verbose output)? Thanks!

@ferventcoder
Copy link
Member

A link to a gist would be fine, or formatted here would also work.

@pascalberger
Copy link
Contributor Author

@ferventcoder The downloaded file is named coderush-vs2015.vsix but is actually a Zip file containing the Vsix file.

Here is the log output.

@ferventcoder
Copy link
Member

Ignoring the checksum issue for just a second, can it actually install?

@ferventcoder
Copy link
Member

as a zip that is

@pascalberger
Copy link
Contributor Author

No, it won't install (since it is a zip and not a vsix)

@ferventcoder
Copy link
Member

That seems to be a much bigger issue than just failing the checksum.

@ferventcoder ferventcoder added this to the 0.10.4 milestone Nov 21, 2016
@pascalberger
Copy link
Contributor Author

@ferventcoder Any info I can provide additionally to help?

@ferventcoder
Copy link
Member

https://devexpress.gallerycdn.vsassets.io - perhaps they are looking at the user agent and giving out a zip instead of a vsix? Are you seeing this same issue with visx files coming from the Microsoft gallery?

@pascalberger
Copy link
Contributor Author

This is actually the Microsoft gallery (they switched recently to the Visual Studio Marketplace). Or did you mean with the old URL?

@ferventcoder
Copy link
Member

Ah! I saw devexpress there and thought it was something else. I did mean the old URLs.

@pascalberger
Copy link
Contributor Author

Mmh, installing https://chocolatey.org/packages/coderush-vs2015/16.1.8, which uses the visualstudiogallery.msdn.microsoft.com URL, seems to work.

@ferventcoder
Copy link
Member

Perhaps we can talk to those folks to see what can be done.

@pascalberger
Copy link
Contributor Author

Sure, I'll contact them about this

@AdmiringWorm
Copy link
Member

@ferventcoder @pascalberger from what I could gather it looks like choco doesn't handle gzip encoding correctly.
No matter the Accept-Encoding Header you set they a gzip stream is returned (which I believe is handled automatically in powershell, but not in the choco helper function).

first time I've seen this happen though.

image

@ferventcoder
Copy link
Member

So this is maybe an issue with choco then.

@AdmiringWorm
Copy link
Member

it's just a possibility, I'm not entirely sure.
but from what I gather, it certainly looks like it is.

@AdmiringWorm
Copy link
Member

found it
looks like there is a need to manually set the HttpWebRequest.AutomaticDecompression to System.Net.DecompressionMethods.GZip manually, but doing that may potentially break packages which require no decompression (I think).

Perhaps allowing a parameter to set the decompression method? (GZip, deflate or none)

@ferventcoder
Copy link
Member

If the response header has GZip, it seems like it would make sense to set it then. Thoughts?

@AdmiringWorm
Copy link
Member

true, it would make sense for that.
but I don't think that is possible without creating multiple web requests,
according to the docs it may throw the following exception

InvalidOperationException -> The object's current state does not allow this property to be set

I'm just guessing this would happen after parsing the content-encoding header.

@ferventcoder
Copy link
Member

@AdmiringWorm We already create multiple requests. 👍

@AdmiringWorm
Copy link
Member

ah yes, I see that now.

the auto detection won't solve this issue entirely.
unless a web browsers user-agent, or an accept-encoding: gzip is passed with the request to the marketplace, the response header content-encoding is not sent.
But that it something that can be set by the maintainer though.

@pascalberger
Copy link
Contributor Author

@AdmiringWorm Where can I set this as the maintainer? I'm actually using the Install-ChocolateyVsixPackage CmdLet, which wrapps all the requests. Or can I pass the options through somehow?

@AdmiringWorm
Copy link
Member

@pascalberger to set the the accept-encoding or any other headers you can use the options parameter:
ie:

$parameters = @{
  # Other parameters here

  Options = @{
    Headers = @{
      'Accept-Encoding' = 'gzip'
      'User-Agent' = 'whatever you want'
      # Other headers, except Connection
    }
  }
}

However this won't solve anything yet, not until the downloader is able to decompress a gzipped stream

@AdmiringWorm
Copy link
Member

Just a small update from me regarding this,
I noticed today that ever since my last comment I had been running choco with the following added to
the `Get-WebFile.ps1' file:

$req.AutomaticDecompression = [System.Net.DecompressionMethods]::GZip -bor [System.Net.DecompressionMethods]::Deflate

That is what I had added to try installing packages from the marketplace (which then worked),
and considering I haven't even noticed anything different when upgrading/installing packages I would believe it to be safe to add.

I can open a PR for it if you're interested.

/cc @ferventcoder

@pascalberger
Copy link
Contributor Author

@ferventcoder @AdmiringWorm Any update on this?

@ferventcoder
Copy link
Member

ferventcoder commented Feb 8, 2017 via email

@pascalberger
Copy link
Contributor Author

@AdmiringWorm Interested in doing the PR? Otherwise I can also give it a try.

@AdmiringWorm
Copy link
Member

@pascalberger Feel free to give it a try if you want.

pascalberger added a commit to pascalberger/choco that referenced this issue Mar 6, 2017
@ferventcoder ferventcoder self-assigned this Mar 9, 2017
ferventcoder added a commit that referenced this issue Mar 9, 2017
When grabbing web headers and the remote file name, also support
automatic decompression to match what is done for Get-WebFile.
ferventcoder added a commit that referenced this issue Mar 9, 2017
* pr1188:
  (GH-1056) Support auto decompression for Headers
  (GH-1056) Add support for automatic decompression to Get-WebFile
ferventcoder added a commit that referenced this issue Mar 9, 2017
* stable:
  (GH-1056) Support auto decompression for Headers
  (GH-1056) Add support for automatic decompression to Get-WebFile
@ferventcoder
Copy link
Member

This will be in 0.10.4

@ferventcoder ferventcoder changed the title Files downloaded using Install-ChocolateyVsixPackage are packed in Zip file Chocolatey doesn't always decompress downloads appropriately (support automatic decompression) Mar 20, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants