Use the raw stream to prevent decoding the response

[dstufft]

No description provided.

[tomprince]

This seems to cause the exact opposite error on other servers.

[dstufft]

Can you give an example?

On Feb 5, 2014, at 2:28 PM, Tom Prince notifications@github.com wrote:

This seems to cause the exact opposite error on other servers.

—
Reply to this email directly or view it on GitHub.

[tomprince]

pip install -U --force-reinstall --no-index --find-links http://data.hybridcluster.net/pip-1435 ujson==1.33

[tomprince]

Exception:
Traceback (most recent call last):
  File "/home/tomprince/dev/b00ef9c48c7082c6/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/home/tomprince/dev/b00ef9c48c7082c6/lib/python2.7/site-packages/pip/commands/install.py", line 274, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/home/tomprince/dev/b00ef9c48c7082c6/lib/python2.7/site-packages/pip/req.py", line 1173, in prepare_files
    self.unpack_url(url, location, self.is_download)
  File "/home/tomprince/dev/b00ef9c48c7082c6/lib/python2.7/site-packages/pip/req.py", line 1320, in unpack_url
    retval = unpack_http_url(link, location, self.download_cache, self.download_dir, self.session)
  File "/home/tomprince/dev/b00ef9c48c7082c6/lib/python2.7/site-packages/pip/download.py", line 587, in unpack_http_url
    unpack_file(temp_location, location, content_type, link)
  File "/home/tomprince/dev/b00ef9c48c7082c6/lib/python2.7/site-packages/pip/util.py", line 621, in unpack_file
    unzip_file(filename, location, flatten=not filename.endswith(('.pybundle', '.whl')))
  File "/home/tomprince/dev/b00ef9c48c7082c6/lib/python2.7/site-packages/pip/util.py", line 491, in unzip_file
    zip = zipfile.ZipFile(zipfp)
  File "/usr/lib/python2.7/zipfile.py", line 766, in __init__
    self._RealGetContents()
  File "/usr/lib/python2.7/zipfile.py", line 807, in _RealGetContents
    raise BadZipfile, "File is not a zip file"
BadZipfile: File is not a zip file

Also:

$ pip install -d . --no-index --find-links http://data.hybridcluster.net/pip-1435 ujson==1.33
$ file usjon.zip
ujson-1.33.zip: gzip compressed data, from Unix
$ zcat ujson-1.33.zip | file -
/dev/stdin: Zip archive data, at least v2.0 to extract
$ curl http://data.hybridcluster.net/pip-1435/ujson-1.33.zip | file -
/dev/stdin: Zip archive data, at least v2.0 to extract

[dstufft]

So, the reason this happens is your server is gzip encoding a zip file and we no longer decode that by default because some servers if you tell them to serve a .tar.gz file they serve an application/tar file with Content-Encoding: gzip. This probably previously worked because prior to this we didn't send the header to accept gzip encoding.

Strictly speaking we should be decoding Content-Encoding I suppose. We'll have to figure out how to solve this better.

[tomprince]

I also put a setuptools tar.gz there, and it gets double compressed and so pip chokes on it as well.

[dstufft]

Probably the right answer is to revert this, and make pip able to tell if we have a plain tarfile or a tar.gz even though the filename will be .tar.gz. Historically we've just used the filename for this.

[dstufft]

Grr, that's not completely right either, because it was breaking checksums too. Probably we're going to have to pick which one to support.

[tomprince]

This is a (I think) somewhat vanilla apache server.

[tomprince]

I would err on the side of not supporting silly server written by people that don't understand what HTTP headers are supposed to mean.

[dstufft]

No your server is doing it correctly. Well gziping a already compressed file is pointless, but it's not wrong. We switched to doing it incorrectly to fix things for one group, and that broke things for another group.

Although I may have a different/better answer. If we can get requests to stop sending the accept header for gzip compression then I think the .tar.gz case this originally was trying to fix will still work and your server should work fine because it won't be compressing on the fly. I need to see if we can do that with requests and if so if it actually does that.

[tomprince]

Yeah. I just turned gzip compression off in the directory we are actually using, for .zip files.

[SmileyChris]

Yeah, also getting double compressed tar.gz with --find-links