HTTPS not working on Google App Engine

[cpavon]

Take a look here: http://stackoverflow.com/questions/21605328/python-requests-on-google-app-engine-not-working-for-https

Thanks!

[Lukasa]

Unfortunately, Requests explicitly doesn't support GAE. It's just not Python. =)

[piotr-dobrogost]

I guess directing OP to some details on why/how GAE is not Python would be appreciated :)

[Lukasa]

Heh, this is a fair point.

The short answer is that GAE doesn't use the same standard library as core Python does. This is particularly significant with the networking sections (e.g. socket, ssl, httplib) which are radically different from their standard library counterparts.

The number of differences is substantial, the differences themselves can often be subtle or only found after substantial use (e.g. rate limiting), and testing on GAE is relatively expensive for the core dev team to do.

For those reasons, it's easier for us simply to not support GAE. We're prepared to make small code changes if they fix an obvious problem for GAE, but we are disinclined to radically change the codebase for any non-standard Python version (see the recent pull request modifying the codebase for Jython, which was rejected).

If Requests doesn't work on GAE, GAE is bugged. We work fine on Python 3.3, Python 2.7, Python 2.6 and PyPy: GAE can sort itself out. =)

[rattrayalex]

Thanks for the explanation!

[rattrayalex]

I understand fixing AppEngine is not something you intend to pursue. In case it helps anyone else, though, I think I have found the cause: line 172 here. It's my impression that get_netrc_auth essentially fails on AppEngine, and without that, authentication will not survive redirects, resulting in infinite recursive redirects to any endpoint that requires Auth (not just POSTs).

I'm not sure, but the absence of the feature mentioned here: https://github.com/kennethreitz/requests/pull/1892#issuecomment-33730591 , which at the time seemed like a "nice-to-have", may be a factor.

@kennethreitz had seemed concerned that making this design decision for the developer might have adverse consequences... I wonder if there are others outside of the GAE ecosystem who would prefer Authentication tokens survive across redirects.

Perhaps an option to allow this would assuage the GAE community as well as those in similar situations?

[Lukasa]

There are absolutely people who would prefer that we keep the auth tokens on over redirects. However, both @sigmavirus24 and I very strongly believe that being insecure by default is dangerous. @kennethreitz has a more nuanced view than we do, and I acknowledge that. However, I think that #1892 is a good change.

However, this bug cannot be #1892 because we haven't released a version with #1892 in it yet. =)

[rattrayalex]

Well that's embarrassing =)
Thanks for your patience Lukasa.

(for the record, what I intended to suggest was opt-out for #1892, not opt-in).

[Lukasa]

There's no need to thank me, it's an easy mistake to make. I have the advantage of having written that particular fix, so I'm keeping track of it. ;)

The fix currently is opt-out: if you pass the argument allow_redirects=False, you can handle the redirect logic yourself and deal with authentication however you like.

[jacobg]

Hi @rattrayalex Your patch does not seem to work anymore, in particular to connect to Google IP addresses, as such IP's are blocked in GAE's new sockets API: https://cloud.google.com/appengine/docs/python/sockets/

[webmaven]

/CC @jonparrott

[theacodes]

urllib3 now has contrib support for GAE. I haven't personally tested this out, but if you configure requests to use AppEngineManager instead of the normal PoolManager it should (in theory) work. I'll be happy to try to take on any issues with getting that to work.

[sigmavirus24]

@Lukasa think that should go into the toolbelt?

[Lukasa]

Seems reasonable to me.