Initial stub of proxito #6226

ericholscher · 2019-10-01T14:27:56Z

This is an initial attempt at adding a method of doc serving that goes through Django, instead of being done by an Nginx server with symlinks managed by Django. It depends on the Read the Docs codebase, and is mostly just a simplification of the existing serving code. It's done in parallel so that we can deploy it alongside the current running application with just settings changes.

Primary Changes

We now require USE_SUBDOMAINS for development, and default the PUBLIC_DOMAIN to dev.readthedocs.io, which points at 127.0.0.1 via public DNS
This deprecates and replaces the middleware in https://github.com/readthedocs/readthedocs.org/blob/7aaef0d9bb2b10e06d177c2feb09592a5d15cf38/readthedocs/core/middleware.py
This deprecates and replaces the urls in https://github.com/readthedocs/readthedocs.org/blob/7aaef0d9bb2b10e06d177c2feb09592a5d15cf38/readthedocs/core/urls/subdomain.py

Current problems

The tests are all running explicitly against the old middleware which is a different style. I don't know the best way to handle this, and probably need to just fork the tests, but that feels bad :/
Fix: I went ahead and just converted the middleware to the old-style class for testing. I got tired of beating my head against it :)
I also want to convert the doc serving code to be class-based, so we can extend it in a nicer fashion on the .com -- currently the logic has diverged pretty dramatically, and I'm hoping we can get them much closer together with just one or two things changed

Ops changes

There will be a corresponding ops update needed, but the gist of it is:

server {
    listen 80;
    server_name dev.readthedocs.io;
   
    location / {
        // proxito
        proxy_pass http://127.0.0.1:8001;
        proxy_buffering off;
        proxy_intercept_errors on;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;

        error_page 404 420 = @fallback;
        add_header X-Served Root always;
    }

    location /proxito/ {
        internal;
        alias /Users/eric/projects/readthedocs.org/media/;

        error_page 404 420 = @fallback;
        add_header X-Served Nginx-Sendfile always;
    }

   
    location @fallback {
        // RTD
        proxy_pass http://127.0.0.1:8000;
        proxy_buffering off;
        proxy_intercept_errors on;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;

        add_header X-Served Nginx-Fallback always;
    }

}

humitos

The code looks that it's going in a good direction. I didn't test it locally, though. I left only nit comments about style or naming.

I think that most of this code comes from what we already have in other places, but it's easier to follow here since it's alone and split in different functions.

readthedocs/proxito/middleware.py

readthedocs/proxito/views.py

davidfischer

This is starting to take shape and that's great.

What other features are needed for this to launch? User defined redirects?

readthedocs/proxito/middleware.py

readthedocs/settings/proxito.py

readthedocs/proxito/urls.py

humitos · 2019-10-07T15:14:21Z

readthedocs/proxito/views.py

+    # Handle a / redirect when we aren't a single version
+    if all([lang_slug is None, version_slug is None, filename == '',
+            not current_project.single_version]):
+        log.info('Proxito redirect: slug=%s', current_project.slug)
+        return redirect_project_slug(
+            request, project=current_project, subproject=None
+        )


I think I found a change in the behavior of the / redirect.

Currently in production, if we hit the .readthedocs.io URL (https://requests.readthedocs.io/) of a project with a Custom Domain defined, our NGINX will serve the documentation under that URL.

On the other hand, El Proxito is redirecting me to the Custom Domain project URL.

The new behavior is what I like the most (and I think @davidfischer as well) but I just wanted to note that there is a change here that we may need to take into account.

Yea, I'm mixed about what the proper behavior is. I think the root redirects are fine to automatically use the custom domain, but I'm guessing we'll actually want this to be configurable in the future. Sometimes I actually want to be able to go to the readthedocs.io domain even when there's a custom domain.

humitos · 2019-10-07T15:15:15Z

readthedocs/proxito/views.py

+        subproject,
+        lang_slug=None,
+        version_slug=None,
+        filename='',


Can we make this default to be None as well? It makes more sense to me.

Because of the way Django processes the URL regex, it always matches to ''. I could leave it without a default actually, but this seems more explicit?

I see --no strong opinion, then.

humitos · 2019-10-07T15:57:49Z

readthedocs/proxito/views.py

+    # TODO: Redirects need to be refactored before we can turn them on
+    # They currently do 1 request per redirect that exists for the project


I understand that the redirects would work properly if they are enabled at this point. Although, the problem here is being really slow because we will be hitting the db for all the redirects the project have all the times even if there is not match.

humitos · 2019-10-07T16:10:07Z

readthedocs/proxito/views.py

+    # Serve from the filesystem if using DEBUG or Testing
+    # Tests require this now since we don't want to check for the file existing in prod


I'm not sure to follow this. Where this code is checking for file existing?

I understand that PYTHON_MEDIA=True is useful to run this locally and get docs served properly instead of just seeing the file served in the response. I'm not seeing why this is needed for testing, though.

The django serve function will check for files existing. This comment is confusing though, I will rewrite it.

ericholscher · 2019-10-07T17:39:10Z

What other features are needed for this to launch? User defined redirects?

Nothing really. We can fallback to the Django app for all the 404's, so we can continue using all the existing app logic for redirects, 404's, and robots.txt handling. The goal will be to port all of that to proxito, but baby steps :)

humitos

Looks good to me! I left some small comments to consider

readthedocs/proxito/tests/test_full.py

readthedocs/proxito/tests/test_middleware.py

stsewd

Didn't review proxito/views.py looks like it's just a copy of the code from core/views/serve.py?

readthedocs/proxito/middleware.py

stsewd · 2019-10-07T20:49:33Z

readthedocs/proxito/middleware.py

+        # Code to be executed for each request before
+        # the view (and later middleware) are called.
+
+        host_project = map_host_to_project_slug(request)


That function doesn't always return a slug, sometimes it returns a response.

Maybe refactor the function to return a tuple slug, response or try..cath exceptions and move the responses to the other part of the code?

Yea, I wasn't sure the best way to refactor this. It does validation, and we have multiple different responses we want (invalid and also CNAME 404).

stsewd · 2019-10-07T21:01:18Z

readthedocs/proxito/middleware.py

+
+class NewStyleProxitoMiddleware:
+
+    """The new style middleware, I can't figure out how to test it."""


I think you can test the request at least by creating a method process_request(self, request), that way you don't depend on self.get_response or mock it to test with localhost and testserver as hostname.

And I think you only want to test is the request

Yea, the get_response thing just feels hard to test. I think having it old-style for now is fine, and we can figure out the migration in the future.

Just going to delete the new-style middleware for now.

ericholscher

❌

ericholscher · 2019-10-08T10:43:30Z

readthedocs/proxito/views.py

+                )
+                subproject = rel.child
+            except (ProjectRelationship.DoesNotExist, KeyError):
+                subproject = get_object_or_404(Project, slug=subproject_slug)


It seems like we aren't even actually validating the subproject relationship here. It seems like we could accidentally serve a random project as a subproject of another project :/

readthedocs/proxito/views.py

humitos · 2019-10-08T14:24:22Z

readthedocs/settings/proxito.py

+
+These settings will eventually be backported into the main settings file,
+but currently we have them to be able to run the site with the old middleware for
+a staged rollout of the proxito code.


Suggested change

a staged rollout of the proxito code.

a staged rollout of El Proxito code.

😄

Co-Authored-By: Manuel Kaufmann <humitos@gmail.com>

ericholscher added 13 commits August 15, 2019 16:14

Add inital spike of Proxito

d43db8e

More interim work.

b7202ca

Cleanup URLs to have less RTD logic.

092cef9

Put old middleware back in for now

7aaef0d

Cleanup some logic and add logging

532084e

Minimum viable proxito settings

624ac72

Much simpler settings file

0c1b846

Cleanup settings

dd7651c

Cleanup settings changes

cb6c0b7

Merge remote-tracking branch 'origin/master' into proxito-stub

5252400

Remove other changed logic

801a240

Remove dev setting chagnes as well

2384035

Add middleware tests

79c9205

humitos reviewed Oct 2, 2019

View reviewed changes

ericholscher added 6 commits October 2, 2019 15:34

Update lots of tests for proxito

2e8cf67

Cleanup more tests

8138da2

Test invalid input

85b4462

Clean up the tests more

f74173c

Review feedback

d78675f

Last bit of review feedback

339f9a9

davidfischer reviewed Oct 4, 2019

View reviewed changes

readthedocs/proxito/middleware.py Outdated Show resolved Hide resolved

readthedocs/settings/proxito.py Show resolved Hide resolved

readthedocs/proxito/urls.py Outdated Show resolved Hide resolved

ericholscher added 4 commits October 7, 2019 14:13

Remove debug_toolbar from URLs.

03f49d7

Only depend on PYTHON_MEDIA for serving, not DEBUG.

091de9c

Fix linting

512dc15

Fix linting once again

af471dc

humitos reviewed Oct 7, 2019

View reviewed changes

ericholscher added 3 commits October 7, 2019 17:54

Fix lint once more :/

21d0e9e

Fix lies in tests.

38bc63a

Don’t set PYTHON_MEDIA in Proxito setting.

3bdce44

humitos reviewed Oct 7, 2019

View reviewed changes

ericholscher marked this pull request as ready for review October 7, 2019 17:47

ericholscher requested a review from a team October 7, 2019 17:47

Review feedback

4acc04f

humitos approved these changes Oct 7, 2019

View reviewed changes

readthedocs/proxito/tests/test_full.py Show resolved Hide resolved

readthedocs/proxito/tests/test_middleware.py Outdated Show resolved Hide resolved

readthedocs/proxito/tests/test_middleware.py Outdated Show resolved Hide resolved

stsewd reviewed Oct 7, 2019

View reviewed changes

ericholscher added 3 commits October 8, 2019 11:59

Address review feedback

1fb4ec7

Clarify use of settings

5ca6d15

Clarify comment and cleanup more logging

596aa85

ericholscher commented Oct 8, 2019

View reviewed changes

ericholscher added 2 commits October 8, 2019 12:46

Fix lint

2db8a4c

Validate subproject is actually a subproject for the project

644c5ba

humitos reviewed Oct 8, 2019

View reviewed changes

readthedocs/proxito/views.py Outdated Show resolved Hide resolved

humitos reviewed Oct 8, 2019

View reviewed changes

stsewd approved these changes Oct 8, 2019

View reviewed changes

ericholscher and others added 2 commits October 9, 2019 08:54

Update readthedocs/proxito/views.py

ed62904

Co-Authored-By: Manuel Kaufmann <humitos@gmail.com>

Fix lint

c6d6d83

ericholscher merged commit 2a7b6d3 into master Oct 9, 2019

ericholscher deleted the proxito-stub branch October 9, 2019 11:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial stub of proxito #6226

Initial stub of proxito #6226

ericholscher commented Oct 1, 2019 •

edited

Loading

humitos left a comment

davidfischer left a comment

humitos Oct 7, 2019

ericholscher Oct 7, 2019

humitos Oct 7, 2019

ericholscher Oct 8, 2019

humitos Oct 8, 2019

humitos Oct 7, 2019

ericholscher Oct 8, 2019

humitos Oct 7, 2019

ericholscher Oct 8, 2019

ericholscher commented Oct 7, 2019

humitos left a comment

stsewd left a comment

stsewd Oct 7, 2019

stsewd Oct 7, 2019

ericholscher Oct 8, 2019

stsewd Oct 7, 2019

stsewd Oct 7, 2019

ericholscher Oct 8, 2019

ericholscher Oct 8, 2019 •

edited

Loading

ericholscher left a comment

ericholscher Oct 8, 2019

humitos Oct 8, 2019

		# TODO: Redirects need to be refactored before we can turn them on
		# They currently do 1 request per redirect that exists for the project

		# Serve from the filesystem if using DEBUG or Testing
		# Tests require this now since we don't want to check for the file existing in prod


		class NewStyleProxitoMiddleware:

		"""The new style middleware, I can't figure out how to test it."""

	a staged rollout of the proxito code.
	a staged rollout of El Proxito code.

Initial stub of proxito #6226

Initial stub of proxito #6226

Conversation

ericholscher commented Oct 1, 2019 • edited Loading

Primary Changes

Current problems

Ops changes

humitos left a comment

Choose a reason for hiding this comment

davidfischer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericholscher commented Oct 7, 2019

humitos left a comment

Choose a reason for hiding this comment

stsewd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericholscher Oct 8, 2019 • edited Loading

Choose a reason for hiding this comment

ericholscher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericholscher commented Oct 1, 2019 •

edited

Loading

ericholscher Oct 8, 2019 •

edited

Loading