-
-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove symlinks and serve static files with Python #4550
Comments
So I'm:
Instead of reimplementing |
My position here is making this in movement in two phases:
|
This only works on internal files. We could proxy to an external file host, but that would add a decent bit of latency. Probably <30ms, but worth thinking about. This is what packages.python.org is doing currently w/ S3, so might be worth asking them how it's working. |
I think we decided to go in this direction all together with "El proxito" (an app that will receive all the requests and translate an URL into the path of that file in blob storage and this file will be proxy). I'm closing this issue here. We can revisit it if we need it when implementing El Proxito. |
Currently all of our documentation pages are served by Nginx. We maintain this with a web of symlinks that is pretty error prone and hard to maintain. We also throw a large number of errors in these code paths, trying to manipulate the filesystem in wonky ways.
We currently already have a pattern that solves this problem: Sendfile. It allows us to handle processing of the request in Python, but still have Nginx serve the file.
This logic would be a combination of our current redirect logic, which doesn't hit the DB, and existing Sendfile support. These nginx docs cover the usage: https://www.nginx.com/resources/wiki/start/topics/examples/x-accel/ -- we already do this in a couple places:
The primary difference is that we need to be able to do it without hitting the database.
Benefits
Considerations
Requirements
Implementation
metadata.json
for each project, allowing us to make more decisions without hitting the database (existing code: https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/projects/tasks.py#L1125)metadata.json
and then served the correct file off disk. We could only keep theuser_builds
directory around, and the Python app would be in charge of translating the URL to the filepath to serve, accounting for subprojects, translations, etc.The text was updated successfully, but these errors were encountered: