A basic HTTP proxy in Python, implementing VeNoMouS' cloudscraper module.
Clone this repo, cd into the repo folder and then use setup.py to install:
sudo python setup.py install
Optionally, install python-prctl. This will alow the proxy to change its process name as displayed in ps, netstat and other such software to make it easier to see what the proxy is doing:
sudo pip install python-prctl
Note: python-prctl requires gcc, libc development headers and libcap development headers to be installed in the operating system. See the python-prctl docs for details.
If you're installing this on a router and are using Entware you may need to install some Python modules with opkg, for example:
opkg install python-requests
opkg install node_legacy
The proxy should run on most linux distros and has been tested on:
- Ubuntu Bionic
- Armbian (Debian) Stretch
- ASUSWRT-Merlin / Entware
The proxy won't run on Windows due to differences in handling network interfaces and sockets.
usage: cfscrape-http-proxy [-h] [-c FILE] [--debug] [-l INTERFACE] [-p PORT]
[-e INTERFACE] [-D] [-P FILE]
CloudFlare Anti-Bot HTTP Proxy
optional arguments:
-h, --help show this help message and exit
-c FILE, --config_file FILE
external configuration file
--debug turn on debug messaging
-l INTERFACE, --listen_if INTERFACE
interface to listen to (use 'any' or '0' to listen to all interfaces) (default 'any')
-p PORT, --listen_port PORT
port to listen on (default 8080)
-e INTERFACE, --exit_if INTERFACE
interface for outgoing connections (default <default>)
-D, --daemonize run the proxy as a daemon
-P FILE, --pidfile FILE
PID file for daemon (default '/tmp/cfscrape-http-proxy.pid')
-R, --no_redirect turn off redirects, prevent adding missing elements to URLs in the browser
Command line parameters override default settings and any settings from a configuration file. An external configuration file specified on the command line will override a configuration file installed in the system config path (typically /etc/, or possibly /opt/etc/ by default).
Once the proxy is running it can be used by accessing it via HTTP with the page you wish to proxy included as the url GET variable:
http://<proxy_server_IP>:<proxy_port>/?url=<target_page>
Example: http://192.168.0.10:8080/?url=www.news.com.au/content-feeds/latest-news-world/
The config file (called cfscrape-http-proxy.conf by default, but can be changed via the command line) uses the same names for paramaters as the command line. As an example, a configuration file for use on a router with distinct LAN (br0) and WAN (eth0) interfaces:
[Defaults]
listen_if=br0
port=8080
exit_if=eth0
daemoinze=True
pidfile=/tmp/cfscrape-http-proxy.pid
no_redirect=False
The proxy will search several paths for a configuration file if none is specified on the command line. If you're running Entware it should be at /opt/etc/cfscrape-http-proxy.conf, otherwise it should probably be at /etc/cfscrape-http-proxy.conf.
The default exit interface (shown as <default> in the usage above), used if nothing is specified in the config file or on the command line, will vary from one system to another. The proxy determines the default by looking at the routing table.
The no_redirect flag will make the proxy faster by not sending HTTP 301 responses to add ?url=http://<domain>/ to the beginning of requests that don't include it (i.e. relative URLs in links on a web page), but it may make browsing web pages through the proxy buggier. It's a trade between speed and robustness, you can experiment to see which you prefer.
Installing with pip (sudo pip install .) ends up putting the config and init files inside Python's site-packages path rather than the root path. This is apparently because it's installing from a bdist, but also because I don't really know how to work setuptools properly. :)