Filtered, throttled HTTP(S)-only networking for Docker jobs #1571
Conversation
|
Not ready for prime time yet because it needs an amd64 build of the gateway image and some tweaks to the tests. But they all do pass with the correct behaviour locally, so something on CI must be different. |
| @@ -139,14 +136,21 @@ ${BINARY_PATH}: ${CMD_FILES} ${PKG_FILES} | |||
| ################################################################################ | |||
| # Target: build-docker-images | |||
| ################################################################################ | |||
| IPFS_FUSE_IMAGE ?= "binocarlos/bacalhau-ipfs-sidecar-image" | |||
There was a problem hiding this comment.
Good question… @binocarlos do you know if we still use the IPFS FUSE sidecar?
| .PHONY: build-ipfs-sidecar-image | ||
| build-ipfs-sidecar-image: | ||
| docker build -t $(IPFS_FUSE_IMAGE):$(IPFS_FUSE_TAG) docker/ipfs-sidecar-image | ||
|
|
||
| HTTP_GATEWAY_IMAGE ?= "ghcr.io/bacalhau-project/http-gateway" |
There was a problem hiding this comment.
I have a docker repo for our project - should we move it over here? or vice versa (move the images over there?)
There was a problem hiding this comment.
Happy with either! Personally I lean towards GitHub just because it’s less things to administer…
| @@ -212,6 +214,10 @@ func newDockerRunCmd() *cobra.Command { //nolint:funlen | |||
| NetworkFlag(&ODR.Networking), "network", | |||
| `Networking capability required by the job`, | |||
| ) | |||
| dockerRunCmd.PersistentFlags().StringArrayVar( | |||
There was a problem hiding this comment.
I'm not super clear on this model - do we really want/need job specifiers to detail this? Shouldn't it be the node operators who light it up - but the job either works or it doesn't?
This is great feedback, thanks! I’ll make it more clear in the documentation. The model is: the job specifier submits a job with the domain(s) it will need to communicate with, the compute provider uses this to make some decision about the risk of the job and bids accordingly, and then at run time the traffic is limited to only the domain(s) specified. As a command, something like: The “risk” for the compute provider is that the job does something that violates its terms, the terms of its hosting provider or ISP, or even the law in it’s jurisdiction (e.g. accessing and spreading illegal content, performing cyber attacks). So the same sort of risk as operating a Tor exit node. The risk for the job specifier is that we are operating in an environment they are paying for, so there is an incentive to hijack that environment (e.g. via a compromised package download that runs a crypto miner on install, and uses up all of the paid-for job time). Having the traffic enforced to only domains specified makes those sorts of attacks much trickier and less valuable. The compute provider might well enforce its limits by other means, but having the domains specified up front allows it to skip bidding on jobs it knows will fail in its executor. So this is hopefully a better UX for job specifiers who can have their job picked up only by someone who will run it successfully. Do you think needing to specify the domains makes the UX too burdensome for the job specifier? If so, we could look at ways to reduce that; some ideas we had include a) allowing jobs to run in an unfiltered mode and produce a list of the domains accessed that can be supplied to apply the limits next time, and/or b) ship with a default allowlist that the Bacalhau team/community maintains of commonly used, generally safe domains that don’t need to be specified up front and are available by default to all jobs. We could also just move on to higher level features that use this capability like e.g. |
simonwo
force-pushed
the
http-filtering
branch
4 times, most recently
from
b8cb234
to
9eaca5b
Compare
|
Honestly, my only feedback is what did the default look like. I really like the entire ability to target very specific domains for set of nodes that you host so you're on the right track. It's only what happens when sets nothing or sets networking on with no other specifications. Like, what if we have a default set of blessed domains that we make available? Not sure! |
Ah right! Yes, at the moment jobs by default still use no networking and compute nodes by default still reject all jobs with networking. Specifying no domains is the same as no networking overall.
Yeah, blessed domains is one way, blessed job types is another, allowing certain domains for jobs signed with a certain trusted key might be another, or even something like jobs submitted by someone who’s real world identity is attested to by some external service, etc. Given that all of these are now possible for CPs to implement using the external probe features, maybe we should start by demonstrating/documenting some of those as CP examples? And configuring our own compute nodes to meet our own immediate use cases and see what solutions fall out of that? |
The Docker executor cleans up the container it creates at the end of a job, and if the executor shuts down, uses a label-based approach to ensure everything has been caught. This commit changes the job cleanup behaviour to also use labels. This makes cleanup more robust and able to handle more than just a single container per job. This is in preparation for jobs having more than a single object (e.g. having a network and auxiliary containers set up).
This commit adds a new “HTTP” network type that jobs can now use to specify they need HTTP networking (as opposed to fully general networking) and the domain names they require access to. The Docker executor supports this new networking type. It enforces the HTTP-only nature of the connection by separately running an HTTP proxy (one per job shard). All traffic can only leave the executor container via the proxy, and hence only HTTP(S) between declared domains are accessible. The HTTP connection is also throttled to ~1 Mbit. This is so that if multiple compute nodes run a job concurrently, there is not an accidental DDoS of remote endpoints. Note that the current default behaviour of compute nodes rejecting all networked jobs by default remains in place. Compute providers still need to opt-in to networking for this feature to be accessible.
Now that we have networking, it is possible for networked jobs to run Bacalhau themselves and trigger new jobs. Whilst this is a powerful feature, it can also lead to a “fork bomb”-style exhaustion attack as jobs could clone themselves with ever increasing concurrency. This is particularly dangerous as it would require taking down a majority of the nodes on the network to prevent the jobs from respawning. Now, we append a header to all outbound HTTP requests made by a job using HTTP networking that signals that the HTTP request is coming from a running job. We configure the submission endpoint to reject any job creation request that contains this header. In this way, we now prevent jobs from creating other jobs. When we have paid jobs, this is less of an issue as the payment channel will eventually exhaust. We can later selectively add the header only for unpaid jobs, but for now it covers all jobs.
simonwo
force-pushed
the
http-filtering
branch
from
9eaca5b
to
677468c
Compare
This PR adds a new “HTTP” network type that jobs can now use to specify they need HTTP networking (as opposed to fully general networking) and the domain names they require access to.
The Docker executor supports this new networking type. It enforces the HTTP-only nature of the connection by separately running an HTTP proxy (one per job shard). All traffic can only leave the executor container via the proxy, and hence only HTTP(S) between declared domains are
accessible.
The HTTP connection is also throttled to ~1 Mbit. This is so that if multiple compute nodes run a job concurrently, there is not an accidental DDoS of remote endpoints.
Now that we have networking, it is possible for networked jobs to run
bacalhauthemselves and trigger new jobs. Whilst this is a powerful feature, it can also lead to a “fork bomb”-style exhaustion attack as jobs could clone themselves with ever increasing concurrency. This isparticularly dangerous as it would require taking down a majority of the nodes on the network to prevent the jobs from respawning.
So we also now append a header to all outbound HTTP requests made by a job using HTTP networking that signals that the HTTP request is coming from a running job. We configure the submission endpoint to reject any job creation request that contains this header. In this way, we now prevent jobs from creating other jobs.
When we have paid jobs, this is less of an issue as the payment channel will eventually exhaust. We can later selectively add the header only for unpaid jobs, but for now it covers all jobs.
Note that the current default behaviour of compute nodes rejecting all networked jobs by default remains in place. Compute providers still need to opt-in to networking for this feature to be accessible.