Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Request collector stores arbitrary Monitoring Requests #419

Closed
palango opened this issue Jul 2, 2019 · 11 comments
Closed

Request collector stores arbitrary Monitoring Requests #419

palango opened this issue Jul 2, 2019 · 11 comments
Assignees
Labels
Bug 🐛 Something isn't working MS 👀 Related to the Monitoring Service
Milestone

Comments

@palango
Copy link
Contributor

palango commented Jul 2, 2019

An attacker can send an arbitrary amount of MonitoringRequests to the request collector, which will - given matching chain id - store it to the shared db. This allows cheap denial of service attacks against the shared database, the request collector and the monitoring service.

@palango palango added Bug 🐛 Something isn't working MS 👀 Related to the Monitoring Service labels Jul 2, 2019
@palango palango added this to the Ithaca milestone Jul 2, 2019
@karlb
Copy link
Contributor

karlb commented Jul 2, 2019

We must accept MonitoringRequests before those channels are confirmed. This makes it hard to just discard MRs in case of a DOS attack without complicating the request collector a lot. I suggest a combination of

  • rate limiting (will be useful for many other attacks, too)
  • regularly checking the DB for MRs that don't have a matching channel and are older than a certain time.

@palango
Copy link
Contributor Author

palango commented Jul 2, 2019

I agree with these two mitigations, @err508 what do you think?

Regarding rate limiting. Is it enough to rate limit in the matrix client when receiving messages, or do we have to put that inside matrix somehow? @ulope @err508 Maybe you can give input on this!

@ulope
Copy link
Contributor

ulope commented Jul 3, 2019

Well that's a bit of a circular problem.

Matrix has server side rate limiting support.
But since we don't want transfers to get throttled we have tuned those to rather high values.
Unfortunately this is a global per-server setting.

But how would an attacker generate arbitrary amounts of requests?
AFAIU only channels that are open on-chain should be allowed to have monitoring requests posted for.

Based on that it should be possible to implement some per-channel-peer rate limiting yourselves, shouldn't it?

@err508
Copy link
Contributor

err508 commented Jul 3, 2019

I'm not really familiar with the client code base, but before a transfer happens there seems to be done this check in the channel which relies on this check in the token_network view.
@karlb could you elaborate on cases where you think the request_collector would have to accept monitoring requests from properly configured clients, where the channel_opened event might not be visible yet?
If it is not to many cases, where valid monitoring requests arrive before the ChannelOpened event is visible, I'd just perform the same check before storing stuff to the MonitoringService db.

@err508
Copy link
Contributor

err508 commented Jul 3, 2019

Also regarding rate limiting or other checks performed in matrix: I think it's good practice to think of a module as an autonomous security perimeter.
MatrixServers might be deployed by some party and I think you don't want to be in the situation where you try to find their phone number in the middle of the night because some other module gets spammed.

@palango
Copy link
Contributor Author

palango commented Jul 3, 2019

AFAIU only channels that are open on-chain should be allowed to have monitoring requests posted for.

We tried that in the beginning, but it might happen that a message is send before the services receive the event. Currently we use the same number of confirmation blocks as the client, but we still run into these cases, so for example #423 , which happened today in a scenario run)
So we want to allow receiving messages before the services receive the channel opening event.

Also regarding rate limiting or other checks performed in matrix: I think it's good practice to think of a module as an autonomous security perimeter.
MatrixServers might be deployed by some party and I think you don't want to be in the situation where you try to find their phone number in the middle of the night because some other module gets spammed.

That makes sense.
My question was more in the direction of, does it make sense to have throttling once the messages are deserialized, or should that be something that's implement in the client side matrix client. Because otherwise people could still send junk messages and overload the services, even though we would later filter out all these messages.

@err508
Copy link
Contributor

err508 commented Jul 5, 2019

We tried that in the beginning, but it might happen that a message is send before the services receive the event.

Correct me if I get this wrong, but this seems to be mostly a issue for tests, as in production a RequestMonitoring would be send only after a balance proof was received, which would only happen for a open, confirmed channel. If this is correct, another approach would be to make the chain-event related sanitization of the Requests configurable and switch them off or mock them for tests.

@palango
Copy link
Contributor Author

palango commented Jul 5, 2019

Correct me if I get this wrong, but this seems to be mostly a issue for tests, as in production a RequestMonitoring would be send only after a balance proof was received, which would only happen for a open, confirmed channel.

The number of confirmed blocks between client and MS might be different. But even if they are the same and block propagation is a bit slower for the eth node of the MS, we can easily run into that scenario.

@err508
Copy link
Contributor

err508 commented Jul 5, 2019

So would using a strictly smaller number of blocks for confirmation in the MS work?

@karlb
Copy link
Contributor

karlb commented Jul 5, 2019

So would using a strictly smaller number of blocks for confirmation in the MS work?

This would increase the chance that the CannelOpened event arrives in time by a lot. Depending on how strict we want to be, that might be sufficient. But it would be nice if clients and services could choose their number of confirmed block independently.

Also the current approach allows the request collector to be very simple and not have any connection to the blockchain at all. We would lose that simplicity if we checked the MR's channel before saving them.

@err508
Copy link
Contributor

err508 commented Jul 5, 2019

If you don't won't to the request collector to depend on events, you can use the MS for that, e.g. provide an internal api to query for events the MS for seen events. Or simply "misuse" the db a little, store channel_identifiers to the db once the MS sees an event, s.t. the request collector can see that the channel actually exists.

But in general it's the services team's call to make if or how to implement a mitigation, just let me know if I can assist.

@karlb karlb self-assigned this Jul 11, 2019
karlb added a commit to karlb/raiden-services that referenced this issue Jul 12, 2019
karlb added a commit to karlb/raiden-services that referenced this issue Jul 15, 2019
karlb added a commit to karlb/raiden-services that referenced this issue Jul 15, 2019
karlb added a commit that referenced this issue Jul 15, 2019
karlb added a commit to karlb/raiden-services that referenced this issue Jul 15, 2019
I wanted to use a pre-built rate limiter, but none really fit well for
this use case.

Should be a sufficient mitigation to close
raiden-network#419.
@karlb karlb closed this as completed in c5ea48b Jul 16, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug 🐛 Something isn't working MS 👀 Related to the Monitoring Service
Projects
None yet
Development

No branches or pull requests

4 participants