Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve throughput for for failed lookups #13

Closed
bradvido opened this issue Oct 2, 2015 · 1 comment
Closed

Improve throughput for for failed lookups #13

bradvido opened this issue Oct 2, 2015 · 1 comment

Comments

@bradvido
Copy link
Contributor

bradvido commented Oct 2, 2015

I use this filter to reverse-lookup every IP in our events, but it has become a huge bottleneck.
When the lookup is successful, it gets cached by our DNS server and is generally not a bottleneck. But when it fails, the timeout parameter hits its limit and the event continues to the next filter in our pipeline (and it never gets cached at the DNS server).

The problem is that a sane timeout of 2 seconds (to allow time for successful lookups) is still way too long. e.g. when you are processing 100 events per second, even if only 5% of those events have failed lookups, it will take 10 seconds to process 1 second worth of events.

I'd propose adding a short-term (expiring) cache that remembers failed lookups and doesn't try them again for a configurable amount of time. This obviously only helps when the IPs being processed are somewhat repetitive (as they are in our case, and I'd guess most other use-cases).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants