Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A disk-based cache for all provider queries to prevent provider API throttling #53

Open
randrej opened this issue May 18, 2024 · 2 comments
Assignees
Labels
enhancement New feature or request

Comments

@randrej
Copy link
Contributor

randrej commented May 18, 2024

If you play around with settings and importing parts too much, you might get yourself throttled or blocked by the providers. Adding caching would prevent you from repeating the same queries to the providers, saving you from this fate.

Might make sense to use something like diskcache (sqlite-based) to cache all provider queries based on the arguments, with a configurable TTL (say 6h for a default).

This would imply that if you have caching enabled, whenever you search for a part or similar, the result is cached. If you search for it again in the next n hours, you'll just get the cached response and you won't hit the API again, lessening the chance of getting yourself throttled.

Might make sense to add a CLI arg for avoiding cache, and a subcommand or flag for clearing it.

@randrej
Copy link
Contributor Author

randrej commented May 18, 2024

Btw I'd like to help on things like this, I'm not just asking for you to do something, but you gotta approve it first.

@30350n
Copy link
Owner

30350n commented May 19, 2024

I've thought about this a while ago, but ultimately decided it's not worth the effort really. With normal use (i.e. not during initial setup/testing) duplicate request shouldn't really happen between runs. So I'm already caching requests during runtime, which imo is sufficient.

you might get yourself throttled or blocked by the providers

This only applies to suppliers that I use crawling for (Mouser, LCSC, reichelt), from those Mouser currently just doesn't work at all, because they hardened their protections (see #49) and I haven't had many problems with blocking from either LCSC or reichelt.

@30350n 30350n self-assigned this May 19, 2024
@30350n 30350n added the enhancement New feature or request label May 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants