Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Find a way to deal with 429 Too Many Requests #53

Open
jyn514 opened this issue Aug 17, 2020 · 3 comments
Open

Find a way to deal with 429 Too Many Requests #53

jyn514 opened this issue Aug 17, 2020 · 3 comments
Labels
A-http-check A: Checking http links C-enhancement Category: This is a new feature

Comments

@jyn514
Copy link
Contributor

jyn514 commented Aug 17, 2020 •
edited
Loading

When validating a lot of URLs with --check-http, it's common to get '429 Too Many Requests'. It would be great to be able to ignore these. Alternatively, maybe cargo-deadlinks could add a rate-limit option that avoids getting the error in the first place.
@ayazhafiz
Copy link

This is a hard thing to fix in a general way because different rate limits (if they have any at all).

@zummenix
Copy link
Contributor

I wonder if #63 mitigates the problem.

@jyn514 jyn514 added A-http-check A: Checking http links C-enhancement Category: This is a new feature labels Nov 17, 2020
@jyn514
Copy link
Contributor Author

jyn514 commented Jan 5, 2021

I tested this out by running deadlinks book on https://github.com/rust-lang/rustc-dev-guide/. As expected, it gave a bunch of 429 errors, so unfortunately #63 didn't help (although I haven't tested with an earlier version of deadlinks, so it might be fewer 429s than before).

This is a hard thing to fix in a general way because different rate limits (if they have any at all).

Right, this is why I think --rate-limit should be passed by the user and not hard-coded by deadlinks. Alternatively, deadlinks could give a warning but not fail the request.

nars1 pushed a commit to YottaDB/YDBDoc that referenced this issue Jun 9, 2021
[#212] Don't check HTTP links
2773698 
Continuous deployments work correctly as long as the pipeline passes.
Unfortunately, the pipeline is not currently passing, and fixing it is
not trivial. Here are the current errors, taken from
https://gitlab.com/jyn514/YDBDoc/-/jobs/791375698 and
https://gitlab.com/jyn514/YDBDoc/-/jobs/793823039:

```
Found invalid urls in /builds/jyn514/YDBDoc/target/AcculturationGuide/acculturation.html:
	Error fetching http://worldvista.org/AboutVistA: http://worldvista.org/AboutVistA: timed out
	Unexpected HTTP status fetching YottaDB/YDB#109: 429 Too Many Requests
	Unexpected HTTP status fetching YottaDB/YDB#161: 429 Too Many Requests
	Unexpected HTTP status fetching YottaDB/YDB#186: 429 Too Many Requests
	Unexpected HTTP status fetching YottaDB/YDB#188: 429 Too Many Requests
	Unexpected HTTP status fetching YottaDB/YDB#145: 429 Too Many Requests
```

Note that all these URLs exist, the only problem is that the pipeline is
rate-limited or the request times out. There are a few possible solutions:

- Don't check HTTP pages at all. This will catch fewer failures, but not have any spurious failures at all.
- Retry pages that fail. This will require upstream support in cargo-deadlinks
  (deadlinks/cargo-deadlinks#53). It will not
  help with rate-limits (since the retries will also count against the limit).

- Ignore failures in some websites. This requires upstream support
  (deadlinks/cargo-deadlinks#57). It will help with both issues.

- Add a GitHub token in the environment to increase the rate limit. This
  will not fix the issues with timeouts, but should fix the vast majority
  of other errors. Unfortunately, it also requires storing secrets in
  pipelines and having a dedicated GitHub account for this. GitLab has
  support for storing secret credentials, so this is feasible.

- Change some of the URLs to point to GitLab. This will help with the
  GitHub rate limits but not with the VistA timeout.

- Remove some URLs from the documentation. This is not a _good_ solution, but it is _a_ solution.

The change implemented here is to not check HTTP pages at all. If in the
future deadlinks has more support for retries, the check could be
re-enabled.
@jordens jordens mentioned this issue Apr 13, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-http-check A: Checking http links C-enhancement Category: This is a new feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@zummenix @ayazhafiz @jyn514