IAFD 403 forbidden error

[DrunkSith]

I can't seem to scrape anything from IAFD.

[wkearney99]

Same here. Get error 403 within AMM. Using same URL in a browser successfully loads the page.

And if I take the title.rme/id= field and use that as a manual ID it also throws the 403 error.

Is the User Agent string editable in AMM? Might that be something iafd is blocking?

[wkearney99]

Is this related? #108 (comment)

AMM utilizes DuckDuckGo for searches on the IAFD if no results are found during the onsite search. Occasionally, DuckDuckGo blocks too many searches, so AMM attempts accessing DuckDuckGo via the Tor network. However, in your case, this was unsuccessful five times. I'll take a look at the DDG search.

[adultmm]

Guys, please attach the corresponding log file. Thanks

[wkearney99]

It's repeatable. No scrapes are successful from iafd. This is a simple test. It has one movie in the database, and ONLY the iafd scraper is enabled.

2024-07-15.csv

[adultmm]

Thank you. I can see the 403 error in the log, but it works for me. I'll try to reproduce.

[wkearney99]

It would really help if we had a bit more indications in the log about how a connection was being made. Using proxies, tor, etc. It'd help point the finger toward the actual problem.

[adultmm]

The actual problem - as usual - is the ClouFlare protection. Me, and many others work on solving these challenges: FlareSolverr, CF-Clearance-Scraper, etc. If you can download the content of the URL with a curl or any other tool, you have solved the problem. Send the solution, and I'll implement it in AMM.

[adultmm]

Fixed in v0.12.12.

[adultmm]

Try out the standalone version that works: v1.0.0-beta1.

[adultmm]

@wkearney99 @DrunkSith v1.0.0-beta1 scrapes IAFD for you? Thanks

[adultmm]

Cody has an issue with the v1.0.0-beta1 version as well: #123 (comment)

[wkearney99]

Nope, 1.0.0.b1 immediate 403 error, same as with 12.11.
2024-07-26.csv

[Fetterbr61]

cant scrape any japanese movies, especially femdom sites.

[wkearney99]

The problem seems to come and go. What's the code logic behind the scenes that would affect this?

[adultmm]

As I can see, IAFD changed someting on their CF settings and now AMM can scrape their pages. Please reopen, if the problem appears again.

[wkearney99]

Seems to be a problem again. 403 errors. Even a paste of the manual ID entry fails with a 403.