Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set bot useragent for Inflator #3490

Merged
merged 2 commits into from
Dec 1, 2021

Conversation

HebaruSan
Copy link
Member

@HebaruSan HebaruSan commented Nov 29, 2021

Background

CKAN and Netkan both use a default User-Agent of Mozilla/4.0 (compatible; CKAN).

Motivation

This User-Agent does not identify the Inflator as a bot, which it most certainly is.

image

See KSP-SpaceDock/SpaceDock#436 for why we might want to do that.

Changes

I don't know if there's a comprehensive reference on how to create a User-Agent string; the best reference I was able to find was:

Based on the examples presented there, the Inflator's User-Agent is now:

Mozilla/5.0 (compatible; Netkanbot/1.0; CKAN; +https://github.com/KSP-CKAN/NetKAN-Infra)

I think parsers will just check whether bot occurs in the browser family.

Blocking external dependencies

The HyperEdit and Graphotron mods use a combination of strange URLs:

... and the User-Agent string to identify Netkan requests. Requesting these URLs with the wrong User-Agent string results in an HTTP status 403 error. Before we merge these changes, we must work with @Ezriilc to update the kerbaltek.com site to support the new User-Agent formats as well as the old format.

@HebaruSan HebaruSan added Enhancement New features or functionality Easy This is easy to fix Pull request In progress We're still working on this Netkan Issues affecting the netkan data Infrastructure Issues affecting everything around CKAN (the GitHub repos, build process, CI, ...) Network Issues affecting internet connections of CKAN labels Nov 29, 2021
@DasSkelett
Copy link
Member

Without having fully understood the syntax, do you think it would be possible to include the word CKAN somewhere? It would be nice to have that reference for people who don't know what Netkan is supposed to be but do know about CKAN.
Technically it's in the link, but having it separately, if possible, makes it stand out more.

@HebaruSan HebaruSan force-pushed the fix/inflator-useragent branch from eceb9a0 to 03113c3 Compare November 29, 2021 18:16
@HebaruSan
Copy link
Member Author

Done in this PR, going to the next now...

Dockerfile.netkan Outdated Show resolved Hide resolved
Co-authored-by: DasSkelett <dasskelett@gmail.com>
Copy link
Member

@DasSkelett DasSkelett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HyperEdit and Graphotron are working again right now, let's see what happens when we deploy this. But it's probably easier for @Ezriilc to work with a stationary target, and with the logs showing the new user-agent string.

@HebaruSan HebaruSan merged commit b2687aa into KSP-CKAN:master Dec 1, 2021
@HebaruSan HebaruSan deleted the fix/inflator-useragent branch December 1, 2021 16:28
@Ezriilc
Copy link

Ezriilc commented Dec 1, 2021

HyperEdit and Graphotron are working again right now, let's see what happens when we deploy this. But it's probably easier for @Ezriilc to work with a stationary target, and with the logs showing the new user-agent string.

However, don't let me become a problem. Y'all do your thing and I'll try to keep up. lol

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Easy This is easy to fix Enhancement New features or functionality In progress We're still working on this Infrastructure Issues affecting everything around CKAN (the GitHub repos, build process, CI, ...) Netkan Issues affecting the netkan data Network Issues affecting internet connections of CKAN
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants