SourceForge kref #4172
Merged
SourceForge kref #4172
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
HebaruSan
added
Enhancement
HebaruSan
force-pushed
the
feature/sourceforge-kref
branch
from
a123c95 to
ef8fc9a
Compare
This comment was marked as resolved.
This comment was marked as resolved.
|
Apparently SourceForge's download link redirection does dynamic load balancing, so we potentially get a different URL each time:
Should figure out a way to stabilize that before we use this more widely... |
This was referenced Sep 5, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Discord user "dan" pointed out that KSP2 mod
KSRe(frozen in KSP-CKAN/KSP2-NetKAN#174 after its author deleted it from SpaceDock) has been posting updates at https://sourceforge.net/projects/ksre/:https://sourceforge.net/projects/ksre/files/
Currently Netkan doesn't have the ability to crawl SourceForge, but while its API documentation looks quite inadequate, it does feature RSS-format (Really Simple Syndication, not Real Solar System) feeds for each mod's files:
https://sourceforge.net/projects/ksre/rss
Changes
Now a netkan with
$kref: '#/ckan/sourceforge/ksre'will be indexed from SourceForge based on its RSS feed. Each<item>with a title ending in.zipis taken to be a download, and the first one is treated as most recent.Retrieving and parsing the RSS feed went smoothly thanks to
System.ServiceModel.Syndication.SyndicationFeed, but getting a download URL was tricky because SourceForge thought our user agent string looked too browser-like and so wasn't returning redirects, so I had to pass in an alternate user agent string.Only a few fields can be set based on the minimal metadata in the feed, but luckily the
swinfo.jsonparser is able to handle the rest.