Support multiple download URLs per module

[HebaruSan]

Motivation

SpaceDock has been having unexplained bandwidth problems recently, which have caused downloads to be very slow and sometimes truncated. This has been reported to the site owner (see KSP-SpaceDock/SpaceDock#463), but so far there's no sign of an improvement on the horizon.

Many mods are available on multiple hosts. Currently we have to choose just one to put in the netkan. This situation with SpaceDock would be easier to handle if we could support alternate download URLs.

Changes

• Now the spec and schema allow download to be a string or a list of strings
  • The first URL in the list always determines the hash for caching purposes
• Now the user can configure which hosts to prioritize and which to de-prioritize when there are multiple
  
• Now when we serialize the registry, properties that allow either a string or a list of strings will be serialized as a string if the list only has one element (initially done for backwards compatibility, but it will also save a little space for author and license)
• When we need to download mods, a breadth-first-search algorithm is used to combine all mods that share one or more URLs into groups, including transitively, then their URL lists are merged and sorted according to the configuration
• If a download fails and there are alternate URLs remaining for that module, then we either start the next URL immediately if there are no active downloads for its host (remember single-download-per-host) or put it back in the queue to be tried again later
• All the various places that display download hosts to the user now list multiple when applicable
• InternetArchive fallback URLs are merged into the URL lists when modules are sent to the downloader, so they will be used as normal and can be prioritized or deprioritized by configuration
• A new Registry.GetAllHosts function scans the available modules in the registry and returns their URLs' hosts in order of frequency
• Netkan is updated to support multi-document format for YAML like this for spec versions v1.34 and later (including for meta-netkans, but not internal .ckans):

  spec_version: v1.34
  identifier: SpaceWarp
  name: SpaceWarp
  $kref: '#/ckan/spacedock/3277'
  $vref: '#/ckan/space-warp'
  license: MIT
  tags:
    - plugin
    - library
  depends:
    - name: BepInEx
    - name: UITKforKSP2
  install:
    - find: BepInEx
      install_to: GameRoot
      include_only:
        - patchers
        - plugins
  ---
  spec_version: v1.34
  identifier: SpaceWarp
  name: SpaceWarp
  $kref: '#/ckan/github/SpaceWarpDev/SpaceWarp'
  x_netkan_version_edit:
    find: 'spacewarp-'
    replace: ''
    strict: false
  $vref: '#/ckan/space-warp'
  license: MIT
  tags:
    - plugin
    - library
  depends:
    - name: BepInEx
    - name: UITKforKSP2
  install:
    - find: BepInEx
      install_to: GameRoot
      include_only:
        - patchers
        - plugins

  Each part of such a netkan will be inflated separately, and then their output modules will be merged by matching up their version attributes. The download properties will be merged into a list.

After this, we will be able to start adding alternate download sources for mods, and users will be able to prioritize GitHub over SpaceDock to avoid SpaceDock's bandwidth problems.

Fixes #487.
Fixes #1760.

[techman83]

Ok, this is super cool! Is there an example of what the outputed ckan will look like? Do we have the extra test coverage for this new functionality?

[HebaruSan]

Here's the output of running netkan on the example netkan in the description:

{
    "spec_version": "v1.34",
    "identifier": "SpaceWarp",
    "name": "SpaceWarp",
    "abstract": "A C# modding API for KSP2 built off of BepInEx",
    "author": "cheese3660",
    "version": "1.4.0",
    "ksp_version_min": "0",
    "ksp_version_max": "0.1.3",
    "license": "MIT",
    "resources": {
        "homepage": "https://forum.kerbalspaceprogram.com/topic/218490-space-warp-v140-for-ksp2-013/",
        "spacedock": "https://spacedock.info/mod/3277/Space%20Warp%20+%20BepInEx",
        "repository": "https://github.com/SpaceWarpDev/SpaceWarp",
        "bugtracker": "https://github.com/SpaceWarpDev/SpaceWarp/issues",
        "remote-swinfo": "https://raw.githubusercontent.com/SpaceWarpDev/SpaceWarp/main/SpaceWarpBuildTemplate/swinfo.json",
        "x_screenshot": "https://spacedock.info/content/cheese3660_103715/Space_Warp__BepInEx/Space_Warp__BepInEx-1678544917.png"
    },
    "tags": [
        "plugin",
        "library"
    ],
    "depends": [
        {
            "name": "BepInEx"
        },
        {
            "name": "UITKforKSP2"
        }
    ],
    "install": [
        {
            "find": "BepInEx",
            "install_to": "GameRoot",
            "include_only": [
                "patchers",
                "plugins"
            ]
        }
    ],
    "download": [
        "https://spacedock.info/mod/3277/Space%20Warp%20+%20BepInEx/download/1.4.0",
        "https://github.com/SpaceWarpDev/SpaceWarp/releases/download/spacewarp-1.4.0/spacewarp-release-1.4.0.zip"
    ],
    "download_size": 10997877,
    "download_hash": {
        "sha1": "44713C69616854626578DD4B0DED54E2DAD3D1A7",
        "sha256": "F4821C66B612074FA8364D6759E7CB21770DFA92C3EE3265EC6BCF69C81775D2"
    },
    "download_content_type": "application/zip",
    "install_size": 26579768,
    "release_date": "2023-07-19T00:16:14Z",
    "x_generated_by": "netkan"
}

Good point about tests. I'll look into that next (but not immediately)...

[techman83]

Awesome. Maybe if we can get at least one on the expected netkan output we should be good enough for now. Our indexing should be ok (it's based around netkan anyway), but maybe need to check the mirrorer, as this is going from a key/value to key/list

[HebaruSan]

OK, each commit now has some tests for the other stuff added in that commit, some of which is now more deterministic (e.g., not relying on HashSet.First()) to be more testable.

Good point, the Infra will probably need some updates.

[techman83]

This is really cool @HebaruSan ! Thanks for adding the extra tests 🙂