Skip to content

openSUSE/mirrorbrain

Repository files navigation

mirrorbrain

This code used to live in Subversion so far, hosted on mirrorbrain.org. Maybe it's time to experiment with moving to GitHub. poeml, Mon 11 May 2015 23:06:47 CEST

What is it?

MirrorBrain is an open source framework to run a content delivery network using mirror servers. It solves a challenge that many popular open source projects face - a flood of download requests, often magnitudes more than any single site could practically handle.

The central (and probably the most obvious) part is a "download redirector" which automatically redirects requests from web browsers or download programs to a mirror server near them.

One example of a running instance is http://download.opensuse.org/

Features

For clients (users):

  • allows to have one central URL for clients to download content
  • uses geolocation and global routing data to find the closest mirror for clients
  • automatically keeps cryptohashes of all files and can serve these on request
  • optionally generates Metalinks (RFC5854) and Torrents in realtime
  • provides automatically generated mirror list for overview (For example: https://mirrors.opensuse.org/)
  • Allows to list available mirrors for single files to choose (example)

For mirrors:

  • load-balancing of mirrors, based on weighting
  • ability to limit requests for a mirror to its own network or country
  • file level granularity (mirrors don't have to mirror the full file tree - they can choose what they want)
  • reliably assess large file support of mirrors

For admins of a MirrorBrain server:

  • has proven to handle hundreds of requests per second
  • ability to NOT redirect certain requests, for security reasons
  • content on mirrors can be protected by URL signing (clients can only download from mirrors if they successfully authenticated with the MirrorBrain server)
  • commandline tools and Python module for maintenance tasks
  • support for running behind a load balancer, using e.g. X-Forwarded-for header for the clients IP address
  • integrated in Apaches module API, for compatibility with numerous other existing Apache modules, e.g. SSL
  • multiple instances are supported to run in one Apache (one per virtual host)
  • flexible logging
  • more than a redirector
    • serve automatically generated cryptohashes (MD5, SHA1, SHA256)
    • generation of RFC5854 Metalinks
    • support for RFC3230 - Instance Digests in HTTP
    • support for RFC6249 - Metalink/HTTP: Mirrors and Hashes
    • generation of Torrents (including the closest mirrors as seeds)
    • support for zsync
    • support for native Yum mirror lists, compatible to Fedora and CentOS

Packages

No packages published

Languages

  • C 44.8%
  • Python 37.1%
  • Perl 9.1%
  • Shell 3.7%
  • PLpgSQL 2.3%
  • TSQL 1.1%
  • Other 1.9%