FilterLists-hosted mirrors

[collinbarrett]

An anonymous user suggested capturing backups of lists in a public repository or similar.

This is a tricky one and something I have certainly considered. Currently, FilterLists has "SnapshotService" that crawls all of the lists and stores all of the rules in the database (in a way that's linked to the list and the time it was Snapshot-ed). These aren't exact copies of the lists, though. Each unique line in the plain text file is ingested as a separate rule. We do not preserve order. So, we could reproduce from this a very similar mirror/backup of the original list, but order would not be preserved and duplicate lines would be de-duplicated (comments, for example, would be completely un-useful). Implementation details of SnapshotService.

I want to be very careful about respecting list maintainers licenses. Some maintainers do not want their lists re-distributed in any way. There are some open GitHub issues for some cool features (such as #33) that could make use of these snapshot-ed rules, but I am not really sure I want FilterLists to be in the "business" of hosting mirrors/backups. If the project ever were to go offline for any reason (no plans to do so, but I do pay monthly out of my own pocket to host it, so it is certainly possible), I don't want people relying on it for their list subscriptions, etc. We could capture backups on a site like GitHub that is free and would theoretically remain online even if FilterLists had to shut down, but that could violate some maintainer's licenses.

related to #6

[Atavic]

Some maintainers do not want their lists re-distributed in any way.

Such data should be totally free.

[collinbarrett]

@Atavic I agree. If I were to maintain a list, I would make it free. That's why FilterLists is under the MIT license. But, there are some maintainers who have a less permissive license. And, there are many lists who do not specify a license, in which case FilterLists treats as "All Rights Reserved" by default. I am certainly not a legal professional, but I do want to keep FilterLists on the right side of the law (U.S., as that's where I'm based) to the best of my knowledge.

[Atavic]

Uh, legalese. To me, if the mirror has zero ads + has big links to the original authors + updates stay a little behind the source (one day? two days?) there should be zero issues.

[elypter]

maybe peer2peer services could do the job.

https://old.reddit.com/r/Rad_Decentralization/comments/7t7oa1/native_support_in_firefox_59_for_decentralized/

[DandelionSprout]

I think that the amount of times that a list that was on FilterLists.com has completely vanished from the internet, can be counted on one hand. As such, this isn't a top-priority thing to implement, in my eyes.

The mirror viewlink system, and the existence of the Wayback Machine, also seems to have worked fairly well as far as I can determine.

[collinbarrett]

Certainly open to revisiting in the future, but closing as won't fix for now.

[collinbarrett]

Re-opening for consideration per NanoAdblocker/NanoCore#220 (comment) and #584

[collinbarrett]

started some work on this here. want to get the service up and building the archives first. then, will maybe expose the mirrored copies to the public. want to consider licensing of lists and only serve FilterLists-hosted mirrors of lists that allow that via licensing.

further down the road, these archived list copies can be used for things like #33 , etc.