Sources and cleanup script update

README.md

@@ -4,6 +4,8 @@ An anti-fascist domains blocklist of QAnon, conspiracy, fake news, far-right and
 
 Distributed under the [anti-fascist licence](https://github.com/rimu/no-qanon/blob/master/LICENSE.txt).
 
+By blocking the websites linked to hate groups and bad actors, it aims to defuse propaganda and slow the spread of false informations (psyops, astroturfing).
+
 A domains blocklist only blocks entire websites: it's unfortunately unable to block anything on social media where radicalization often happens. It will still definitely reduce exposition to harmful information. Fascists prey on the vulnerable so stay vigilant and consider also using a social media blocklist if needed.
 
 ## Hosts format

scripts/update.sh

@@ -7,9 +7,9 @@
 # Cleanup sources:
 ## Remove `www.` subdomains.
 find ./sources -type f -name "*.txt" -exec sed -i 's/^www\.//i' {} \;
-## Remove duplicates from each source file. (TO DO: keep "duplicate" line jumps and comments)
+## Remove duplicate domains from each source file (keeps repeated comments and empty lines for organization).
 find ./sources -type f -name "*.txt" -exec bash -c '
-    cat -n "$0" | sort -uk2 | sort -nk1 | cut -f2- > "$0_temp.txt";
+    awk "(\$0 ~ /^#/ || NF == 0 || !seen[\$0]++)" "$0" > "$0_temp.txt";
     mv "$0_temp.txt" "$0";
 ' {} \;
 

sources/Cults/Falun Gong.txt

@@ -17,6 +17,7 @@ learnfalungong.in
 learnfalungong.org.au
 es-learnfalungong.com
 faluninfo.net
+
 # The Epoch Times
 theepochtimes.com
 epochtimes.com
@@ -46,11 +47,13 @@ epochtimes.es
 es.theepochtimes.com
 kr.theepochtimes.com
 subscribe.theepochtimes.com
+
 # Shen Yun
 shenyun.com
 shenyunperformingarts.org
 shenyuncollections.com
 shenyundancer.com
+
 # Minghui
 minghui.org
 mhpublishing.org

sources/France/Partis d'extrême droite.txt

@@ -1,12 +1,18 @@
 # Principalement les nationalistes et fascistes.
+
 # Front National, Rassemblement National, famille Le Pen
 les-patriotes.fr
 rassemblementnational.fr
+comitejeanne54.wordpress.com
+jeanmarielepen.com
+
 # Reconquête, Éric Zemmour
 parti-reconquete.fr
 ericzemmour.fr
+
 # Marion Maréchal Le Pen
 marionmarechal.info
 issep.fr
+
 # Debout la France, Nicolas Dupont-Aignan
 debout-la-france.fr

sources/France/README.md

@@ -0,0 +1,3 @@
+Good sources:
+[La Horde's map of the far right in France](https://lahorde.samizdat.net/)Schema-de-l-extreme-droite-maj-12-2022
+About conspiracies: [Conspiracy Watch](https://www.conspiracywatch.info/) BE CAREFUL it's founded and directed by Rudy Reichstadt who is part of the Printemps Républicain ("laicity" but actually conservative, anti-immigration, xenophobia, islamophobia and embezzlement via the Marianne funds)

sources/Nazis/Nazis.txt

@@ -1,9 +1,22 @@
-therightstuff.biz
+# Germany
+der-dritte-weg.info
+asatru.de
+die-heimat.de
+die-rechte.net
 
-# "I Hate Illinois Nazis."
+# Europe
+apfeurope.eu
+
+# USA "I Hate Illinois Nazis."
 americannaziparty.com
 nationaljusticeparty.com
 patriotfront.us
+nsm88.org
+nsm88records.com
+
+# Japan
+nsjap.com
+
 # The Daily Stormer
 dailystormer.red
 dailystormer.cn
@@ -16,5 +29,13 @@ punishedstormer.com
 dailystormer.ai
 dailystormer.ws
 dailystormer.top
+
 # Stormfront
 stormfront.org
+
+# Autonomous Nationalists (Nazis who have adopted some of the far-left and Antifa's organizational concepts (autonomous activism), demonstration tactics (black bloc), symbolism, and elements of clothing, including Che Guevara T-shirts and keffiyehs.)
+## Poland
+autonom.pl
+
+# Unsorted
+therightstuff.biz

sources/Police and army/Recruitment/United States of America.txt

@@ -1,5 +1,8 @@
 # Army
 recruiting.army.mil
 goarmy.com
+## America's Army video game
+aao25.com
+
 # Police
 joinlapd.com

sources/default.txt

@@ -1311,8 +1311,6 @@ ihr.org
 leagueofthesouth.com
 mjolnirmagazine.blogspot.com
 britishdemocrats.uk
-nsm88.org
-nsm88records.com
 national-justice.com
 nationalistwomensfront.org
 nationalist.ca