First release

.gitignore

@@ -1,3 +1,4 @@
+.idea
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]

NODES/.gitignore

@@ -0,0 +1 @@
+nodes.*

NODES/readme.md

@@ -0,0 +1,5 @@
+nodes.1
+..
+nodes.5
+
+5 nodes files from different in cycle verifiers

README.md

@@ -1,2 +1,142 @@
 # Nyzo-Q
-Nyzo Queue issues, tests and practical proposals
+
+Nyzo Queue issues, tests and practical proposals.  
+
+This repo is under active development. Posting in current raw state anyway, under the Unlicence.
+
+## Current Nyzo queue issues
+
+Current Queue displays 2 issues, related to heavy queue cost optimizations.  
+
+- Unreachable verifiers, non stock or running part time, merely spamming nodejoins  
+  (not dealt with here - Will be the subject of a github issue)
+- Large groups of ips all owned by a very few entities (A few/single large OVH customers, One Ukrainian ISP with 6000 ips, Amazon)
+
+The large ip classes owned by a few owners represents like 0.5% of all ip groups but do account for 50% of new node joins.  
+This is happening since months (*insert first seen timestamp of full c-class*) and is growing. 
+As a result, the diversity of in-cycle verifier can not be assumed anymore and the recent NCFP-10 votes (can't reach 50% "YES", but no "NO") is quite worrysome. 
+
+*insert table from open nyzo*
+
+Instead of "proof of diversity", Nyzo queue in fact now runs a "proof of quantity".
+
+It's time to wonder if nyzo aims to be run by a few large operators, or wants to give control and chances back to individuals.
+
+> See cache/countries.csv and cache/owners.csv for current repartition of queue ip, built by stats_global.py
+
+## Current lottery
+
+Current lottery mechanism relies on the fact that queue nodes need a unique ip, however that ip is **not** used at all in the scoring process.    
+Only the verifier id is used after some hashing.
+
+Thus, the more IPs you own - no matter their *diversity* - the more chances you have.   
+
+Imagine a lottery wheel, where every eligible ip has the same size and chance to win. Double the IPs, all size are divided by 2, every eligible ip has the same chance.
+
+The drawback of this is encouraging IP cost optimization, no matter the diversity.  
+
+- Big players end up on the cheapest recurring cost provider (OVH, setup fees but no recurring)
+- Some players do use IPs they already own for other purposes (ISP, but also any service provider needing ip for existing services like proxies, vpn, scraping, bots...)
+
+Looking at the queue, this is obvious: on a regular vps provider, where clients do rent vps and ips, you may have a few ip per c-class running a verifier.  
+On big players verifiers, you have full blocks of 32, 64, 128 and 256 or more contiguous ips with queue verifiers.
+
+## My approach
+
+To ensure in-cycle diversity, we have to favor in-queue diversity.    
+We don't want KYC or one person - one node, we just want stronger guarantees that the queue - hence the cycle - is diverse.
+
+c-class (or smaller) filled with queue verifiers are the track left by a few big players.  
+This can be mitigated by scoring on ip data, not just random verifier identifier.
+
+Imagine a lottery wheel, where every possible ip is drawn. Some parts are dense in queue verifiers (full c-class), other are sparse (regular vps users, home users)  
+When you spin the wheel, either you end up on a lucky queue ip directly, either the closest queue ip is the winner.
+
+This lottery is based upon ip diversity, no more ip quantity. 
+You still have more chances if you have more ips, but you don't end up with the same leverage as now.
+
+Such a lottery, instead of concentrating all queue nodes to OVH and a few service providers, incitates to use many various providers instead, and gives regular users their chances to join the cycle.
+
+## The concerns
+
+It's all a question of balance.  
+Current lottery removes all bias in verifier data.  
+If we remove all bias in ips structure as well, we can no more ensure diversity and we go back to quantity only.
+
+The thing is to find the good balance between diversity and bias. Roughly: how to draw all possible ips on the wheel so that everyone gets a fair chance, and diversity still is ensured.
+
+## Simulations
+
+Nyzo does not have a consensus on the queue nodes.  
+Every in-cycle verifier has its own list of queue nodes, with related ip and first seen timestamp.  
+This makes it harder to elect a common verifier in a safe way, since 2 different in-cycle will have a different queue list and data.
+
+To account for that and make sure the various options do not add more scattering, I ran simulations from 5 nodes files from 5 different in-cycles nodes.  
+This can be run on more than that of course.
+
+### current_lottery
+
+see simulations/current_lottery/stats.json and related CSVs
+```
+{
+  "Simulation": "current_lottery",
+  "Total": 20000, "Consensus": 19718,
+  "Consensus_PC": "98.59",
+  "Queue": {"127": 64, "63": 28, "31": 45, "15": 17, "1": 6221},
+  "Classes": {"127": 10236, "63": 1682, "31": 1311, "15": 261, "1": 6510},
+  "Classes_PC": {"127": "51.18", "63": "8.41", "31": "6.55", "15": "1.31", "1": "32.55"},
+  "Classes_global_PC": {"127": "0.80", "63": "0.30", "31": "0.15", "15": "0.08", "1": "0.01"},
+}
+```
+
+**Total**: 20000 - Number of entry simulations
+**Consensus** and **Consensus_PC**: how many times the 5 different nodes files gave the same winner.
+
+> Note: this can't be 100% and does not need to. New entrant can be voted in even if all do not agree. If votes are split 50/50 between two winners, then after 50 blocks a new lottery will take place anyway.
+
+**Queue**: How many queue candidates for each class in the first node file.    
+- "1" is a group of 1 to 14 ips in their c-class
+- "15" is a group of 15 to 30 ips in their c-class
+- "31" is a group of 31 to 62 ips in their c-class
+- "63" is a group of 63 to 126 ips in their c-class
+- "127" is a group of 127 to 256 ips in their c-class  
+We can see that classes with less than 15 ips per c-class are the vast majority (6221 classes).    
+classes with 127 and up ips are low (64) but account for a lot of ips (64*256).  
+This data is the same for all simulations and lottery mechanisms.
+
+**Classes**: How many winners for each class.  
+
+**Classes_PC**: % of times an ip of that class won.
+
+**Classes_global_PC**: individual odd for an ip of that class to win.  
+Here, if you have an average ip that is in the 127 class (say, one ip from a full c-class) then you have 0.08% odds to win.  
+If you have an ip in the "1" class, majority one, you only have 0.01% chances.
+
+**Current lottery and queue favors quantity over quality by a factor 8** 
+
+> Note: sim.csv line content: index, Consensus(True|False), C-Class Winner, Number of IPs in that class
+
+Current lottery scoring is ported from official nyzo code
+
+```
+def current_score(cycle_hash: bytes, identifier: bytes, ip: str) -> int:
+    """
+    Nyzo Score computation, see
+    https://github.com/n-y-z-o/nyzoVerifier/blob/75786060a822443154bcdaaa371fe8696d54a201/src/main/java/co/nyzo/verifier/NewVerifierQueueManager.java#L214
+    """
+    score = sys.maxsize
+    if len(cycle_hash) != 32 or len(identifier) != 32:
+        return score
+
+    combined_array = b''
+    for i in range(32):
+        combined_array += ((cycle_hash[i] + identifier[i]) & 0xff).to_bytes(1, byteorder='big')
+    hashed_identifier = sha256(combined_array).digest()
+
+    score = 0
+    for i in range(32):
+        hash_value = cycle_hash[i] & 0xff
+        identifier_value = hashed_identifier[i] & 0xff
+        score += abs(hash_value - identifier_value)
+    return score
+```

Ref/readme.md

@@ -0,0 +1,30 @@
+# Random notes
+
+https://github.com/n-y-z-o/nyzoVerifier/blob/master/src/main/java/co/nyzo/verifier/Node.java
+
+
+private static final int communicationFailureInactiveThreshold = 6;
+
+private byte[] identifier;                    // wallet public key (32 bytes)
+private byte[] ipAddress;                     // IPv4 address, stored as bytes to keep memory predictable (4 bytes)
+private int portTcp;                          // TCP port number
+private int portUdp;                          // UDP port number, if available
+private long queueTimestamp;                  // this is the timestamp that determines queue placement -- it is
+                                              // when the verifier joined the mesh or when the verifier was last
+                                              // updated
+private long inactiveTimestamp;               // when the verifier was marked as inactive; -1 for active verifiers
+private long communicationFailureCount;       // consecutive communication failures before marking inactive
+
+
+communicationFailureCount is inc by markFailedConnection(), reset by markSuccessfulConnection()
+
+markFailedConnection is called by NodeManager, from its own markFailedConnection(), called by  Message:
+https://github.com/n-y-z-o/nyzoVerifier/blob/75786060a822443154bcdaaa371fe8696d54a201/src/main/java/co/nyzo/verifier/Message.java#L213
+
+markSuccessfulConnection is called by NodeManager.
+Either by a successful message fetch... either from updateNode at https://github.com/n-y-z-o/nyzoVerifier/blob/75786060a822443154bcdaaa371fe8696d54a201/src/main/java/co/nyzo/verifier/NodeManager.java#L99
+That one is called from a successful response to nodejoin, **or** a NodeJoinV2_43.
+
+https://github.com/n-y-z-o/nyzoVerifier/blob/75786060a822443154bcdaaa371fe8696d54a201/src/main/java/co/nyzo/verifier/MeshListener.java#L506
+
+

cache/nodes1.json

@@ -0,0 +1 @@
+{"Simulation": "nodes.1", "Total": 6375, "Classes": {"127": 64, "63": 28, "31": 45, "15": 17, "1": 6221}}

cache/stats.json

@@ -0,0 +1 @@
+{"Owners": {"OVH": 10094, "AGROSVIT": 6144, "AMAZON-02": 3701, "HETZNER-AS": 1449, "DIGITALOCEAN-ASN": 621, "AMAZON-AES": 611, "AS-CHOOPA": 395, "WZCOM-": 358, "ONEANDONE-AS Brauerstrasse 48": 328, "CNNIC-ALIBABA-US-NET-AP Alibaba (US) Technology Co., Ltd.": 191, "BAIDU Beijing Baidu Netcom Science and Technology Co., Ltd.": 142, "GOOGLE": 136, "RAMNODE": 89, "REBACOM-AS": 65, "PONYNET": 60, "CONTABO": 59, "XSSERVER": 58, "NETCUP-AS netcup GmbH": 57, "CLOUVIDER Clouvider - Global ASN": 44, "IOFLOOD": 40, "ASN-ROUTELABEL": 33, "ASN-QUADRANET-GLOBAL": 31, "RACKRAY UAB Rakrejus": 26, "NETZBETRIEB-GMBH": 25, "UHGL-AS-AP UCloud (HK) Holdings Group Limited": 24, "M247": 18, "Online SAS": 17, "ADVANCEDHOSTERS-AS": 15, "MICROSOFT-CORP-MSN-AS-BLOCK": 11, "TKPSA-AS": 11, "LINODE-AP Linode, LLC": 11, "AS-COLOCROSSING": 10, "SPECTRAIP SpectraIP B.V.": 10, "CHINATELECOM-GUANGDONG-IDC Guangdong": 10, "CNNIC-ALIBABA-CN-NET-AP Hangzhou Alibaba Advertising Co.,Ltd.": 10, "CHINA169-BACKBONE CHINA UNICOM China169 Backbone": 9, "CNNIC-TENCENT-NET-AP Shenzhen Tencent Computer Systems Company Limited": 9, "BELCLOUD": 6, "CHINATELECOM-CTCLOUD Cloud Computing Corporation": 5, "CHINANET-SH-AP China Telecom (Group)": 4, "CHINANET-JIANGSU-PROVINCE-IDC AS Number for CHINANET jiangsu province backbone": 4, "MULTA-ASN1": 3, "INTERNET-CZ Ktis 2, 384 03 Ktis": 2, "WII": 2, "STRATO STRATO AG": 2, "BKVG-AS": 2, "ANYNODE": 2, "BRISBANETECHSERVICES-AS-AP Brisbane Tech Services": 1, "SERVERHUB-NL": 1, "ASN-DCS-01": 1, "CLOUDIE-AS-AP Cloudie Limited": 1, "LEASEWEB-USA-SFO-12": 1, "CLOUVIDER-NETWORK-OPTIMISATION": 1, "CHINANET-BACKBONE No.31,Jin-rong Street": 1, "GENESIS-HOSTING-SOLUTIONS-LLC": 1, "ECOTEL": 1, "FIRSTHEBERG": 1, "LDCOMNET": 1, "NOCIX": 1, "AS-HOSTINGER": 1, "ALTIBOX_AS Norway": 1, "IPAX-AS": 1, "HINET Data Communication Business Group": 1, "EONIX-COMMUNICATIONS-ASBLOCK-62904": 1, "XIAONIAOYUN Shenzhen Qianhai bird cloud computing Co. Ltd.": 1, "CNSERVERS": 1, "MTW-AS": 1, "UKSERVERS-AS UK Dedicated Servers, Hosting and Co-Location": 1, "ASN-M3NET": 1, "IT7NET": 1, "TENCENT-NET-AP-CN Tencent Building, Kejizhongyi Avenue": 1, "IPSERVER-RU-NET Fiord": 1, "IKOULA": 1, "BOUYGTEL-ISP": 1, "PROXAD": 1, "MYLOC-AS IP Backbone of myLoc managed IT AG": 1, "LIQUID-AS": 1, "INFOMANIAK-AS": 1, "ARUBA-ASN": 1, "BACOM": 1, "COMCAST-7922": 1, "ECRITEL-FRANCE ISP and web hosting.": 1, "ATT-INTERNET4": 1, "HOSTUS-GLOBAL-AS HostUS": 1, "TENET-AS": 1, "TILAA": 1}, "Countries": {"FR": 10117, "UA": 6145, "US": 6089, "DE": 1925, "CN": 387, "NL": 182, "GB": 64, "LT": 27, "HK": 26, "PL": 13, "BG": 6, "CZ": 2, "RU": 2, "CA": 2, "AU": 1, "NO": 1, "AT": 1, "TW": 1, "CH": 1, "IT": 1}}

chart.py

@@ -0,0 +1,70 @@
+"""
+Create charts for simulation CSVs
+"""
+
+import json
+import pathlib
+
+SIMULATION = "current_lottery"
+SIMULATION = "ip_lottery"
+SIMULATION = "raw_ip_lottery"
+SIMULATION = "shuffle_ip_lottery"
+SIMULATION = "linear_ip_lottery"
+SIMULATION = "linear_ip_lottery2"
+
+
+DIR = pathlib.Path('./simulations/{}'.format(SIMULATION))
+
+STATS = {"Simulation": SIMULATION,
+         "Total": 0, "Consensus": 0,
+         "Classes": {
+             "127": 0,
+             "63": 0,
+             "31": 0,
+             "15": 0,
+             "1": 0},
+         "Classes_PC": {},
+         "Classes_global_PC": {}
+         }
+
+
+def process(file_name):
+    global STATS
+    with open(file_name) as fp:
+        for line in fp:
+            if "DIVERGE" in line:
+                continue
+            _, consensus, ip_class, ip_count = line.strip().split(",")
+            STATS["Total"] += 1
+            ip_count = int(ip_count)
+            if consensus == 'True':
+                STATS["Consensus"] += 1
+            if ip_count >= 127:
+                STATS["Classes"]["127"] += 1
+            elif ip_count >= 63:
+                STATS["Classes"]["63"] += 1
+            elif ip_count >= 31:
+                STATS["Classes"]["31"] += 1
+            elif ip_count >= 15:
+                STATS["Classes"]["15"] += 1
+            else:
+                STATS["Classes"]["1"] += 1
+
+
+if __name__ == "__main__":
+    with open("cache/nodes1.json") as fp:
+        STATS["Queue"] = json.load(fp)['Classes']
+    with open("cache/nodes1.json") as fp:
+        total_classes_in_queue = json.load(fp)['Total']
+
+    for file_name in DIR.glob("*.csv"):
+        process(file_name)
+    STATS["Consensus_PC"] = "{:0.2f}".format(STATS["Consensus"] / STATS["Total"] * 100)
+    for classe in ("127", "63", "31", "15", "1"):
+        STATS["Classes_PC"][classe] = "{:0.2f}".format(STATS["Classes"][classe] / STATS["Total"] * 100)
+
+        STATS["Classes_global_PC"][classe] = "{:0.2f}".format(float(STATS["Classes_PC"][classe]) / STATS["Queue"][classe])
+
+    print(STATS)
+    with open('./simulations/{}/stats.json'.format(SIMULATION), "w") as fp:
+        json.dump(STATS, fp)

libs/nodesreader.py

@@ -0,0 +1,52 @@
+"""
+Nyzo Nodes file reader
+"""
+
+from libs.utils import identifier_to_bytes, ip2class, ip_whois
+from libs.utils import current_score, ip_score
+from sys import maxsize
+
+
+class NodesReader:
+
+    def __init__(self, filename: str):
+        """Loads nodes file into memory and index by identifier and c ip class"""
+        self.verifiers = {}
+        self.ip_classes = {}
+        with open(filename) as f:
+            for line in f:
+                verifier, ip, tcp, udp, queue_ts, void, inactive_ts = line.strip().split(":")
+                if inactive_ts == '-1':
+                    verifier_bytes = identifier_to_bytes(verifier)
+                    ip_class = ip2class(ip)
+                    # print(verifier_bytes.hex(), ip, ip_class, tcp, udp, queue_ts, void, inactive_ts)
+                    self.verifiers[verifier_bytes] = [ip, ip_class, False, False, False]
+                    if ip_class in self.ip_classes:
+                        self.ip_classes[ip_class][0] += 1
+                        self.ip_classes[ip_class][1].append((verifier_bytes, ip))
+                    else:
+                        whois = ip_whois(ip)
+                        # whois = ""
+                        self.ip_classes[ip_class] = [1, [(verifier_bytes, ip)], whois]
+
+    def winner(self, cycle_hash: bytes, scoring=None)-> bytes:
+        """Simplified version of calculateVoteLotteryMethod() for all nodes of the file,
+        regardless of their timestamp"""
+        if scoring is None:
+            scoring = current_score
+        winning_score = maxsize
+        winning_identifier = b''
+        for verifier in self.verifiers:
+            score = scoring(cycle_hash, verifier, self.verifiers[verifier][0])
+            # print(verifier.hex(), score)
+            if score < winning_score:
+                winning_score = score
+                winning_identifier = verifier
+        return winning_identifier
+
+    def calc_ends(self):
+        """Parse data and add flags for "first in range" and "last in range", to estimate bias"""
+        pass
+
+
+