Implement FEC for files

[george-hopkins]

This patch implements a FEC decoder for files (based on ondd v2.2.0). As I don't have any knowledge about LDPC codes, the implementation is most probably non-optimal (I'm not even sure whether it's actually a LDPC code or they just named it ldpc 😉).

The decoder mimics the same behaviour as ondd: If a file is reconstructable for sure, do it. If not, wait until a new file starts and no new packets are expected to be received. If you want to decode a file as soon as possible, you might want to try to reconstruct earlier. When tested with the four KISS samples, ondd and this implementation seem show the same behaviour.

In case there is already a library which implements such a decoder, I'm happy to incorporate it and ditch my own implementation (same goes for the PRNG).

[daniestevez]

Amazing job! I'll try to review and merge this during the next few days. For the time being, some comments and references:

• RFC5170 This is the RFC describing the LDPC code that we believe that Outernet uses. I'm pretty sure that it is indeed this LDPC code, not only because the FEC parameters in the file announcement XML make sense for this code, but because of a brief look at the PRNG and parity check matrix generator in ondd seemed to match the C implementations in the RFC. You have looked at the ondd binary much more in depth than me, so I'll go check if your code does indeed implement this RFC.

• It is indeed an LDPC code, but this is a rather loose term that only means that the parity check matrix is sparse. This "sparsness" condition is not formally defined, but an informal concept. The thing about LDPC codes is that they are usually decoded using the same message passing algorithms, which works pretty well for them. Thus, they are usually treated in the same way, although there are very different LDPC codes in the wild.

• I've done some tests with ldpc_v2.1, an open source C++ library that implements RFC5170. I don't know if there is something similar for Python. In any case, it would be good to take another look at this library to see if the decoding algorithm is similar to yours.

• I did some tests a couple months ago with some python implementation of what I suspected to be the PRNG and parity check matrix generator. I didn't get this to match what Outernet was using, probably because some stupid small difference. I'll go and compare this with your code. At first glance, the PRNGs look very different, but I believe they are equivalent (mine is much simpler by making use of arbitrary precission arithmetic in Python).

[george-hopkins]

Thank you for your notes, they really helped me to get started. I just compared your matrix generator with mine and only difference are basically the variable names (quite amazing considering the fact that one is completely derived from assembler).

So I think they just did what's explicitly noted should be avoided:

[...] one must keep the most significant bits of the value returned by the PRNG (the least significant bits are known to be less random, and modulo-based solutions should be avoided [PTVF92]).

With this small modification incorporated, both generators return the same matrices:

def pmms_rand(N):
    global seed
    seed = (7**5 * seed) % (2**31 - 1)
    return seed % N

[daniestevez]

I've just tested the code and it works well, so I'm merging it. In fact, from the outernet-f4gmu.kiss and outernet-primesh.kiss, which had frequent packet loss, the new code is able to reconstruct most files, while the older code without FEC got none.

Regarding the PRNG, amazing! I can't believe that this silly mistake by Outernet is what made my attempts at FEC decoding fail. It is really basic stuff that with almost PRNGs that you should divide instead of taking modulo to reduce the size of your random output.