General Code Overview

This is an excerpt of an email the dev team used to try and explain the current state of the vault code. It will change quickly but is here as an aid memoir

Above the RUDP layer we have the Routing layer which between them implement most of the peer-to-peer DHT (it's a wee bit like Kademlia). Routing-level messages contain information solely related to peers' contact details (Node ID, IPs, Ports, etc.). Above Routing we have Clients and Vaults. The Vaults form a P2P overlay - I guess they're closest to servers in a more traditional network. They implement the security, account mangement, and perform the actual storage of the users' data on the network.

We have several "Personas" which a Vault/Client implements. Clients only have a single persona (we have 3 different types of client in the pipeline), and a single Vault type which is composed of 5 Personas. The Personas are possibly best thought of as different services, so one Persona might be responsible for looking after Clients' accounts, while another is responsible for maintaining the availability/integrity of data.

We have a lot of Vault-level messages which the Client doesn't need to be exposed to, so we have come up with a "Network-Filesystem" (NFS) layer to provide the message interface to the Vault network. The NFS library is intended to be used by Clients and Vaults alike, but means that the Client library shouldn't need to include the Vault library.

We've struggled in the past with debugging the network, and in the last few months we've been pushing very hard to achieve as much type-safety as possible. Hence the Routing message (https://github.com/maidsafe/MaidSafe-Routing/blob/next/include/maidsafe/routing/message.h#L57) is templated on Sender and Receiver.

All Vault-level and NFS messages are wrapped into Routing messages. This means every NFS/Vault message needs to also define these 2 Routing template args. The NFS message actually has 6 template args: https://github.com/maidsafe/MaidSafe-Network-Filesystem/blob/next/include/maidsafe/nfs/message_wrapper.h#L47. In order, these are: an enum defining the action, sender Persona type, sender Routing type, receiver Persona type, receiver Routing type, and contents type. The contents are all fairly small structs which wrap Google protocol buffer classes. We could just use protobufs here, but we're loathe to make them part of our API.

Now for the real strangeness :-) To avoid hand-writing a lot of boiler-plate, we use CMake to generate source files from custom input files (I chose CMake simply because I know it better than Python). An example input file is https://github.com/maidsafe/MaidSafe-Network-Filesystem/blob/next/cmake/vault/maid_manager.message_types.meta Each row essentially defines the 6 template args for an nfs::WrapperMessage. The result is: https://gist.github.com/Fraser999/6776083

So we have one such generated header for the NFS messages and another for the Vault ones. Each Persona can then define the list of message types it handles in terms of a single NFS variant and/or a single Vault variant. Demultiplexing becomes a relatively straightforward matter of applying a static_visitor. It's not quite as simple as I'd like, for a couple of reasons.

First reason: we define the Service class in NFS: (https://github.com/maidsafe/MaidSafe-Network-Filesystem/blob/next/include/maidsafe/nfs/service.h) This is templated on the Persona Service (which provides the implementation details for that Persona). However, since each service can handle either one or two types of variant, we've had to require each Persona service to provide typedefs for the Public (i.e. NFS) variant and the Vault variant.

This is a bit frustrating, since we're already handling concatenating 2 variants into a new single variant for those Personas which have both types (e.g. (https://github.com/maidsafe/MaidSafe-Vault/blob/next/src/maidsafe/vault/maid_manager/service.h#L160)). It would be better to have the Service class in NFS just deal with one variant. The problem is that the generated boilerplate defines overloaded functions "GetVariant" for each variant. We'd need to combine 2 separate such functions into a new one - and this doesn't seem to have a trivial answer to me.

The second reason is that since we want to maintain separation between the NFS and Vault libraries, the Service class in NFS doesn't include the vault header in which the Vault variants are declared. This means that in Vaults, we always have to ensure that we place the #include "maidsafe/vault/message_types" before the #include "maidsafe/nfs/service.h". Having a required inclusion order smacks of bad design. Again, I don't see an easy fix.

OK - so I think that maybe begins to explain the NFS/Vault insanity! Or maybe it's made it worse? :-) Anyway, while we're keen to hear your and Joel's thoughts on this, I think we'd still prefer that RUDP is given priority, unless you feel there is no more you can do in the allotted timescale?