Merge pull request ethereum#46 from ethersphere/swinx-nagydani

Additional discussion of long-term storage incentives and history

swarm/docs/sw^3/history.rst

@@ -109,10 +109,15 @@ related:
 Towards Web 3.0
 -----------------------------
 
-In order to enable responsive distributed web applications (called dapps
-in Web 3.0 communities), :dfn:`IPFS` (:cite:`ipfs2014`) had to introduce a few major improvements over Bittorrent. The most immediately apparent novelty is the highly web-compatible URL-based retrieval. In addition, the directory (also organized as a DHT) has been vastly
-improved, making it possible to search for any part of any file (called
-*chunk*). It has also been made very flexible and pluggable in order to work with any kind of storage backend, be it a laptop with intermittent wifi, or a sophisticated HA cluster in a fiber-optic connected datacenter.
+The straightforward approach of using bittorrent as a distribution mechanism for web content has been successfully implemented by
+:dfn:`Zeronet`. However, because of the aforementioned issues with bittorrent, Zeronet fails to provide the same kind of responsive 
+experience that users of web services came to expect.
+
+In order to enable responsive distributed web applications (called dapps in Web 3.0 communities), :dfn:`IPFS` (:cite:`ipfs2014`) had to 
+introduce a few major improvements over Bittorrent. The most immediately apparent novelty is the highly web-compatible URL-based 
+retrieval. In addition, the directory (also organized as a DHT) has been vastly improved, making it possible to search for any part of 
+any file (called *chunk*). It has also been made very flexible and pluggable in order to work with any kind of storage backend, be it a 
+laptop with intermittent wifi, or a sophisticated HA cluster in a fiber-optic connected datacenter.
 
 A further important innovation is that IPFS has incentivisation factored out into pluggable modules. Modules such as bitswap for example establish that it is in the interest of greedy downloaders to balance the load they impose on other nodes, and also that it is in every node's interest to host popular content. Bitswap or no bitswap, IPFS largely solves the problem of content consumers helping shouldering the costs of information dissemination.
 
@@ -126,7 +131,7 @@ A further important innovation is that IPFS has incentivisation factored out int
 ..
   While there is not much to gain for the user by choking uploads, or falsely advertizing content, without bitswap there is not much penalty for it either. However, bitswap incentivizes the hosting of popular content, since the constraint of swapped bits coming from the same piece of content are gone in IPFS. If you host popular content, bitswap-guarded nodes will be nice to you. There aren't that many of them, though. In this early stage of abundance, while supplied disk and bandwidth vastly outstrip demand, the system works fine as it is. If bottlenecks emerge either due to increased use or malicious intent, bitswap can be expected to become more popular as a security measure against widespread freeriding. Bitswap or no bitswap, IPFS largely solves the problem of content consumers helping shouldering the costs of information dissemination.
 
-The same problem with lack of incentives is apparent in various other projects such as zeronet, i2p or MAIDSAFE.
+The same problem with lack of incentives is apparent in various other projects such as zeronet or MAIDSAFE.
 Incentivization for distributed document storage is still a relatively new research field. Especially in the light of the blockchain. The Tor network has seen suggestions (:cite:`jansen2014onions`, :cite:`ghoshetal2014tor`) but these schemes are largely academic, they are not built in at the heart of the underlying system. Bitcoin has also been repurposed to drive other systems like Permacoin (:cite:`miller2014permacoin`) or Sia (:cite:`vorick2014sia`), some use their own blockchain, altcoin such as Metadisk (:cite:`wilkinson2014metadisk`) for Storj (:cite:`wilkinsonetal2014storj`) or Filecoin (:cite:`filecoin2014`) for IPFS :cite:`ipfs2014`.
 
 What is still missing from the above incentive system, is the possibility to rent out

swarm/docs/sw^3/storage.rst

@@ -42,25 +42,7 @@ especially if the user explicitly requires that in the fashion of 'upload and di
 :dfn:`Filecoin` (:cite:`filecoin2014`), an incentivised p2p storage network using IPFS (:cite:`ipfs2014`) offers an interesting solution. Nodes participating in its network also mine
 on the filecoin blockchain. Filecoin can be earned (mined) through replicating other people's content and spent on having one's content replicated.
 Filecoin's proof of work is defined to include proof that the miner possesses a set of randomly chosen units of storage depending on the parent block.
-Using a strong proof of retrievability scheme, Filecoin ensures that the winning miner had relevant data. As miners compete, they will find that their chances of winning will be proportional to the percentage of the existing storage units they actually store. This is because the missing ones need to be retrieved from other nodes and thus delaying nodes chance to respond.
-
-We see a whole range of issues with this particular approach:
-
-* It is not clear that network latency cannot be masked by the parallel calculation of the ordinary proof of work component in the algorithm.
-* If the set of chunks are not selected differently for each node, mining will resemble a DDOS on nodes that actually store the data needed for the round.
-* Even if the selection of data to prove varies depending on the miner, normal operation incurs huge network traffic.
-* As the network grows, the expected proportion of the data that needs to be retrieved increases. In fact given a practical maximum limit on a node's storage capacity, this proportion reaches a ceiling. If that happens miners will end up effectively competing on bandwidth.
-* In order to check proof of retrievability responses as part of block validation, existing data needs to be recorded on the blockchain. This leads to excessive use of the blockchain as the network grows and is unlikely to scale.
-* Competing miners working on the same task mean redundant use of resources.
-* If content is known to be popular, checking their integrity is spurious. But if choice of storage data to audit for the next block is truely random, there is no distinction between rarely accessed content and popular ones stored by many nodes resulting in wasted resouces.
-* Similarly, users originating the content have also no way to indicate directly that some documents are important and not to be lost, while other temporary or derived data they can afford to lose.
-
-Due to excessive use of blockchain and generated network traffic, these issues make the approach suspect: at best hugely wasteful, at worst infeasible on the large scale.
-
-More importantly, however, Filecoin provides only a scheme to collectively incentivise the network to store content. This brings in a 'tragedy of the commons' problem in that losing any particular data will have no negative consequence to any one storer node. This lack of individual accountability means the solution is rather limited as a security measure against lost content.
-
-To summarise, we consider positive incentivisation in itself insufficient for ensured archival. In addition to that collective positive incentivisation implemented by competitive proof of retrievability mining is wasteful in terms of network traffic, computational resources as well as blockchain storage. In the subsequent sections we will introduce a different approach.
-
+Using a strong proof of retrievability scheme, Filecoin ensures that the winning miner had relevant data. As miners compete, they will find that their chances of winning will be proportional to the percentage of the existing storage units they actually store. This is because the missing ones need to be retrieved from other nodes, which is hopelessly slow for the purpose of mining.
 
 Compensation for storage and guarantees for long-term data preservation
 ===========================================================================
@@ -78,6 +60,30 @@ The flipside of using only this incentive on it own is that chunks that are rare
 
 A long-term storage incentivisation scheme faces unique challenges. For example, unlike in the case of bandwidth incentives where retrievals are immediately accounted and settled, long-term storage guarantees are promisory in nature and deciding if the promise was kept can only be decided at the end of its validity. Loss of reputation is not an available deterrent against foul play in these instances: since new nodes need to be allowed to provide services right away, cheaters could just resort to new identities and keep selling (empty) storage promises.
 
+In the context of an owner paying one or more entities for long-term storage of data where losing even a small part of it renders it 
+useless, but large enough for it not to be practical to regularly verify the availability of every bit, particual attention is required 
+in the design of the incentive system to make the failure to store every last bit unprofitable. In the typical case, the storer 
+receives some payment upon successfully passing a probabilistic audit. Such audits are structured in such a way that the storer always 
+passes the audit, if she indeed has the entire data set, but there is also some non-zero probability of passing the audit if some data 
+is lost. This probability, of course, depends on what data is lost, but in any design there is a maximum to it, typically attained by 
+losing some very small portion of data. In case of systems that break up the date into chunks with a fixed maximum size, the probability 
+of passing the audit with incomplete data is maximized in the case when only one chunk is lost.
+
+Let us denote this maximum probability by :math:`p`. To keep failure to store all data unprofitable, the expected payout :math:`E` of subjecting 
+oneself to an audit should be negative, if the probability of failure is greater or equal to :math:`1 - p`. Formally: :math:`E = pR - (1-p)P < 0` 
+where :math:`R` is the reward for passing the audit and :math:`P` is the penalty for failing it. This yields the following important constraint on 
+the relative amounts of reward and penalty: :math:`P > pR/(1 - p)`. If this constraint is not met, storers doing a less than perfect --
+and thus unacceptable -- job can still be profitable, resulting in misaligned incentives.
+
+In the simplest case, the audit is a securely randomized spot check of having one random chunk of data in custody, with the proof of
+custody being the chunk itself. If the data consists of :math:`n` chunks, the probability of passing the audit with one chunk missing
+:math:`p = (n - 1)/n`. This yields :math:`P > (n - 1)R`. For different audits, :math:`p` is calculated differently.
+
+Thus, in order to pay :math:`R` for a successful audit, the auditor (be it the owner, a third party or even a smart contract) needs to verify
+whether or not the storer has comitted to losing at least :math:`P` in case of failure before the audit.
+
+
+
 ..  index::
   reputation
   punative measures
@@ -513,8 +519,16 @@ When the network grows, it can happen that a custodian finds a new registered no
 Multiple receipts and local replication
 ----------------------------------------------
 
-As discussed above owners can manage the desired security level by using erasure coding with arbitrary degree of redundancy. However, it still makes sense to require that more than one node actually store the chunk. Although the cloud industry is trying to get away from the explicit x-fold redundancy model because it is very wasteful and incurs high costs – erasure coding can guarantee the same level of security using only a fraction of the storage space. However, in a data center, redundancy is interpreted in the context of hard drives whose failure rates are low, independent and predictable and their connectivity is almost guaranteed at highest possible speed due to proximity. In a peer-to-peer network scenario, nodes could disappear much more frequently than hard drives fail. In order to guarantee robust operation, we need to require several local replicas of each chunk (commonly 3, see :cite:`wilkinson2014metadisk`). Since the syncing protocol already provides replication across the proximate bin, regular resyncing of the chunk may be sufficient to ensure availability in case the custodian drops off. If this proves too weak in practice we may require the custodian to get receipts from two proximate peers who act as cocustodians. The added benefit of this extra complexity is unclear.
-There is another interesting idea where we allow fixed permutations of each chunks hash to encode the same hash from the start (similar to the discussion in section :ref:`sec:erasure`). Redundancy produced his way can also counterbalance the decreased reliability of storage for large file sizes which we introduced with fixing the size of datablobs which are CRS encoded (restricting it to a node, i.e., a data blob of size :math:`b\cdot m` = branching times redundancy) at the cost of extra linear storage.
+As discussed above, owners can manage the desired security level by using erasure coding with arbitrary degree of redundancy. However, 
+it still makes sense to require that more than one node actually store the chunk. Although the cloud industry is trying to get away 
+from the explicit x-fold redundancy model because it is very wasteful and incurs high costs – erasure coding can guarantee the same 
+level of security using only a fraction of the storage space. However, in a data center, redundancy is interpreted in the context of 
+hard drives whose failure rates are low, independent and predictable and their connectivity is almost guaranteed at highest possible 
+speed due to proximity. In a peer-to-peer network scenario, nodes could disappear much more frequently than hard drives fail. In order 
+to guarantee robust operation, we need to require several local replicas of each chunk (commonly 3, see :cite:`wilkinson2014metadisk`). 
+Since the syncing protocol already provides replication across the proximate bin, regular resyncing of the chunk may be sufficient to 
+ensure availability in case the custodian drops off. If this proves too weak in practice we may require the custodian to get receipts 
+from two proximate peers who act as cocustodians. The added benefit of this extra complexity is unclear.
 
 
 .. _sec:accounting:
@@ -601,4 +615,4 @@ Various ways of escrow conditions on the release of funds are able to capture qu
 We presented a technique of erasure coding applicable to the swarm hash tree, which makes it possible for clients to manage the level of storage security and redundancy within their own remit.
 
 .. bibliography:: ../refs.bib
-  :style: plain
+  :style: plain