IP allocation requirements

Functional requirements

From the user's POV, the key requirement is the ability to join containers to a multi-host group without having to specify an IP address (and hence w/o having to think of an IP address, making sure it's unique, etc.).

The user should be able to fire up the system and add containers without any static configuration, i.e. it should ideally be as simple as:

$ weave launch --dns --iprange 10.1.0.1/16
$ docker run -h shell.weave.local ubuntu /bin/bash

Users should be able to:

• Run the ipam system on a single node
• Incrementally add new nodes to the system
• Remove nodes from the system

Operations on a given host:

• Main mode of use will be to request IP addresses for new containers before they are running, i.e. we need to operate in a different way to that envisaged for DHCP-like protocols that are used by nodes already on the network
• Should also be able to claim a specific IP address, for backwards-compatibility with existing weave run command syntax
• Finally, given a container name, the system should be able to remove its allocation. This should be done automatically when containers go away.

These operations should be integrated into the existing Weave Network product such that the user does not need to explicitly invoke them.

Non-functional requirements

"The system" refers to a collection of hosts working together to issue IP addresses.

1. Consistency - There should be no conflict in IP address assignment, i.e., at any given instant of time there should not be two or more containers with the same IP address in the system. The corollary of this is the system should guarantee never to issue an IP address to a new container if that IP is in use elsewhere in the system.
2. In the absence of network partition or host failure, the system should be able to issue IP address allocations if there are free addresses elsewhere in the system.
3. That being said, under partitions and failures, and the system should attempt continue to operate where possible, without breaching requirement 1. The protocol should also be able to handle network partitioning and merging - a single system should be able to be split into two or more parts that are unable to communicate across the partition, then recover when communication is restored. Note during said partition requirement 1 must be held.
4. An IP address is assigned only for the duration the container stays in the network. When the container exits or departs the network, its IP address should become available for assignment to other containers.
5. Robustness - These requirements must hold true across restarts of the system. The system needs to be able to recover from a crash of the process managing IP addresses without invalidating these requirements.
6. Scalability: The protocol should avoid significant performance degradation as the size of the network increases.

• "Normal" sizing expected to be up to hundreds of containers per host, and up to 100 hosts in a network. Should be able to install and run on a single host.
• Start/stop rate of containers on one host expected to be up to 10/sec
• Time to request an IP address should be less than 10ms in 95% of cases

Dimensions to consider:

1. total node count
   • low (< 10)
   • medium (20-100)
   • high (>100)
2. total IP count average per node
   • low (<10)
   • medium (20-100)
   • high (>100)
3. distribution of IPs between nodes
   • even
   • uneven and static
   • uneven and dynamic
4. churn of IPs
   • low
   • high
5. utilisation of IP space
   • low (<20%)
   • medium (20-80%)
   • high (>80%)
6. node count
   • largely static
   • gradual increase and decrease
   • spikey
7. node churn
   • low
   • high
8. node failure rate
   • low
   • high

Non-requirements

• We do not require the system to be able to take two independently-running Weave networks that have existing IP addresses allocated, and merge the reservations so as to work as one bigger network.
• Automatically removing failed nodes and their associated allocations.
  • This is included as it is impossible to differentiate transient failures from hosts being completely removed; therefore, it is to possible to automatically remove failed hosts and still satisfy the consistency guarantees.

Future requirements

• Subnet configuration: Administrator should be able to set up multiple subnets for isolation and join containers to specific subnets.
• From @yaronr in issue #22: "The ability to lease IPs, CIDR ranges, and maybe even Strings with regex (server names?) and reclaim them"
• From @yaronr in issue #22: "The specifications of this service also reflect on multi-tenancy capabilities, since I could envision multiple customers each with a different CIDR range, and these CIDR ranges are also an asset that needs proper administration"
• Ability to specify static addresses for some containers, which can also communicate with IPAM-allocated containers (i.e. they are on the same subnet).