Interplanetary Test Lab

[whyrusleeping]

I've been doing some thinking on a program i want to exist to make testing easier, heres the writeup so far, note that this is a work in progress.

Interplanetary Lab

An application that allows us to script and control large ipfs testnets.

What it looks like

Two parts:

• A daemon that runs on each client machine, connects to 'master' node for instructions.
• A global 'master' daemon that gives out tasks for clients to perform and reports metrics

The client

The client connects the the configured 'master' node on startup and will keep
trying to stay connected until it is manually shut down. Clients generate an
'id' that the master node can choose to authenticate, helping to prevent random
clients from joining the network and messing with things. (this is a soft layer
of protection, if more is needed we can figure something better out).

The client might also want to use a tool like: https://github.com/whyrusleeping/natest
to determine its NAT situation and report that to the master node

Commands (rpc)

The client can be asked via rpc to perform any of the following set of commands by the master

• install
  • Install a particular version of ipfs (can be a git ref, or a tagged version)
  • Restarts the daemon if one is running
• init
  • Initializes an ipfs repo on this client
  • parameters:
    • ConfigTweaks
    • InitFlags
• start
  • Starts an IPFS node on this machine, and keeps it running until told not to. Any crashes should be reported to the master and the node should be restarted.
  • parameters:
    • DaemonFlags
    • EnvVars
• stop
  • Kills the IPFS node on this machine.
• exec
  • Runs an ipfs command, Only runs ipfs <cmd>
• ping
  • Ping a specified host, report latency
• fetch
  • ipfs cat a specific file, output goes to /dev/null, and report statistics about the operation
• mkfile
  • Create a file of a specifed size and add it, returning the hash
  • parameters:
    • Size
• clean
  • Delete repo for the ipfs node on this client
• stat
  • Return statistics about this client

The Master

The master node is a daemon that keeps track of clients and runs commands from the user

Commands (cli)

• list
  • List all connected clients
  • list should also accept filters to allow large networks to be more easily worked with
  • should also have sorting options
• install <client> <version>
  • Start an install of the given ipfs version on the specified client, block until complete.
• clean <client>
  • Kills the daemon running on the given client, and deletes its repo and all associated data.
• stat <client>
  • Print statistics about a given client
• lat <client> <client>
  • Check the latency between the two given clients
• run <client> <cmd>
  • Run an ipfs command on a given client
• auth
  • Auth run with no arguments should list all new, unauthorized clients trying to connect
  • It can be run with an argument to authenticate the selected client
• view
  • view (or a better named command) can be used to view stack traces or other error reports from clients
• tag
  • tag allows the master to add or remove a filterable tag to the selected client
  • this is used for organizing clients in a large network
  • Some tags may get added automatically, for example:
    • The installed version of ipfs
    • running for all clients whose daemons are online
    • idle for all clients whose daemons are offline
• prof-bundle
  • Outputs a bundle of profiler information from the given node, including:
    • The ipfs binary that is running
    • cpu profile (30s)
    • heap profile
    • goroutine stack dump
• log
  • Prints a log of recent events, including:
    • Node crashes (should be noted with a crash ID that can be queried)
    • Node Installs
    • All master node operations (fetches, execs, etc)
    • Information about client {dis,re,}connects

Metrics

The master node can also take advantage of prometheus metrics exposed by each of the clients daemons. (harder, can come later)

Some example usages

On a client machine, should just be:

> iplab-client --master=iplab.i.ipfs.io

Starting the master node:

> iplab --daemon

Running commands on the master:

> iplab list
Client           IP             Status     Uptime   Alerts
foo              10.4.62.34     Idle
whyshome         19.53.102.22   Running    31h17m   (2!)

> iplab install foo v0.4.4
Installing ipfs v0.4.4 on 'foo'...
Binary downloaded, running verification tests...
Binary verified.
ipfs v0.4.4 installed on 'foo'

> iplab init foo
Initialized an ipfs repo on 'foo'

> iplab start foo
Started ipfs daemon on 'foo'

> iplab mkfile whyshome --size=100MB
Creating 100MB file on 'whyshome'...
QmUv48t6NUyFEzPUb2mQ6S2QVv9nFKpuHpAUtFJRtDwYzZ

> iplab fetch foo QmUv48t6NUyFEzPUb2mQ6S2QVv9nFKpuHpAUtFJRtDwYzZ
Fetching 'QmUv48t6NUyFEzPUb2mQ6S2QVv9nFKpuHpAUtFJRtDwYzZ' on 'foo'
# print progress statistics?
Fetch completed in: 12:31s
Time to first byte: 17s
Average Tx Speed: 9.2Mb/s

Commands should be able to also accept tag groups as client name parameters, for example:

> iplab install @blue v0.4.5-rc4

Should install the given version on all nodes with the tag 'blue'

Existing work

iptb

A program that manages a local cluster of ipfs nodes. Basically this tool, but only for a single system.
And lacking the ability to install ipfs versions

https://github.com/whyrusleeping/iptb

ipfs-update

A tool that manages installing and checking different versions of ipfs

https://github.com/ipfs/ipfs-update

[SidHarder]

@whyrusleeping This looks cool. Do you intend for this to work with both js-ipf and go-ipfs or just go? How long do you think it would take complete this?

[whyrusleeping]

@SidHarder I think this could work for go-ipfs, js-ipfs, ipfs-cluster, and maybe even be able to work with other tools in the ecosystem. I don't want to sink myself into just having things for for a single usecase. But I also don't want to sink into overgeneralization hell

[daviddias]

Testground is here!

https://github.com/testground/testground