9-nodes cluster in VMware on Windows PC

[andriytk]

Managed to start 9-nodes cluster (7 server nodes with the disks + 1 cmu + 1 client node) in VMware+Vagrant environment on my Windows PC desktop with 8+2 EC-configuration and run a smoke performance test:

$ m0vg env
M0_VM_SSU_NR=7
M0_VM_CMU_CPUS=8
M0_VM_CMU_MEM_MB=8192
M0_VM_SSU_CPUS=4
M0_VM_CLIENT_CPUS=4
M0_VM_CLIENT_MEM_MB=2048
M0_VM_CMU_ENABLE_NFS_SERVER=yes
M0_VM_CMU_ENABLE_PUBLIC_NET=yes
M0_VM_SHARE_TYPE=smb
$
$ m0vg up cmu ssu{1..7} client1
Bringing machine 'cmu' up with 'vmware_desktop' provider...
Bringing machine 'ssu1' up with 'vmware_desktop' provider...
Bringing machine 'ssu2' up with 'vmware_desktop' provider...
Bringing machine 'ssu3' up with 'vmware_desktop' provider...
Bringing machine 'ssu4' up with 'vmware_desktop' provider...
Bringing machine 'ssu5' up with 'vmware_desktop' provider...
Bringing machine 'ssu6' up with 'vmware_desktop' provider...
Bringing machine 'ssu7' up with 'vmware_desktop' provider...
Bringing machine 'client1' up with 'vmware_desktop' provider...
...
$ m0vg status
Current machine states:

cmu                       running (vmware_desktop)
ssu1                      running (vmware_desktop)
ssu2                      running (vmware_desktop)
ssu3                      running (vmware_desktop)
ssu4                      running (vmware_desktop)
ssu5                      running (vmware_desktop)
ssu6                      running (vmware_desktop)
ssu7                      running (vmware_desktop)
client1                   running (vmware_desktop)

This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.

$
$ ssh vagrant@192.168.85.132
Last login: Mon Dec 14 18:16:11 2020 from 192.168.85.1
[vagrant@cmu ~]$
[vagrant@cmu ~]$ hctl bootstrap --mkfs ~/cmu-ssu7-cleint1-cdf.yaml
2020-12-14 18:45:51: Generating cluster configuration...Warning: Permanently added the ECDSA host key for IP address '192.168.1.64' to the list of known hosts.
 OK
2020-12-14 18:46:03: Starting Consul server on this node......... OK
2020-12-14 18:46:10: Importing configuration into the KV store... OK
2020-12-14 18:46:11: Starting Consul on other nodes..... OK
2020-12-14 18:46:14: Updating Consul configuraton from the KV store... OK
2020-12-14 18:46:17: Waiting for the RC Leader to get elected... OK
2020-12-14 18:46:17: Starting Motr (phase1, mkfs)... OK
2020-12-14 18:46:32: Starting Motr (phase1, m0d)... OK
2020-12-14 18:46:37: Starting Motr (phase2, mkfs)... OK
2020-12-14 18:47:01: Starting Motr (phase2, m0d)... OK
2020-12-14 18:47:12: Checking health of services... OK
[vagrant@cmu ~]$
[vagrant@cmu ~]$ hctl status
Data pool:
    # fid name
    0x6f00000000000001:0x11b 'default'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x199 'all': 'default' None None
Services:
    client1.local
    [started]  hax        0x7200000000000001:0x10f  192.168.85.141@tcp:12345:1:1
    [unknown]  m0_client  0x7200000000000001:0x112  192.168.85.141@tcp:12345:4:1
    [unknown]  m0_client  0x7200000000000001:0x115  192.168.85.141@tcp:12345:4:2
    [unknown]  m0_client  0x7200000000000001:0x118  192.168.85.141@tcp:12345:4:3
    ssu1.local
    [started]  hax        0x7200000000000001:0x16  192.168.85.133@tcp:12345:1:1
    [started]  confd      0x7200000000000001:0x19  192.168.85.133@tcp:12345:2:1
    [started]  ioservice  0x7200000000000001:0x1c  192.168.85.133@tcp:12345:2:2
    [unknown]  m0_client  0x7200000000000001:0x30  192.168.85.133@tcp:12345:4:1
    [unknown]  m0_client  0x7200000000000001:0x33  192.168.85.133@tcp:12345:4:2
    [unknown]  m0_client  0x7200000000000001:0x36  192.168.85.133@tcp:12345:4:3
    cmu.local  (RC)
    [started]  hax        0x7200000000000001:0x4   192.168.85.132@tcp:12345:1:1
    [started]  confd      0x7200000000000001:0x7   192.168.85.132@tcp:12345:2:1
    [unknown]  m0_client  0x7200000000000001:0xa   192.168.85.132@tcp:12345:4:1
    [unknown]  m0_client  0x7200000000000001:0xd   192.168.85.132@tcp:12345:4:2
    [unknown]  m0_client  0x7200000000000001:0x10  192.168.85.132@tcp:12345:4:3
    ssu2.local
    [started]  hax        0x7200000000000001:0x3c  192.168.85.134@tcp:12345:1:1
    [started]  confd      0x7200000000000001:0x3f  192.168.85.134@tcp:12345:2:1
    [started]  ioservice  0x7200000000000001:0x42  192.168.85.134@tcp:12345:2:2
    [unknown]  m0_client  0x7200000000000001:0x56  192.168.85.134@tcp:12345:4:1
    [unknown]  m0_client  0x7200000000000001:0x59  192.168.85.134@tcp:12345:4:2
    [unknown]  m0_client  0x7200000000000001:0x5c  192.168.85.134@tcp:12345:4:3
    ssu3.local
    [started]  hax        0x7200000000000001:0x62  192.168.85.135@tcp:12345:1:1
    [started]  ioservice  0x7200000000000001:0x65  192.168.85.135@tcp:12345:2:1
    [unknown]  m0_client  0x7200000000000001:0x79  192.168.85.135@tcp:12345:4:1
    [unknown]  m0_client  0x7200000000000001:0x7c  192.168.85.135@tcp:12345:4:2
    [unknown]  m0_client  0x7200000000000001:0x7f  192.168.85.135@tcp:12345:4:3
    ssu4.local
    [started]  hax        0x7200000000000001:0x85  192.168.85.136@tcp:12345:1:1
    [started]  ioservice  0x7200000000000001:0x88  192.168.85.136@tcp:12345:2:1
    [unknown]  m0_client  0x7200000000000001:0x9c  192.168.85.136@tcp:12345:4:1
    [unknown]  m0_client  0x7200000000000001:0x9f  192.168.85.136@tcp:12345:4:2
    [unknown]  m0_client  0x7200000000000001:0xa2  192.168.85.136@tcp:12345:4:3
    ssu5.local
    [started]  hax        0x7200000000000001:0xa8  192.168.85.137@tcp:12345:1:1
    [started]  ioservice  0x7200000000000001:0xab  192.168.85.137@tcp:12345:2:1
    [unknown]  m0_client  0x7200000000000001:0xbf  192.168.85.137@tcp:12345:4:1
    [unknown]  m0_client  0x7200000000000001:0xc2  192.168.85.137@tcp:12345:4:2
    [unknown]  m0_client  0x7200000000000001:0xc5  192.168.85.137@tcp:12345:4:3
    ssu6.local
    [started]  hax        0x7200000000000001:0xcb  192.168.85.138@tcp:12345:1:1
    [started]  ioservice  0x7200000000000001:0xce  192.168.85.138@tcp:12345:2:1
    [unknown]  m0_client  0x7200000000000001:0xe2  192.168.85.138@tcp:12345:4:1
    [unknown]  m0_client  0x7200000000000001:0xe5  192.168.85.138@tcp:12345:4:2
    [unknown]  m0_client  0x7200000000000001:0xe8  192.168.85.138@tcp:12345:4:3
    ssu7.local
    [started]  hax        0x7200000000000001:0xee  192.168.85.139@tcp:12345:1:1
    [started]  ioservice  0x7200000000000001:0xf1  192.168.85.139@tcp:12345:2:1
    [unknown]  m0_client  0x7200000000000001:0x105  192.168.85.139@tcp:12345:4:1
    [unknown]  m0_client  0x7200000000000001:0x108  192.168.85.139@tcp:12345:4:2
    [unknown]  m0_client  0x7200000000000001:0x10b  192.168.85.139@tcp:12345:4:3
[vagrant@cmu ~]$
[vagrant@cmu ~]$
[vagrant@cmu ~]$
[vagrant@cmu ~]$ cd /data/motr/bindings/go/mcp/
[vagrant@cmu mcp]$ go build
[vagrant@cmu mcp]$ hctl status | head
Data pool:
    # fid name
    0x6f00000000000001:0x11b 'default'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x199 'all': 'default' None None
Services:
    client1.local
    [started]  hax        0x7200000000000001:0x10f  192.168.85.141@tcp:12345:1:1
    [unknown]  m0_client  0x7200000000000001:0x112  192.168.85.141@tcp:12345:4:1
[vagrant@cmu mcp]$ hctl status | grep -A5 `hostname`
    cmu.local  (RC)
    [started]  hax        0x7200000000000001:0x4   192.168.85.132@tcp:12345:1:1
    [started]  confd      0x7200000000000001:0x7   192.168.85.132@tcp:12345:2:1
    [unknown]  m0_client  0x7200000000000001:0xa   192.168.85.132@tcp:12345:4:1
    [unknown]  m0_client  0x7200000000000001:0xd   192.168.85.132@tcp:12345:4:2
    [unknown]  m0_client  0x7200000000000001:0x10  192.168.85.132@tcp:12345:4:3
[vagrant@cmu mcp]$ export LD_LIBRARY_PATH=../../../motr/.libs/
[vagrant@cmu mcp]$ pdsh -w cmu.local,ssu[1-7].local,client1.local sudo chmod g+rw /dev/m0lnet
[vagrant@cmu mcp]$ pdsh -w cmu.local,ssu[1-7].local,client1.local sudo chgrp vagrant /dev/m0lnet
[vagrant@cmu mcp]$ ./mcp -prof 0x7000000000000001:0x199 -hax 192.168.85.132@tcp:12345:1:1 -proc 0x7200000000000001:0xa -ep 192.168.85.132@tcp:12345:4:1 -osz $((128*1024)) -v -bsz 256 -threads 2 /dev/zero 12345:123456700
2020/12/14 18:49:40 mio.go:477: W: off=0 len=268435456 bs=134217728 gs=33554432 speed=77 (Mbytes/sec)
2020/12/14 18:49:42 mio.go:477: W: off=268435456 len=268435456 bs=134217728 gs=33554432 speed=184 (Mbytes/sec)
2020/12/14 18:49:43 mio.go:477: W: off=536870912 len=268435456 bs=134217728 gs=33554432 speed=185 (Mbytes/sec)
2020/12/14 18:49:45 mio.go:477: W: off=805306368 len=268435456 bs=134217728 gs=33554432 speed=195 (Mbytes/sec)
2020/12/14 18:49:46 mio.go:477: W: off=1073741824 len=268435456 bs=134217728 gs=33554432 speed=172 (Mbytes/sec)
2020/12/14 18:49:48 mio.go:477: W: off=1342177280 len=268435456 bs=134217728 gs=33554432 speed=176 (Mbytes/sec)
2020/12/14 18:49:49 mio.go:477: W: off=1610612736 len=268435456 bs=134217728 gs=33554432 speed=191 (Mbytes/sec)
2020/12/14 18:49:50 mio.go:477: W: off=1879048192 len=268435456 bs=134217728 gs=33554432 speed=179 (Mbytes/sec)
2020/12/14 18:49:52 mio.go:477: W: off=2147483648 len=268435456 bs=134217728 gs=33554432 speed=185 (Mbytes/sec)
2020/12/14 18:49:53 mio.go:477: W: off=2415919104 len=268435456 bs=134217728 gs=33554432 speed=186 (Mbytes/sec)
2020/12/14 18:49:55 mio.go:477: W: off=2684354560 len=268435456 bs=134217728 gs=33554432 speed=176 (Mbytes/sec)
2020/12/14 18:49:56 mio.go:477: W: off=2952790016 len=268435456 bs=134217728 gs=33554432 speed=179 (Mbytes/sec)
2020/12/14 18:49:58 mio.go:477: W: off=3221225472 len=268435456 bs=134217728 gs=33554432 speed=165 (Mbytes/sec)
2020/12/14 18:49:59 mio.go:477: W: off=3489660928 len=268435456 bs=134217728 gs=33554432 speed=168 (Mbytes/sec)
2020/12/14 18:50:01 mio.go:477: W: off=3758096384 len=268435456 bs=134217728 gs=33554432 speed=185 (Mbytes/sec)
2020/12/14 18:50:02 mio.go:477: W: off=4026531840 len=268435456 bs=134217728 gs=33554432 speed=191 (Mbytes/sec)
2020/12/14 18:50:04 mio.go:477: W: off=4294967296 len=268435456 bs=134217728 gs=33554432 speed=188 (Mbytes/sec)
2020/12/14 18:50:05 mio.go:477: W: off=4563402752 len=268435456 bs=134217728 gs=33554432 speed=195 (Mbytes/sec)
2020/12/14 18:50:06 mio.go:477: W: off=4831838208 len=268435456 bs=134217728 gs=33554432 speed=197 (Mbytes/sec)
2020/12/14 18:50:08 mio.go:477: W: off=5100273664 len=268435456 bs=134217728 gs=33554432 speed=191 (Mbytes/sec)
2020/12/14 18:50:09 mio.go:477: W: off=5368709120 len=268435456 bs=134217728 gs=33554432 speed=193 (Mbytes/sec)
2020/12/14 18:50:10 mio.go:477: W: off=5637144576 len=268435456 bs=134217728 gs=33554432 speed=189 (Mbytes/sec)
2020/12/14 18:50:12 mio.go:477: W: off=5905580032 len=268435456 bs=134217728 gs=33554432 speed=192 (Mbytes/sec)
2020/12/14 18:50:13 mio.go:477: W: off=6174015488 len=268435456 bs=134217728 gs=33554432 speed=188 (Mbytes/sec)
2020/12/14 18:50:14 mio.go:477: W: off=6442450944 len=268435456 bs=134217728 gs=33554432 speed=193 (Mbytes/sec)
2020/12/14 18:50:16 mio.go:477: W: off=6710886400 len=268435456 bs=134217728 gs=33554432 speed=197 (Mbytes/sec)
2020/12/14 18:50:17 mio.go:477: W: off=6979321856 len=268435456 bs=134217728 gs=33554432 speed=197 (Mbytes/sec)
2020/12/14 18:50:18 mio.go:477: W: off=7247757312 len=268435456 bs=134217728 gs=33554432 speed=192 (Mbytes/sec)
2020/12/14 18:50:20 mio.go:477: W: off=7516192768 len=268435456 bs=134217728 gs=33554432 speed=197 (Mbytes/sec)
2020/12/14 18:50:21 mio.go:477: W: off=7784628224 len=268435456 bs=134217728 gs=33554432 speed=194 (Mbytes/sec)
2020/12/14 18:50:22 mio.go:477: W: off=8053063680 len=268435456 bs=134217728 gs=33554432 speed=193 (Mbytes/sec)
2020/12/14 18:50:24 mio.go:477: W: off=8321499136 len=268435456 bs=134217728 gs=33554432 speed=187 (Mbytes/sec)
2020/12/14 18:50:25 mio.go:477: W: off=8589934592 len=268435456 bs=134217728 gs=33554432 speed=185 (Mbytes/sec)
2020/12/14 18:50:27 mio.go:477: W: off=8858370048 len=268435456 bs=134217728 gs=33554432 speed=179 (Mbytes/sec)
2020/12/14 18:50:28 mio.go:477: W: off=9126805504 len=268435456 bs=134217728 gs=33554432 speed=188 (Mbytes/sec)
2020/12/14 18:50:29 mio.go:477: W: off=9395240960 len=268435456 bs=134217728 gs=33554432 speed=183 (Mbytes/sec)
2020/12/14 18:50:31 mio.go:477: W: off=9663676416 len=268435456 bs=134217728 gs=33554432 speed=164 (Mbytes/sec)
2020/12/14 18:50:32 mio.go:477: W: off=9932111872 len=268435456 bs=134217728 gs=33554432 speed=179 (Mbytes/sec)
2020/12/14 18:50:34 mio.go:477: W: off=10200547328 len=268435456 bs=134217728 gs=33554432 speed=196 (Mbytes/sec)
2020/12/14 18:50:35 mio.go:477: W: off=10468982784 len=268435456 bs=134217728 gs=33554432 speed=195 (Mbytes/sec)
2020/12/14 18:50:36 mio.go:477: W: off=10737418240 len=268435456 bs=134217728 gs=33554432 speed=195 (Mbytes/sec)
2020/12/14 18:50:38 mio.go:477: W: off=11005853696 len=268435456 bs=134217728 gs=33554432 speed=194 (Mbytes/sec)
2020/12/14 18:50:39 mio.go:477: W: off=11274289152 len=268435456 bs=134217728 gs=33554432 speed=196 (Mbytes/sec)
2020/12/14 18:50:40 mio.go:477: W: off=11542724608 len=268435456 bs=134217728 gs=33554432 speed=197 (Mbytes/sec)
2020/12/14 18:50:42 mio.go:477: W: off=11811160064 len=268435456 bs=134217728 gs=33554432 speed=199 (Mbytes/sec)
2020/12/14 18:50:43 mio.go:477: W: off=12079595520 len=268435456 bs=134217728 gs=33554432 speed=196 (Mbytes/sec)
2020/12/14 18:50:45 mio.go:477: W: off=12348030976 len=268435456 bs=134217728 gs=33554432 speed=190 (Mbytes/sec)
2020/12/14 18:50:46 mio.go:477: W: off=12616466432 len=268435456 bs=134217728 gs=33554432 speed=196 (Mbytes/sec)
2020/12/14 18:50:47 mio.go:477: W: off=12884901888 len=268435456 bs=134217728 gs=33554432 speed=193 (Mbytes/sec)
^C
[vagrant@cmu mcp]$

Memory footprint and CPU load during the test:

Note: in case Motr is installed not from rpm, but compiled manually - make sure to configure it with --disable-expensive-checks to get some reasonable perf numbers.

Versions:

• Motr - commit fb19a78
• Hare - commit 571ae39

Cluster configuration file (CDF): cmu-ssu7-cleint1-cdf.yaml.txt.

[huanghua78]

Fantastic, @andriytk .
How much memory is configured for each VM?
How many CPU cores are configured for each VM?
How much disk space for each internal dick?

[andriytk]

Thanks @huanghua78 for reminding me mentioning it, let me update it...

[andriytk]

So, I've added the output of m0vg env command there in the beginning which shows that:

1. 8GB RAM is configured for cmu and 2GBs (the default value) is configured for ssu and client1 nodes.
2. 8 CPU cores is configured for cmu and 4 cores is configured for ssu and client1 nodes.
3. By default 6 2GB-disks are configured per each ssu node (6 * 2 * 7 == 84 GB in total).

I've configured more resources in cmu to speed up the building of Motr mainly. But in VMware environment (in contrast to VirtualBox) the RAM is shared among host and VMs - unused memory can be used by host or another VMs. So if the memory is not used by cmu - it won't affect the others. Also cmu can be used as a client node which calculates the parity and might need some computational resources for this.