This repository has been archived by the owner on Feb 8, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 80
Hare provisioning for 3 node deployment
Shreya-18 edited this page Aug 16, 2021
·
14 revisions
Provisioning for 3 node cluster is similar to that of single node deployment mentioned here.
Below are the details of additional things needs to be done for provisioning of 3 node cluster
Pre-requisites are similar to that of single node deployment. Refer this
NOTE: These steps need to run on all 3 Nodes.
In addition to steps mentioned in above link we need to have password less configuration between all three nodes.
- cd /root
- ssh-keygen -o -t rsa -b 4096
- press enter 3 times and key will be generated in /root/.ssh folder
- then cat ~/.ssh/id_rsa.pub
- copy data from files /.ssh/id_rsa.pub of other two nodes to every node's ~/.ssh/authorized_keys
- After above step add below line in /etc/ssh/ssh_config
StrictHostKeyChecking no
- once this is successful then you should be able to access all nodes from each other using command ssh root@ without any password
Most of the functionality remains the same as single node provisioning.
Init and Test sections run hctl bootstrap --mkfs command to format motr data storage devices. Thus it is important that they are executed only on one of the nodes in the cluster. Hare takes care of the same implicitly. Init and test sections automatically execute on only the first node of the cluster and is a no-operation method on other nodes. This is presently done by checking if the node being provisioned is a first node in the cluster or not.
- Every phase accepts a separate conf-store configuration.
- Every phase has its own conf-store template that lists the conf-store keys required only in a given section.
- Available sample files, e.g. config sample file, can be used as a reference.
- User needs to make sure that total number of data devices in a pool (storage set) are >=
cluster>CLUSTER_ID>storage_set[0]>durability>sns>data+cluster>CLUSTER_ID>storage_set[0]>durability>sns>parity+cluster>CLUSTER_ID>storage_set[0]>durability>sns>spare.
Respective conf-store templates can be populated with the appropriate values and used to execute the corresponding phases as follows,
[root@ssc-vm-1623:root] /opt/seagate/cortx/hare/bin/hare_setup post_install --config 'json:///root/hare.post_install.conf.tmpl.3-node'
[root@ssc-vm-1623:root]
[root@ssc-vm-1623:root] /opt/seagate/cortx/hare/bin/hare_setup prepare --config 'json:///root/hare.prepare.conf.tmpl.3-node'
[root@ssc-vm-1623:root]
[root@ssc-vm-1623:root] /opt/seagate/cortx/hare/bin/hare_setup upgrade --config 'json:///root/hare.rpm'
[root@ssc-vm-1623:root]
Prerequisites:
- cortx-py-utils package must be installed.
- Motr must be installed.
- S3server must be installed.
- Data interfaces must be registered with lnet. Refer this
- lnet kernel module inserted and lnet service must be running.
- Add/Set srvnode-1.data.private as Private IP of VM-1 in /etc/hosts file of VM-1
- Add/Set srvnode-2.data.private as Private IP of VM-2 in /etc/hosts file of VM-2
- Add/Set srvnode-3.data.private as Private IP of VM-3 in /etc/hosts file of VM-3
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.57.236 srvnode-1.data.private
- Motr data and metadata volumes must be configured. Refer this
- Follow the mini-provisioning post-install, prepare and config steps of cortx-motr. This generates the shared motr-hare confstore file at path
/opt/seagate/cortx/motr/conf/motr_hare_keys.json. Follow this on all the 3 nodes. - Copy the template file hare.config.conf.tmpl.3-node in /root/. and populate following keys appropriately for nodes in a storage set.
- Copy this populated hare.config.conf.tmpl.3-node file on all 3 nodes in the cluster in root directory.
TMPL_CLUSTER_ID
TMPL_STORAGESET_COUNT
TMPL_DATA_UNITS_COUNT (4)
TMPL_PARITY_UNITS_COUNT (2)
TMPL_SPARE_UNITS_COUNT (2)
TMPL_STORAGESET_NAME
TMPL_MACHINE_ID_1 (/etc/machine-id)
TMPL_MACHINE_ID_2 (/etc/machine-id)
TMPL_MACHINE_ID_3 (/etc/machine-id)
TMPL_HOSTNAME
TMPL_SERVER_NODE_NAME
TMPL_DATA_INTERFACE_TYPE (tcp)
TMPL_DATA_PRIVATE_FQDN_1 (srvnode-1.data.private)
TMPL_DATA_PRIVATE_FQDN_2 (srvnode-2.data.private)
TMPL_DATA_PRIVATE_FQDN_3 (srvnode-3.data.private)
TMPL_PRIVATE_DATA_INTERFACE_1 (eth0)
TMPL_PRIVATE_DATA_INTERFACE_2 (optional)
TMPL_S3SERVER_INSTANCES_COUNT
Note: Total number of data devices in a storage set must be >= TMPL_DATA_UNITS_COUNT + TMPL_PARITY_UNITS_COUNT + TMPL_SPARE_UNITS_COUNT.
Also number of IO services in CDF will be equal to number of CVG in confstore
TMPL_DATA_DEVICE_11(In the format TMPL_DATA_DEVICE_<IO service number><device number>)
TMPL_DATA_DEVICE_12
TMPL_METADATA_DEVICE_1(In the format TMPL_METADATA_DEVICE_<IO service number>)
{
"cluster": {
"my-cluster": {
"site": {
"storage_set_count": "1"
},
"storage_set": [
{
"durability": {
"sns": {
"data": "1",
"parity": "0",
"spare": "0"
}
},
"name": "storage1",
"server_nodes": [
"7650c0e8d27ffb4f44b8ac3c26f29a45",
"7650c0e8d27ffb4f44b8ac3c26f29a46",
"7650c0e8d27ffb4f44b8ac3c26f29a47"
]
}
]
}
},
"server_node": {
"7650c0e8d27ffb4f44b8ac3c26f29a45": {
"cluster_id": "my-cluster",
"hostname": "ssc-vm-3351.colo.seagate.com",
"name": "srvnode-1",
"network": {
"data": {
"interface_type": "tcp",
"private_fqdn": "srvnode-1.data.private",
"private_interfaces": [
"eth1",
"eno2"
]
}
},
"s3_instances": "1",
"storage": {
"cvg": [
{
"data_devices": [
"/dev/loop0"
],
"metadata_devices": [
"/dev/loop1",
"/dev/loop2",
"/dev/loop3"
]
},
{
"data_devices": [
"/dev/loop4"
],
"metadata_devices": [
"/dev/loop6",
"/dev/loop7",
"/dev/loop8"
]
}
]
}
},
"7650c0e8d27ffb4f44b8ac3c26f29a46": {
"cluster_id": "my-cluster",
"hostname": "ssc-vm-3352.colo.seagate.com",
"name": "srvnode-2",
"network": {
"data": {
"interface_type": "tcp",
"private_fqdn": "srvnode-2.data.private",
"private_interfaces": [
"eth1",
"eno2"
]
}
},
"s3_instances": "1",
"storage": {
"cvg": [
{
"data_devices": [
"/dev/loop0"
],
"metadata_devices": [
"/dev/loop1",
"/dev/loop2",
"/dev/loop3"
]
},
{
"data_devices": [
"/dev/loop4"
],
"metadata_devices": [
"/dev/loop6",
"/dev/loop7",
"/dev/loop8"
]
}
]
}
},
"7650c0e8d27ffb4f44b8ac3c26f29a47": {
"cluster_id": "my-cluster",
"hostname": "ssc-vm-3353.colo.seagate.com",
"name": "srvnode-3",
"network": {
"data": {
"interface_type": "tcp",
"private_fqdn": "srvnode-3.data.private",
"private_interfaces": [
"eth1",
"eno2"
]
}
},
"s3_instances": "1",
"storage": {
"cvg": [
{
"data_devices": [
"/dev/loop0"
],
"metadata_devices": [
"/dev/loop1",
"/dev/loop2",
"/dev/loop3"
]
},
{
"data_devices": [
"/dev/loop4"
],
"metadata_devices": [
"/dev/loop6",
"/dev/loop7",
"/dev/loop8"
]
}
]
}
}
}
}
[root@ssc-vm-1623:root] /opt/seagate/cortx/hare/bin/hare_setup config --config 'json:///root/hare.config.conf.tmpl.3-node.sample' --file /var/lib/hare/cluster.yaml
[root@ssc-vm-1623:root]
Note that, the CDF is generated as /var/lib/hare/cluster.yaml in this case.
[root@ssc-vm-1623:root] cat /var/lib/hare/cluster.yaml
pools:
- allowed_failures: null
name: storage1
data_units: 4
parity_units: 2
disk_refs:
- path: /dev/sdb
node: srvnode-1.data.private
- path: /dev/sdc
node: srvnode-1.data.private
- path: /dev/sdd
node: srvnode-1.data.private
- path: /dev/sde
node: srvnode-1.data.private
- path: /dev/sdf
node: srvnode-1.data.private
- path: /dev/sdg
node: srvnode-1.data.private
- path: /dev/sdh
node: srvnode-1.data.private
- path: /dev/sdi
node: srvnode-1.data.private
- path: /dev/sdb
node: srvnode-2.data.private
- path: /dev/sdc
node: srvnode-2.data.private
- path: /dev/sdd
node: srvnode-2.data.private
- path: /dev/sde
node: srvnode-2.data.private
- path: /dev/sdf
node: srvnode-2.data.private
- path: /dev/sdg
node: srvnode-2.data.private
- path: /dev/sdh
node: srvnode-2.data.private
- path: /dev/sdi
node: srvnode-2.data.private
- path: /dev/sdb
node: srvnode-3.data.private
- path: /dev/sdc
node: srvnode-3.data.private
- path: /dev/sdd
node: srvnode-3.data.private
- path: /dev/sde
node: srvnode-3.data.private
- path: /dev/sdf
node: srvnode-3.data.private
- path: /dev/sdg
node: srvnode-3.data.private
- path: /dev/sdh
node: srvnode-3.data.private
- path: /dev/sdi
node: srvnode-3.data.private
type: sns
profiles:
- pools:
- storage1
name: Profile_the_pool
nodes:
- m0_servers:
- io_disks:
data: []
meta_data: /dev/vg_srvnode-1_md1/lv_raw_md1
runs_confd: true
- io_disks:
data:
- /dev/sdb
- /dev/sdc
- /dev/sdd
- /dev/sde
- /dev/sdf
- /dev/sdg
- /dev/sdh
- /dev/sdi
meta_data: /dev/vg_srvnode-1_md2/lv_raw_md2
runs_confd: null
data_iface: eth0
hostname: srvnode-1.data.private
m0_clients:
other: 3
s3: 1
data_iface_type: tcp
- m0_servers:
- io_disks:
data: []
meta_data: /dev/vg_srvnode-2_md1/lv_raw_md1
runs_confd: true
- io_disks:
data:
- /dev/sdb
- /dev/sdc
- /dev/sdd
- /dev/sde
- /dev/sdf
- /dev/sdg
- /dev/sdh
- /dev/sdi
meta_data: /dev/vg_srvnode-2_md2/lv_raw_md2
runs_confd: null
data_iface: eth0
hostname: srvnode-2.data.private
m0_clients:
other: 3
s3: 1
data_iface_type: tcp
- m0_servers:
- io_disks:
data: []
meta_data: /dev/vg_srvnode-3_md1/lv_raw_md1
runs_confd: true
- io_disks:
data:
- /dev/sdb
- /dev/sdc
- /dev/sdd
- /dev/sde
- /dev/sdf
- /dev/sdg
- /dev/sdh
- /dev/sdi
meta_data: /dev/vg_srvnode-3_md2/lv_raw_md2
runs_confd: null
data_iface: eth0
hostname: srvnode-3.data.private
m0_clients:
other: 3
s3: 1
data_iface_type: tcp
Prerequisites:
- cortx-py-utils package must be installed.
- Consul must be installed.
- Motr must be installed.
- S3server must be installed.
- Data interfaces must be registered with lnet. Refer this
- lnet kernel module inserted and lnet service must be running.
- Motr data and metadata volumes must be configured. Refer this
- Copy the template file hare.init.conf.tmpl.3-node in /root/. and populate following keys appropriately for nodes in a storage set.
- Copy this populated hare.init.conf.tmpl.3-node file on all 3 nodes in the cluster in root directory.
TMPL_CLUSTER_ID
TMPL_STORAGESET_COUNT
TMPL_DATA_UNITS_COUNT (4)
TMPL_PARITY_UNITS_COUNT (2)
TMPL_SPARE_UNITS_COUNT (2)
TMPL_STORAGESET_NAME
TMPL_MACHINE_ID_1 (/etc/machine-id)
TMPL_MACHINE_ID_2 (/etc/machine-id)
TMPL_MACHINE_ID_3 (/etc/machine-id)
TMPL_HOSTNAME
TMPL_SERVER_NODE_NAME
TMPL_DATA_INTERFACE_TYPE (tcp)
TMPL_PRIVATE_DATA_INTERFACE_1 (eth0)
TMPL_PRIVATE_DATA_INTERFACE_2 (optional)
TMPL_S3SERVER_INSTANCES_COUNT
Note: Total number of data devices in a storage set must be >= TMPL_DATA_UNITS_COUNT + TMPL_PARITY_UNITS_COUNT + TMPL_SPARE_UNITS_COUNT.
Also number of IO services in CDF will be equal to number of CVG in confstore
TMPL_DATA_DEVICE_11(In the format TMPL_DATA_DEVICE_<IO service number><device number>)
TMPL_DATA_DEVICE_12
TMPL_METADATA_DEVICE_1(In the format TMPL_METADATA_DEVICE_<IO service number>)
[root@ssc-vm-1623:root] /opt/seagate/cortx/hare/bin/hare_setup init --config 'json:///root/hare.init.conf.tmpl.3-node.sample' --file /var/lib/hare/cluster.yaml
Cluster is not running
2021-04-01 15:05:09: Generating cluster configuration... OK
2021-04-01 15:05:12: Starting Consul server on this node.................................. OK
2021-04-01 15:05:46: Importing configuration into the KV store... OK
2021-04-01 15:05:46: Starting Consul on other nodes.... OK
2021-04-01 15:05:47: Updating Consul configuraton from the KV store... OK
2021-04-01 15:05:49: Waiting for the RC Leader to get elected.... OK
2021-04-01 15:05:50: Starting Motr (phase1, mkfs)... OK
2021-04-01 15:05:58: Starting Motr (phase1, m0d)... OK
2021-04-01 15:06:10: Starting Motr (phase2, mkfs)... OK
2021-04-01 15:06:16: Starting Motr (phase2, m0d)... OK
2021-04-01 15:06:31: Starting S3 servers (phase3)... OK
2021-04-01 15:06:32: Checking health of services... OK
Stopping s3server@0x7200000000000001:0x25 at ssc-vm-1623.colo.seagate.com...
Stopping s3server@0x7200000000000001:0x53 at ssc-vm-1624.colo.seagate.com...
Stopping s3server@0x7200000000000001:0x81 at ssc-vm-1625.colo.seagate.com...
Stopped s3server@0x7200000000000001:0x25 at ssc-vm-1623.colo.seagate.com
Stopped s3server@0x7200000000000001:0x53 at ssc-vm-1624.colo.seagate.com
Stopped s3server@0x7200000000000001:0x81 at ssc-vm-1625.colo.seagate.com
Stopping m0d@0x7200000000000001:0xc (ios) at ssc-vm-1623.colo.seagate.com...
Stopping m0d@0x7200000000000001:0x3a (ios) at ssc-vm-1624.colo.seagate.com...
Stopping m0d@0x7200000000000001:0x68 (ios) at ssc-vm-1625.colo.seagate.com...
Stopped m0d@0x7200000000000001:0xc (ios) at ssc-vm-1623.colo.seagate.com
Stopped m0d@0x7200000000000001:0x3a (ios) at ssc-vm-1624.colo.seagate.com
Stopped m0d@0x7200000000000001:0x68 (ios) at ssc-vm-1625.colo.seagate.com
Stopping m0d@0x7200000000000001:0x9 (confd) at ssc-vm-1623.colo.seagate.com...
Stopping m0d@0x7200000000000001:0x37 (confd) at ssc-vm-1624.colo.seagate.com...
Stopping m0d@0x7200000000000001:0x65 (confd) at ssc-vm-1625.colo.seagate.com...
Stopped m0d@0x7200000000000001:0x9 (confd) at ssc-vm-1623.colo.seagate.com
Stopped m0d@0x7200000000000001:0x65 (confd) at ssc-vm-1625.colo.seagate.com
Stopped m0d@0x7200000000000001:0x37 (confd) at ssc-vm-1624.colo.seagate.com
Stopping hare-hax at ssc-vm-1623.colo.seagate.com...
Stopping hare-hax at ssc-vm-1624.colo.seagate.com...
Stopping hare-hax at ssc-vm-1625.colo.seagate.com...
Stopped hare-hax at ssc-vm-1624.colo.seagate.com
Stopped hare-hax at ssc-vm-1623.colo.seagate.com
Stopped hare-hax at ssc-vm-1625.colo.seagate.com
Stopping hare-consul-agent at ssc-vm-1624.colo.seagate.com...
Stopping hare-consul-agent at ssc-vm-1625.colo.seagate.com...
Stopped hare-consul-agent at ssc-vm-1625.colo.seagate.com
Stopped hare-consul-agent at ssc-vm-1624.colo.seagate.com
Stopping hare-consul-agent at ssc-vm-1623.colo.seagate.com...
Stopped hare-consul-agent at ssc-vm-1623.colo.seagate.com
Shutting down RC Leader at ssc-vm-1623.colo.seagate.com...
Cluster is not running
[root@ssc-vm-1624 ~]# /opt/seagate/cortx/hare/bin/hare_setup init --config 'json:///root/hare.init.conf.tmpl.3-node' --file /var/lib/hare/cluster.yaml
[root@ssc-vm-1624 ~]#
Prerequisites:
- Cluster must be running for executing tests since 'test' will be used in IVT after cluster deployment is completed. Check output of 'hctl status' to confirm that cluster is running.
- Copy the template file hare.test.conf.tmpl.3-node in /root/. and populate following keys appropriately for nodes in a storage set.
- Copy this populated hare.test.conf.tmpl.3-node file on all 3 nodes in the cluster in root directory.
TMPL_CLUSTER_ID
TMPL_STORAGESET_COUNT
TMPL_DATA_UNITS_COUNT (4)
TMPL_PARITY_UNITS_COUNT (2)
TMPL_SPARE_UNITS_COUNT (2)
TMPL_STORAGESET_NAME
TMPL_MACHINE_ID_1 (/etc/machine-id)
TMPL_MACHINE_ID_2 (/etc/machine-id)
TMPL_MACHINE_ID_3 (/etc/machine-id)
TMPL_HOSTNAME
TMPL_SERVER_NODE_NAME
TMPL_DATA_INTERFACE_TYPE (tcp)
TMPL_PRIVATE_DATA_INTERFACE_1 (eth0)
TMPL_PRIVATE_DATA_INTERFACE_2 (optional)
TMPL_S3SERVER_INSTANCES_COUNT
Note: Total number of data devices in a storage set must be >= TMPL_DATA_UNITS_COUNT + TMPL_PARITY_UNITS_COUNT + TMPL_SPARE_UNITS_COUNT.
Also number of IO services in CDF will be equal to number of CVG in confstore
TMPL_DATA_DEVICE_11(In the format TMPL_DATA_DEVICE_<IO service number><device number>)
TMPL_DATA_DEVICE_12
TMPL_METADATA_DEVICE_1(In the format TMPL_METADATA_DEVICE_<IO service number>)
[root@ssc-vm-1623:root] /opt/seagate/cortx/hare/bin/hare_setup test --config 'json:///root/hare.test.conf.tmpl.3-node.sample' --file /var/lib/hare/cluster.yaml --plan sanity
2021-06-30 11:08:48,359 [INFO] Running test plan: sanity
2021-06-30 11:08:49,111 [INFO] Tests executed successfully
Without bundle-id and destination directory path options, hare_setup support_bundle command generates Hare support bundle at /tmp/hare.
[root@ssc-vm-1623:root] /opt/seagate/cortx/hare/bin/hare_setup support_bundle
[root@ssc-vm-1623:root] ls /tmp/hare
hare_ssc-vm-1623.tar.gz
With bundle-id and destination directory arguments to hare_setup support_bundle.
[root@ssc-vm-1623:root] /opt/seagate/cortx/hare/bin/hare_setup support_bundle SB12345 /root
[root@ssc-vm-1623:root] ls hare
hare_SB12345.tar.gz
Runs hctl reportbug command to generate hare support bundle.
Note that hare_setup support_bundle command generates support bundle for the local node only. Thus, for multi-node, the command needs to be executed on every node individually.