Skip to content
This repository has been archived by the owner on May 3, 2024. It is now read-only.

Handling 'short' write in some cases, e.g. disk full. #1888

Merged
merged 1 commit into from
Aug 17, 2022

Conversation

huanghua78
Copy link

@huanghua78 huanghua78 commented Jun 16, 2022

Handling 'short write' in some cases, e.g. disk full.

In cases, e.g. disk full, the write request may return
with 'short write': the written count is less than requested count.
The assertion of requested write count equal to returned count is removed.
In this case, the rc value of this request is still 0. And next write
request may return -ENOSPC because of disk full.
This can be triggered with small disks and writing a lot of data into
the system.

Signed-off-by: Hua Huang <hua.huang@seagate.com>

Problem Statement

Sometimes, the write request may return short write result, especially
in case of disk full. Please see: #1868

Design

We don't require the I/O request to write/read all data in the request.

Coding

Checklist for Author

  • Coding conventions are followed and code is consistent

Testing

Checklist for Author

  • Unit and System Tests are added
  • Test Cases cover Happy Path, Non-Happy Path and Scalability
  • Testing was performed with RPM

Impact Analysis

Checklist for Author/Reviewer/GateKeeper

  • Interface change (if any) are documented
  • Side effects on other features (deployment/upgrade)
  • Dependencies on other component(s)

Review Checklist

Checklist for Author

  • JIRA number/GitHub Issue added to PR
  • PR is self reviewed
  • Jira and state/status is updated and JIRA is updated with PR link
  • Check if the description is clear and explained

Documentation

Checklist for Author

  • Changes done to WIKI / Confluence page / Quick Start Guide

@cortx-admin
Copy link

Jenkins CI Result : Motr#1342

Motr Test Summary

Test ResultCountInfo
❌Failed2
📁

03motr-single-node/71spiel-sns-motr-repair
01motr-single-node/00userspace-tests

🏁Skipped32
📁

01motr-single-node/28sys-kvs
01motr-single-node/35m0singlenode
01motr-single-node/04initscripts
01motr-single-node/37protocol
02motr-single-node/51kem
02motr-single-node/20rpc-session-cancel
02motr-single-node/10pver-assign
02motr-single-node/21fsync-single-node
02motr-single-node/13dgmode-io
02motr-single-node/14poolmach
02motr-single-node/11m0t1fs
02motr-single-node/26motr-user-kernel-tests
02motr-single-node/08spiel
03motr-single-node/06conf
03motr-single-node/36spare-reservation
04motr-single-node/34sns-repair-1n-1f
04motr-single-node/08spiel-sns-repair-quiesce
04motr-single-node/28sys-kvs-kernel
04motr-single-node/11m0t1fs-rconfc-fail
04motr-single-node/08spiel-sns-repair
04motr-single-node/19sns-repair-abort
04motr-single-node/22sns-repair-ios-fail
05motr-single-node/18sns-repair-quiesce
05motr-single-node/12fwait
05motr-single-node/16sns-repair-multi
05motr-single-node/07mount-fail
05motr-single-node/15sns-repair-single
05motr-single-node/23sns-abort-quiesce
05motr-single-node/17sns-repair-concurrent-io
05motr-single-node/07mount
05motr-single-node/07mount-multiple
05motr-single-node/12fsync

✔️Passed42
📁

01motr-single-node/43m0crate
01motr-single-node/05confgen
01motr-single-node/06hagen
01motr-single-node/52motr-singlenode-sanity
01motr-single-node/01net
01motr-single-node/01kernel-tests
01motr-single-node/03console
01motr-single-node/02rpcping
02motr-single-node/07m0d-fatal
02motr-single-node/67fdmi-plugin-multi-filters
02motr-single-node/53clusterusage-alert
02motr-single-node/41motr-conf-update
03motr-single-node/61sns-repair-motr-1n-1f
03motr-single-node/72spiel-sns-motr-repair-quiesce
03motr-single-node/08spiel-multi-confd
03motr-single-node/69sns-repair-motr-quiesce
03motr-single-node/62sns-repair-motr-mf
03motr-single-node/70sns-failure-after-repair-quiesce
03motr-single-node/63sns-repair-motr-1k-1f
03motr-single-node/60sns-repair-motr-1f
03motr-single-node/66sns-repair-motr-abort-quiesce
03motr-single-node/24motr-dix-repair-lookup-insert-spiel
03motr-single-node/68sns-repair-motr-shutdown
03motr-single-node/64sns-repair-motr-ios-fail
03motr-single-node/24motr-dix-repair-lookup-insert-m0repair
03motr-single-node/04sss
03motr-single-node/65sns-repair-motr-abort
04motr-single-node/73motr-io-small-disks
04motr-single-node/48motr-raid0-io
04motr-single-node/49motr-rpc-cancel
04motr-single-node/25m0kv
04motr-single-node/44motr-rm-lock-cc-io
04motr-single-node/45motr-rmw
05motr-single-node/23dix-repair-m0repair
05motr-single-node/43motr-sync-replication
05motr-single-node/42motr-utils
05motr-single-node/45motr-sns-repair-N-1
05motr-single-node/40motr-dgmode
05motr-single-node/23dix-repair-quiesce-m0repair
05motr-single-node/23spiel-dix-repair-quiesce
05motr-single-node/44motr-sns-repair
05motr-single-node/23spiel-dix-repair

Total76🔗

CppCheck Summary

   Cppcheck: No new warnings found 👍

@stale
Copy link

stale bot commented Jun 22, 2022

This issue/pull request has been marked as needs attention as it has been left pending without new activity for 4 days. Tagging @nkommuri @mehjoshi @huanghua78 for appropriate assignment. Sorry for the delay & Thank you for contributing to CORTX. We will get back to you as soon as possible.

@huanghua78
Copy link
Author

huanghua78 commented Jul 19, 2022

The Jenkins test result:
https://eos-jenkins.colo.seagate.com/job/Cortx-PR-Build/job/Motr/1499/

It is NOT automatically posted here.

@stale stale bot removed the needs-attention label Jul 19, 2022
motr/st/utils/copy.c Outdated Show resolved Hide resolved
@rkothiya
Copy link
Contributor

Jenkins CI Result : Motr#1522

Motr Test Summary

Test ResultCountInfo
❌Failed2
📁

04motr-single-node/49motr-rpc-cancel
01motr-single-node/00userspace-tests

🏁Skipped32
📁

01motr-single-node/28sys-kvs
01motr-single-node/35m0singlenode
01motr-single-node/04initscripts
01motr-single-node/37protocol
02motr-single-node/51kem
02motr-single-node/20rpc-session-cancel
02motr-single-node/10pver-assign
02motr-single-node/21fsync-single-node
02motr-single-node/13dgmode-io
02motr-single-node/14poolmach
02motr-single-node/11m0t1fs
02motr-single-node/26motr-user-kernel-tests
02motr-single-node/08spiel
03motr-single-node/06conf
03motr-single-node/36spare-reservation
04motr-single-node/34sns-repair-1n-1f
04motr-single-node/08spiel-sns-repair-quiesce
04motr-single-node/28sys-kvs-kernel
04motr-single-node/11m0t1fs-rconfc-fail
04motr-single-node/08spiel-sns-repair
04motr-single-node/19sns-repair-abort
04motr-single-node/22sns-repair-ios-fail
05motr-single-node/18sns-repair-quiesce
05motr-single-node/12fwait
05motr-single-node/16sns-repair-multi
05motr-single-node/07mount-fail
05motr-single-node/15sns-repair-single
05motr-single-node/23sns-abort-quiesce
05motr-single-node/17sns-repair-concurrent-io
05motr-single-node/07mount
05motr-single-node/07mount-multiple
05motr-single-node/12fsync

✔️Passed42
📁

01motr-single-node/43m0crate
01motr-single-node/05confgen
01motr-single-node/06hagen
01motr-single-node/52motr-singlenode-sanity
01motr-single-node/01net
01motr-single-node/01kernel-tests
01motr-single-node/03console
01motr-single-node/02rpcping
02motr-single-node/07m0d-fatal
02motr-single-node/67fdmi-plugin-multi-filters
02motr-single-node/53clusterusage-alert
02motr-single-node/41motr-conf-update
03motr-single-node/61sns-repair-motr-1n-1f
03motr-single-node/72spiel-sns-motr-repair-quiesce
03motr-single-node/08spiel-multi-confd
03motr-single-node/69sns-repair-motr-quiesce
03motr-single-node/62sns-repair-motr-mf
03motr-single-node/70sns-failure-after-repair-quiesce
03motr-single-node/63sns-repair-motr-1k-1f
03motr-single-node/60sns-repair-motr-1f
03motr-single-node/66sns-repair-motr-abort-quiesce
03motr-single-node/24motr-dix-repair-lookup-insert-spiel
03motr-single-node/68sns-repair-motr-shutdown
03motr-single-node/64sns-repair-motr-ios-fail
03motr-single-node/71spiel-sns-motr-repair
03motr-single-node/24motr-dix-repair-lookup-insert-m0repair
03motr-single-node/04sss
03motr-single-node/65sns-repair-motr-abort
04motr-single-node/73motr-io-small-disks
04motr-single-node/48motr-raid0-io
04motr-single-node/25m0kv
04motr-single-node/44motr-rm-lock-cc-io
04motr-single-node/45motr-rmw
05motr-single-node/23dix-repair-m0repair
05motr-single-node/43motr-sync-replication
05motr-single-node/42motr-utils
05motr-single-node/45motr-sns-repair-N-1
05motr-single-node/40motr-dgmode
05motr-single-node/23dix-repair-quiesce-m0repair
05motr-single-node/23spiel-dix-repair-quiesce
05motr-single-node/44motr-sns-repair
05motr-single-node/23spiel-dix-repair

Total76🔗

CppCheck Summary

   Cppcheck: No new warnings found 👍

@cortx-admin
Copy link

Can one of the admins verify this patch?

@hessio hessio added the Status: Waiting to be Reviewed PR is waiting for reviewers to review the PR label Aug 9, 2022
@huanghua78
Copy link
Author

retest this please

Copy link
Contributor

@yatin-mahajan yatin-mahajan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

In cases, e.g. disk full, the write request may return
with 'short write': the written count is less than requested count.
The assertion of requested write count equal to returned count is removed.
In this case, the rc value of this request is still 0. And next write
request may return -ENOSPC because of disk full.
This can be triggered with small disks and writing a lot of data into
the system.

Signed-off-by: Hua Huang <hua.huang@seagate.com>
@huanghua78
Copy link
Author

@rkothiya
Copy link
Contributor

Jenkins CI Result : Motr#1586

Motr Test Summary

Test ResultCountInfo
❌Failed3
📁

04motr-single-node/45motr-rmw
04motr-single-node/49motr-rpc-cancel
01motr-single-node/00userspace-tests

🏁Skipped32
📁

01motr-single-node/28sys-kvs
01motr-single-node/35m0singlenode
01motr-single-node/04initscripts
01motr-single-node/37protocol
02motr-single-node/51kem
02motr-single-node/20rpc-session-cancel
02motr-single-node/10pver-assign
02motr-single-node/21fsync-single-node
02motr-single-node/13dgmode-io
02motr-single-node/14poolmach
02motr-single-node/11m0t1fs
02motr-single-node/26motr-user-kernel-tests
02motr-single-node/08spiel
03motr-single-node/06conf
03motr-single-node/36spare-reservation
04motr-single-node/34sns-repair-1n-1f
04motr-single-node/08spiel-sns-repair-quiesce
04motr-single-node/28sys-kvs-kernel
04motr-single-node/11m0t1fs-rconfc-fail
04motr-single-node/08spiel-sns-repair
04motr-single-node/19sns-repair-abort
04motr-single-node/22sns-repair-ios-fail
05motr-single-node/18sns-repair-quiesce
05motr-single-node/12fwait
05motr-single-node/16sns-repair-multi
05motr-single-node/07mount-fail
05motr-single-node/15sns-repair-single
05motr-single-node/23sns-abort-quiesce
05motr-single-node/17sns-repair-concurrent-io
05motr-single-node/07mount
05motr-single-node/07mount-multiple
05motr-single-node/12fsync

✔️Passed41
📁

01motr-single-node/43m0crate
01motr-single-node/05confgen
01motr-single-node/06hagen
01motr-single-node/52motr-singlenode-sanity
01motr-single-node/01net
01motr-single-node/01kernel-tests
01motr-single-node/03console
01motr-single-node/02rpcping
02motr-single-node/07m0d-fatal
02motr-single-node/67fdmi-plugin-multi-filters
02motr-single-node/53clusterusage-alert
02motr-single-node/41motr-conf-update
03motr-single-node/61sns-repair-motr-1n-1f
03motr-single-node/72spiel-sns-motr-repair-quiesce
03motr-single-node/08spiel-multi-confd
03motr-single-node/69sns-repair-motr-quiesce
03motr-single-node/62sns-repair-motr-mf
03motr-single-node/70sns-failure-after-repair-quiesce
03motr-single-node/63sns-repair-motr-1k-1f
03motr-single-node/60sns-repair-motr-1f
03motr-single-node/66sns-repair-motr-abort-quiesce
03motr-single-node/24motr-dix-repair-lookup-insert-spiel
03motr-single-node/68sns-repair-motr-shutdown
03motr-single-node/64sns-repair-motr-ios-fail
03motr-single-node/71spiel-sns-motr-repair
03motr-single-node/24motr-dix-repair-lookup-insert-m0repair
03motr-single-node/04sss
03motr-single-node/65sns-repair-motr-abort
04motr-single-node/73motr-io-small-disks
04motr-single-node/48motr-raid0-io
04motr-single-node/25m0kv
04motr-single-node/44motr-rm-lock-cc-io
05motr-single-node/23dix-repair-m0repair
05motr-single-node/43motr-sync-replication
05motr-single-node/42motr-utils
05motr-single-node/45motr-sns-repair-N-1
05motr-single-node/40motr-dgmode
05motr-single-node/23dix-repair-quiesce-m0repair
05motr-single-node/23spiel-dix-repair-quiesce
05motr-single-node/44motr-sns-repair
05motr-single-node/23spiel-dix-repair

Total76🔗

CppCheck Summary

   Cppcheck: No new warnings found 👍

@mehjoshi
Copy link

mehjoshi commented Aug 16, 2022

@huanghua78 , In case of short writes, how will the user know how much data was written?

By returning the actually written bytes of data.

@huanghua78
Copy link
Author

I ran the 45motr-rmw manually, and it passed:

bash-4.4$ time sudo ./scripts/m0 run-st 45motr-rmw
[sudo] password for 520428:
----- run_st 45motr-rmw -----
<< 45motr-rmw >>
false
Motr RMW IO Test ...
Testing RMW in Erasure Coded layout
n k s p:3 2 2 20
vm.max_map_count = 30000000
motr_service_start: (N,K,S,P)=(3,2,2,20) nr_ios=4 multiple_pools=0
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0025595 s, 410 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00232151 s, 452 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00189034 s, 555 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00184147 s, 569 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00173459 s, 605 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00139585 s, 751 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00232784 s, 450 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00282888 s, 371 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00187356 s, 560 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00195552 s, 536 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0020965 s, 500 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00175601 s, 597 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00218554 s, 480 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00241801 s, 434 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00335099 s, 313 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00254078 s, 413 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00379544 s, 276 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0019621 s, 534 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00182263 s, 575 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00305449 s, 343 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00214201 s, 490 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00209937 s, 499 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00155432 s, 675 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00178998 s, 586 MB/s

[152:
  {0x74| ((^t|1:0), 1, (11, 22), ^o|2:9, ^v|1:20, 1,
          [1: "20 3 2 2"],
          [1: ^n|1:2],
          [1: ^S|1:6],
          [3: ^o|1:9, ^o|20:1, ^o|2:9],
          [1: ^p|1:0], [0])},
  {0x70| ((^p|1:0), [3: ^o|1:9, ^o|20:1, ^o|2:9])},
  {0x6e| ((^n|1:2), 16000, 2, 3, 2, [8: ^r|1:100, ^r|1:0, ^r|1:1, ^r|1:2, ^r|1:3, ^r|1:4, ^r|1:5, ^r|1:6])},

  {0x72| ((^r|1:100), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:1", [1: ^s|1:101])},
  {0x72| ((^r|1:0), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:900", [8: ^s|1:0, ^s|11:0, ^s|6:0, ^s|7:0,                     ^s|3:0, ^s|13:0, ^s|15:0, ^s|16:0])},
  {0x72| ((^r|1:1), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:901", [8: ^s|1:1, ^s|11:1, ^s|6:1, ^s|7:1,                     ^s|3:1, ^s|13:1, ^s|15:1, ^s|16:1])},
  {0x72| ((^r|1:2), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:902", [8: ^s|1:2, ^s|11:2, ^s|6:2, ^s|7:2,                     ^s|3:2, ^s|13:2, ^s|15:2, ^s|16:2])},
  {0x72| ((^r|1:3), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:903", [8: ^s|1:3, ^s|11:3, ^s|6:3, ^s|7:3,                     ^s|3:3, ^s|13:3, ^s|15:3, ^s|16:3])},
 {0x72| ((^r|1:4), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:800", [3: ^s|2:0, ^s|12:0, ^s|3:4])},
 {0x72| ((^r|1:5), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:34:1",
                          [3: ^s|1:6, ^s|1:7, ^s|3:6])},
 {0x72| ((^r|1:6), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:100",
                          [2: ^s|8:0, ^s|3:7])},
  {0x73| ((^s|8:0), @M0_CST_CONFD, [1: "10.230.240.203@tcp:12345:33:100"], [0], [0])},
  {0x73| ((^s|1:6), @M0_CST_HA, [1: "10.230.240.203@tcp:12345:34:1"], [0], [0])},
  {0x73| ((^s|1:7), @M0_CST_FIS, [1: "10.230.240.203@tcp:12345:34:1"], [0], [0])},
  {0x73| ((^s|1:101), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:1"], [0], [0])},




 {0x73| ((^s|2:0), @M0_CST_MDS, [1: "10.230.240.203@tcp:12345:33:800"], [0], [0])},
  {0x73| ((^s|12:0), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:800"], [0], [0])},
  {0x73| ((^s|3:4), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:800"], [0], [0])},

  {0x73| ((^s|1:0), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:900"], [0], [5: ^d|1:1, ^d|1:2, ^d|1:3, ^d|1:4, ^d|1:5])},
  {0x73| ((^s|11:0), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
 {0x73| ((^s|6:0), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
  {0x73| ((^s|7:0), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
 {0x73| ((^s|3:0), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
 {0x73| ((^s|13:0), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:900"], [0], [1: ^d|20:0])},
  {0x73| ((^s|15:0), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
  {0x73| ((^s|16:0), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
  {0x73| ((^s|1:1), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:901"], [0], [5: ^d|1:6, ^d|1:7, ^d|1:8, ^d|1:9, ^d|1:10])},
  {0x73| ((^s|11:1), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
 {0x73| ((^s|6:1), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
  {0x73| ((^s|7:1), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
 {0x73| ((^s|3:1), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
 {0x73| ((^s|13:1), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:901"], [0], [1: ^d|20:1])},
  {0x73| ((^s|15:1), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
  {0x73| ((^s|16:1), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
  {0x73| ((^s|1:2), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:902"], [0], [5: ^d|1:11, ^d|1:12, ^d|1:13, ^d|1:14, ^d|1:15])},
  {0x73| ((^s|11:2), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
 {0x73| ((^s|6:2), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
  {0x73| ((^s|7:2), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
 {0x73| ((^s|3:2), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
 {0x73| ((^s|13:2), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:902"], [0], [1: ^d|20:2])},
  {0x73| ((^s|15:2), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
  {0x73| ((^s|16:2), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
  {0x73| ((^s|1:3), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:903"], [0], [5: ^d|1:16, ^d|1:17, ^d|1:18, ^d|1:19, ^d|1:20])},
  {0x73| ((^s|11:3), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
 {0x73| ((^s|6:3), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
  {0x73| ((^s|7:3), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
 {0x73| ((^s|3:3), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
 {0x73| ((^s|13:3), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:903"], [0], [1: ^d|20:3])},
  {0x73| ((^s|15:3), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
  {0x73| ((^s|16:3), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},

 {0x73| ((^s|3:6), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:34:1"], [0], [0])},
 {0x73| ((^s|3:7), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:100"], [0], [0])},
  {0x64| ((^d|1:1), 0, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop1")},
 {0x6b| ((^k|1:1), ^d|1:1, [1: ^v|1:10])},
 {0x6a| ((^j|1:1), ^k|1:1, [0])},
 {0x64| ((^d|1:2), 1, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop2")},
 {0x6b| ((^k|1:2), ^d|1:2, [1: ^v|1:10])},
 {0x6a| ((^j|1:2), ^k|1:2, [0])},
 {0x64| ((^d|1:3), 2, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop3")},
 {0x6b| ((^k|1:3), ^d|1:3, [1: ^v|1:10])},
 {0x6a| ((^j|1:3), ^k|1:3, [0])},
 {0x64| ((^d|1:4), 3, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop4")},
 {0x6b| ((^k|1:4), ^d|1:4, [1: ^v|1:10])},
 {0x6a| ((^j|1:4), ^k|1:4, [0])},
 {0x64| ((^d|1:5), 4, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop5")},
 {0x6b| ((^k|1:5), ^d|1:5, [1: ^v|1:10])},
 {0x6a| ((^j|1:5), ^k|1:5, [0])},
 {0x64| ((^d|1:6), 5, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop6")},
 {0x6b| ((^k|1:6), ^d|1:6, [1: ^v|1:10])},
 {0x6a| ((^j|1:6), ^k|1:6, [0])},
 {0x64| ((^d|1:7), 6, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop7")},
 {0x6b| ((^k|1:7), ^d|1:7, [1: ^v|1:10])},
 {0x6a| ((^j|1:7), ^k|1:7, [0])},
 {0x64| ((^d|1:8), 7, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop8")},
 {0x6b| ((^k|1:8), ^d|1:8, [1: ^v|1:10])},
 {0x6a| ((^j|1:8), ^k|1:8, [0])},
 {0x64| ((^d|1:9), 8, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop9")},
 {0x6b| ((^k|1:9), ^d|1:9, [1: ^v|1:10])},
 {0x6a| ((^j|1:9), ^k|1:9, [0])},
 {0x64| ((^d|1:10), 9, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop10")},
 {0x6b| ((^k|1:10), ^d|1:10, [1: ^v|1:10])},
 {0x6a| ((^j|1:10), ^k|1:10, [0])},
 {0x64| ((^d|1:11), 10, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop11")},
 {0x6b| ((^k|1:11), ^d|1:11, [1: ^v|1:10])},
 {0x6a| ((^j|1:11), ^k|1:11, [0])},
 {0x64| ((^d|1:12), 11, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop12")},
 {0x6b| ((^k|1:12), ^d|1:12, [1: ^v|1:10])},
 {0x6a| ((^j|1:12), ^k|1:12, [0])},
 {0x64| ((^d|1:13), 12, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop13")},
 {0x6b| ((^k|1:13), ^d|1:13, [1: ^v|1:10])},
 {0x6a| ((^j|1:13), ^k|1:13, [0])},
 {0x64| ((^d|1:14), 13, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop14")},
 {0x6b| ((^k|1:14), ^d|1:14, [1: ^v|1:10])},
 {0x6a| ((^j|1:14), ^k|1:14, [0])},
 {0x64| ((^d|1:15), 14, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop15")},
 {0x6b| ((^k|1:15), ^d|1:15, [1: ^v|1:10])},
 {0x6a| ((^j|1:15), ^k|1:15, [0])},
 {0x64| ((^d|1:16), 15, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop16")},
 {0x6b| ((^k|1:16), ^d|1:16, [1: ^v|1:10])},
 {0x6a| ((^j|1:16), ^k|1:16, [0])},
 {0x64| ((^d|1:17), 16, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop17")},
 {0x6b| ((^k|1:17), ^d|1:17, [1: ^v|1:10])},
 {0x6a| ((^j|1:17), ^k|1:17, [0])},
 {0x64| ((^d|1:18), 17, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop18")},
 {0x6b| ((^k|1:18), ^d|1:18, [1: ^v|1:10])},
 {0x6a| ((^j|1:18), ^k|1:18, [0])},
 {0x64| ((^d|1:19), 18, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop19")},
 {0x6b| ((^k|1:19), ^d|1:19, [1: ^v|1:10])},
 {0x6a| ((^j|1:19), ^k|1:19, [0])},
 {0x64| ((^d|1:20), 19, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop20")},
 {0x6b| ((^k|1:20), ^d|1:20, [1: ^v|1:10])},
 {0x6a| ((^j|1:20), ^k|1:20, [0])},
  {0x53| ((^S|1:6), [1: ^a|1:6], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x61| ((^a|1:6), [1: ^e|1:7], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x65| ((^e|1:7), ^n|1:2, [1: ^c|1:8], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x63| ((^c|1:8), [24: ^k|1:1, ^k|1:2, ^k|1:3, ^k|1:4, ^k|1:5, ^k|1:6, ^k|1:7, ^k|1:8, ^k|1:9, ^k|1:10, ^k|1:11, ^k|1:12, ^k|1:13, ^k|1:14, ^k|1:15, ^k|1:16, ^k|1:17, ^k|1:18, ^k|1:19, ^k|1:20, ^k|20:0, ^k|20:1, ^k|20:2, ^k|20:3],
                            [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x6f| ((^o|1:9), 0, [3: ^v|1:10, ^v|0x40000000000001:11, ^v|0x40000000000001:12])},
  {0x76| ((^v|1:10), {0| (3, 2,
                                            2, 20,
                                            [5: 0, 0, 0, 0, 2],
                                            [1: ^j|1:21])})},
  {0x76| ((^v|0x40000000000001:11), {1| (0, ^v|1:10, [5: 0, 0, 0, 0, 1])})},
  {0x76| ((^v|0x40000000000001:12), {1| (1, ^v|1:10, [5: 0, 0, 0, 0, 2])})},
  {0x6a| ((^j|1:21), ^S|1:6, [1: ^j|1:22])},
  {0x6a| ((^j|1:22), ^a|1:6, [1: ^j|1:23])},
  {0x6a| ((^j|1:23), ^e|1:7, [1: ^j|1:24])},
  {0x6a| ((^j|1:24), ^c|1:8, [20: ^j|1:1, ^j|1:2, ^j|1:3, ^j|1:4, ^j|1:5, ^j|1:6, ^j|1:7, ^j|1:8, ^j|1:9, ^j|1:10, ^j|1:11, ^j|1:12, ^j|1:13, ^j|1:14, ^j|1:15, ^j|1:16, ^j|1:17, ^j|1:18, ^j|1:19, ^j|1:20])} ,
{0x6f| ((^o|2:9), 0, [1: ^v|2:10])},
{0x76| ((^v|2:10), {0| (4, 0, 0, 4, [5: 0, 0, 0, 0, 1], [1: ^j|2:21])})},
{0x6a| ((^j|2:21), ^S|1:6, [1: ^j|2:22])},
{0x6a| ((^j|2:22), ^a|1:6, [1: ^j|2:23])},
{0x6a| ((^j|2:23), ^e|1:7, [1: ^j|2:24])},
{0x6a| ((^j|2:24), ^c|1:8, [4: ^j|2:1, ^j|2:6, ^j|2:11, ^j|2:16])},
{0x6a| ((^j|2:1), ^k|1:1, [0])},
 {0x6a| ((^j|2:6), ^k|1:6, [0])},
 {0x6a| ((^j|2:11), ^k|1:11, [0])},
 {0x6a| ((^j|2:16), ^k|1:16, [0])}  ,
{0x64| ((^d|20:0), 20, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop25")},
{0x6b| ((^k|20:0), ^d|20:0, [1: ^v|1:20])},
{0x6a| ((^j|20:100), ^k|20:0, [0])},
{0x64| ((^d|20:1), 21, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop26")},
{0x6b| ((^k|20:1), ^d|20:1, [1: ^v|1:20])},
{0x6a| ((^j|20:101), ^k|20:1, [0])},
{0x64| ((^d|20:2), 22, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop27")},
{0x6b| ((^k|20:2), ^d|20:2, [1: ^v|1:20])},
{0x6a| ((^j|20:102), ^k|20:2, [0])},
{0x64| ((^d|20:3), 23, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop28")},
{0x6b| ((^k|20:3), ^d|20:3, [1: ^v|1:20])},
{0x6a| ((^j|20:103), ^k|20:3, [0])},
{0x6f| ((^o|20:1), 0, [1: ^v|1:20])},
{0x76| ((^v|1:20), {0| (1, 1, 1,
                                                    4,
                                                    [5: 0, 0, 0, 0, 1],
                                                    [1: ^j|20:1])})},
{0x6a| ((^j|20:1), ^S|1:6, [1: ^j|20:2])},
{0x6a| ((^j|20:2), ^a|1:6, [1: ^j|20:3])},
{0x6a| ((^j|20:3), ^e|1:7, [1: ^j|20:4])},
{0x6a| ((^j|20:4), ^c|1:8, [4:  ^j|20:100, ^j|20:101, ^j|20:102, ^j|20:103])}]
cd /var/motr/root/sandbox.st-45motr-rmw/confd && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -D db -S stobs -A linuxstob:addb-stobs                       -w 20 -m 65536      -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T linux -e libfab:10.230.240.203@tcp:12345:35:1                      -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/confd && exec /home/520428/work/cortx-motr/motr/m0d -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -f '<0x7200000000000001:6>' -T linux -e libfab:10.230.240.203@tcp:12345:33:100 -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0d.log
cd /var/motr/root/sandbox.st-45motr-rmw/ha && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T ad -e libfab:10.230.240.203@tcp:12345:35:1 -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
motr[1670204]:  1070  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
cd /var/motr/root/sandbox.st-45motr-rmw/mds1 && exec                    /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad                         -D db -S stobs -A linuxstob:addb-stobs                            -w 20 -m 65536                          -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:800          -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios1 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:900 -f '<0x7200000000000001:0>' -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios2 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:901 -f '<0x7200000000000001:1>' -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios3 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:902 -f '<0x7200000000000001:2>' -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios4 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:903 -f '<0x7200000000000001:3>' -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ha && exec /home/520428/work/cortx-motr/motr/m0d -D db -S stobs -A linuxstob:addb-stobs                              -w 20 -m 65536              -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T ad -e libfab:10.230.240.203@tcp:12345:34:1                 -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc -f '<0x7200000000000001:5>'  -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
motr[1670721]:  3ac0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
Motr HA agent started.
cd /var/motr/root/sandbox.st-45motr-rmw/mds1 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:800 -f '<0x7200000000000001:4>' -H 10.230.240.203@tcp:12345:34:1 -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0d.log
motr[1670795]:  c200  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680494a0 dest_ep=10.230.240.203@tcp:12345:33:900 ri_session=0xac0620 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680440f0, item 0x7f82680494a0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f8268047ad0 dest_ep=10.230.240.203@tcp:12345:33:900 ri_session=0xac7740 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680499d0, item 0x7f8268047ad0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680472c0 dest_ep=10.230.240.203@tcp:12345:33:900 ri_session=0xace860 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680d5f50, item 0x7f82680472c0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680483b0 dest_ep=10.230.240.203@tcp:12345:33:900 ri_session=0xad5980 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680e9f70, item 0x7f82680483b0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680462e0 dest_ep=10.230.240.203@tcp:12345:33:900 ri_session=0xaeace0 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680dd000, item 0x7f82680462e0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680d1a40 dest_ep=10.230.240.203@tcp:12345:33:900 ri_session=0xae3bc0 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f8268046810, item 0x7f82680d1a40[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680da910 dest_ep=10.230.240.203@tcp:12345:33:900 ri_session=0xadcaa0 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680fefa0, item 0x7f82680da910[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680d0950 dest_ep=10.230.240.203@tcp:12345:33:901 ri_session=0xb07df0 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680e7870, item 0x7f82680d0950[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680450e0 dest_ep=10.230.240.203@tcp:12345:33:901 ri_session=0xaf2a90 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680d9870, item 0x7f82680450e0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680d40c0 dest_ep=10.230.240.203@tcp:12345:33:901 ri_session=0xb00cd0 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680dfb40, item 0x7f82680d40c0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680da1d0 dest_ep=10.230.240.203@tcp:12345:33:901 ri_session=0xb0ef10 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680dd9c0, item 0x7f82680da1d0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680d2320 dest_ep=10.230.240.203@tcp:12345:33:901 ri_session=0xaf9bb0 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680e9580, item 0x7f82680d2320[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680dc8f0 dest_ep=10.230.240.203@tcp:12345:33:901 ri_session=0xb16030 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f8268101810, item 0x7f82680dc8f0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680dead0 dest_ep=10.230.240.203@tcp:12345:33:901 ri_session=0xb1d150 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680fe970, item 0x7f82680dead0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680d98f0 dest_ep=10.230.240.203@tcp:12345:33:902 ri_session=0xb24ed0 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680fdd80, item 0x7f82680d98f0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680458f0 dest_ep=10.230.240.203@tcp:12345:33:902 ri_session=0xb2bff0 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680e9260, item 0x7f82680458f0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680e78f0 dest_ep=10.230.240.203@tcp:12345:33:902 ri_session=0xb4f590 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680fdf50, item 0x7f82680e78f0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680f19e0 dest_ep=10.230.240.203@tcp:12345:33:903 ri_session=0xb5e430 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680dfe80, item 0x7f82680f19e0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f8268103910 dest_ep=10.230.240.203@tcp:12345:33:902 ri_session=0xb41350 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680f3e20, item 0x7f8268103910[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680d59e0 dest_ep=10.230.240.203@tcp:12345:33:902 ri_session=0xb3a230 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82681048e0, item 0x7f82680d59e0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680ea8f0 dest_ep=10.230.240.203@tcp:12345:33:902 ri_session=0xb33110 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f8268104ce0, item 0x7f82680ea8f0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680e60e0 dest_ep=10.230.240.203@tcp:12345:33:902 ri_session=0xb48470 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680e4880, item 0x7f82680e60e0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680e30e0 dest_ep=10.230.240.203@tcp:12345:33:903 ri_session=0xb65550 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680e4bd0, item 0x7f82680e30e0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82681019c0 dest_ep=10.230.240.203@tcp:12345:33:903 ri_session=0xb6c670 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680e4c90, item 0x7f82681019c0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680ec9e0 dest_ep=10.230.240.203@tcp:12345:33:903 ri_session=0xb73790 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680fd940, item 0x7f82680ec9e0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f8268100910 dest_ep=10.230.240.203@tcp:12345:33:903 ri_session=0xb819d0 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680d48d0, item 0x7f8268100910[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680edad0 dest_ep=10.230.240.203@tcp:12345:33:903 ri_session=0xb57310 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680ee050, item 0x7f82680edad0[32] failed with ri_error=-110
cd /var/motr/root/sandbox.st-45motr-rmw/ios1 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:900 -f '<0x7200000000000001:0>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios2 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:901 -f '<0x7200000000000001:1>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios3 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:902 -f '<0x7200000000000001:2>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios4 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:903 -f '<0x7200000000000001:3>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
Motr confd started.
Motr mdservices started.
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f8268047ad0 dest_ep=10.230.240.203@tcp:12345:33:903 ri_session=0xb7a8b0 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f8268045e90, item 0x7f8268047ad0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680472c0 dest_ep=10.230.240.203@tcp:12345:33:903 ri_session=0xb6c670 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680dd000, item 0x7f82680472c0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680d59e0 dest_ep=10.230.240.203@tcp:12345:33:903 ri_session=0xb73790 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680d6e10, item 0x7f82680d59e0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680da910 dest_ep=10.230.240.203@tcp:12345:33:903 ri_session=0xb65550 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680440f0, item 0x7f82680da910[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680494a0 dest_ep=10.230.240.203@tcp:12345:33:902 ri_session=0xb48470 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680d5f50, item 0x7f82680494a0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680483b0 dest_ep=10.230.240.203@tcp:12345:33:902 ri_session=0xb33110 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680e9f70, item 0x7f82680483b0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680e8e40 dest_ep=10.230.240.203@tcp:12345:33:902 ri_session=0xb3a230 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680499d0, item 0x7f82680e8e40[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680d0950 dest_ep=10.230.240.203@tcp:12345:33:902 ri_session=0xb4f590 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680fdec0, item 0x7f82680d0950[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680dc8f0 dest_ep=10.230.240.203@tcp:12345:33:902 ri_session=0xb41350 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680ddc00, item 0x7f82680dc8f0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680450e0 dest_ep=10.230.240.203@tcp:12345:33:902 ri_session=0xb2bff0 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680dfc30, item 0x7f82680450e0[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680ea830 dest_ep=10.230.240.203@tcp:12345:33:903 ri_session=0xb5e430 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680dfad0, item 0x7f82680ea830[32] failed with ri_error=-110
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f82680e7830 dest_ep=10.230.240.203@tcp:12345:33:901 ri_session=0xb1d150 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670795]:  97e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f82680e9c60, item 0x7f82680e7830[32] failed with ri_error=-110
motr[1670883]:   c70  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
motr[1670876]:  1670  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
motr[1670878]:  1d30  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
motr[1670867]:  d2c0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
Motr ioservices started.
motr service started
*** m0dixinit is omitted. Mkfs creates meta indices now.
Testing with layout ID 3 in Healthy mode
m0cp
motr[1670204]:  da30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1671169]:  4f60  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cp update
motr[1670204]:  da30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1671202]:  c3d0  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cat
pver is : <0:0>
motr[1670204]:  da30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1671230]:  2590  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0unlink
motr[1670204]:  da30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1671263]:  b430  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
diff
Testing with layout ID 5 in Healthy mode
m0cp
motr[1670204]:  da30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1671302]:  fd80  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cp update
motr[1670204]:  da30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1671330]:  c6b0  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cat
pver is : <0:0>
motr[1670204]:  da30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1671358]:  bc50  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0unlink
motr[1670204]:  da30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1671386]:  3e30  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
diff
Testing with layout ID 7 in Healthy mode
m0cp
motr[1670204]:  da30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1671420]:  6440  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cp update
motr[1670204]:  da30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1671449]:   8d0  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cat
pver is : <0:0>
motr[1670204]:  da30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1671477]:  ca20  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0unlink
motr[1670204]:  da30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1671505]:  bc80  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
diff
Testing with layout ID 9 in Healthy mode
m0cp
motr[1670204]:  da30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1671539]:  79d0  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cp update
motr[1670204]:  da30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1671567]:  4090  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cat
pver is : <0:0>
motr[1670204]:  da30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1671595]:  7320  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0unlink
motr[1670204]:  da30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1671623]:  6370  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
diff
Testing with layout ID 11 in Healthy mode
m0cp
motr[1670204]:  da30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1671660]:  a9d0  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cp update
motr[1670204]:  da30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1671689]:   890  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cat
pver is : <0:0>
motr[1670204]:  da30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1671717]:  ca90  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0unlink
motr[1670204]:  da30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1671745]:  2e90  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
diff
stopping /home/520428/work/cortx-motr/motr/m0d processes...
=== pids of services: 1670204 1670721 1670795 1670867 1670876 1670878 1670883 ===
Shutting down services one by one. mdservice is the last.
----- 1670204 stopping--------lt-m0d: got signal 1
motr[1670204]:  1770  FATAL  [lib/assert.c:50:m0_panic]  panic: (!m0_clink_is_armed(&revc->rcf_disc_wait)) at m0_rpc_service_reverse_session_put() (rpc/service.c:241)  [git: 2.0.0-895-5-gdc9f62213] /var/motr/root/sandbox.st-45motr-rmw/confd/m0trace.1670204.2022-08-16-16:55:02
Motr panic: (!m0_clink_is_armed(&revc->rcf_disc_wait)) at m0_rpc_service_reverse_session_put() rpc/service.c:241 (errno: 4) (last failed: none) [git: 2.0.0-895-5-gdc9f62213] pid: 1670204  /var/motr/root/sandbox.st-45motr-rmw/confd/m0trace.1670204.2022-08-16-16:55:02
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_arch_backtrace+0x20)[0x7ff715d3aa60]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_arch_panic+0xdf)[0x7ff715d3ac0f]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_panic+0x127)[0x7ff715d28bf7]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(+0x45fe1e)[0x7ff715dc7e1e]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_rpc_service_reverse_sessions_cleanup+0x9f)[0x7ff715dc7ebf]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(+0x45ffdc)[0x7ff715dc7fdc]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_reqh_service_prepare_to_stop+0xc6)[0x7ff715da38f6]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_reqh_services_prepare_to_stop+0x6e)[0x7ff715da212e]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_reqh_shutdown_wait+0x11)[0x7ff715da2221]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(+0x3e0f0e)[0x7ff715d48f0e]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_module_fini+0x70)[0x7ff715d7b7d0]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_cs_fini+0x2d)[0x7ff715d4cd3d]
/home/520428/work/cortx-motr/motr/.libs/lt-m0d[0x4011cc]
/lib64/libc.so.6(__libc_start_main+0xf3)[0x7ff713bfa4a3]
/home/520428/work/cortx-motr/motr/.libs/lt-m0d[0x40145e]
----- 1670204 stopped --------
----- 1670721 stopping--------lt-m0d: got signal 1
motr[1670721]:  2810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1670721]:  2810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1670721]:  2810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1670721]:  2810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1670721]:  2810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1670721]:  2810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1670721]:  2810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1670721]:  2810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1670721]:  2810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1670721]:  2810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1670721]:  2810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1670721]:  2810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1670721]:  57e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7fc49c0458f0 dest_ep=10.230.240.203@tcp:12345:34:101 ri_session=0x7fc4a0184e10 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670721]:  57e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7fc49c0ea980, item 0x7fc49c0458f0[38] failed with ri_error=-110
motr[1670721]:  2810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1670721]:  57e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7fc49c0484d0 dest_ep=10.230.240.203@tcp:12345:34:101 ri_session=0x7fc4a0200860 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670721]:  57e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7fc49c0cf620, item 0x7fc49c0484d0[38] failed with ri_error=-110
motr[1670721]:  57e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7fc49c0469e0 dest_ep=10.230.240.203@tcp:12345:34:101 ri_session=0x7fc4a01bdfa0 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670721]:  57e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7fc49c044af0, item 0x7fc49c0469e0[38] failed with ri_error=-110
motr[1670721]:  2810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1670721]:  57e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7fc49c0d5020 dest_ep=10.230.240.203@tcp:12345:34:101 ri_session=0x7fc4a0200be0 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670721]:  57e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7fc49c0ded00, item 0x7fc49c0d5020[38] failed with ri_error=-110
motr[1670721]:  2810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1670721]:  2810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1670721]:  57e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7fc49c048ab0 dest_ep=10.230.240.203@tcp:12345:34:101 ri_session=0x7fc4a02009b0 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670721]:  57e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7fc49c046870, item 0x7fc49c048ab0[38] failed with ri_error=-110
motr[1670721]:  57e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7fc49c0d58f0 dest_ep=10.230.240.203@tcp:12345:34:101 ri_session=0x7fc4a024ae50 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670721]:  57e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7fc49c0e9f20, item 0x7fc49c0d58f0[38] failed with ri_error=-110
motr[1670721]:  2810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1670721]:  57e0  ERROR  [rpc/frmops.c:482:item_done]  motr[1670721]:  2810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
 item: 0x7fc49c0eaa00 dest_ep=10.230.240.203@tcp:12345:34:101 ri_session=0x7fc4a026f2d0 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670721]:  57e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7fc49c0ea810, item 0x7fc49c0eaa00[38] failed with ri_error=-110
motr[1670721]:  2810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1670721]:  57e0  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7fc49c0e7a00 dest_ep=10.230.240.203@tcp:12345:34:101 ri_session=0x7fc4a01ebe00 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[1670721]:  57e0  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7fc49c0e6f10, item 0x7fc49c0e7a00[38] failed with ri_error=-110
motr[1670721]:  2810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
----- 1670721 stopped --------
----- 1670795 stopping--------lt-m0d: got signal 1
motr[1670795]:  6810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x973c20 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=98 delivered=98 next=98 assign=124)
motr[1670795]:  6810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=49 hl=0x973c20 ep=10.230.240.203@tcp:12345:34:1
motr[1670795]:  6a70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1670795 stopped --------
----- 1670867 stopping--------lt-m0d: got signal 1
motr[1670867]:  8810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0xcd0fb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=64 delivered=64 next=64 assign=104)
motr[1670867]:  8810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=33 hl=0xcd0fb0 ep=10.230.240.203@tcp:12345:34:1
motr[1670867]:  8a70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1670867 stopped --------
----- 1670876 stopping--------lt-m0d: got signal 1
motr[1670876]:  2810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x1e99fb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=64 delivered=64 next=64 assign=118)
motr[1670876]:  2810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=33 hl=0x1e99fb0 ep=10.230.240.203@tcp:12345:34:1
motr[1670876]:  2a70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1670876 stopped --------
----- 1670878 stopping--------lt-m0d: got signal 1
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:3> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:3> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:3> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:3> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:3> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:3> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:3> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:2> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:2> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:2> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:2> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:2> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:2> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:2> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:3> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:3> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:3> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:3> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:3> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:3> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:3> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:2> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:2> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:2> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:2> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:2> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:2> not found in cache
motr[1670878]:  4020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:2> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:1> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:1> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:1> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000c:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000002:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:1> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:1> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:1> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:1> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000003:7> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:1> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:1> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:1> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000c:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000002:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:1> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:1> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:1> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:1> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000003:7> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:1> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:1> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:1> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000c:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000002:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:1> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:1> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:1> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:1> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:0> not found in cache
motr[1670878]:  6a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000003:7> not found in cache
motr[1670878]:  f810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x7c2fb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=66 delivered=66 next=66 assign=100)
motr[1670878]:  f810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=36 hl=0x7c2fb0 ep=10.230.240.203@tcp:12345:34:1
motr[1670878]:  fa70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1670878 stopped --------
----- 1670883 stopping--------lt-m0d: got signal 1
motr[1670883]:  2810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x1cb5fb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=64 delivered=64 next=64 assign=146)
motr[1670883]:  2810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=34 hl=0x1cb5fb0 ep=10.230.240.203@tcp:12345:34:1
motr[1670883]:  2a70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1670883 stopped --------
Motr services stopped.
Testing RMW in Replicaetd layout
n k s p:3 2 2 20
vm.max_map_count = 30000000
motr_service_start: (N,K,S,P)=(1,2,2,20) nr_ios=4 multiple_pools=0
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0035032 s, 299 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0025083 s, 418 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00409266 s, 256 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00271786 s, 386 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.001708 s, 614 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00202877 s, 517 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00294998 s, 355 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00199765 s, 525 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00183775 s, 571 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00426872 s, 246 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00145513 s, 721 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0016589 s, 632 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00232651 s, 451 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00145229 s, 722 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00460235 s, 228 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00377964 s, 277 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0016807 s, 624 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00192063 s, 546 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00221281 s, 474 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.001341 s, 782 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00149171 s, 703 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00143471 s, 731 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00336186 s, 312 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00178894 s, 586 MB/s

[152:
  {0x74| ((^t|1:0), 1, (11, 22), ^o|2:9, ^v|1:20, 1,
          [1: "20 1 2 2"],
          [1: ^n|1:2],
          [1: ^S|1:6],
          [3: ^o|1:9, ^o|20:1, ^o|2:9],
          [1: ^p|1:0], [0])},
  {0x70| ((^p|1:0), [3: ^o|1:9, ^o|20:1, ^o|2:9])},
  {0x6e| ((^n|1:2), 16000, 2, 3, 2, [8: ^r|1:100, ^r|1:0, ^r|1:1, ^r|1:2, ^r|1:3, ^r|1:4, ^r|1:5, ^r|1:6])},

  {0x72| ((^r|1:100), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:1", [1: ^s|1:101])},
  {0x72| ((^r|1:0), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:900", [8: ^s|1:0, ^s|11:0, ^s|6:0, ^s|7:0,                     ^s|3:0, ^s|13:0, ^s|15:0, ^s|16:0])},
  {0x72| ((^r|1:1), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:901", [8: ^s|1:1, ^s|11:1, ^s|6:1, ^s|7:1,                     ^s|3:1, ^s|13:1, ^s|15:1, ^s|16:1])},
  {0x72| ((^r|1:2), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:902", [8: ^s|1:2, ^s|11:2, ^s|6:2, ^s|7:2,                     ^s|3:2, ^s|13:2, ^s|15:2, ^s|16:2])},
  {0x72| ((^r|1:3), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:903", [8: ^s|1:3, ^s|11:3, ^s|6:3, ^s|7:3,                     ^s|3:3, ^s|13:3, ^s|15:3, ^s|16:3])},
 {0x72| ((^r|1:4), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:800", [3: ^s|2:0, ^s|12:0, ^s|3:4])},
 {0x72| ((^r|1:5), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:34:1",
                          [3: ^s|1:6, ^s|1:7, ^s|3:6])},
 {0x72| ((^r|1:6), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:100",
                          [2: ^s|8:0, ^s|3:7])},
  {0x73| ((^s|8:0), @M0_CST_CONFD, [1: "10.230.240.203@tcp:12345:33:100"], [0], [0])},
  {0x73| ((^s|1:6), @M0_CST_HA, [1: "10.230.240.203@tcp:12345:34:1"], [0], [0])},
  {0x73| ((^s|1:7), @M0_CST_FIS, [1: "10.230.240.203@tcp:12345:34:1"], [0], [0])},
  {0x73| ((^s|1:101), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:1"], [0], [0])},




 {0x73| ((^s|2:0), @M0_CST_MDS, [1: "10.230.240.203@tcp:12345:33:800"], [0], [0])},
  {0x73| ((^s|12:0), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:800"], [0], [0])},
  {0x73| ((^s|3:4), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:800"], [0], [0])},

  {0x73| ((^s|1:0), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:900"], [0], [5: ^d|1:1, ^d|1:2, ^d|1:3, ^d|1:4, ^d|1:5])},
  {0x73| ((^s|11:0), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
 {0x73| ((^s|6:0), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
  {0x73| ((^s|7:0), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
 {0x73| ((^s|3:0), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
 {0x73| ((^s|13:0), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:900"], [0], [1: ^d|20:0])},
  {0x73| ((^s|15:0), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
  {0x73| ((^s|16:0), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
  {0x73| ((^s|1:1), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:901"], [0], [5: ^d|1:6, ^d|1:7, ^d|1:8, ^d|1:9, ^d|1:10])},
  {0x73| ((^s|11:1), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
 {0x73| ((^s|6:1), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
  {0x73| ((^s|7:1), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
 {0x73| ((^s|3:1), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
 {0x73| ((^s|13:1), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:901"], [0], [1: ^d|20:1])},
  {0x73| ((^s|15:1), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
  {0x73| ((^s|16:1), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
  {0x73| ((^s|1:2), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:902"], [0], [5: ^d|1:11, ^d|1:12, ^d|1:13, ^d|1:14, ^d|1:15])},
  {0x73| ((^s|11:2), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
 {0x73| ((^s|6:2), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
  {0x73| ((^s|7:2), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
 {0x73| ((^s|3:2), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
 {0x73| ((^s|13:2), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:902"], [0], [1: ^d|20:2])},
  {0x73| ((^s|15:2), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
  {0x73| ((^s|16:2), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
  {0x73| ((^s|1:3), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:903"], [0], [5: ^d|1:16, ^d|1:17, ^d|1:18, ^d|1:19, ^d|1:20])},
  {0x73| ((^s|11:3), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
 {0x73| ((^s|6:3), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
  {0x73| ((^s|7:3), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
 {0x73| ((^s|3:3), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
 {0x73| ((^s|13:3), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:903"], [0], [1: ^d|20:3])},
  {0x73| ((^s|15:3), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
  {0x73| ((^s|16:3), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},

 {0x73| ((^s|3:6), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:34:1"], [0], [0])},
 {0x73| ((^s|3:7), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:100"], [0], [0])},
  {0x64| ((^d|1:1), 0, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop1")},
 {0x6b| ((^k|1:1), ^d|1:1, [1: ^v|1:10])},
 {0x6a| ((^j|1:1), ^k|1:1, [0])},
 {0x64| ((^d|1:2), 1, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop2")},
 {0x6b| ((^k|1:2), ^d|1:2, [1: ^v|1:10])},
 {0x6a| ((^j|1:2), ^k|1:2, [0])},
 {0x64| ((^d|1:3), 2, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop3")},
 {0x6b| ((^k|1:3), ^d|1:3, [1: ^v|1:10])},
 {0x6a| ((^j|1:3), ^k|1:3, [0])},
 {0x64| ((^d|1:4), 3, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop4")},
 {0x6b| ((^k|1:4), ^d|1:4, [1: ^v|1:10])},
 {0x6a| ((^j|1:4), ^k|1:4, [0])},
 {0x64| ((^d|1:5), 4, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop5")},
 {0x6b| ((^k|1:5), ^d|1:5, [1: ^v|1:10])},
 {0x6a| ((^j|1:5), ^k|1:5, [0])},
 {0x64| ((^d|1:6), 5, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop6")},
 {0x6b| ((^k|1:6), ^d|1:6, [1: ^v|1:10])},
 {0x6a| ((^j|1:6), ^k|1:6, [0])},
 {0x64| ((^d|1:7), 6, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop7")},
 {0x6b| ((^k|1:7), ^d|1:7, [1: ^v|1:10])},
 {0x6a| ((^j|1:7), ^k|1:7, [0])},
 {0x64| ((^d|1:8), 7, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop8")},
 {0x6b| ((^k|1:8), ^d|1:8, [1: ^v|1:10])},
 {0x6a| ((^j|1:8), ^k|1:8, [0])},
 {0x64| ((^d|1:9), 8, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop9")},
 {0x6b| ((^k|1:9), ^d|1:9, [1: ^v|1:10])},
 {0x6a| ((^j|1:9), ^k|1:9, [0])},
 {0x64| ((^d|1:10), 9, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop10")},
 {0x6b| ((^k|1:10), ^d|1:10, [1: ^v|1:10])},
 {0x6a| ((^j|1:10), ^k|1:10, [0])},
 {0x64| ((^d|1:11), 10, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop11")},
 {0x6b| ((^k|1:11), ^d|1:11, [1: ^v|1:10])},
 {0x6a| ((^j|1:11), ^k|1:11, [0])},
 {0x64| ((^d|1:12), 11, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop12")},
 {0x6b| ((^k|1:12), ^d|1:12, [1: ^v|1:10])},
 {0x6a| ((^j|1:12), ^k|1:12, [0])},
 {0x64| ((^d|1:13), 12, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop13")},
 {0x6b| ((^k|1:13), ^d|1:13, [1: ^v|1:10])},
 {0x6a| ((^j|1:13), ^k|1:13, [0])},
 {0x64| ((^d|1:14), 13, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop14")},
 {0x6b| ((^k|1:14), ^d|1:14, [1: ^v|1:10])},
 {0x6a| ((^j|1:14), ^k|1:14, [0])},
 {0x64| ((^d|1:15), 14, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop15")},
 {0x6b| ((^k|1:15), ^d|1:15, [1: ^v|1:10])},
 {0x6a| ((^j|1:15), ^k|1:15, [0])},
 {0x64| ((^d|1:16), 15, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop16")},
 {0x6b| ((^k|1:16), ^d|1:16, [1: ^v|1:10])},
 {0x6a| ((^j|1:16), ^k|1:16, [0])},
 {0x64| ((^d|1:17), 16, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop17")},
 {0x6b| ((^k|1:17), ^d|1:17, [1: ^v|1:10])},
 {0x6a| ((^j|1:17), ^k|1:17, [0])},
 {0x64| ((^d|1:18), 17, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop18")},
 {0x6b| ((^k|1:18), ^d|1:18, [1: ^v|1:10])},
 {0x6a| ((^j|1:18), ^k|1:18, [0])},
 {0x64| ((^d|1:19), 18, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop19")},
 {0x6b| ((^k|1:19), ^d|1:19, [1: ^v|1:10])},
 {0x6a| ((^j|1:19), ^k|1:19, [0])},
 {0x64| ((^d|1:20), 19, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop20")},
 {0x6b| ((^k|1:20), ^d|1:20, [1: ^v|1:10])},
 {0x6a| ((^j|1:20), ^k|1:20, [0])},
  {0x53| ((^S|1:6), [1: ^a|1:6], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x61| ((^a|1:6), [1: ^e|1:7], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x65| ((^e|1:7), ^n|1:2, [1: ^c|1:8], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x63| ((^c|1:8), [24: ^k|1:1, ^k|1:2, ^k|1:3, ^k|1:4, ^k|1:5, ^k|1:6, ^k|1:7, ^k|1:8, ^k|1:9, ^k|1:10, ^k|1:11, ^k|1:12, ^k|1:13, ^k|1:14, ^k|1:15, ^k|1:16, ^k|1:17, ^k|1:18, ^k|1:19, ^k|1:20, ^k|20:0, ^k|20:1, ^k|20:2, ^k|20:3],
                            [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x6f| ((^o|1:9), 0, [3: ^v|1:10, ^v|0x40000000000001:11, ^v|0x40000000000001:12])},
  {0x76| ((^v|1:10), {0| (1, 2,
                                            2, 20,
                                            [5: 0, 0, 0, 0, 2],
                                            [1: ^j|1:21])})},
  {0x76| ((^v|0x40000000000001:11), {1| (0, ^v|1:10, [5: 0, 0, 0, 0, 1])})},
  {0x76| ((^v|0x40000000000001:12), {1| (1, ^v|1:10, [5: 0, 0, 0, 0, 2])})},
  {0x6a| ((^j|1:21), ^S|1:6, [1: ^j|1:22])},
  {0x6a| ((^j|1:22), ^a|1:6, [1: ^j|1:23])},
  {0x6a| ((^j|1:23), ^e|1:7, [1: ^j|1:24])},
  {0x6a| ((^j|1:24), ^c|1:8, [20: ^j|1:1, ^j|1:2, ^j|1:3, ^j|1:4, ^j|1:5, ^j|1:6, ^j|1:7, ^j|1:8, ^j|1:9, ^j|1:10, ^j|1:11, ^j|1:12, ^j|1:13, ^j|1:14, ^j|1:15, ^j|1:16, ^j|1:17, ^j|1:18, ^j|1:19, ^j|1:20])} ,
{0x6f| ((^o|2:9), 0, [1: ^v|2:10])},
{0x76| ((^v|2:10), {0| (4, 0, 0, 4, [5: 0, 0, 0, 0, 1], [1: ^j|2:21])})},
{0x6a| ((^j|2:21), ^S|1:6, [1: ^j|2:22])},
{0x6a| ((^j|2:22), ^a|1:6, [1: ^j|2:23])},
{0x6a| ((^j|2:23), ^e|1:7, [1: ^j|2:24])},
{0x6a| ((^j|2:24), ^c|1:8, [4: ^j|2:1, ^j|2:6, ^j|2:11, ^j|2:16])},
{0x6a| ((^j|2:1), ^k|1:1, [0])},
 {0x6a| ((^j|2:6), ^k|1:6, [0])},
 {0x6a| ((^j|2:11), ^k|1:11, [0])},
 {0x6a| ((^j|2:16), ^k|1:16, [0])}  ,
{0x64| ((^d|20:0), 20, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop25")},
{0x6b| ((^k|20:0), ^d|20:0, [1: ^v|1:20])},
{0x6a| ((^j|20:100), ^k|20:0, [0])},
{0x64| ((^d|20:1), 21, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop26")},
{0x6b| ((^k|20:1), ^d|20:1, [1: ^v|1:20])},
{0x6a| ((^j|20:101), ^k|20:1, [0])},
{0x64| ((^d|20:2), 22, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop27")},
{0x6b| ((^k|20:2), ^d|20:2, [1: ^v|1:20])},
{0x6a| ((^j|20:102), ^k|20:2, [0])},
{0x64| ((^d|20:3), 23, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop28")},
{0x6b| ((^k|20:3), ^d|20:3, [1: ^v|1:20])},
{0x6a| ((^j|20:103), ^k|20:3, [0])},
{0x6f| ((^o|20:1), 0, [1: ^v|1:20])},
{0x76| ((^v|1:20), {0| (1, 1, 1,
                                                    4,
                                                    [5: 0, 0, 0, 0, 1],
                                                    [1: ^j|20:1])})},
{0x6a| ((^j|20:1), ^S|1:6, [1: ^j|20:2])},
{0x6a| ((^j|20:2), ^a|1:6, [1: ^j|20:3])},
{0x6a| ((^j|20:3), ^e|1:7, [1: ^j|20:4])},
{0x6a| ((^j|20:4), ^c|1:8, [4:  ^j|20:100, ^j|20:101, ^j|20:102, ^j|20:103])}]
cd /var/motr/root/sandbox.st-45motr-rmw/confd && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -D db -S stobs -A linuxstob:addb-stobs                       -w 20 -m 65536      -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T linux -e libfab:10.230.240.203@tcp:12345:35:1                      -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/confd && exec /home/520428/work/cortx-motr/motr/m0d -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -f '<0x7200000000000001:6>' -T linux -e libfab:10.230.240.203@tcp:12345:33:100 -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0d.log
cd /var/motr/root/sandbox.st-45motr-rmw/ha && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T ad -e libfab:10.230.240.203@tcp:12345:35:1 -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
motr[1672099]:  ae60  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
cd /var/motr/root/sandbox.st-45motr-rmw/mds1 && exec                    /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad                         -D db -S stobs -A linuxstob:addb-stobs                            -w 20 -m 65536                          -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:800          -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios1 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:900 -f '<0x7200000000000001:0>' -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios2 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:901 -f '<0x7200000000000001:1>' -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios3 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:902 -f '<0x7200000000000001:2>' -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios4 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:903 -f '<0x7200000000000001:3>' -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ha && exec /home/520428/work/cortx-motr/motr/m0d -D db -S stobs -A linuxstob:addb-stobs                              -w 20 -m 65536              -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T ad -e libfab:10.230.240.203@tcp:12345:34:1                 -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc -f '<0x7200000000000001:5>'  -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
motr[1672610]:  a8c0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
Motr HA agent started.
cd /var/motr/root/sandbox.st-45motr-rmw/mds1 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:800 -f '<0x7200000000000001:4>' -H 10.230.240.203@tcp:12345:34:1 -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0d.log
motr[1672685]:  3430  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
cd /var/motr/root/sandbox.st-45motr-rmw/ios1 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:900 -f '<0x7200000000000001:0>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios2 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:901 -f '<0x7200000000000001:1>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios3 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:902 -f '<0x7200000000000001:2>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios4 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:903 -f '<0x7200000000000001:3>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
Motr confd started.
Motr mdservices started.
motr[1672757]:  1920  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
motr[1672777]:  4590  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
motr[1672765]:  8110  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
motr[1672766]:   150  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
Motr ioservices started.
motr service started
*** m0dixinit is omitted. Mkfs creates meta indices now.
Testing with layout ID 3 in Healthy mode
m0cp
motr[1672099]:  ca30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1673058]:  d240  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cp update
motr[1672099]:  ca30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1673086]:  da00  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cat
pver is : <0:0>
motr[1672099]:  ca30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1673114]:  8e40  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0unlink
motr[1672099]:  ca30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1673142]:  8f10  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
diff
Testing with layout ID 5 in Healthy mode
m0cp
motr[1672099]:  ca30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1673176]:  3af0  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cp update
motr[1672099]:  ca30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1673204]:  a280  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cat
pver is : <0:0>
motr[1672099]:  ca30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1673232]:  4180  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0unlink
motr[1672099]:  ca30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1673260]:  4000  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
diff
Testing with layout ID 7 in Healthy mode
m0cp
motr[1672099]:  ca30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1673294]:  b820  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cp update
motr[1672099]:  ca30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1673322]:  3df0  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cat
pver is : <0:0>
motr[1672099]:  ca30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1673352]:  3470  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0unlink
motr[1672099]:  ca30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1673380]:  df90  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
diff
Testing with layout ID 9 in Healthy mode
m0cp
motr[1672099]:  ca30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1673414]:  3fe0  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cp update
motr[1672099]:  ca30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1673442]:  6d00  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cat
pver is : <0:0>
motr[1672099]:  ca30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1673470]:  ecc0  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0unlink
motr[1672099]:  ca30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1673498]:  b300  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
diff
Testing with layout ID 11 in Healthy mode
m0cp
motr[1672099]:  ca30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1673532]:  c890  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cp update
motr[1672099]:  ca30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1673560]:  d7b0  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cat
pver is : <0:0>
motr[1672099]:  ca30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1673588]:  92b0  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0unlink
motr[1672099]:  ca30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1673616]:  1570  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
diff
stopping /home/520428/work/cortx-motr/motr/m0d processes...
=== pids of services: 1672099 1672610 1672685 1672757 1672765 1672766 1672777 ===
Shutting down services one by one. mdservice is the last.
----- 1672099 stopping--------lt-m0d: got signal 1
motr[1672099]:  b560  FATAL  [lib/assert.c:50:m0_panic]  panic: (!m0_clink_is_armed(&revc->rcf_disc_wait)) at m0_rpc_service_reverse_session_put() (rpc/service.c:241)  [git: 2.0.0-895-5-gdc9f62213] /var/motr/root/sandbox.st-45motr-rmw/confd/m0trace.1672099.2022-08-16-16:59:02
Motr panic: (!m0_clink_is_armed(&revc->rcf_disc_wait)) at m0_rpc_service_reverse_session_put() rpc/service.c:241 (errno: 4) (last failed: none) [git: 2.0.0-895-5-gdc9f62213] pid: 1672099  /var/motr/root/sandbox.st-45motr-rmw/confd/m0trace.1672099.2022-08-16-16:59:02
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_arch_backtrace+0x20)[0x7f5e0d2d8a60]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_arch_panic+0xdf)[0x7f5e0d2d8c0f]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_panic+0x127)[0x7f5e0d2c6bf7]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(+0x45fe1e)[0x7f5e0d365e1e]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_rpc_service_reverse_sessions_cleanup+0x9f)[0x7f5e0d365ebf]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(+0x45ffdc)[0x7f5e0d365fdc]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_reqh_service_prepare_to_stop+0xc6)[0x7f5e0d3418f6]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_reqh_services_prepare_to_stop+0x6e)[0x7f5e0d34012e]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_reqh_shutdown_wait+0x11)[0x7f5e0d340221]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(+0x3e0f0e)[0x7f5e0d2e6f0e]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_module_fini+0x70)[0x7f5e0d3197d0]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_cs_fini+0x2d)[0x7f5e0d2ead3d]
/home/520428/work/cortx-motr/motr/.libs/lt-m0d[0x4011cc]
/lib64/libc.so.6(__libc_start_main+0xf3)[0x7f5e0b1984a3]
/home/520428/work/cortx-motr/motr/.libs/lt-m0d[0x40145e]
----- 1672099 stopped --------
----- 1672610 stopping--------lt-m0d: got signal 1
motr[1672610]:  8810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1672610]:  8810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1672610]:  8810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1672610]:  8810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1672610]:  8810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1672610]:  8810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1672610]:  8810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1672610]:  8810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1672610]:  8810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1672610]:  8810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1672610]:  8810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1672610]:  8810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1672610]:  8810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1672610]:  8810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1672610]:  8810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1672610]:  8810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1672610]:  8810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1672610]:  8810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1672610]:  8810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1672610]:  8810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
----- 1672610 stopped --------
----- 1672685 stopping--------lt-m0d: got signal 1
motr[1672685]:  d810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0xa59c20 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=98 delivered=98 next=98 assign=124)
motr[1672685]:  d810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=48 hl=0xa59c20 ep=10.230.240.203@tcp:12345:34:1
motr[1672685]:  da70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1672685 stopped --------
----- 1672757 stopping--------lt-m0d: got signal 1
motr[1672757]:  5810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x1fcbfb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=64 delivered=64 next=64 assign=104)
motr[1672757]:  5810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=32 hl=0x1fcbfb0 ep=10.230.240.203@tcp:12345:34:1
motr[1672757]:  5a70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1672757 stopped --------
----- 1672765 stopping--------lt-m0d: got signal 1
motr[1672765]:  b810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x19fefb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=64 delivered=64 next=64 assign=118)
motr[1672765]:  b810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=33 hl=0x19fefb0 ep=10.230.240.203@tcp:12345:34:1
motr[1672765]:  ba70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1672765 stopped --------
----- 1672766 stopping--------lt-m0d: got signal 1
motr[1672766]:  c810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x1e23fb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=64 delivered=64 next=64 assign=132)
motr[1672766]:  c810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=34 hl=0x1e23fb0 ep=10.230.240.203@tcp:12345:34:1
motr[1672766]:  ca70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1672766 stopped --------
----- 1672777 stopping--------lt-m0d: got signal 1
motr[1672777]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:3> not found in cache
motr[1672777]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:3> not found in cache
motr[1672777]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:3> not found in cache
motr[1672777]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:3> not found in cache
motr[1672777]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:3> not found in cache
motr[1672777]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:3> not found in cache
motr[1672777]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:3> not found in cache
motr[1672777]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:3> not found in cache
motr[1672777]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:3> not found in cache
motr[1672777]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:3> not found in cache
motr[1672777]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:3> not found in cache
motr[1672777]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:3> not found in cache
motr[1672777]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:3> not found in cache
motr[1672777]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:3> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000003:7> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000c:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000002:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:2> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:2> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:2> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:2> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:2> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:2> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:2> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:1> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:1> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:1> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:1> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:1> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:1> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:1> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000003:7> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000c:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000002:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:2> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:2> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:2> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:2> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:2> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:1> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:1> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:1> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:1> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:1> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:1> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:1> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:2> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:2> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000003:7> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000c:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000002:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:2> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:2> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:2> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:2> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:2> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:1> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:1> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:1> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:1> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:1> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:1> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:1> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:2> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:0> not found in cache
motr[1672777]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:2> not found in cache
motr[1672777]:  b810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x1eb9fb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=66 delivered=66 next=66 assign=100)
motr[1672777]:  b810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=35 hl=0x1eb9fb0 ep=10.230.240.203@tcp:12345:34:1
motr[1672777]:  ba70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1672777 stopped --------
Motr services stopped.
Testing RMW when update size < unit size
n k s p:3 2 2 20
vm.max_map_count = 30000000
motr_service_start: (N,K,S,P)=(1,4,4,20) nr_ios=4 multiple_pools=0
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00331531 s, 316 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00315277 s, 333 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00219938 s, 477 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00226621 s, 463 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0086003 s, 122 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00258602 s, 405 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00762735 s, 137 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00208991 s, 502 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00611027 s, 172 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00133959 s, 783 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00150814 s, 695 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00167543 s, 626 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0020466 s, 512 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00141129 s, 743 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00218518 s, 480 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00204132 s, 514 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00168497 s, 622 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00209209 s, 501 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00716205 s, 146 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00155124 s, 676 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00408958 s, 256 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00367885 s, 285 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00174697 s, 600 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00194405 s, 539 MB/s

[152:
  {0x74| ((^t|1:0), 1, (11, 22), ^o|2:9, ^v|1:20, 1,
          [1: "20 1 4 4"],
          [1: ^n|1:2],
          [1: ^S|1:6],
          [3: ^o|1:9, ^o|20:1, ^o|2:9],
          [1: ^p|1:0], [0])},
  {0x70| ((^p|1:0), [3: ^o|1:9, ^o|20:1, ^o|2:9])},
  {0x6e| ((^n|1:2), 16000, 2, 3, 2, [8: ^r|1:100, ^r|1:0, ^r|1:1, ^r|1:2, ^r|1:3, ^r|1:4, ^r|1:5, ^r|1:6])},

  {0x72| ((^r|1:100), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:1", [1: ^s|1:101])},
  {0x72| ((^r|1:0), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:900", [8: ^s|1:0, ^s|11:0, ^s|6:0, ^s|7:0,                     ^s|3:0, ^s|13:0, ^s|15:0, ^s|16:0])},
  {0x72| ((^r|1:1), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:901", [8: ^s|1:1, ^s|11:1, ^s|6:1, ^s|7:1,                     ^s|3:1, ^s|13:1, ^s|15:1, ^s|16:1])},
  {0x72| ((^r|1:2), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:902", [8: ^s|1:2, ^s|11:2, ^s|6:2, ^s|7:2,                     ^s|3:2, ^s|13:2, ^s|15:2, ^s|16:2])},
  {0x72| ((^r|1:3), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:903", [8: ^s|1:3, ^s|11:3, ^s|6:3, ^s|7:3,                     ^s|3:3, ^s|13:3, ^s|15:3, ^s|16:3])},
 {0x72| ((^r|1:4), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:800", [3: ^s|2:0, ^s|12:0, ^s|3:4])},
 {0x72| ((^r|1:5), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:34:1",
                          [3: ^s|1:6, ^s|1:7, ^s|3:6])},
 {0x72| ((^r|1:6), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:100",
                          [2: ^s|8:0, ^s|3:7])},
  {0x73| ((^s|8:0), @M0_CST_CONFD, [1: "10.230.240.203@tcp:12345:33:100"], [0], [0])},
  {0x73| ((^s|1:6), @M0_CST_HA, [1: "10.230.240.203@tcp:12345:34:1"], [0], [0])},
  {0x73| ((^s|1:7), @M0_CST_FIS, [1: "10.230.240.203@tcp:12345:34:1"], [0], [0])},
  {0x73| ((^s|1:101), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:1"], [0], [0])},




 {0x73| ((^s|2:0), @M0_CST_MDS, [1: "10.230.240.203@tcp:12345:33:800"], [0], [0])},
  {0x73| ((^s|12:0), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:800"], [0], [0])},
  {0x73| ((^s|3:4), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:800"], [0], [0])},

  {0x73| ((^s|1:0), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:900"], [0], [5: ^d|1:1, ^d|1:2, ^d|1:3, ^d|1:4, ^d|1:5])},
  {0x73| ((^s|11:0), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
 {0x73| ((^s|6:0), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
  {0x73| ((^s|7:0), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
 {0x73| ((^s|3:0), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
 {0x73| ((^s|13:0), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:900"], [0], [1: ^d|20:0])},
  {0x73| ((^s|15:0), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
  {0x73| ((^s|16:0), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
  {0x73| ((^s|1:1), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:901"], [0], [5: ^d|1:6, ^d|1:7, ^d|1:8, ^d|1:9, ^d|1:10])},
  {0x73| ((^s|11:1), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
 {0x73| ((^s|6:1), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
  {0x73| ((^s|7:1), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
 {0x73| ((^s|3:1), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
 {0x73| ((^s|13:1), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:901"], [0], [1: ^d|20:1])},
  {0x73| ((^s|15:1), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
  {0x73| ((^s|16:1), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
  {0x73| ((^s|1:2), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:902"], [0], [5: ^d|1:11, ^d|1:12, ^d|1:13, ^d|1:14, ^d|1:15])},
  {0x73| ((^s|11:2), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
 {0x73| ((^s|6:2), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
  {0x73| ((^s|7:2), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
 {0x73| ((^s|3:2), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
 {0x73| ((^s|13:2), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:902"], [0], [1: ^d|20:2])},
  {0x73| ((^s|15:2), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
  {0x73| ((^s|16:2), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
  {0x73| ((^s|1:3), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:903"], [0], [5: ^d|1:16, ^d|1:17, ^d|1:18, ^d|1:19, ^d|1:20])},
  {0x73| ((^s|11:3), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
 {0x73| ((^s|6:3), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
  {0x73| ((^s|7:3), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
 {0x73| ((^s|3:3), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
 {0x73| ((^s|13:3), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:903"], [0], [1: ^d|20:3])},
  {0x73| ((^s|15:3), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
  {0x73| ((^s|16:3), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},

 {0x73| ((^s|3:6), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:34:1"], [0], [0])},
 {0x73| ((^s|3:7), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:100"], [0], [0])},
  {0x64| ((^d|1:1), 0, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop1")},
 {0x6b| ((^k|1:1), ^d|1:1, [1: ^v|1:10])},
 {0x6a| ((^j|1:1), ^k|1:1, [0])},
 {0x64| ((^d|1:2), 1, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop2")},
 {0x6b| ((^k|1:2), ^d|1:2, [1: ^v|1:10])},
 {0x6a| ((^j|1:2), ^k|1:2, [0])},
 {0x64| ((^d|1:3), 2, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop3")},
 {0x6b| ((^k|1:3), ^d|1:3, [1: ^v|1:10])},
 {0x6a| ((^j|1:3), ^k|1:3, [0])},
 {0x64| ((^d|1:4), 3, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop4")},
 {0x6b| ((^k|1:4), ^d|1:4, [1: ^v|1:10])},
 {0x6a| ((^j|1:4), ^k|1:4, [0])},
 {0x64| ((^d|1:5), 4, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop5")},
 {0x6b| ((^k|1:5), ^d|1:5, [1: ^v|1:10])},
 {0x6a| ((^j|1:5), ^k|1:5, [0])},
 {0x64| ((^d|1:6), 5, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop6")},
 {0x6b| ((^k|1:6), ^d|1:6, [1: ^v|1:10])},
 {0x6a| ((^j|1:6), ^k|1:6, [0])},
 {0x64| ((^d|1:7), 6, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop7")},
 {0x6b| ((^k|1:7), ^d|1:7, [1: ^v|1:10])},
 {0x6a| ((^j|1:7), ^k|1:7, [0])},
 {0x64| ((^d|1:8), 7, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop8")},
 {0x6b| ((^k|1:8), ^d|1:8, [1: ^v|1:10])},
 {0x6a| ((^j|1:8), ^k|1:8, [0])},
 {0x64| ((^d|1:9), 8, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop9")},
 {0x6b| ((^k|1:9), ^d|1:9, [1: ^v|1:10])},
 {0x6a| ((^j|1:9), ^k|1:9, [0])},
 {0x64| ((^d|1:10), 9, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop10")},
 {0x6b| ((^k|1:10), ^d|1:10, [1: ^v|1:10])},
 {0x6a| ((^j|1:10), ^k|1:10, [0])},
 {0x64| ((^d|1:11), 10, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop11")},
 {0x6b| ((^k|1:11), ^d|1:11, [1: ^v|1:10])},
 {0x6a| ((^j|1:11), ^k|1:11, [0])},
 {0x64| ((^d|1:12), 11, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop12")},
 {0x6b| ((^k|1:12), ^d|1:12, [1: ^v|1:10])},
 {0x6a| ((^j|1:12), ^k|1:12, [0])},
 {0x64| ((^d|1:13), 12, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop13")},
 {0x6b| ((^k|1:13), ^d|1:13, [1: ^v|1:10])},
 {0x6a| ((^j|1:13), ^k|1:13, [0])},
 {0x64| ((^d|1:14), 13, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop14")},
 {0x6b| ((^k|1:14), ^d|1:14, [1: ^v|1:10])},
 {0x6a| ((^j|1:14), ^k|1:14, [0])},
 {0x64| ((^d|1:15), 14, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop15")},
 {0x6b| ((^k|1:15), ^d|1:15, [1: ^v|1:10])},
 {0x6a| ((^j|1:15), ^k|1:15, [0])},
 {0x64| ((^d|1:16), 15, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop16")},
 {0x6b| ((^k|1:16), ^d|1:16, [1: ^v|1:10])},
 {0x6a| ((^j|1:16), ^k|1:16, [0])},
 {0x64| ((^d|1:17), 16, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop17")},
 {0x6b| ((^k|1:17), ^d|1:17, [1: ^v|1:10])},
 {0x6a| ((^j|1:17), ^k|1:17, [0])},
 {0x64| ((^d|1:18), 17, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop18")},
 {0x6b| ((^k|1:18), ^d|1:18, [1: ^v|1:10])},
 {0x6a| ((^j|1:18), ^k|1:18, [0])},
 {0x64| ((^d|1:19), 18, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop19")},
 {0x6b| ((^k|1:19), ^d|1:19, [1: ^v|1:10])},
 {0x6a| ((^j|1:19), ^k|1:19, [0])},
 {0x64| ((^d|1:20), 19, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop20")},
 {0x6b| ((^k|1:20), ^d|1:20, [1: ^v|1:10])},
 {0x6a| ((^j|1:20), ^k|1:20, [0])},
  {0x53| ((^S|1:6), [1: ^a|1:6], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x61| ((^a|1:6), [1: ^e|1:7], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x65| ((^e|1:7), ^n|1:2, [1: ^c|1:8], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x63| ((^c|1:8), [24: ^k|1:1, ^k|1:2, ^k|1:3, ^k|1:4, ^k|1:5, ^k|1:6, ^k|1:7, ^k|1:8, ^k|1:9, ^k|1:10, ^k|1:11, ^k|1:12, ^k|1:13, ^k|1:14, ^k|1:15, ^k|1:16, ^k|1:17, ^k|1:18, ^k|1:19, ^k|1:20, ^k|20:0, ^k|20:1, ^k|20:2, ^k|20:3],
                            [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x6f| ((^o|1:9), 0, [3: ^v|1:10, ^v|0x40000000000001:11, ^v|0x40000000000001:12])},
  {0x76| ((^v|1:10), {0| (1, 4,
                                            4, 20,
                                            [5: 0, 0, 0, 0, 4],
                                            [1: ^j|1:21])})},
  {0x76| ((^v|0x40000000000001:11), {1| (0, ^v|1:10, [5: 0, 0, 0, 0, 1])})},
  {0x76| ((^v|0x40000000000001:12), {1| (1, ^v|1:10, [5: 0, 0, 0, 0, 2])})},
  {0x6a| ((^j|1:21), ^S|1:6, [1: ^j|1:22])},
  {0x6a| ((^j|1:22), ^a|1:6, [1: ^j|1:23])},
  {0x6a| ((^j|1:23), ^e|1:7, [1: ^j|1:24])},
  {0x6a| ((^j|1:24), ^c|1:8, [20: ^j|1:1, ^j|1:2, ^j|1:3, ^j|1:4, ^j|1:5, ^j|1:6, ^j|1:7, ^j|1:8, ^j|1:9, ^j|1:10, ^j|1:11, ^j|1:12, ^j|1:13, ^j|1:14, ^j|1:15, ^j|1:16, ^j|1:17, ^j|1:18, ^j|1:19, ^j|1:20])} ,
{0x6f| ((^o|2:9), 0, [1: ^v|2:10])},
{0x76| ((^v|2:10), {0| (4, 0, 0, 4, [5: 0, 0, 0, 0, 1], [1: ^j|2:21])})},
{0x6a| ((^j|2:21), ^S|1:6, [1: ^j|2:22])},
{0x6a| ((^j|2:22), ^a|1:6, [1: ^j|2:23])},
{0x6a| ((^j|2:23), ^e|1:7, [1: ^j|2:24])},
{0x6a| ((^j|2:24), ^c|1:8, [4: ^j|2:1, ^j|2:6, ^j|2:11, ^j|2:16])},
{0x6a| ((^j|2:1), ^k|1:1, [0])},
 {0x6a| ((^j|2:6), ^k|1:6, [0])},
 {0x6a| ((^j|2:11), ^k|1:11, [0])},
 {0x6a| ((^j|2:16), ^k|1:16, [0])}  ,
{0x64| ((^d|20:0), 20, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop25")},
{0x6b| ((^k|20:0), ^d|20:0, [1: ^v|1:20])},
{0x6a| ((^j|20:100), ^k|20:0, [0])},
{0x64| ((^d|20:1), 21, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop26")},
{0x6b| ((^k|20:1), ^d|20:1, [1: ^v|1:20])},
{0x6a| ((^j|20:101), ^k|20:1, [0])},
{0x64| ((^d|20:2), 22, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop27")},
{0x6b| ((^k|20:2), ^d|20:2, [1: ^v|1:20])},
{0x6a| ((^j|20:102), ^k|20:2, [0])},
{0x64| ((^d|20:3), 23, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop28")},
{0x6b| ((^k|20:3), ^d|20:3, [1: ^v|1:20])},
{0x6a| ((^j|20:103), ^k|20:3, [0])},
{0x6f| ((^o|20:1), 0, [1: ^v|1:20])},
{0x76| ((^v|1:20), {0| (1, 1, 1,
                                                    4,
                                                    [5: 0, 0, 0, 0, 1],
                                                    [1: ^j|20:1])})},
{0x6a| ((^j|20:1), ^S|1:6, [1: ^j|20:2])},
{0x6a| ((^j|20:2), ^a|1:6, [1: ^j|20:3])},
{0x6a| ((^j|20:3), ^e|1:7, [1: ^j|20:4])},
{0x6a| ((^j|20:4), ^c|1:8, [4:  ^j|20:100, ^j|20:101, ^j|20:102, ^j|20:103])}]
cd /var/motr/root/sandbox.st-45motr-rmw/confd && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -D db -S stobs -A linuxstob:addb-stobs                       -w 20 -m 65536      -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T linux -e libfab:10.230.240.203@tcp:12345:35:1                      -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/confd && exec /home/520428/work/cortx-motr/motr/m0d -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -f '<0x7200000000000001:6>' -T linux -e libfab:10.230.240.203@tcp:12345:33:100 -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0d.log
cd /var/motr/root/sandbox.st-45motr-rmw/ha && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T ad -e libfab:10.230.240.203@tcp:12345:35:1 -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
motr[1674011]:  4f00  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
cd /var/motr/root/sandbox.st-45motr-rmw/mds1 && exec                    /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad                         -D db -S stobs -A linuxstob:addb-stobs                            -w 20 -m 65536                          -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:800          -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios1 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:900 -f '<0x7200000000000001:0>' -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios2 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:901 -f '<0x7200000000000001:1>' -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios3 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:902 -f '<0x7200000000000001:2>' -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios4 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:903 -f '<0x7200000000000001:3>' -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ha && exec /home/520428/work/cortx-motr/motr/m0d -D db -S stobs -A linuxstob:addb-stobs                              -w 20 -m 65536              -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T ad -e libfab:10.230.240.203@tcp:12345:34:1                 -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc -f '<0x7200000000000001:5>'  -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
motr[1674523]:  80b0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
Motr HA agent started.
cd /var/motr/root/sandbox.st-45motr-rmw/mds1 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:800 -f '<0x7200000000000001:4>' -H 10.230.240.203@tcp:12345:34:1 -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0d.log
motr[1674597]:  9fb0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
cd /var/motr/root/sandbox.st-45motr-rmw/ios1 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:900 -f '<0x7200000000000001:0>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios2 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:901 -f '<0x7200000000000001:1>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios3 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:902 -f '<0x7200000000000001:2>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios4 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:903 -f '<0x7200000000000001:3>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
Motr confd started.
Motr mdservices started.
motr[1674685]:  a890  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
motr[1674679]:  5a30  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
motr[1674677]:  9770  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
motr[1674668]:  4ec0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
Motr ioservices started.
motr service started
*** m0dixinit is omitted. Mkfs creates meta indices now.
stopping /home/520428/work/cortx-motr/motr/m0d processes...
=== pids of services: 1674011 1674523 1674597 1674668 1674677 1674679 1674685 ===
Shutting down services one by one. mdservice is the last.
----- 1674011 stopping--------lt-m0d: got signal 1
motr[1674011]:  5600  FATAL  [lib/assert.c:50:m0_panic]  panic: (!m0_clink_is_armed(&revc->rcf_disc_wait)) at m0_rpc_service_reverse_session_put() (rpc/service.c:241)  [git: 2.0.0-895-5-gdc9f62213] /var/motr/root/sandbox.st-45motr-rmw/confd/m0trace.1674011.2022-08-16-17:02:30
Motr panic: (!m0_clink_is_armed(&revc->rcf_disc_wait)) at m0_rpc_service_reverse_session_put() rpc/service.c:241 (errno: 4) (last failed: none) [git: 2.0.0-895-5-gdc9f62213] pid: 1674011  /var/motr/root/sandbox.st-45motr-rmw/confd/m0trace.1674011.2022-08-16-17:02:30
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_arch_backtrace+0x20)[0x7fd20cac9a60]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_arch_panic+0xdf)[0x7fd20cac9c0f]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_panic+0x127)[0x7fd20cab7bf7]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(+0x45fe1e)[0x7fd20cb56e1e]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_rpc_service_reverse_sessions_cleanup+0x9f)[0x7fd20cb56ebf]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(+0x45ffdc)[0x7fd20cb56fdc]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_reqh_service_prepare_to_stop+0xc6)[0x7fd20cb328f6]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_reqh_services_prepare_to_stop+0x6e)[0x7fd20cb3112e]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_reqh_shutdown_wait+0x11)[0x7fd20cb31221]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(+0x3e0f0e)[0x7fd20cad7f0e]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_module_fini+0x70)[0x7fd20cb0a7d0]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_cs_fini+0x2d)[0x7fd20cadbd3d]
/home/520428/work/cortx-motr/motr/.libs/lt-m0d[0x4011cc]
/lib64/libc.so.6(__libc_start_main+0xf3)[0x7fd20a9894a3]
/home/520428/work/cortx-motr/motr/.libs/lt-m0d[0x40145e]
----- 1674011 stopped --------
----- 1674523 stopping--------lt-m0d: got signal 1
----- 1674523 stopped --------
----- 1674597 stopping--------lt-m0d: got signal 1
motr[1674597]:  3810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x1f9dc20 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=98 delivered=98 next=98 assign=124)
motr[1674597]:  3810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=50 hl=0x1f9dc20 ep=10.230.240.203@tcp:12345:34:1
motr[1674597]:  3a70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1674597 stopped --------
----- 1674668 stopping--------lt-m0d: got signal 1
motr[1674668]:  f810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x11dafb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=64 delivered=64 next=64 assign=104)
motr[1674668]:  f810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=33 hl=0x11dafb0 ep=10.230.240.203@tcp:12345:34:1
motr[1674668]:  fa70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1674668 stopped --------
----- 1674677 stopping--------lt-m0d: got signal 1
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:3> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:3> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:3> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:3> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:2> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:2> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:2> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:2> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:2> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:2> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:2> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:3> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:3> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:3> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:3> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:1> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:1> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:1> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:1> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:1> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:1> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:2> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:2> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:2> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:2> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:2> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:2> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:2> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:3> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:3> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:3> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:3> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:3> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:3> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:1> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:1> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:1> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:1> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:1> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:1> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:1> not found in cache
motr[1674677]:  3020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:1> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000003:7> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000c:0> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000002:0> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:0> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:0> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:0> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:0> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:0> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:0> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:0> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000003:7> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000c:0> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000002:0> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:0> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:0> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:0> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:0> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:0> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:0> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:0> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000003:7> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000002:0> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000c:0> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:0> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:0> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:0> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:0> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:0> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:0> not found in cache
motr[1674677]:  5a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:0> not found in cache
motr[1674677]:  9810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x2462fb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=66 delivered=66 next=66 assign=100)
motr[1674677]:  9810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=34 hl=0x2462fb0 ep=10.230.240.203@tcp:12345:34:1
motr[1674677]:  9a70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1674677 stopped --------
----- 1674679 stopping--------lt-m0d: got signal 1
motr[1674679]:  e810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x26d5fb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=64 delivered=64 next=64 assign=132)
motr[1674679]:  e810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=33 hl=0x26d5fb0 ep=10.230.240.203@tcp:12345:34:1
motr[1674679]:  ea70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1674679 stopped --------
----- 1674685 stopping--------lt-m0d: got signal 1
motr[1674685]:  2810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x1182fb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=64 delivered=64 next=64 assign=146)
motr[1674685]:  2810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=34 hl=0x1182fb0 ep=10.230.240.203@tcp:12345:34:1
motr[1674685]:  2a70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1674685 stopped --------
Motr services stopped.
motr_rmw_IO: test status: SUCCESS

real    9m29.919s
user    5m59.772s
sys     2m29.785s

real    9m34.295s
user    5m59.826s
sys     2m29.852s
bash-4.4$

Please note the panic happens while doing finalization, which does not impact this patch, and will be fixed soon in another PR.

@huanghua78
Copy link
Author

Another round of successful 45motr-rmw:

bash-4.4$ sudo ./scripts/m0 run-st 45motr-rmw 2>&1 | tee 45.txt
[sudo] password for 520428:
sudo: timed out reading password
bash-4.4$ sudo ./scripts/m0 run-st 45motr-rmw 2>&1 | tee 45.txt
[sudo] password for 520428:
----- run_st 45motr-rmw -----
<< 45motr-rmw >>
false
Motr RMW IO Test ...
Testing RMW in Erasure Coded layout
n k s p:3 2 2 20
vm.max_map_count = 30000000
motr_service_start: (N,K,S,P)=(3,2,2,20) nr_ios=4 multiple_pools=0
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00142728 s, 735 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00132843 s, 789 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00151921 s, 690 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00129742 s, 808 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00153122 s, 685 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00506152 s, 207 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00128584 s, 815 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00181311 s, 578 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00224703 s, 467 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00141464 s, 741 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00145204 s, 722 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00171816 s, 610 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00213687 s, 491 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00148539 s, 706 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00188029 s, 558 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00182943 s, 573 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00147107 s, 713 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00154741 s, 678 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00131568 s, 797 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00126296 s, 830 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00214587 s, 489 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00260204 s, 403 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00450896 s, 233 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00175207 s, 598 MB/s

[152:
  {0x74| ((^t|1:0), 1, (11, 22), ^o|2:9, ^v|1:20, 1,
          [1: "20 3 2 2"],
          [1: ^n|1:2],
          [1: ^S|1:6],
          [3: ^o|1:9, ^o|20:1, ^o|2:9],
          [1: ^p|1:0], [0])},
  {0x70| ((^p|1:0), [3: ^o|1:9, ^o|20:1, ^o|2:9])},
  {0x6e| ((^n|1:2), 16000, 2, 3, 2, [8: ^r|1:100, ^r|1:0, ^r|1:1, ^r|1:2, ^r|1:3, ^r|1:4, ^r|1:5, ^r|1:6])},

  {0x72| ((^r|1:100), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:1", [1: ^s|1:101])},
  {0x72| ((^r|1:0), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:900", [8: ^s|1:0, ^s|11:0, ^s|6:0, ^s|7:0,                     ^s|3:0, ^s|13:0, ^s|15:0, ^s|16:0])},
  {0x72| ((^r|1:1), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:901", [8: ^s|1:1, ^s|11:1, ^s|6:1, ^s|7:1,                     ^s|3:1, ^s|13:1, ^s|15:1, ^s|16:1])},
  {0x72| ((^r|1:2), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:902", [8: ^s|1:2, ^s|11:2, ^s|6:2, ^s|7:2,                     ^s|3:2, ^s|13:2, ^s|15:2, ^s|16:2])},
  {0x72| ((^r|1:3), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:903", [8: ^s|1:3, ^s|11:3, ^s|6:3, ^s|7:3,                     ^s|3:3, ^s|13:3, ^s|15:3, ^s|16:3])},
 {0x72| ((^r|1:4), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:800", [3: ^s|2:0, ^s|12:0, ^s|3:4])},
 {0x72| ((^r|1:5), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:34:1",
                          [3: ^s|1:6, ^s|1:7, ^s|3:6])},
 {0x72| ((^r|1:6), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:100",
                          [2: ^s|8:0, ^s|3:7])},
  {0x73| ((^s|8:0), @M0_CST_CONFD, [1: "10.230.240.203@tcp:12345:33:100"], [0], [0])},
  {0x73| ((^s|1:6), @M0_CST_HA, [1: "10.230.240.203@tcp:12345:34:1"], [0], [0])},
  {0x73| ((^s|1:7), @M0_CST_FIS, [1: "10.230.240.203@tcp:12345:34:1"], [0], [0])},
  {0x73| ((^s|1:101), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:1"], [0], [0])},




 {0x73| ((^s|2:0), @M0_CST_MDS, [1: "10.230.240.203@tcp:12345:33:800"], [0], [0])},
  {0x73| ((^s|12:0), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:800"], [0], [0])},
  {0x73| ((^s|3:4), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:800"], [0], [0])},

  {0x73| ((^s|1:0), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:900"], [0], [5: ^d|1:1, ^d|1:2, ^d|1:3, ^d|1:4, ^d|1:5])},
  {0x73| ((^s|11:0), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
 {0x73| ((^s|6:0), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
  {0x73| ((^s|7:0), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
 {0x73| ((^s|3:0), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
 {0x73| ((^s|13:0), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:900"], [0], [1: ^d|20:0])},
  {0x73| ((^s|15:0), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
  {0x73| ((^s|16:0), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
  {0x73| ((^s|1:1), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:901"], [0], [5: ^d|1:6, ^d|1:7, ^d|1:8, ^d|1:9, ^d|1:10])},
  {0x73| ((^s|11:1), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
 {0x73| ((^s|6:1), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
  {0x73| ((^s|7:1), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
 {0x73| ((^s|3:1), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
 {0x73| ((^s|13:1), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:901"], [0], [1: ^d|20:1])},
  {0x73| ((^s|15:1), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
  {0x73| ((^s|16:1), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
  {0x73| ((^s|1:2), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:902"], [0], [5: ^d|1:11, ^d|1:12, ^d|1:13, ^d|1:14, ^d|1:15])},
  {0x73| ((^s|11:2), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
 {0x73| ((^s|6:2), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
  {0x73| ((^s|7:2), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
 {0x73| ((^s|3:2), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
 {0x73| ((^s|13:2), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:902"], [0], [1: ^d|20:2])},
  {0x73| ((^s|15:2), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
  {0x73| ((^s|16:2), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
  {0x73| ((^s|1:3), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:903"], [0], [5: ^d|1:16, ^d|1:17, ^d|1:18, ^d|1:19, ^d|1:20])},
  {0x73| ((^s|11:3), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
 {0x73| ((^s|6:3), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
  {0x73| ((^s|7:3), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
 {0x73| ((^s|3:3), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
 {0x73| ((^s|13:3), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:903"], [0], [1: ^d|20:3])},
  {0x73| ((^s|15:3), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
  {0x73| ((^s|16:3), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},

 {0x73| ((^s|3:6), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:34:1"], [0], [0])},
 {0x73| ((^s|3:7), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:100"], [0], [0])},
  {0x64| ((^d|1:1), 0, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop1")},
 {0x6b| ((^k|1:1), ^d|1:1, [1: ^v|1:10])},
 {0x6a| ((^j|1:1), ^k|1:1, [0])},
 {0x64| ((^d|1:2), 1, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop2")},
 {0x6b| ((^k|1:2), ^d|1:2, [1: ^v|1:10])},
 {0x6a| ((^j|1:2), ^k|1:2, [0])},
 {0x64| ((^d|1:3), 2, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop3")},
 {0x6b| ((^k|1:3), ^d|1:3, [1: ^v|1:10])},
 {0x6a| ((^j|1:3), ^k|1:3, [0])},
 {0x64| ((^d|1:4), 3, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop4")},
 {0x6b| ((^k|1:4), ^d|1:4, [1: ^v|1:10])},
 {0x6a| ((^j|1:4), ^k|1:4, [0])},
 {0x64| ((^d|1:5), 4, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop5")},
 {0x6b| ((^k|1:5), ^d|1:5, [1: ^v|1:10])},
 {0x6a| ((^j|1:5), ^k|1:5, [0])},
 {0x64| ((^d|1:6), 5, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop6")},
 {0x6b| ((^k|1:6), ^d|1:6, [1: ^v|1:10])},
 {0x6a| ((^j|1:6), ^k|1:6, [0])},
 {0x64| ((^d|1:7), 6, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop7")},
 {0x6b| ((^k|1:7), ^d|1:7, [1: ^v|1:10])},
 {0x6a| ((^j|1:7), ^k|1:7, [0])},
 {0x64| ((^d|1:8), 7, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop8")},
 {0x6b| ((^k|1:8), ^d|1:8, [1: ^v|1:10])},
 {0x6a| ((^j|1:8), ^k|1:8, [0])},
 {0x64| ((^d|1:9), 8, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop9")},
 {0x6b| ((^k|1:9), ^d|1:9, [1: ^v|1:10])},
 {0x6a| ((^j|1:9), ^k|1:9, [0])},
 {0x64| ((^d|1:10), 9, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop10")},
 {0x6b| ((^k|1:10), ^d|1:10, [1: ^v|1:10])},
 {0x6a| ((^j|1:10), ^k|1:10, [0])},
 {0x64| ((^d|1:11), 10, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop11")},
 {0x6b| ((^k|1:11), ^d|1:11, [1: ^v|1:10])},
 {0x6a| ((^j|1:11), ^k|1:11, [0])},
 {0x64| ((^d|1:12), 11, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop12")},
 {0x6b| ((^k|1:12), ^d|1:12, [1: ^v|1:10])},
 {0x6a| ((^j|1:12), ^k|1:12, [0])},
 {0x64| ((^d|1:13), 12, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop13")},
 {0x6b| ((^k|1:13), ^d|1:13, [1: ^v|1:10])},
 {0x6a| ((^j|1:13), ^k|1:13, [0])},
 {0x64| ((^d|1:14), 13, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop14")},
 {0x6b| ((^k|1:14), ^d|1:14, [1: ^v|1:10])},
 {0x6a| ((^j|1:14), ^k|1:14, [0])},
 {0x64| ((^d|1:15), 14, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop15")},
 {0x6b| ((^k|1:15), ^d|1:15, [1: ^v|1:10])},
 {0x6a| ((^j|1:15), ^k|1:15, [0])},
 {0x64| ((^d|1:16), 15, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop16")},
 {0x6b| ((^k|1:16), ^d|1:16, [1: ^v|1:10])},
 {0x6a| ((^j|1:16), ^k|1:16, [0])},
 {0x64| ((^d|1:17), 16, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop17")},
 {0x6b| ((^k|1:17), ^d|1:17, [1: ^v|1:10])},
 {0x6a| ((^j|1:17), ^k|1:17, [0])},
 {0x64| ((^d|1:18), 17, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop18")},
 {0x6b| ((^k|1:18), ^d|1:18, [1: ^v|1:10])},
 {0x6a| ((^j|1:18), ^k|1:18, [0])},
 {0x64| ((^d|1:19), 18, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop19")},
 {0x6b| ((^k|1:19), ^d|1:19, [1: ^v|1:10])},
 {0x6a| ((^j|1:19), ^k|1:19, [0])},
 {0x64| ((^d|1:20), 19, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop20")},
 {0x6b| ((^k|1:20), ^d|1:20, [1: ^v|1:10])},
 {0x6a| ((^j|1:20), ^k|1:20, [0])},
  {0x53| ((^S|1:6), [1: ^a|1:6], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x61| ((^a|1:6), [1: ^e|1:7], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x65| ((^e|1:7), ^n|1:2, [1: ^c|1:8], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x63| ((^c|1:8), [24: ^k|1:1, ^k|1:2, ^k|1:3, ^k|1:4, ^k|1:5, ^k|1:6, ^k|1:7, ^k|1:8, ^k|1:9, ^k|1:10, ^k|1:11, ^k|1:12, ^k|1:13, ^k|1:14, ^k|1:15, ^k|1:16, ^k|1:17, ^k|1:18, ^k|1:19, ^k|1:20, ^k|20:0, ^k|20:1, ^k|20:2, ^k|20:3],
                            [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x6f| ((^o|1:9), 0, [3: ^v|1:10, ^v|0x40000000000001:11, ^v|0x40000000000001:12])},
  {0x76| ((^v|1:10), {0| (3, 2,
                                            2, 20,
                                            [5: 0, 0, 0, 0, 2],
                                            [1: ^j|1:21])})},
  {0x76| ((^v|0x40000000000001:11), {1| (0, ^v|1:10, [5: 0, 0, 0, 0, 1])})},
  {0x76| ((^v|0x40000000000001:12), {1| (1, ^v|1:10, [5: 0, 0, 0, 0, 2])})},
  {0x6a| ((^j|1:21), ^S|1:6, [1: ^j|1:22])},
  {0x6a| ((^j|1:22), ^a|1:6, [1: ^j|1:23])},
  {0x6a| ((^j|1:23), ^e|1:7, [1: ^j|1:24])},
  {0x6a| ((^j|1:24), ^c|1:8, [20: ^j|1:1, ^j|1:2, ^j|1:3, ^j|1:4, ^j|1:5, ^j|1:6, ^j|1:7, ^j|1:8, ^j|1:9, ^j|1:10, ^j|1:11, ^j|1:12, ^j|1:13, ^j|1:14, ^j|1:15, ^j|1:16, ^j|1:17, ^j|1:18, ^j|1:19, ^j|1:20])} ,
{0x6f| ((^o|2:9), 0, [1: ^v|2:10])},
{0x76| ((^v|2:10), {0| (4, 0, 0, 4, [5: 0, 0, 0, 0, 1], [1: ^j|2:21])})},
{0x6a| ((^j|2:21), ^S|1:6, [1: ^j|2:22])},
{0x6a| ((^j|2:22), ^a|1:6, [1: ^j|2:23])},
{0x6a| ((^j|2:23), ^e|1:7, [1: ^j|2:24])},
{0x6a| ((^j|2:24), ^c|1:8, [4: ^j|2:1, ^j|2:6, ^j|2:11, ^j|2:16])},
{0x6a| ((^j|2:1), ^k|1:1, [0])},
 {0x6a| ((^j|2:6), ^k|1:6, [0])},
 {0x6a| ((^j|2:11), ^k|1:11, [0])},
 {0x6a| ((^j|2:16), ^k|1:16, [0])}  ,
{0x64| ((^d|20:0), 20, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop25")},
{0x6b| ((^k|20:0), ^d|20:0, [1: ^v|1:20])},
{0x6a| ((^j|20:100), ^k|20:0, [0])},
{0x64| ((^d|20:1), 21, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop26")},
{0x6b| ((^k|20:1), ^d|20:1, [1: ^v|1:20])},
{0x6a| ((^j|20:101), ^k|20:1, [0])},
{0x64| ((^d|20:2), 22, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop27")},
{0x6b| ((^k|20:2), ^d|20:2, [1: ^v|1:20])},
{0x6a| ((^j|20:102), ^k|20:2, [0])},
{0x64| ((^d|20:3), 23, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop28")},
{0x6b| ((^k|20:3), ^d|20:3, [1: ^v|1:20])},
{0x6a| ((^j|20:103), ^k|20:3, [0])},
{0x6f| ((^o|20:1), 0, [1: ^v|1:20])},
{0x76| ((^v|1:20), {0| (1, 1, 1,
                                                    4,
                                                    [5: 0, 0, 0, 0, 1],
                                                    [1: ^j|20:1])})},
{0x6a| ((^j|20:1), ^S|1:6, [1: ^j|20:2])},
{0x6a| ((^j|20:2), ^a|1:6, [1: ^j|20:3])},
{0x6a| ((^j|20:3), ^e|1:7, [1: ^j|20:4])},
{0x6a| ((^j|20:4), ^c|1:8, [4:  ^j|20:100, ^j|20:101, ^j|20:102, ^j|20:103])}]
cd /var/motr/root/sandbox.st-45motr-rmw/confd && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -D db -S stobs -A linuxstob:addb-stobs                       -w 20 -m 65536      -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T linux -e libfab:10.230.240.203@tcp:12345:35:1                      -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/confd && exec /home/520428/work/cortx-motr/motr/m0d -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -f '<0x7200000000000001:6>' -T linux -e libfab:10.230.240.203@tcp:12345:33:100 -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0d.log
cd /var/motr/root/sandbox.st-45motr-rmw/ha && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T ad -e libfab:10.230.240.203@tcp:12345:35:1 -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
motr[1677487]:  ef70  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
cd /var/motr/root/sandbox.st-45motr-rmw/mds1 && exec                    /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad                         -D db -S stobs -A linuxstob:addb-stobs                            -w 20 -m 65536                          -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:800          -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios1 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:900 -f '<0x7200000000000001:0>' -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios2 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:901 -f '<0x7200000000000001:1>' -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios3 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:902 -f '<0x7200000000000001:2>' -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios4 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:903 -f '<0x7200000000000001:3>' -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ha && exec /home/520428/work/cortx-motr/motr/m0d -D db -S stobs -A linuxstob:addb-stobs                              -w 20 -m 65536              -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T ad -e libfab:10.230.240.203@tcp:12345:34:1                 -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc -f '<0x7200000000000001:5>'  -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
motr[1677997]:  1730  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
Motr HA agent started.
cd /var/motr/root/sandbox.st-45motr-rmw/mds1 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:800 -f '<0x7200000000000001:4>' -H 10.230.240.203@tcp:12345:34:1 -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0d.log
motr[1678071]:  63c0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
cd /var/motr/root/sandbox.st-45motr-rmw/ios1 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:900 -f '<0x7200000000000001:0>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios2 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:901 -f '<0x7200000000000001:1>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios3 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:902 -f '<0x7200000000000001:2>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios4 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:903 -f '<0x7200000000000001:3>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
Motr confd started.
Motr mdservices started.
motr[1678156]:   b90  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
motr[1678142]:  6280  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
motr[1678152]:  9070  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
motr[1678150]:  bf10  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
Motr ioservices started.
motr service started
*** m0dixinit is omitted. Mkfs creates meta indices now.
Testing with layout ID 3 in Healthy mode
m0cp
motr[1677487]:  fa30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1678444]:  2720  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cp update
motr[1677487]:  fa30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1678472]:  c450  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cat
pver is : <0:0>
motr[1677487]:  fa30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1678500]:  cc20  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0unlink
motr[1677487]:  fa30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1678528]:  f290  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
diff
Testing with layout ID 5 in Healthy mode
m0cp
motr[1677487]:  fa30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1678562]:   e30  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cp update
motr[1677487]:  fa30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1678590]:  2b20  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cat
pver is : <0:0>
motr[1677487]:  fa30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1678618]:  b360  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0unlink
motr[1677487]:  fa30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1678646]:  aad0  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
diff
Testing with layout ID 7 in Healthy mode
m0cp
motr[1677487]:  fa30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1678680]:  5610  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cp update
motr[1677487]:  fa30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1678709]:  80f0  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cat
pver is : <0:0>
motr[1677487]:  fa30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1678737]:  fa40  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0unlink
motr[1677487]:  fa30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1678765]:   ef0  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
diff
Testing with layout ID 9 in Healthy mode
m0cp
motr[1677487]:  fa30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1678799]:  c2a0  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cp update
motr[1677487]:  fa30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1678917]:  77d0  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cat
pver is : <0:0>
motr[1677487]:  fa30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1679022]:  2950  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0unlink
motr[1677487]:  fa30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1679050]:  2780  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
diff
Testing with layout ID 11 in Healthy mode
m0cp
motr[1677487]:  fa30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1679084]:  4330  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cp update
motr[1677487]:  fa30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1679119]:  26f0  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cat
pver is : <0:0>
motr[1677487]:  fa30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1679147]:  9210  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0unlink
motr[1677487]:  fa30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1679175]:  9770  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
diff
stopping /home/520428/work/cortx-motr/motr/m0d processes...
=== pids of services: 1677487 1677997 1678071 1678142 1678150 1678152 1678156 ===
Shutting down services one by one. mdservice is the last.
----- 1677487 stopping--------lt-m0d: got signal 1
motr[1677487]:  f670  FATAL  [lib/assert.c:50:m0_panic]  panic: (!m0_clink_is_armed(&revc->rcf_disc_wait)) at m0_rpc_service_reverse_session_put() (rpc/service.c:241)  [git: 2.0.0-895-5-gdc9f62213] /var/motr/root/sandbox.st-45motr-rmw/confd/m0trace.1677487.2022-08-16-21:07:14
Motr panic: (!m0_clink_is_armed(&revc->rcf_disc_wait)) at m0_rpc_service_reverse_session_put() rpc/service.c:241 (errno: 4) (last failed: none) [git: 2.0.0-895-5-gdc9f62213] pid: 1677487  /var/motr/root/sandbox.st-45motr-rmw/confd/m0trace.1677487.2022-08-16-21:07:14
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_arch_backtrace+0x20)[0x7f3b32eeaa60]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_arch_panic+0xdf)[0x7f3b32eeac0f]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_panic+0x127)[0x7f3b32ed8bf7]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(+0x45fe1e)[0x7f3b32f77e1e]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_rpc_service_reverse_sessions_cleanup+0x9f)[0x7f3b32f77ebf]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(+0x45ffdc)[0x7f3b32f77fdc]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_reqh_service_prepare_to_stop+0xc6)[0x7f3b32f538f6]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_reqh_services_prepare_to_stop+0x6e)[0x7f3b32f5212e]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_reqh_shutdown_wait+0x11)[0x7f3b32f52221]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(+0x3e0f0e)[0x7f3b32ef8f0e]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_module_fini+0x70)[0x7f3b32f2b7d0]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_cs_fini+0x2d)[0x7f3b32efcd3d]
/home/520428/work/cortx-motr/motr/.libs/lt-m0d[0x4011cc]
/lib64/libc.so.6(__libc_start_main+0xf3)[0x7f3b30daa4a3]
/home/520428/work/cortx-motr/motr/.libs/lt-m0d[0x40145e]
----- 1677487 stopped --------
----- 1677997 stopping--------lt-m0d: got signal 1
motr[1677997]:  e810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1677997]:  e810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1677997]:  e810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1677997]:  e810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1677997]:  e810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1677997]:  e810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1677997]:  e810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1677997]:  e810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1677997]:  e810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1677997]:  e810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1677997]:  e810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1677997]:  e810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1677997]:  e810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1677997]:  e810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1677997]:  e810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1677997]:  e810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1677997]:  e810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1677997]:  e810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1677997]:  e810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1677997]:  e810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
----- 1677997 stopped --------
----- 1678071 stopping--------lt-m0d: got signal 1
motr[1678071]:  d810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x1afbc20 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=98 delivered=98 next=98 assign=124)
motr[1678071]:  d810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=50 hl=0x1afbc20 ep=10.230.240.203@tcp:12345:34:1
motr[1678071]:  da70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1678071 stopped --------
----- 1678142 stopping--------lt-m0d: got signal 1
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:2> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:2> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:2> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:2> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:2> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:2> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:2> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:3> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:3> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:3> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:3> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:3> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:3> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:3> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:0> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:0> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:0> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:0> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:0> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:0> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:1> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:1> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:1> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:1> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:1> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:1> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:1> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:2> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:2> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:2> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:2> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:2> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:2> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:2> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:3> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:3> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:3> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:3> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:3> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:3> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:0> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:1> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:1> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:1> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:1> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:1> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:1> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:1> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:3> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:0> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:0> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:0> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:0> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:0> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:0> not found in cache
motr[1678142]:  8020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:0> not found in cache
motr[1678142]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000c:0> not found in cache
motr[1678142]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000002:0> not found in cache
motr[1678142]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000003:7> not found in cache
motr[1678142]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000c:0> not found in cache
motr[1678142]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000002:0> not found in cache
motr[1678142]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000003:7> not found in cache
motr[1678142]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000c:0> not found in cache
motr[1678142]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000002:0> not found in cache
motr[1678142]:  aa40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000003:7> not found in cache
motr[1678142]:  7810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0xc2efb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=66 delivered=66 next=66 assign=100)
motr[1678142]:  7810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=36 hl=0xc2efb0 ep=10.230.240.203@tcp:12345:34:1
motr[1678142]:  7a70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1678142 stopped --------
----- 1678150 stopping--------lt-m0d: got signal 1
motr[1678150]:  9810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x1579fb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=64 delivered=64 next=64 assign=118)
motr[1678150]:  9810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=33 hl=0x1579fb0 ep=10.230.240.203@tcp:12345:34:1
motr[1678150]:  9a70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1678150 stopped --------
----- 1678152 stopping--------lt-m0d: got signal 1
motr[1678152]:  d810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x23e5fb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=64 delivered=64 next=64 assign=132)
motr[1678152]:  d810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=32 hl=0x23e5fb0 ep=10.230.240.203@tcp:12345:34:1
motr[1678152]:  da70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1678152 stopped --------
----- 1678156 stopping--------lt-m0d: got signal 1
motr[1678156]:  c810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x2153fb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=64 delivered=64 next=64 assign=146)
motr[1678156]:  c810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=32 hl=0x2153fb0 ep=10.230.240.203@tcp:12345:34:1
motr[1678156]:  ca70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1678156 stopped --------
Motr services stopped.
Testing RMW in Replicaetd layout
n k s p:3 2 2 20
vm.max_map_count = 30000000
motr_service_start: (N,K,S,P)=(1,2,2,20) nr_ios=4 multiple_pools=0
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00338732 s, 310 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00254604 s, 412 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00189119 s, 554 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00191923 s, 546 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00630285 s, 166 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00187299 s, 560 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00292624 s, 358 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00211133 s, 497 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00164628 s, 637 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00631249 s, 166 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.001653 s, 634 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00441246 s, 238 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00383327 s, 274 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00210947 s, 497 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00143758 s, 729 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00263744 s, 398 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00205093 s, 511 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00488078 s, 215 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00448263 s, 234 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00177703 s, 590 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00139488 s, 752 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00138562 s, 757 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00175306 s, 598 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00511633 s, 205 MB/s

[152:
  {0x74| ((^t|1:0), 1, (11, 22), ^o|2:9, ^v|1:20, 1,
          [1: "20 1 2 2"],
          [1: ^n|1:2],
          [1: ^S|1:6],
          [3: ^o|1:9, ^o|20:1, ^o|2:9],
          [1: ^p|1:0], [0])},
  {0x70| ((^p|1:0), [3: ^o|1:9, ^o|20:1, ^o|2:9])},
  {0x6e| ((^n|1:2), 16000, 2, 3, 2, [8: ^r|1:100, ^r|1:0, ^r|1:1, ^r|1:2, ^r|1:3, ^r|1:4, ^r|1:5, ^r|1:6])},

  {0x72| ((^r|1:100), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:1", [1: ^s|1:101])},
  {0x72| ((^r|1:0), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:900", [8: ^s|1:0, ^s|11:0, ^s|6:0, ^s|7:0,                     ^s|3:0, ^s|13:0, ^s|15:0, ^s|16:0])},
  {0x72| ((^r|1:1), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:901", [8: ^s|1:1, ^s|11:1, ^s|6:1, ^s|7:1,                     ^s|3:1, ^s|13:1, ^s|15:1, ^s|16:1])},
  {0x72| ((^r|1:2), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:902", [8: ^s|1:2, ^s|11:2, ^s|6:2, ^s|7:2,                     ^s|3:2, ^s|13:2, ^s|15:2, ^s|16:2])},
  {0x72| ((^r|1:3), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:903", [8: ^s|1:3, ^s|11:3, ^s|6:3, ^s|7:3,                     ^s|3:3, ^s|13:3, ^s|15:3, ^s|16:3])},
 {0x72| ((^r|1:4), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:800", [3: ^s|2:0, ^s|12:0, ^s|3:4])},
 {0x72| ((^r|1:5), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:34:1",
                          [3: ^s|1:6, ^s|1:7, ^s|3:6])},
 {0x72| ((^r|1:6), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:100",
                          [2: ^s|8:0, ^s|3:7])},
  {0x73| ((^s|8:0), @M0_CST_CONFD, [1: "10.230.240.203@tcp:12345:33:100"], [0], [0])},
  {0x73| ((^s|1:6), @M0_CST_HA, [1: "10.230.240.203@tcp:12345:34:1"], [0], [0])},
  {0x73| ((^s|1:7), @M0_CST_FIS, [1: "10.230.240.203@tcp:12345:34:1"], [0], [0])},
  {0x73| ((^s|1:101), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:1"], [0], [0])},




 {0x73| ((^s|2:0), @M0_CST_MDS, [1: "10.230.240.203@tcp:12345:33:800"], [0], [0])},
  {0x73| ((^s|12:0), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:800"], [0], [0])},
  {0x73| ((^s|3:4), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:800"], [0], [0])},

  {0x73| ((^s|1:0), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:900"], [0], [5: ^d|1:1, ^d|1:2, ^d|1:3, ^d|1:4, ^d|1:5])},
  {0x73| ((^s|11:0), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
 {0x73| ((^s|6:0), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
  {0x73| ((^s|7:0), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
 {0x73| ((^s|3:0), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
 {0x73| ((^s|13:0), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:900"], [0], [1: ^d|20:0])},
  {0x73| ((^s|15:0), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
  {0x73| ((^s|16:0), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
  {0x73| ((^s|1:1), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:901"], [0], [5: ^d|1:6, ^d|1:7, ^d|1:8, ^d|1:9, ^d|1:10])},
  {0x73| ((^s|11:1), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
 {0x73| ((^s|6:1), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
  {0x73| ((^s|7:1), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
 {0x73| ((^s|3:1), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
 {0x73| ((^s|13:1), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:901"], [0], [1: ^d|20:1])},
  {0x73| ((^s|15:1), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
  {0x73| ((^s|16:1), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
  {0x73| ((^s|1:2), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:902"], [0], [5: ^d|1:11, ^d|1:12, ^d|1:13, ^d|1:14, ^d|1:15])},
  {0x73| ((^s|11:2), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
 {0x73| ((^s|6:2), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
  {0x73| ((^s|7:2), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
 {0x73| ((^s|3:2), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
 {0x73| ((^s|13:2), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:902"], [0], [1: ^d|20:2])},
  {0x73| ((^s|15:2), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
  {0x73| ((^s|16:2), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
  {0x73| ((^s|1:3), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:903"], [0], [5: ^d|1:16, ^d|1:17, ^d|1:18, ^d|1:19, ^d|1:20])},
  {0x73| ((^s|11:3), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
 {0x73| ((^s|6:3), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
  {0x73| ((^s|7:3), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
 {0x73| ((^s|3:3), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
 {0x73| ((^s|13:3), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:903"], [0], [1: ^d|20:3])},
  {0x73| ((^s|15:3), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
  {0x73| ((^s|16:3), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},

 {0x73| ((^s|3:6), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:34:1"], [0], [0])},
 {0x73| ((^s|3:7), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:100"], [0], [0])},
  {0x64| ((^d|1:1), 0, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop1")},
 {0x6b| ((^k|1:1), ^d|1:1, [1: ^v|1:10])},
 {0x6a| ((^j|1:1), ^k|1:1, [0])},
 {0x64| ((^d|1:2), 1, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop2")},
 {0x6b| ((^k|1:2), ^d|1:2, [1: ^v|1:10])},
 {0x6a| ((^j|1:2), ^k|1:2, [0])},
 {0x64| ((^d|1:3), 2, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop3")},
 {0x6b| ((^k|1:3), ^d|1:3, [1: ^v|1:10])},
 {0x6a| ((^j|1:3), ^k|1:3, [0])},
 {0x64| ((^d|1:4), 3, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop4")},
 {0x6b| ((^k|1:4), ^d|1:4, [1: ^v|1:10])},
 {0x6a| ((^j|1:4), ^k|1:4, [0])},
 {0x64| ((^d|1:5), 4, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop5")},
 {0x6b| ((^k|1:5), ^d|1:5, [1: ^v|1:10])},
 {0x6a| ((^j|1:5), ^k|1:5, [0])},
 {0x64| ((^d|1:6), 5, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop6")},
 {0x6b| ((^k|1:6), ^d|1:6, [1: ^v|1:10])},
 {0x6a| ((^j|1:6), ^k|1:6, [0])},
 {0x64| ((^d|1:7), 6, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop7")},
 {0x6b| ((^k|1:7), ^d|1:7, [1: ^v|1:10])},
 {0x6a| ((^j|1:7), ^k|1:7, [0])},
 {0x64| ((^d|1:8), 7, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop8")},
 {0x6b| ((^k|1:8), ^d|1:8, [1: ^v|1:10])},
 {0x6a| ((^j|1:8), ^k|1:8, [0])},
 {0x64| ((^d|1:9), 8, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop9")},
 {0x6b| ((^k|1:9), ^d|1:9, [1: ^v|1:10])},
 {0x6a| ((^j|1:9), ^k|1:9, [0])},
 {0x64| ((^d|1:10), 9, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop10")},
 {0x6b| ((^k|1:10), ^d|1:10, [1: ^v|1:10])},
 {0x6a| ((^j|1:10), ^k|1:10, [0])},
 {0x64| ((^d|1:11), 10, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop11")},
 {0x6b| ((^k|1:11), ^d|1:11, [1: ^v|1:10])},
 {0x6a| ((^j|1:11), ^k|1:11, [0])},
 {0x64| ((^d|1:12), 11, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop12")},
 {0x6b| ((^k|1:12), ^d|1:12, [1: ^v|1:10])},
 {0x6a| ((^j|1:12), ^k|1:12, [0])},
 {0x64| ((^d|1:13), 12, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop13")},
 {0x6b| ((^k|1:13), ^d|1:13, [1: ^v|1:10])},
 {0x6a| ((^j|1:13), ^k|1:13, [0])},
 {0x64| ((^d|1:14), 13, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop14")},
 {0x6b| ((^k|1:14), ^d|1:14, [1: ^v|1:10])},
 {0x6a| ((^j|1:14), ^k|1:14, [0])},
 {0x64| ((^d|1:15), 14, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop15")},
 {0x6b| ((^k|1:15), ^d|1:15, [1: ^v|1:10])},
 {0x6a| ((^j|1:15), ^k|1:15, [0])},
 {0x64| ((^d|1:16), 15, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop16")},
 {0x6b| ((^k|1:16), ^d|1:16, [1: ^v|1:10])},
 {0x6a| ((^j|1:16), ^k|1:16, [0])},
 {0x64| ((^d|1:17), 16, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop17")},
 {0x6b| ((^k|1:17), ^d|1:17, [1: ^v|1:10])},
 {0x6a| ((^j|1:17), ^k|1:17, [0])},
 {0x64| ((^d|1:18), 17, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop18")},
 {0x6b| ((^k|1:18), ^d|1:18, [1: ^v|1:10])},
 {0x6a| ((^j|1:18), ^k|1:18, [0])},
 {0x64| ((^d|1:19), 18, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop19")},
 {0x6b| ((^k|1:19), ^d|1:19, [1: ^v|1:10])},
 {0x6a| ((^j|1:19), ^k|1:19, [0])},
 {0x64| ((^d|1:20), 19, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop20")},
 {0x6b| ((^k|1:20), ^d|1:20, [1: ^v|1:10])},
 {0x6a| ((^j|1:20), ^k|1:20, [0])},
  {0x53| ((^S|1:6), [1: ^a|1:6], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x61| ((^a|1:6), [1: ^e|1:7], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x65| ((^e|1:7), ^n|1:2, [1: ^c|1:8], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x63| ((^c|1:8), [24: ^k|1:1, ^k|1:2, ^k|1:3, ^k|1:4, ^k|1:5, ^k|1:6, ^k|1:7, ^k|1:8, ^k|1:9, ^k|1:10, ^k|1:11, ^k|1:12, ^k|1:13, ^k|1:14, ^k|1:15, ^k|1:16, ^k|1:17, ^k|1:18, ^k|1:19, ^k|1:20, ^k|20:0, ^k|20:1, ^k|20:2, ^k|20:3],
                            [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x6f| ((^o|1:9), 0, [3: ^v|1:10, ^v|0x40000000000001:11, ^v|0x40000000000001:12])},
  {0x76| ((^v|1:10), {0| (1, 2,
                                            2, 20,
                                            [5: 0, 0, 0, 0, 2],
                                            [1: ^j|1:21])})},
  {0x76| ((^v|0x40000000000001:11), {1| (0, ^v|1:10, [5: 0, 0, 0, 0, 1])})},
  {0x76| ((^v|0x40000000000001:12), {1| (1, ^v|1:10, [5: 0, 0, 0, 0, 2])})},
  {0x6a| ((^j|1:21), ^S|1:6, [1: ^j|1:22])},
  {0x6a| ((^j|1:22), ^a|1:6, [1: ^j|1:23])},
  {0x6a| ((^j|1:23), ^e|1:7, [1: ^j|1:24])},
  {0x6a| ((^j|1:24), ^c|1:8, [20: ^j|1:1, ^j|1:2, ^j|1:3, ^j|1:4, ^j|1:5, ^j|1:6, ^j|1:7, ^j|1:8, ^j|1:9, ^j|1:10, ^j|1:11, ^j|1:12, ^j|1:13, ^j|1:14, ^j|1:15, ^j|1:16, ^j|1:17, ^j|1:18, ^j|1:19, ^j|1:20])} ,
{0x6f| ((^o|2:9), 0, [1: ^v|2:10])},
{0x76| ((^v|2:10), {0| (4, 0, 0, 4, [5: 0, 0, 0, 0, 1], [1: ^j|2:21])})},
{0x6a| ((^j|2:21), ^S|1:6, [1: ^j|2:22])},
{0x6a| ((^j|2:22), ^a|1:6, [1: ^j|2:23])},
{0x6a| ((^j|2:23), ^e|1:7, [1: ^j|2:24])},
{0x6a| ((^j|2:24), ^c|1:8, [4: ^j|2:1, ^j|2:6, ^j|2:11, ^j|2:16])},
{0x6a| ((^j|2:1), ^k|1:1, [0])},
 {0x6a| ((^j|2:6), ^k|1:6, [0])},
 {0x6a| ((^j|2:11), ^k|1:11, [0])},
 {0x6a| ((^j|2:16), ^k|1:16, [0])}  ,
{0x64| ((^d|20:0), 20, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop25")},
{0x6b| ((^k|20:0), ^d|20:0, [1: ^v|1:20])},
{0x6a| ((^j|20:100), ^k|20:0, [0])},
{0x64| ((^d|20:1), 21, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop26")},
{0x6b| ((^k|20:1), ^d|20:1, [1: ^v|1:20])},
{0x6a| ((^j|20:101), ^k|20:1, [0])},
{0x64| ((^d|20:2), 22, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop27")},
{0x6b| ((^k|20:2), ^d|20:2, [1: ^v|1:20])},
{0x6a| ((^j|20:102), ^k|20:2, [0])},
{0x64| ((^d|20:3), 23, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop28")},
{0x6b| ((^k|20:3), ^d|20:3, [1: ^v|1:20])},
{0x6a| ((^j|20:103), ^k|20:3, [0])},
{0x6f| ((^o|20:1), 0, [1: ^v|1:20])},
{0x76| ((^v|1:20), {0| (1, 1, 1,
                                                    4,
                                                    [5: 0, 0, 0, 0, 1],
                                                    [1: ^j|20:1])})},
{0x6a| ((^j|20:1), ^S|1:6, [1: ^j|20:2])},
{0x6a| ((^j|20:2), ^a|1:6, [1: ^j|20:3])},
{0x6a| ((^j|20:3), ^e|1:7, [1: ^j|20:4])},
{0x6a| ((^j|20:4), ^c|1:8, [4:  ^j|20:100, ^j|20:101, ^j|20:102, ^j|20:103])}]
cd /var/motr/root/sandbox.st-45motr-rmw/confd && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -D db -S stobs -A linuxstob:addb-stobs                       -w 20 -m 65536      -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T linux -e libfab:10.230.240.203@tcp:12345:35:1                      -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/confd && exec /home/520428/work/cortx-motr/motr/m0d -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -f '<0x7200000000000001:6>' -T linux -e libfab:10.230.240.203@tcp:12345:33:100 -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0d.log
cd /var/motr/root/sandbox.st-45motr-rmw/ha && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T ad -e libfab:10.230.240.203@tcp:12345:35:1 -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
motr[1679538]:  ebc0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
cd /var/motr/root/sandbox.st-45motr-rmw/mds1 && exec                    /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad                         -D db -S stobs -A linuxstob:addb-stobs                            -w 20 -m 65536                          -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:800          -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios1 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:900 -f '<0x7200000000000001:0>' -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios2 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:901 -f '<0x7200000000000001:1>' -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios3 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:902 -f '<0x7200000000000001:2>' -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios4 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:903 -f '<0x7200000000000001:3>' -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ha && exec /home/520428/work/cortx-motr/motr/m0d -D db -S stobs -A linuxstob:addb-stobs                              -w 20 -m 65536              -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T ad -e libfab:10.230.240.203@tcp:12345:34:1                 -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc -f '<0x7200000000000001:5>'  -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
motr[1680049]:  2530  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
Motr HA agent started.
cd /var/motr/root/sandbox.st-45motr-rmw/mds1 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:800 -f '<0x7200000000000001:4>' -H 10.230.240.203@tcp:12345:34:1 -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0d.log
motr[1680123]:  c5b0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
cd /var/motr/root/sandbox.st-45motr-rmw/ios1 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:900 -f '<0x7200000000000001:0>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios2 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:901 -f '<0x7200000000000001:1>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios3 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:902 -f '<0x7200000000000001:2>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios4 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:903 -f '<0x7200000000000001:3>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
Motr confd started.
Motr mdservices started.
motr[1680203]:  cda0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
motr[1680210]:  c0f0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
motr[1680194]:  e190  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
motr[1680205]:  fcf0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
Motr ioservices started.
motr service started
*** m0dixinit is omitted. Mkfs creates meta indices now.
Testing with layout ID 3 in Healthy mode
m0cp
motr[1679538]:  ea30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1680496]:  f210  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cp update
motr[1679538]:  ea30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1680524]:  e2b0  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cat
pver is : <0:0>
motr[1679538]:  ea30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1680552]:  c0e0  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0unlink
motr[1679538]:  ea30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1680580]:  b1f0  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
diff
Testing with layout ID 5 in Healthy mode
m0cp
motr[1679538]:  ea30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1680614]:  a790  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cp update
motr[1679538]:  ea30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1680642]:  6980  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cat
pver is : <0:0>
motr[1679538]:  ea30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1680670]:  6b60  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0unlink
motr[1679538]:  ea30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1680700]:  2a20  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
diff
Testing with layout ID 7 in Healthy mode
m0cp
motr[1679538]:  ea30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1680734]:  2d90  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cp update
motr[1679538]:  ea30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1680762]:  4a80  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cat
pver is : <0:0>
motr[1679538]:  ea30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1680790]:  3f00  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0unlink
motr[1679538]:  ea30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1680818]:  fb50  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
diff
Testing with layout ID 9 in Healthy mode
m0cp
motr[1679538]:  ea30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1680852]:  fef0  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cp update
motr[1679538]:  ea30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1680880]:   370  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cat
pver is : <0:0>
motr[1679538]:  ea30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1680908]:  63c0  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0unlink
motr[1679538]:  ea30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1680936]:  13c0  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
diff
Testing with layout ID 11 in Healthy mode
m0cp
motr[1679538]:  ea30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1680970]:  67c0  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cp update
motr[1679538]:  ea30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1680998]:  2100  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0cat
pver is : <0:0>
motr[1679538]:  ea30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1681026]:  ebf0  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
m0unlink
motr[1679538]:  ea30  ERROR  [conf/confd_fom.c:287:confx_populate]  <! rc=-2 origin=<7200000000000000:0>
motr[1681054]:   dc0  ERROR  [conf/helpers.c:532:m0_conf_process2service_get]  <! rc=-2 process=<7200000000000000:0>
diff
stopping /home/520428/work/cortx-motr/motr/m0d processes...
=== pids of services: 1679538 1680049 1680123 1680194 1680203 1680205 1680210 ===
Shutting down services one by one. mdservice is the last.
----- 1679538 stopping--------lt-m0d: got signal 1
motr[1679538]:  f2c0  FATAL  [lib/assert.c:50:m0_panic]  panic: (!m0_clink_is_armed(&revc->rcf_disc_wait)) at m0_rpc_service_reverse_session_put() (rpc/service.c:241)  [git: 2.0.0-895-5-gdc9f62213] /var/motr/root/sandbox.st-45motr-rmw/confd/m0trace.1679538.2022-08-16-21:11:18
Motr panic: (!m0_clink_is_armed(&revc->rcf_disc_wait)) at m0_rpc_service_reverse_session_put() rpc/service.c:241 (errno: 4) (last failed: none) [git: 2.0.0-895-5-gdc9f62213] pid: 1679538  /var/motr/root/sandbox.st-45motr-rmw/confd/m0trace.1679538.2022-08-16-21:11:18
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_arch_backtrace+0x20)[0x7f3e83dc9a60]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_arch_panic+0xdf)[0x7f3e83dc9c0f]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_panic+0x127)[0x7f3e83db7bf7]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(+0x45fe1e)[0x7f3e83e56e1e]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_rpc_service_reverse_sessions_cleanup+0x9f)[0x7f3e83e56ebf]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(+0x45ffdc)[0x7f3e83e56fdc]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_reqh_service_prepare_to_stop+0xc6)[0x7f3e83e328f6]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_reqh_services_prepare_to_stop+0x6e)[0x7f3e83e3112e]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_reqh_shutdown_wait+0x11)[0x7f3e83e31221]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(+0x3e0f0e)[0x7f3e83dd7f0e]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_module_fini+0x70)[0x7f3e83e0a7d0]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_cs_fini+0x2d)[0x7f3e83ddbd3d]
/home/520428/work/cortx-motr/motr/.libs/lt-m0d[0x4011cc]
/lib64/libc.so.6(__libc_start_main+0xf3)[0x7f3e81c894a3]
/home/520428/work/cortx-motr/motr/.libs/lt-m0d[0x40145e]
----- 1679538 stopped --------
----- 1680049 stopping--------lt-m0d: got signal 1
motr[1680049]:  f810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1680049]:  f810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1680049]:  f810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1680049]:  f810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1680049]:  f810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1680049]:  f810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1680049]:  f810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1680049]:  f810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1680049]:  f810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1680049]:  f810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1680049]:  f810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1680049]:  f810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1680049]:  f810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1680049]:  f810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1680049]:  f810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1680049]:  f810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1680049]:  f810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1680049]:  f810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1680049]:  f810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
motr[1680049]:  f810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:34:101
----- 1680049 stopped --------
----- 1680123 stopping--------lt-m0d: got signal 1
motr[1680123]:  9810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x1d83c20 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=98 delivered=98 next=98 assign=124)
motr[1680123]:  9810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=50 hl=0x1d83c20 ep=10.230.240.203@tcp:12345:34:1
motr[1680123]:  9a70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1680123 stopped --------
----- 1680194 stopping--------lt-m0d: got signal 1
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:2> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:2> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:2> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:3> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:3> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:3> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:3> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:1> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:1> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:1> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:1> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:1> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:1> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:0> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:0> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:0> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:2> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:2> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:2> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:2> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:2> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:2> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:2> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:3> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:3> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:3> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:3> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:3> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:3> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:3> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:1> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:1> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:1> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:1> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:1> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:1> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:1> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:0> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:0> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:0> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:0> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:2> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:2> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:2> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:2> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:3> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:3> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:3> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:1> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:0> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:0> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:0> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:0> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:0> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:0> not found in cache
motr[1680194]:  6020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:0> not found in cache
motr[1680194]:  8a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000c:0> not found in cache
motr[1680194]:  8a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000002:0> not found in cache
motr[1680194]:  8a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000003:7> not found in cache
motr[1680194]:  8a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000c:0> not found in cache
motr[1680194]:  8a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000002:0> not found in cache
motr[1680194]:  8a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000003:7> not found in cache
motr[1680194]:  8a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000c:0> not found in cache
motr[1680194]:  8a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000002:0> not found in cache
motr[1680194]:  8a40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000003:7> not found in cache
motr[1680194]:  e810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x18c7fb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=66 delivered=66 next=66 assign=100)
motr[1680194]:  e810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=34 hl=0x18c7fb0 ep=10.230.240.203@tcp:12345:34:1
motr[1680194]:  ea70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1680194 stopped --------
----- 1680203 stopping--------lt-m0d: got signal 1
motr[1680203]:   810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0xba3fb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=64 delivered=64 next=64 assign=118)
motr[1680203]:   810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=32 hl=0xba3fb0 ep=10.230.240.203@tcp:12345:34:1
motr[1680203]:   a70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1680203 stopped --------
----- 1680205 stopping--------lt-m0d: got signal 1
motr[1680205]:  6810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x2786fb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=64 delivered=64 next=64 assign=132)
motr[1680205]:  6810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=33 hl=0x2786fb0 ep=10.230.240.203@tcp:12345:34:1
motr[1680205]:  6a70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1680205 stopped --------
----- 1680210 stopping--------lt-m0d: got signal 1
motr[1680210]:  1810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0xfe2fb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=64 delivered=64 next=64 assign=146)
motr[1680210]:  1810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=32 hl=0xfe2fb0 ep=10.230.240.203@tcp:12345:34:1
motr[1680210]:  1a70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1680210 stopped --------
Motr services stopped.
Testing RMW when update size < unit size
n k s p:3 2 2 20
vm.max_map_count = 30000000
motr_service_start: (N,K,S,P)=(1,4,4,20) nr_ios=4 multiple_pools=0
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00325897 s, 322 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0022978 s, 456 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00180343 s, 581 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00765251 s, 137 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00466387 s, 225 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00162515 s, 645 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00269046 s, 390 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0018792 s, 558 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00175086 s, 599 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00196265 s, 534 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00194335 s, 540 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00135722 s, 773 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00195538 s, 536 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0022484 s, 466 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00183893 s, 570 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00163193 s, 643 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00162279 s, 646 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00801748 s, 131 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00191455 s, 548 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00142484 s, 736 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00914315 s, 115 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00546809 s, 192 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00432439 s, 242 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00361968 s, 290 MB/s

[152:
  {0x74| ((^t|1:0), 1, (11, 22), ^o|2:9, ^v|1:20, 1,
          [1: "20 1 4 4"],
          [1: ^n|1:2],
          [1: ^S|1:6],
          [3: ^o|1:9, ^o|20:1, ^o|2:9],
          [1: ^p|1:0], [0])},
  {0x70| ((^p|1:0), [3: ^o|1:9, ^o|20:1, ^o|2:9])},
  {0x6e| ((^n|1:2), 16000, 2, 3, 2, [8: ^r|1:100, ^r|1:0, ^r|1:1, ^r|1:2, ^r|1:3, ^r|1:4, ^r|1:5, ^r|1:6])},

  {0x72| ((^r|1:100), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:1", [1: ^s|1:101])},
  {0x72| ((^r|1:0), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:900", [8: ^s|1:0, ^s|11:0, ^s|6:0, ^s|7:0,                     ^s|3:0, ^s|13:0, ^s|15:0, ^s|16:0])},
  {0x72| ((^r|1:1), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:901", [8: ^s|1:1, ^s|11:1, ^s|6:1, ^s|7:1,                     ^s|3:1, ^s|13:1, ^s|15:1, ^s|16:1])},
  {0x72| ((^r|1:2), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:902", [8: ^s|1:2, ^s|11:2, ^s|6:2, ^s|7:2,                     ^s|3:2, ^s|13:2, ^s|15:2, ^s|16:2])},
  {0x72| ((^r|1:3), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:903", [8: ^s|1:3, ^s|11:3, ^s|6:3, ^s|7:3,                     ^s|3:3, ^s|13:3, ^s|15:3, ^s|16:3])},
 {0x72| ((^r|1:4), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:800", [3: ^s|2:0, ^s|12:0, ^s|3:4])},
 {0x72| ((^r|1:5), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:34:1",
                          [3: ^s|1:6, ^s|1:7, ^s|3:6])},
 {0x72| ((^r|1:6), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:100",
                          [2: ^s|8:0, ^s|3:7])},
  {0x73| ((^s|8:0), @M0_CST_CONFD, [1: "10.230.240.203@tcp:12345:33:100"], [0], [0])},
  {0x73| ((^s|1:6), @M0_CST_HA, [1: "10.230.240.203@tcp:12345:34:1"], [0], [0])},
  {0x73| ((^s|1:7), @M0_CST_FIS, [1: "10.230.240.203@tcp:12345:34:1"], [0], [0])},
  {0x73| ((^s|1:101), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:1"], [0], [0])},




 {0x73| ((^s|2:0), @M0_CST_MDS, [1: "10.230.240.203@tcp:12345:33:800"], [0], [0])},
  {0x73| ((^s|12:0), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:800"], [0], [0])},
  {0x73| ((^s|3:4), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:800"], [0], [0])},

  {0x73| ((^s|1:0), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:900"], [0], [5: ^d|1:1, ^d|1:2, ^d|1:3, ^d|1:4, ^d|1:5])},
  {0x73| ((^s|11:0), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
 {0x73| ((^s|6:0), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
  {0x73| ((^s|7:0), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
 {0x73| ((^s|3:0), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
 {0x73| ((^s|13:0), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:900"], [0], [1: ^d|20:0])},
  {0x73| ((^s|15:0), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
  {0x73| ((^s|16:0), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
  {0x73| ((^s|1:1), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:901"], [0], [5: ^d|1:6, ^d|1:7, ^d|1:8, ^d|1:9, ^d|1:10])},
  {0x73| ((^s|11:1), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
 {0x73| ((^s|6:1), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
  {0x73| ((^s|7:1), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
 {0x73| ((^s|3:1), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
 {0x73| ((^s|13:1), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:901"], [0], [1: ^d|20:1])},
  {0x73| ((^s|15:1), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
  {0x73| ((^s|16:1), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
  {0x73| ((^s|1:2), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:902"], [0], [5: ^d|1:11, ^d|1:12, ^d|1:13, ^d|1:14, ^d|1:15])},
  {0x73| ((^s|11:2), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
 {0x73| ((^s|6:2), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
  {0x73| ((^s|7:2), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
 {0x73| ((^s|3:2), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
 {0x73| ((^s|13:2), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:902"], [0], [1: ^d|20:2])},
  {0x73| ((^s|15:2), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
  {0x73| ((^s|16:2), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
  {0x73| ((^s|1:3), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:903"], [0], [5: ^d|1:16, ^d|1:17, ^d|1:18, ^d|1:19, ^d|1:20])},
  {0x73| ((^s|11:3), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
 {0x73| ((^s|6:3), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
  {0x73| ((^s|7:3), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
 {0x73| ((^s|3:3), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
 {0x73| ((^s|13:3), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:903"], [0], [1: ^d|20:3])},
  {0x73| ((^s|15:3), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
  {0x73| ((^s|16:3), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},

 {0x73| ((^s|3:6), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:34:1"], [0], [0])},
 {0x73| ((^s|3:7), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:100"], [0], [0])},
  {0x64| ((^d|1:1), 0, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop1")},
 {0x6b| ((^k|1:1), ^d|1:1, [1: ^v|1:10])},
 {0x6a| ((^j|1:1), ^k|1:1, [0])},
 {0x64| ((^d|1:2), 1, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop2")},
 {0x6b| ((^k|1:2), ^d|1:2, [1: ^v|1:10])},
 {0x6a| ((^j|1:2), ^k|1:2, [0])},
 {0x64| ((^d|1:3), 2, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop3")},
 {0x6b| ((^k|1:3), ^d|1:3, [1: ^v|1:10])},
 {0x6a| ((^j|1:3), ^k|1:3, [0])},
 {0x64| ((^d|1:4), 3, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop4")},
 {0x6b| ((^k|1:4), ^d|1:4, [1: ^v|1:10])},
 {0x6a| ((^j|1:4), ^k|1:4, [0])},
 {0x64| ((^d|1:5), 4, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop5")},
 {0x6b| ((^k|1:5), ^d|1:5, [1: ^v|1:10])},
 {0x6a| ((^j|1:5), ^k|1:5, [0])},
 {0x64| ((^d|1:6), 5, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop6")},
 {0x6b| ((^k|1:6), ^d|1:6, [1: ^v|1:10])},
 {0x6a| ((^j|1:6), ^k|1:6, [0])},
 {0x64| ((^d|1:7), 6, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop7")},
 {0x6b| ((^k|1:7), ^d|1:7, [1: ^v|1:10])},
 {0x6a| ((^j|1:7), ^k|1:7, [0])},
 {0x64| ((^d|1:8), 7, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop8")},
 {0x6b| ((^k|1:8), ^d|1:8, [1: ^v|1:10])},
 {0x6a| ((^j|1:8), ^k|1:8, [0])},
 {0x64| ((^d|1:9), 8, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop9")},
 {0x6b| ((^k|1:9), ^d|1:9, [1: ^v|1:10])},
 {0x6a| ((^j|1:9), ^k|1:9, [0])},
 {0x64| ((^d|1:10), 9, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop10")},
 {0x6b| ((^k|1:10), ^d|1:10, [1: ^v|1:10])},
 {0x6a| ((^j|1:10), ^k|1:10, [0])},
 {0x64| ((^d|1:11), 10, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop11")},
 {0x6b| ((^k|1:11), ^d|1:11, [1: ^v|1:10])},
 {0x6a| ((^j|1:11), ^k|1:11, [0])},
 {0x64| ((^d|1:12), 11, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop12")},
 {0x6b| ((^k|1:12), ^d|1:12, [1: ^v|1:10])},
 {0x6a| ((^j|1:12), ^k|1:12, [0])},
 {0x64| ((^d|1:13), 12, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop13")},
 {0x6b| ((^k|1:13), ^d|1:13, [1: ^v|1:10])},
 {0x6a| ((^j|1:13), ^k|1:13, [0])},
 {0x64| ((^d|1:14), 13, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop14")},
 {0x6b| ((^k|1:14), ^d|1:14, [1: ^v|1:10])},
 {0x6a| ((^j|1:14), ^k|1:14, [0])},
 {0x64| ((^d|1:15), 14, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop15")},
 {0x6b| ((^k|1:15), ^d|1:15, [1: ^v|1:10])},
 {0x6a| ((^j|1:15), ^k|1:15, [0])},
 {0x64| ((^d|1:16), 15, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop16")},
 {0x6b| ((^k|1:16), ^d|1:16, [1: ^v|1:10])},
 {0x6a| ((^j|1:16), ^k|1:16, [0])},
 {0x64| ((^d|1:17), 16, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop17")},
 {0x6b| ((^k|1:17), ^d|1:17, [1: ^v|1:10])},
 {0x6a| ((^j|1:17), ^k|1:17, [0])},
 {0x64| ((^d|1:18), 17, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop18")},
 {0x6b| ((^k|1:18), ^d|1:18, [1: ^v|1:10])},
 {0x6a| ((^j|1:18), ^k|1:18, [0])},
 {0x64| ((^d|1:19), 18, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop19")},
 {0x6b| ((^k|1:19), ^d|1:19, [1: ^v|1:10])},
 {0x6a| ((^j|1:19), ^k|1:19, [0])},
 {0x64| ((^d|1:20), 19, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop20")},
 {0x6b| ((^k|1:20), ^d|1:20, [1: ^v|1:10])},
 {0x6a| ((^j|1:20), ^k|1:20, [0])},
  {0x53| ((^S|1:6), [1: ^a|1:6], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x61| ((^a|1:6), [1: ^e|1:7], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x65| ((^e|1:7), ^n|1:2, [1: ^c|1:8], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x63| ((^c|1:8), [24: ^k|1:1, ^k|1:2, ^k|1:3, ^k|1:4, ^k|1:5, ^k|1:6, ^k|1:7, ^k|1:8, ^k|1:9, ^k|1:10, ^k|1:11, ^k|1:12, ^k|1:13, ^k|1:14, ^k|1:15, ^k|1:16, ^k|1:17, ^k|1:18, ^k|1:19, ^k|1:20, ^k|20:0, ^k|20:1, ^k|20:2, ^k|20:3],
                            [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x6f| ((^o|1:9), 0, [3: ^v|1:10, ^v|0x40000000000001:11, ^v|0x40000000000001:12])},
  {0x76| ((^v|1:10), {0| (1, 4,
                                            4, 20,
                                            [5: 0, 0, 0, 0, 4],
                                            [1: ^j|1:21])})},
  {0x76| ((^v|0x40000000000001:11), {1| (0, ^v|1:10, [5: 0, 0, 0, 0, 1])})},
  {0x76| ((^v|0x40000000000001:12), {1| (1, ^v|1:10, [5: 0, 0, 0, 0, 2])})},
  {0x6a| ((^j|1:21), ^S|1:6, [1: ^j|1:22])},
  {0x6a| ((^j|1:22), ^a|1:6, [1: ^j|1:23])},
  {0x6a| ((^j|1:23), ^e|1:7, [1: ^j|1:24])},
  {0x6a| ((^j|1:24), ^c|1:8, [20: ^j|1:1, ^j|1:2, ^j|1:3, ^j|1:4, ^j|1:5, ^j|1:6, ^j|1:7, ^j|1:8, ^j|1:9, ^j|1:10, ^j|1:11, ^j|1:12, ^j|1:13, ^j|1:14, ^j|1:15, ^j|1:16, ^j|1:17, ^j|1:18, ^j|1:19, ^j|1:20])} ,
{0x6f| ((^o|2:9), 0, [1: ^v|2:10])},
{0x76| ((^v|2:10), {0| (4, 0, 0, 4, [5: 0, 0, 0, 0, 1], [1: ^j|2:21])})},
{0x6a| ((^j|2:21), ^S|1:6, [1: ^j|2:22])},
{0x6a| ((^j|2:22), ^a|1:6, [1: ^j|2:23])},
{0x6a| ((^j|2:23), ^e|1:7, [1: ^j|2:24])},
{0x6a| ((^j|2:24), ^c|1:8, [4: ^j|2:1, ^j|2:6, ^j|2:11, ^j|2:16])},
{0x6a| ((^j|2:1), ^k|1:1, [0])},
 {0x6a| ((^j|2:6), ^k|1:6, [0])},
 {0x6a| ((^j|2:11), ^k|1:11, [0])},
 {0x6a| ((^j|2:16), ^k|1:16, [0])}  ,
{0x64| ((^d|20:0), 20, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop25")},
{0x6b| ((^k|20:0), ^d|20:0, [1: ^v|1:20])},
{0x6a| ((^j|20:100), ^k|20:0, [0])},
{0x64| ((^d|20:1), 21, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop26")},
{0x6b| ((^k|20:1), ^d|20:1, [1: ^v|1:20])},
{0x6a| ((^j|20:101), ^k|20:1, [0])},
{0x64| ((^d|20:2), 22, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop27")},
{0x6b| ((^k|20:2), ^d|20:2, [1: ^v|1:20])},
{0x6a| ((^j|20:102), ^k|20:2, [0])},
{0x64| ((^d|20:3), 23, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop28")},
{0x6b| ((^k|20:3), ^d|20:3, [1: ^v|1:20])},
{0x6a| ((^j|20:103), ^k|20:3, [0])},
{0x6f| ((^o|20:1), 0, [1: ^v|1:20])},
{0x76| ((^v|1:20), {0| (1, 1, 1,
                                                    4,
                                                    [5: 0, 0, 0, 0, 1],
                                                    [1: ^j|20:1])})},
{0x6a| ((^j|20:1), ^S|1:6, [1: ^j|20:2])},
{0x6a| ((^j|20:2), ^a|1:6, [1: ^j|20:3])},
{0x6a| ((^j|20:3), ^e|1:7, [1: ^j|20:4])},
{0x6a| ((^j|20:4), ^c|1:8, [4:  ^j|20:100, ^j|20:101, ^j|20:102, ^j|20:103])}]
cd /var/motr/root/sandbox.st-45motr-rmw/confd && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -D db -S stobs -A linuxstob:addb-stobs                       -w 20 -m 65536      -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T linux -e libfab:10.230.240.203@tcp:12345:35:1                      -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/confd && exec /home/520428/work/cortx-motr/motr/m0d -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -f '<0x7200000000000001:6>' -T linux -e libfab:10.230.240.203@tcp:12345:33:100 -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0d.log
cd /var/motr/root/sandbox.st-45motr-rmw/ha && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T ad -e libfab:10.230.240.203@tcp:12345:35:1 -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
motr[1681419]:  3bb0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
cd /var/motr/root/sandbox.st-45motr-rmw/mds1 && exec                    /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad                         -D db -S stobs -A linuxstob:addb-stobs                            -w 20 -m 65536                          -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:800          -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios1 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:900 -f '<0x7200000000000001:0>' -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios2 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:901 -f '<0x7200000000000001:1>' -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios3 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:902 -f '<0x7200000000000001:2>' -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios4 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:903 -f '<0x7200000000000001:3>' -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/root/sandbox.st-45motr-rmw/ha && exec /home/520428/work/cortx-motr/motr/m0d -D db -S stobs -A linuxstob:addb-stobs                              -w 20 -m 65536              -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T ad -e libfab:10.230.240.203@tcp:12345:34:1                 -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc -f '<0x7200000000000001:5>'  -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
motr[1681929]:  8ac0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
Motr HA agent started.
cd /var/motr/root/sandbox.st-45motr-rmw/mds1 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:800 -f '<0x7200000000000001:4>' -H 10.230.240.203@tcp:12345:34:1 -c /var/motr/root/sandbox.st-45motr-rmw/confd/conf.xc |& tee -a m0d.log
motr[1682003]:  a4f0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
cd /var/motr/root/sandbox.st-45motr-rmw/ios1 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:900 -f '<0x7200000000000001:0>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios2 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:901 -f '<0x7200000000000001:1>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios3 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:902 -f '<0x7200000000000001:2>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/root/sandbox.st-45motr-rmw/ios4 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 20 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:903 -f '<0x7200000000000001:3>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
Motr confd started.
Motr mdservices started.
motr[1682089]:  bde0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
motr[1682082]:  d910  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
motr[1682074]:  9700  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
motr[1682085]:  3ee0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
Motr ioservices started.
motr service started
*** m0dixinit is omitted. Mkfs creates meta indices now.
stopping /home/520428/work/cortx-motr/motr/m0d processes...
=== pids of services: 1681419 1681929 1682003 1682074 1682082 1682085 1682089 ===
Shutting down services one by one. mdservice is the last.
----- 1681419 stopping--------lt-m0d: got signal 1
motr[1681419]:  42b0  FATAL  [lib/assert.c:50:m0_panic]  panic: (!m0_clink_is_armed(&revc->rcf_disc_wait)) at m0_rpc_service_reverse_session_put() (rpc/service.c:241)  [git: 2.0.0-895-5-gdc9f62213] /var/motr/root/sandbox.st-45motr-rmw/confd/m0trace.1681419.2022-08-16-21:14:30
Motr panic: (!m0_clink_is_armed(&revc->rcf_disc_wait)) at m0_rpc_service_reverse_session_put() rpc/service.c:241 (errno: 4) (last failed: none) [git: 2.0.0-895-5-gdc9f62213] pid: 1681419  /var/motr/root/sandbox.st-45motr-rmw/confd/m0trace.1681419.2022-08-16-21:14:30
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_arch_backtrace+0x20)[0x7f8640eaea60]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_arch_panic+0xdf)[0x7f8640eaec0f]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_panic+0x127)[0x7f8640e9cbf7]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(+0x45fe1e)[0x7f8640f3be1e]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_rpc_service_reverse_sessions_cleanup+0x9f)[0x7f8640f3bebf]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(+0x45ffdc)[0x7f8640f3bfdc]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_reqh_service_prepare_to_stop+0xc6)[0x7f8640f178f6]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_reqh_services_prepare_to_stop+0x6e)[0x7f8640f1612e]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_reqh_shutdown_wait+0x11)[0x7f8640f16221]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(+0x3e0f0e)[0x7f8640ebcf0e]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_module_fini+0x70)[0x7f8640eef7d0]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_cs_fini+0x2d)[0x7f8640ec0d3d]
/home/520428/work/cortx-motr/motr/.libs/lt-m0d[0x4011cc]
/lib64/libc.so.6(__libc_start_main+0xf3)[0x7f863ed6e4a3]
/home/520428/work/cortx-motr/motr/.libs/lt-m0d[0x40145e]
----- 1681419 stopped --------
----- 1681929 stopping--------lt-m0d: got signal 1
----- 1681929 stopped --------
----- 1682003 stopping--------lt-m0d: got signal 1
motr[1682003]:  5810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x1db9c20 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=98 delivered=98 next=98 assign=124)
motr[1682003]:  5810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=49 hl=0x1db9c20 ep=10.230.240.203@tcp:12345:34:1
motr[1682003]:  5a70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1682003 stopped --------
----- 1682074 stopping--------lt-m0d: got signal 1
motr[1682074]:  8810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x1e5afb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=64 delivered=64 next=64 assign=104)
motr[1682074]:  8810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=32 hl=0x1e5afb0 ep=10.230.240.203@tcp:12345:34:1
motr[1682074]:  8a70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1682074 stopped --------
----- 1682082 stopping--------lt-m0d: got signal 1
motr[1682082]:  7810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x1224fb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=64 delivered=64 next=64 assign=118)
motr[1682082]:  7810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=33 hl=0x1224fb0 ep=10.230.240.203@tcp:12345:34:1
motr[1682082]:  7a70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1682082 stopped --------
----- 1682085 stopping--------lt-m0d: got signal 1
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:3> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:3> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:3> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:3> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:3> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:3> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:3> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:3> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:3> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:3> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:3> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:3> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:3> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:3> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:2> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:2> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:2> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:2> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:2> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:2> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:2> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:2> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:2> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:2> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:2> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:2> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:2> not found in cache
motr[1682085]:  c020   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:2> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:0> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:0> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000003:7> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:0> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:0> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:1> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:1> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:1> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:1> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000c:0> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000002:0> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:1> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:1> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:1> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:0> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:0> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:0> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:0> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:0> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000003:7> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:0> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:0> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:1> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:1> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:1> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:1> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000c:0> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000002:0> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:1> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:1> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:0> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:1> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:0> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:0> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:0> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:0> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:0> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:0> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000003:7> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000010:1> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000007:1> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000b:1> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:1> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000c:0> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000002:0> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:1> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:1> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000f:0> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000001:1> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <730000000000000d:0> not found in cache
motr[1682085]:  ea40   WARN  [rpc/conn.c:361:m0_rpc_conn2svc]  <7300000000000006:0> not found in cache
motr[1682085]:  4810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x279dfb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=66 delivered=66 next=66 assign=100)
motr[1682085]:  4810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=34 hl=0x279dfb0 ep=10.230.240.203@tcp:12345:34:1
motr[1682085]:  4a70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1682085 stopped --------
----- 1682089 stopping--------lt-m0d: got signal 1
motr[1682089]:  8810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x16c4fb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=64 delivered=64 next=64 assign=146)
motr[1682089]:  8810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=32 hl=0x16c4fb0 ep=10.230.240.203@tcp:12345:34:1
motr[1682089]:  8a70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1682089 stopped --------
Motr services stopped.
motr_rmw_IO: test status: SUCCESS

real    9m16.985s
user    5m55.477s
sys     2m27.952s
bash-4.4$
bash-4.4$
bash-4.4$
bash-4.4$
bash-4.4$
bash-4.4$

@rkothiya
Copy link
Contributor

Jenkins CI Result : Motr#1593

Motr Test Summary

Test ResultCountInfo
❌Failed2
📁

04motr-single-node/49motr-rpc-cancel
01motr-single-node/00userspace-tests

🏁Skipped32
📁

01motr-single-node/28sys-kvs
01motr-single-node/35m0singlenode
01motr-single-node/04initscripts
01motr-single-node/37protocol
02motr-single-node/51kem
02motr-single-node/20rpc-session-cancel
02motr-single-node/10pver-assign
02motr-single-node/21fsync-single-node
02motr-single-node/13dgmode-io
02motr-single-node/14poolmach
02motr-single-node/11m0t1fs
02motr-single-node/26motr-user-kernel-tests
02motr-single-node/08spiel
03motr-single-node/06conf
03motr-single-node/36spare-reservation
04motr-single-node/34sns-repair-1n-1f
04motr-single-node/08spiel-sns-repair-quiesce
04motr-single-node/28sys-kvs-kernel
04motr-single-node/11m0t1fs-rconfc-fail
04motr-single-node/08spiel-sns-repair
04motr-single-node/19sns-repair-abort
04motr-single-node/22sns-repair-ios-fail
05motr-single-node/18sns-repair-quiesce
05motr-single-node/12fwait
05motr-single-node/16sns-repair-multi
05motr-single-node/07mount-fail
05motr-single-node/15sns-repair-single
05motr-single-node/23sns-abort-quiesce
05motr-single-node/17sns-repair-concurrent-io
05motr-single-node/07mount
05motr-single-node/07mount-multiple
05motr-single-node/12fsync

✔️Passed42
📁

01motr-single-node/43m0crate
01motr-single-node/05confgen
01motr-single-node/06hagen
01motr-single-node/52motr-singlenode-sanity
01motr-single-node/01net
01motr-single-node/01kernel-tests
01motr-single-node/03console
01motr-single-node/02rpcping
02motr-single-node/07m0d-fatal
02motr-single-node/67fdmi-plugin-multi-filters
02motr-single-node/53clusterusage-alert
02motr-single-node/41motr-conf-update
03motr-single-node/61sns-repair-motr-1n-1f
03motr-single-node/72spiel-sns-motr-repair-quiesce
03motr-single-node/08spiel-multi-confd
03motr-single-node/69sns-repair-motr-quiesce
03motr-single-node/62sns-repair-motr-mf
03motr-single-node/70sns-failure-after-repair-quiesce
03motr-single-node/63sns-repair-motr-1k-1f
03motr-single-node/60sns-repair-motr-1f
03motr-single-node/66sns-repair-motr-abort-quiesce
03motr-single-node/24motr-dix-repair-lookup-insert-spiel
03motr-single-node/68sns-repair-motr-shutdown
03motr-single-node/64sns-repair-motr-ios-fail
03motr-single-node/71spiel-sns-motr-repair
03motr-single-node/24motr-dix-repair-lookup-insert-m0repair
03motr-single-node/04sss
03motr-single-node/65sns-repair-motr-abort
04motr-single-node/73motr-io-small-disks
04motr-single-node/48motr-raid0-io
04motr-single-node/25m0kv
04motr-single-node/44motr-rm-lock-cc-io
04motr-single-node/45motr-rmw
05motr-single-node/23dix-repair-m0repair
05motr-single-node/43motr-sync-replication
05motr-single-node/42motr-utils
05motr-single-node/45motr-sns-repair-N-1
05motr-single-node/40motr-dgmode
05motr-single-node/23dix-repair-quiesce-m0repair
05motr-single-node/23spiel-dix-repair-quiesce
05motr-single-node/44motr-sns-repair
05motr-single-node/23spiel-dix-repair

Total76🔗

CppCheck Summary

   Cppcheck: No new warnings found 👍

@rkothiya rkothiya merged commit 6abf2c5 into Seagate:main Aug 17, 2022
@rkothiya
Copy link
Contributor

Jenkins CI Result : Motr#1597

Motr Test Summary

Test ResultCountInfo
❌Failed3
📁

01motr-single-node/52motr-singlenode-sanity
04motr-single-node/49motr-rpc-cancel
01motr-single-node/00userspace-tests

🏁Skipped32
📁

01motr-single-node/28sys-kvs
01motr-single-node/35m0singlenode
01motr-single-node/04initscripts
01motr-single-node/37protocol
02motr-single-node/51kem
02motr-single-node/20rpc-session-cancel
02motr-single-node/10pver-assign
02motr-single-node/21fsync-single-node
02motr-single-node/13dgmode-io
02motr-single-node/14poolmach
02motr-single-node/11m0t1fs
02motr-single-node/26motr-user-kernel-tests
02motr-single-node/08spiel
03motr-single-node/06conf
03motr-single-node/36spare-reservation
04motr-single-node/34sns-repair-1n-1f
04motr-single-node/08spiel-sns-repair-quiesce
04motr-single-node/28sys-kvs-kernel
04motr-single-node/11m0t1fs-rconfc-fail
04motr-single-node/08spiel-sns-repair
04motr-single-node/19sns-repair-abort
04motr-single-node/22sns-repair-ios-fail
05motr-single-node/18sns-repair-quiesce
05motr-single-node/12fwait
05motr-single-node/16sns-repair-multi
05motr-single-node/07mount-fail
05motr-single-node/15sns-repair-single
05motr-single-node/23sns-abort-quiesce
05motr-single-node/17sns-repair-concurrent-io
05motr-single-node/07mount
05motr-single-node/07mount-multiple
05motr-single-node/12fsync

✔️Passed41
📁

01motr-single-node/43m0crate
01motr-single-node/05confgen
01motr-single-node/06hagen
01motr-single-node/01net
01motr-single-node/01kernel-tests
01motr-single-node/03console
01motr-single-node/02rpcping
02motr-single-node/07m0d-fatal
02motr-single-node/67fdmi-plugin-multi-filters
02motr-single-node/53clusterusage-alert
02motr-single-node/41motr-conf-update
03motr-single-node/61sns-repair-motr-1n-1f
03motr-single-node/72spiel-sns-motr-repair-quiesce
03motr-single-node/08spiel-multi-confd
03motr-single-node/69sns-repair-motr-quiesce
03motr-single-node/62sns-repair-motr-mf
03motr-single-node/70sns-failure-after-repair-quiesce
03motr-single-node/63sns-repair-motr-1k-1f
03motr-single-node/60sns-repair-motr-1f
03motr-single-node/66sns-repair-motr-abort-quiesce
03motr-single-node/24motr-dix-repair-lookup-insert-spiel
03motr-single-node/68sns-repair-motr-shutdown
03motr-single-node/64sns-repair-motr-ios-fail
03motr-single-node/71spiel-sns-motr-repair
03motr-single-node/24motr-dix-repair-lookup-insert-m0repair
03motr-single-node/04sss
03motr-single-node/65sns-repair-motr-abort
04motr-single-node/73motr-io-small-disks
04motr-single-node/48motr-raid0-io
04motr-single-node/25m0kv
04motr-single-node/44motr-rm-lock-cc-io
04motr-single-node/45motr-rmw
05motr-single-node/23dix-repair-m0repair
05motr-single-node/43motr-sync-replication
05motr-single-node/42motr-utils
05motr-single-node/45motr-sns-repair-N-1
05motr-single-node/40motr-dgmode
05motr-single-node/23dix-repair-quiesce-m0repair
05motr-single-node/23spiel-dix-repair-quiesce
05motr-single-node/44motr-sns-repair
05motr-single-node/23spiel-dix-repair

Total76🔗

CppCheck Summary

   Cppcheck: No new warnings found 👍

kiwionly2 pushed a commit to kiwionly2/cortx-motr that referenced this pull request Aug 30, 2022
…eagate#1888)

In cases, e.g. disk full, the write request may return
with 'short write': the written count is less than requested count.
The assertion of requested write count equal to returned count is removed.
In this case, the rc value of this request is still 0. And next write
request may return -ENOSPC because of disk full.
This can be triggered with small disks and writing a lot of data into
the system.

Signed-off-by: Hua Huang <hua.huang@seagate.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
cla-signed Status: Waiting to be Reviewed PR is waiting for reviewers to review the PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants