Fix handling nuances of `src_mbuffer` #632

jimklimov · 2024-01-17T15:01:38Z

Follow up for PR #630 for issue #629

Example znapzendzetup list output with these fixes:

[2024-01-17 15:28:36.79858] [1624189] [info] WARNING: property 'src_mbuffer' not set on backup for rpool/ROOT, defaulting to 'off'
[2024-01-17 15:28:36.79925] [1624189] [info] WARNING: property 'src_mbuffer_size' not set on backup for rpool/ROOT, inheriting from legacy 'mbuffer_size': 128M
[2024-01-17 15:28:37.09402] [1624189] [info] WARNING: property 'dst_0_mbuffer' not set on backup for rpool/ROOT, inheriting path[:port] from legacy 'mbuffer': mbuffer
[2024-01-17 15:28:37.09426] [1624189] [info] WARNING: property 'dst_0_mbuffer_size' not set on backup for rpool/ROOT, inheriting from legacy 'mbuffer_size': 128M

...

*** backup plan: rpool/ROOT ***
           dst_0 = znapzend:pond/export/DUMP/NUTCI/znapzend/ci-deb/rpool/ROOT
   dst_0_mbuffer = mbuffer
dst_0_mbuffer_size = 128M
      dst_0_plan = 1month=>1week,1year=>1month,10years=>6months
         enabled = on
         mbuffer = mbuffer
    mbuffer_size = 128M
   post_znap_cmd = off
    pre_znap_cmd = off
       recursive = on
             src = rpool/ROOT
     src_mbuffer = off
src_mbuffer_size = 128M
        src_plan = 1month=>1week,1year=>1month,10years=>6months
        tsformat = znapzend-auto-%Y-%m-%dT%H:%M:%SZ
      zend_delay = 0

Current properties in the dataset do not convey any mbuffer customizations (setup was created before these changes):

# zfs get -s local all rpool/ROOT | sort | grep znap
rpool/ROOT  org.znapzend:dst_0          znapzend:pond/export/DUMP/NUTCI/znapzend/ci-deb/rpool/ROOT  local
rpool/ROOT  org.znapzend:dst_0_plan     1months=>1weeks,1years=>1months,10years=>6months            local
rpool/ROOT  org.znapzend:enabled        on                                                          local
rpool/ROOT  org.znapzend:mbuffer        mbuffer                                                     local
rpool/ROOT  org.znapzend:mbuffer_size   128M                                                        local
rpool/ROOT  org.znapzend:post_znap_cmd  off                                                         local
rpool/ROOT  org.znapzend:pre_znap_cmd   off                                                         local
rpool/ROOT  org.znapzend:recursive      on                                                          local
rpool/ROOT  org.znapzend:src_plan       1months=>1weeks,1years=>1months,10years=>6months            local
rpool/ROOT  org.znapzend:tsformat       znapzend-auto-%Y-%m-%dT%H:%M:%SZ                            local
rpool/ROOT  org.znapzend:zend_delay     0                                                           local

On another system, there is no local mbuffer and we state that explicitly (as "off" in both legacy and SRC); related values like buffer sizes are successfully customized:

# znapzendzetup create --recursive --mbuffer=off --mbuffersize=256M --tsformat='znapzend-auto-%Y-%m-%dT%H:%M:%SZ' SRC '1month=>1week,1year=>1month,10years=>6months' zroot off 128M DST '1month=>1week,1year=>1month,10years=>6months' znapzend:pond/export/DUMP/NUTCI/znapzend/ci-bsd/zroot off off mbuffer 1G
*** backup plan: zroot ***
           dst_0 = znapzend:pond/export/DUMP/NUTCI/znapzend/ci-bsd/zroot
   dst_0_mbuffer = mbuffer
dst_0_mbuffer_size = 1G
      dst_0_plan = 1month=>1week,1year=>1month,10years=>6months
    dst_0_precmd = off
    dst_0_pstcmd = off
         enabled = on
         mbuffer = off
    mbuffer_size = 256M
   post_znap_cmd = off
    pre_znap_cmd = off
       recursive = on
             src = zroot
     src_mbuffer = off
src_mbuffer_size = 128M
        src_plan = 1month=>1week,1year=>1month,10years=>6months
        tsformat = znapzend-auto-%Y-%m-%dT%H:%M:%SZ
      zend_delay = 0

Do you want to save this backup set [y/N]? y
NOTE: if you have modified your configuration, send a HUP signal
(pkill -HUP znapzend) to your znapzend daemon for it to notice the change.


# zfs get -s local all zroot | sort | grep znap
zroot  org.znapzend:dst_0               znapzend:pond/export/DUMP/NUTCI/znapzend/ci-bsd/zroot  local
zroot  org.znapzend:dst_0_mbuffer       mbuffer                                                local
zroot  org.znapzend:dst_0_mbuffer_size  1G                                                     local
zroot  org.znapzend:dst_0_plan          1months=>1weeks,1years=>1months,10years=>6months       local
zroot  org.znapzend:dst_0_precmd        off                                                    local
zroot  org.znapzend:dst_0_pstcmd        off                                                    local
zroot  org.znapzend:enabled             on                                                     local
zroot  org.znapzend:mbuffer             off                                                    local
zroot  org.znapzend:mbuffer_size        256M                                                   local
zroot  org.znapzend:post_znap_cmd       off                                                    local
zroot  org.znapzend:pre_znap_cmd        off                                                    local
zroot  org.znapzend:recursive           on                                                     local
zroot  org.znapzend:src_mbuffer         off                                                    local
zroot  org.znapzend:src_mbuffer_size    128M                                                   local
zroot  org.znapzend:src_plan            1months=>1weeks,1years=>1months,10years=>6months       local
zroot  org.znapzend:tsformat            znapzend-auto-%Y-%m-%dT%H:%M:%SZ                       local
zroot  org.znapzend:zend_delay          0                                                      local


# znapzendzetup list
*** backup plan: zroot ***
           dst_0 = znapzend:pond/export/DUMP/NUTCI/znapzend/ci-bsd/zroot
   dst_0_mbuffer = mbuffer
dst_0_mbuffer_size = 1G
      dst_0_plan = 1month=>1week,1year=>1month,10years=>6months
    dst_0_precmd = off
    dst_0_pstcmd = off
         enabled = on
         mbuffer = off
    mbuffer_size = 256M
   post_znap_cmd = off
    pre_znap_cmd = off
       recursive = on
             src = zroot
     src_mbuffer = off
src_mbuffer_size = 128M
        src_plan = 1month=>1week,1year=>1month,10years=>6months
        tsformat = znapzend-auto-%Y-%m-%dT%H:%M:%SZ
      zend_delay = 0

…off [oetiker#629] Signed-off-by: Jim Klimov <jimklimov@gmail.com>

…mbuffer, fall back to "off" [oetiker#629] Signed-off-by: Jim Klimov <jimklimov@gmail.com>

jimklimov · 2024-01-17T15:49:01Z

Further experiments:

not setting legacy options (mbuffer gets defaulted to off somewhere automatically) and not setting it in SRC; setting in DST though:

# znapzendzetup create --recursive --tsformat='znapzend-auto-%Y-%m-%dT%H:%M:%SZ' SRC '1month=>1week,1year=>1month,10years=>6months' zroot DST '1month=>1week,1year=>1month,10years=>6months' znapzend:pond/export/DUMP/NUTCI/znapzend/ci-bsd/zroot off off mbuffer 1G
[2024-01-17 15:41:39.18773] [49409] [info] WARNING: property 'src_mbuffer' not set on backup for zroot, inheriting 'off' from legacy 'mbuffer'
[2024-01-17 15:41:39.18809] [49409] [info] WARNING: property 'src_mbuffer_size' not set on backup for zroot, inheriting from legacy 'mbuffer_size': 1G
*** backup plan: zroot ***
           dst_0 = znapzend:pond/export/DUMP/NUTCI/znapzend/ci-bsd/zroot
   dst_0_mbuffer = mbuffer
dst_0_mbuffer_size = 1G
      dst_0_plan = 1month=>1week,1year=>1month,10years=>6months
    dst_0_precmd = off
    dst_0_pstcmd = off
         enabled = on
         mbuffer = off
    mbuffer_size = 1G
   post_znap_cmd = off
    pre_znap_cmd = off
       recursive = on
             src = zroot
     src_mbuffer = off
src_mbuffer_size = 1G
        src_plan = 1month=>1week,1year=>1month,10years=>6months
        tsformat = znapzend-auto-%Y-%m-%dT%H:%M:%SZ
      zend_delay = 0

Do you want to save this backup set [y/N]? y
NOTE: if you have modified your configuration, send a HUP signal
(pkill -HUP znapzend) to your znapzend daemon for it to notice the change.


# zfs get -s local all zroot | sort | grep znap
zroot  org.znapzend:dst_0               znapzend:pond/export/DUMP/NUTCI/znapzend/ci-bsd/zroot  local
zroot  org.znapzend:dst_0_mbuffer       mbuffer                                                local
zroot  org.znapzend:dst_0_mbuffer_size  1G                                                     local
zroot  org.znapzend:dst_0_plan          1months=>1weeks,1years=>1months,10years=>6months       local
zroot  org.znapzend:dst_0_precmd        off                                                    local
zroot  org.znapzend:dst_0_pstcmd        off                                                    local
zroot  org.znapzend:enabled             on                                                     local
zroot  org.znapzend:mbuffer             off                                                    local
zroot  org.znapzend:mbuffer_size        1G                                                     local
zroot  org.znapzend:post_znap_cmd       off                                                    local
zroot  org.znapzend:pre_znap_cmd        off                                                    local
zroot  org.znapzend:recursive           on                                                     local
zroot  org.znapzend:src_mbuffer         off                                                    local
zroot  org.znapzend:src_mbuffer_size    1G                                                     local
zroot  org.znapzend:src_plan            1months=>1weeks,1years=>1months,10years=>6months       local
zroot  org.znapzend:tsformat            znapzend-auto-%Y-%m-%dT%H:%M:%SZ                       local
zroot  org.znapzend:zend_delay          0                                                      local

legacy-style setup, with --mbuffer=mbuffer (no port part) and no mention in SRC -- src_mbuffer is decided as off per legacy default for non-networked mode:

# znapzendzetup create --recursive --mbuffer=mbuffer --mbuffersize=256M --tsformat='znapzend-auto-%Y-%m-%dT%H:%M:%SZ' SRC '1month=>1week,1year=>1month,10years=>6months' zroot DST '1month=>1week,1year=>1month,10years=>6months' znapzend:pond/export/DUMP/NUTCI/znapzend/ci-bsd/zroot off off mbuffer 1G
[2024-01-17 15:44:06.55401] [49469] [info] WARNING: property 'src_mbuffer' not set on backup for zroot, defaulting to 'off'
[2024-01-17 15:44:06.55438] [49469] [info] WARNING: property 'src_mbuffer_size' not set on backup for zroot, inheriting from legacy 'mbuffer_size': 256M
*** backup plan: zroot ***
           dst_0 = znapzend:pond/export/DUMP/NUTCI/znapzend/ci-bsd/zroot
   dst_0_mbuffer = mbuffer
dst_0_mbuffer_size = 1G
      dst_0_plan = 1month=>1week,1year=>1month,10years=>6months
    dst_0_precmd = off
    dst_0_pstcmd = off
         enabled = on
         mbuffer = mbuffer
    mbuffer_size = 256M
   post_znap_cmd = off
    pre_znap_cmd = off
       recursive = on
             src = zroot
     src_mbuffer = off
src_mbuffer_size = 256M
        src_plan = 1month=>1week,1year=>1month,10years=>6months
        tsformat = znapzend-auto-%Y-%m-%dT%H:%M:%SZ
      zend_delay = 0

Do you want to save this backup set [y/N]?
...

# zfs get -s local all zroot | sort | grep znap
zroot  org.znapzend:dst_0               znapzend:pond/export/DUMP/NUTCI/znapzend/ci-bsd/zroot  local
zroot  org.znapzend:dst_0_mbuffer       mbuffer                                                local
zroot  org.znapzend:dst_0_mbuffer_size  1G                                                     local
zroot  org.znapzend:dst_0_plan          1months=>1weeks,1years=>1months,10years=>6months       local
zroot  org.znapzend:dst_0_precmd        off                                                    local
zroot  org.znapzend:dst_0_pstcmd        off                                                    local
zroot  org.znapzend:enabled             on                                                     local
zroot  org.znapzend:mbuffer             mbuffer                                                local
zroot  org.znapzend:mbuffer_size        256M                                                   local
zroot  org.znapzend:post_znap_cmd       off                                                    local
zroot  org.znapzend:pre_znap_cmd        off                                                    local
zroot  org.znapzend:recursive           on                                                     local
zroot  org.znapzend:src_mbuffer         off                                                    local
zroot  org.znapzend:src_mbuffer_size    256M                                                   local
zroot  org.znapzend:src_plan            1months=>1weeks,1years=>1months,10years=>6months       local
zroot  org.znapzend:tsformat            znapzend-auto-%Y-%m-%dT%H:%M:%SZ                       local
zroot  org.znapzend:zend_delay          0                                                      local

lib/ZnapZend/Config.pm

lib/ZnapZend/Config.pm: checkBackupSets(): do not ignore src_mbuffer=…

42c6efe

…off [oetiker#629] Signed-off-by: Jim Klimov <jimklimov@gmail.com>

jimklimov mentioned this pull request Jan 17, 2024

Allow configuring mbuffer path and size separately for each destination (and source) system #630

Merged

jimklimov force-pushed the issue-629 branch from 75fa769 to 5d9d6a9 Compare January 17, 2024 15:23

lib/ZnapZend/Config.pm: checkBackupSets(): do not assign "undef" src_…

714e8f6

…mbuffer, fall back to "off" [oetiker#629] Signed-off-by: Jim Klimov <jimklimov@gmail.com>

jimklimov force-pushed the issue-629 branch from 5d9d6a9 to 714e8f6 Compare January 17, 2024 15:27

jimklimov added 2 commits February 7, 2024 09:43

Merge branch 'master' into issue-629

192829b

Merge branch 'master' into issue-629

c2b0e18

This was referenced Feb 7, 2024

[RFE] znapzendsetup : support dst_N_enable setting #634

Closed

Dockerfile: fail fast if mbuffer is not compatible/fetchable [#589] #635

Merged

oetiker requested changes Mar 11, 2024

View reviewed changes

lib/ZnapZend/Config.pm Show resolved Hide resolved

jimklimov and others added 3 commits March 12, 2024 10:34

Merge branch 'master' into issue-629

4fa171a

Merge branch 'master' into issue-629

b4ee92a

Merge branch 'master' into issue-629

335dec7

oetiker merged commit 8ae2bf1 into oetiker:master Mar 13, 2024
4 checks passed

jimklimov deleted the issue-629 branch March 13, 2024 09:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix handling nuances of `src_mbuffer` #632

Fix handling nuances of `src_mbuffer` #632

jimklimov commented Jan 17, 2024 •

edited

Loading

jimklimov commented Jan 17, 2024 •

edited

Loading

Fix handling nuances of src_mbuffer #632

Fix handling nuances of src_mbuffer #632

Conversation

jimklimov commented Jan 17, 2024 • edited Loading

jimklimov commented Jan 17, 2024 • edited Loading

Fix handling nuances of `src_mbuffer` #632

Fix handling nuances of `src_mbuffer` #632

jimklimov commented Jan 17, 2024 •

edited

Loading

jimklimov commented Jan 17, 2024 •

edited

Loading