[systemd-generator]: Fix dependency update for multi-asic platform #4820

SuvarnaMeenakshi · 2020-06-19T22:14:53Z

- Why I did it
First Issue: Issue seen while updating .service file for host services in multi-asic platform:
if host service file has the line:

After=updategraph.service swss.service syncd.service

For multi-asic platform, with 3 asics, where there are 3 syncd and swss services, the dependency should be modified as:

After=updategraph.service
After=swss@0.service
After=swss@1.service
After=swss@2.service
After=syncd@0.service
After=syncd@1.service
After=syncd@2.service

But currently, the updated service files looks like:

After=updategraph.service

On VS platform, the string formed is not clean and corrupted characters are getting added.

second issue: Currently .timer files are not updated with the right dependencies for multi-asic platform. Like snmp.timer has WantedBy swss.service, this should get updated as WantedBy swss@0.service, swss@1.service etc.

- How I did it

systemd-generator is parsing .service file and updating the dependency. The code uses nested strtok. This is causing string corruption. To fix this, used strtok_r(), to make sure that context of each strotok is saved.
Added code to make sure that .timer files are also updated with the right dependencies for mutli-asic platform.

- How to verify it

Checked the generated service files in VS platform to make sure that the string is not corrupted and the right dependencies are added.
timer file is also updated with the right dependency.

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

of host services are generated correctly for multi-asic platforms. Add code to make sure that systemd timer files are also modified to add the correct service dependency for multi-asic platforms. Signed-off-by: SuvarnaMeenakshi <sumeenak@microsoft.com>

remove unused variable. Signed-off-by: SuvarnaMeenakshi <sumeenak@microsoft.com>

Signed-off-by: SuvarnaMeenakshi <sumeenak@microsoft.com>

jleveque · 2020-06-19T22:27:32Z

Interesting. So the strtok() call must be getting interrupted somehow. Should we also change all the other strtok() calls to strtok_r() as well to be safe?

SuvarnaMeenakshi · 2020-06-19T22:30:59Z

Interesting. So the strtok() call must be getting interrupted somehow. Should we also change all the other strtok() calls to strtok_r() as well to be safe?

From my understanding, strtok() invokes strtok_r() internally. There will be an issue only if there is nested strtok(), else it should be fine.

jleveque · 2020-06-19T22:41:15Z

files/build_templates/sonic_debian_extension.j2

-sudo LANG=C chroot $FILESYSTEM_ROOT systemctl enable snmp.timer
+echo "snmp.timer" | sudo tee -a $GENERATED_SERVICE_FILE
 {% if enable_system_telemetry == 'y' %}
 sudo cp $BUILD_TEMPLATES/telemetry.timer $FILESYSTEM_ROOT_USR_LIB_SYSTEMD_SYSTEM
-sudo LANG=C chroot $FILESYSTEM_ROOT systemctl enable telemetry.timer
+echo "telemetry.timer" | sudo tee -a $GENERATED_SERVICE_FILE


Why no longer calling systemctl enable on the timers?

systemctl enable will create a mapping of the service in /etc/systemd/system/.wants directory. This can cause issue in multi-asic platform. The dependencies that are created in systemd generator are present in /var/run/systemd/generator/.wants directory.
To explain better, I have directory structure from VS platform, with systemctl enable and without it.
Without enable: single asic VS platform

admin@sonic:**/var/run/systemd/generator**$ ls timers.target.wants/ snmp.timer telemetry.timer admin@sonic:**/var/run/systemd/generator**$ ls swss.service.wants/ nat.service snmp.timer admin@sonic:/etc/systemd/system/timers.target.wants$ ls apt-daily.timer logrotate.timer apt-daily-upgrade.timer process-reboot-cause.timer admin@sonic:/var/run/systemd/generator/swss.service.wants$ .timer â—� snmp.timer - Delays snmp container until SONiC has started Loaded: loaded (/lib/systemd/system/snmp.timer; **enabled-runtime;** vendor prese Active: inactive (dead) since Mon 2020-06-22 00:35:35 UTC; 4min 40s ago multi-asic: admin@sonic:**/var/run/systemd/generator$ ls swss@0.service.wants/** nat.service snmp.timer admin@sonic:/var/run/systemd/generator$ ls timers.target.wants/ snmp.timer telemetry.timer

Though timers.target.wants directory is present in /etc/systemd/system and /var/run/systemd/generator, systemctl -r shows all under timers.target:

timers.target loaded active active Timers --------------------------------------------------------------------------------- apt-daily-upgrade.timer loaded active waiting Daily apt upgrade a apt-daily.timer loaded active waiting Daily apt download fstrim.timer loaded active waiting Discard unused bloc logrotate.timer loaded active waiting Daily rotation of l process-reboot-cause.timer loaded active waiting Delays process-rebo snmp.timer loaded active waiting Delays snmp contain systemd-tmpfiles-clean.timer loaded active waiting Daily Cleanup of telemetry.timer loaded active waiting Delays telemetry co

If we keep sysctl enable, then in mutli-asic platform, /etc/systemd/system/swss.service.wants directory is created with snmp.timer in it. /var/run/systemd/generator/swss@0.service.wants also has snmp.timer. So, this is not inline with the required dependency.

admin@sonic:/etc/systemd/system/timers.target.wants$ ls apt-daily.timer logrotate.timer snmp.timer apt-daily-upgrade.timer process-reboot-cause.timer telemetry.timer admin@sonic:/etc/systemd/system/swss.service.wants$ ls snmp.timer

SuvarnaMeenakshi · 2020-06-22T00:42:30Z

retest vsimage please

qiluo-msft · 2020-06-22T22:58:30Z

src/systemd-sonic-generator/systemd-sonic-generator.c

            fputs(line,fp_tmp);
        } else {
            line_copy = strdup(line);
-            token = strtok(line_copy, "=");
-            while ((word = strtok(NULL, " "))){
+            token = strtok_r(line_copy, "=", &save_ptr1);


strtok_r [](start = 20, length = 8)

Rewrite everything by python?

The initial change to add systemd-generator had a discussion on this after which it was decided that C will be used. Guohan had pointed to a reference: "If you are careful, you can implement generators in shell scripts. We do recommend C code however, since generators are executed synchronously and hence delay the entire boot if they are slow." https://www.freedesktop.org/software/systemd/man/systemd.generator.html.

Thanks for sharing the history discussion.

I don't understand why it is on boot critical path. Could you elaborate? It seems relating the ASIC numbers, and only need to run once on first cold boot.

Can we remove explicit knowledge on ASIC numbers, just like depending on all syncd@N.service ?

Can we use C++ (as high performance as C) string and stream to split strings?

In reply to: 443869780 [](ancestors = 443869780)

systemd-generator runs before systemd starts, hence it is on boot critical path.
Yes, we will ideally need to run this only once on first cold boot. Currently this is always run upon each boot before systemd starts.

systemd generator does 2 things based on number of asics: 1. create the number of instances to be started, ex: swss@0, swss@1 etc. 2. Change the dependency of host services to reflect this list of actual running services. if we specify syncd@N - systemd does not convert that to 0 - n-1, instead looks for syncd@N service, which is not running.

Agreed that c++ will provide in built functions for string, but as service files are going to have some set of strings which will not vary much, c string functions should satisfy that requirement. Was there any other reason to use c++ other than ease of use with string functions?

Did a comparison on runtime between c and c++. On a mulit-asic platform, C program takes 2643 micro seconds and c++ take 2651 micro seconds. As there is not much difference, we can move to c++. Will raise another PR to move to code c++ with stream functions for string operations to make it less error prone.

…4820) * [systemd-generator]: Fix the code to make sure that dependencies of host services are generated correctly for multi-asic platforms. Add code to make sure that systemd timer files are also modified to add the correct service dependency for multi-asic platforms. Signed-off-by: SuvarnaMeenakshi <sumeenak@microsoft.com> * [systemd-generator]: Minor fix, remove debug code and remove unused variable.

SuvarnaMeenakshi added 3 commits June 19, 2020 14:23

[systemd-generator]: Minor fix, remove debug code and

f30b817

remove unused variable. Signed-off-by: SuvarnaMeenakshi <sumeenak@microsoft.com>

Minor fix to remove tabs.

71c07f0

Signed-off-by: SuvarnaMeenakshi <sumeenak@microsoft.com>

SuvarnaMeenakshi requested review from jleveque and pavel-shirshov June 19, 2020 22:15

SuvarnaMeenakshi added Bug 🐛 Request for 201911 Branch labels Jun 19, 2020

jleveque reviewed Jun 19, 2020

View reviewed changes

jleveque requested a review from qiluo-msft June 22, 2020 17:31

qiluo-msft reviewed Jun 22, 2020

View reviewed changes

jleveque approved these changes Jun 26, 2020

View reviewed changes

SuvarnaMeenakshi merged commit ab2177b into sonic-net:master Jun 29, 2020

SuvarnaMeenakshi added Multi-ASIC Multi-NPU 🔢 labels Jul 2, 2020

abdosi added the Included in 201911 Branch label Jul 5, 2020

SuvarnaMeenakshi deleted the systemd_generator branch September 2, 2022 17:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[systemd-generator]: Fix dependency update for multi-asic platform #4820

[systemd-generator]: Fix dependency update for multi-asic platform #4820

SuvarnaMeenakshi commented Jun 19, 2020

jleveque commented Jun 19, 2020

SuvarnaMeenakshi commented Jun 19, 2020

jleveque Jun 19, 2020

SuvarnaMeenakshi Jun 22, 2020

SuvarnaMeenakshi commented Jun 22, 2020

qiluo-msft Jun 22, 2020

SuvarnaMeenakshi Jun 22, 2020

qiluo-msft Jun 23, 2020

SuvarnaMeenakshi Jun 23, 2020

SuvarnaMeenakshi Jun 24, 2020

[systemd-generator]: Fix dependency update for multi-asic platform #4820

[systemd-generator]: Fix dependency update for multi-asic platform #4820

Conversation

SuvarnaMeenakshi commented Jun 19, 2020

jleveque commented Jun 19, 2020

SuvarnaMeenakshi commented Jun 19, 2020

jleveque Jun 19, 2020

Choose a reason for hiding this comment

SuvarnaMeenakshi Jun 22, 2020

Choose a reason for hiding this comment

SuvarnaMeenakshi commented Jun 22, 2020

qiluo-msft Jun 22, 2020

Choose a reason for hiding this comment

SuvarnaMeenakshi Jun 22, 2020

Choose a reason for hiding this comment

qiluo-msft Jun 23, 2020

Choose a reason for hiding this comment

SuvarnaMeenakshi Jun 23, 2020

Choose a reason for hiding this comment

SuvarnaMeenakshi Jun 24, 2020

Choose a reason for hiding this comment