All front-end debugging messages suppressed by early initial message

[rountree]

Calls to spindle_debug_printf_impl() (usually via a wrapper) will trigger logger initialization via init_spindle_debugging() as needed. Correct initialization requires the SPINDLE_DEBUG environment variable to be already set. However, this does not happen until several lines into sp_init() (when running under Flux).

When that environment variable is not yet set, the initialization process assumes the absence of the variable is intentional and short-circuits further attempts at initialization.

A possible two-part fix:

Force sp_init() to always set SPINDLE_DEBUG in the shell environment, setting it to zero if the variable was not present in the flux shell environment. Force init_spindle_debugging() to check for the existence of SPINDLE_DEBUG. If the variable is not present, take that as an indication that shell initialization isn't yet complete. The message should probably be logged regardless of log level; alternatively, it could be buffered and logged later, but that gets complicated quickly. Subsequent calls to spindle_debug_printf_impl() can check if the environment variable has been set yet, and reset initialization if needed.

[mplegendre]

Can we just move the SPINDLE_DEBUG environment variable setting to the top of sp_init(), so it's before any spindle calls?

[rountree]

Well, that'll work until some newbie adds code to the top of every function to generate a call tree...

We'd also have to make flux_plugin_init() off-limits (all ten lines of it). It's not a big deal one way or the other. I've got a PR that let me generate the call trace below. If it's too intrusive, I have a couple of alternatives in mind. If you'd rather let this go, that's fine, too.

[Launcher.1165773@flux-spindle.c:499] flux_plugin_init - flux_plugin_init:499 BLR ENTRY
[Launcher.1165773@flux-spindle.c:336] sp_init - sp_init:336 BLR ENTRY
[Launcher.1165773@flux-spindle.c:97] spindle_ctx_create - spindle_ctx_create:97 BLR ENTRY
[Launcher.1165773@flux-spindle.c:51] R_to_hosts - R_to_hosts:51 BLR ENTRY
[Launcher.1165773@flux-spindle.c:230] sp_getopts - sp_getopts:230 BLR ENTRY
[Launcher.1165773@flux-spindle.c:463] sp_task - sp_task:463 BLR ENTRY
[Launcher.1165773@flux-spindle.c:463] sp_task - sp_task:463 BLR ENTRY
[Launcher.1165773@flux-spindle.c:181] wait_for_shell_init - wait_for_shell_init:181 BLR ENTRY
[Launcher.1165773@flux-spindle.c:181] wait_for_shell_init - wait_for_shell_init:181 BLR ENTRY
[Launcher.1165773@flux-spindle.c:181] wait_for_shell_init - wait_for_shell_init:181 BLR ENTRY
[Launcher.1165773@flux-spindle.c:145] run_spindle_backend - run_spindle_backend:145 BLR ENTRY
[Launcher.1165773@flux-spindle.c:168] run_spindle_frontend - run_spindle_frontend:168 BLR ENTRY
[Launcher.1165773@flux-spindle.c:490] sp_exit - sp_exit:490 BLR ENTRY
[Launcher.1165773@flux-spindle.c:131] spindle_ctx_destroy - spindle_ctx_destroy:131 BLR ENTRY
[Launcher.1165773@flux-spindle.c:38] free_argv - free_argv:38 BLR ENTRY
[Launcher.1165773@flux-spindle.c:38] free_argv - free_argv:38 BLR ENTRY