Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logging stops on all backend when overloaded for a time #75736

Closed
thales-nascimento opened this issue Jul 10, 2024 · 2 comments · Fixed by #75806
Closed

Logging stops on all backend when overloaded for a time #75736

thales-nascimento opened this issue Jul 10, 2024 · 2 comments · Fixed by #75806
Assignees
Labels
area: Logging bug The issue is a bug, or the PR is fixing a bug priority: medium Medium impact/importance bug

Comments

@thales-nascimento
Copy link

Describe the bug
Logging stops on all backend when overloaded for a time. After stopped no more messages are produced, even when the overload ceases. By overloaded I mean to drop 99% of the messages.

Platform: STM32H743
Version: tag zephyr-v3.5.0

Additional info:
I have two backends, one logs on UART and other logs to an external flash with filesystem littlefs and file rotation. All my threas are same priority 5.

Workarround: patching log_core.c to trigger the logging thread when above the configured threshold seems to work.
With the workarround the system behaves normally after the overload passes.

KConfigs
#
# Logging
#
CONFIG_LOG=y
CONFIG_LOG_CORE_INIT_PRIORITY=0
CONFIG_LOG_MODE_DEFERRED=y
# CONFIG_LOG_MODE_IMMEDIATE is not set
# CONFIG_LOG_MODE_MINIMAL is not set
# CONFIG_LOG_FRONTEND is not set
# CONFIG_LOG_CUSTOM_HEADER is not set
# CONFIG_LOG_MULTIDOMAIN is not set

#
# Logging levels filtering
#
# CONFIG_LOG_RUNTIME_FILTERING is not set
CONFIG_LOG_DEFAULT_LEVEL=3
CONFIG_LOG_OVERRIDE_LEVEL=0
CONFIG_LOG_MAX_LEVEL=4
# end of Logging levels filtering

#
# Processing
#
CONFIG_LOG_PRINTK=y
CONFIG_LOG_MODE_OVERFLOW=y
# CONFIG_LOG_BLOCK_IN_THREAD is not set
CONFIG_LOG_PROCESS_TRIGGER_THRESHOLD=10
CONFIG_LOG_PROCESS_THREAD=y
CONFIG_LOG_PROCESS_THREAD_STARTUP_DELAY_MS=1000
CONFIG_LOG_PROCESS_THREAD_SLEEP_MS=1000
CONFIG_LOG_PROCESS_THREAD_STACK_SIZE=4096
# CONFIG_LOG_PROCESS_THREAD_CUSTOM_PRIORITY is not set
CONFIG_LOG_BUFFER_SIZE=8192
CONFIG_LOG_TRACE_SHORT_TIMESTAMP=y
# CONFIG_LOG_TIMESTAMP_64BIT is not set
# CONFIG_LOG_SPEED is not set
# end of Processing

#
# Output Formatting
#

#
# Prepend non-hexdump log message with function name
#
# CONFIG_LOG_FUNC_NAME_PREFIX_ERR is not set
# CONFIG_LOG_FUNC_NAME_PREFIX_WRN is not set
# CONFIG_LOG_FUNC_NAME_PREFIX_INF is not set
# CONFIG_LOG_FUNC_NAME_PREFIX_DBG is not set
# end of Prepend non-hexdump log message with function name

# CONFIG_LOG_MIPI_SYST_ENABLE is not set
# CONFIG_LOG_THREAD_ID_PREFIX is not set
# CONFIG_LOG_CUSTOM_FORMAT_SUPPORT is not set
CONFIG_LOG_BACKEND_SHOW_COLOR=y
# CONFIG_LOG_INFO_COLOR_GREEN is not set
CONFIG_LOG_TAG_MAX_LEN=0
CONFIG_LOG_BACKEND_FORMAT_TIMESTAMP=y
# CONFIG_LOG_OUTPUT_FORMAT_LINUX_TIMESTAMP is not set
# CONFIG_LOG_OUTPUT_FORMAT_CUSTOM_TIMESTAMP is not set
# end of Output Formatting

#
# Backends
#
# CONFIG_LOG_BACKEND_FS is not set
# CONFIG_LOG_BACKEND_NET is not set
# CONFIG_LOG_BACKEND_SWO is not set
CONFIG_LOG_BACKEND_UART=y
# CONFIG_LOG_BACKEND_UART_ASYNC is not set
CONFIG_LOG_BACKEND_UART_BUFFER_SIZE=1
CONFIG_LOG_BACKEND_UART_AUTOSTART=y
CONFIG_LOG_BACKEND_UART_OUTPUT_TEXT=y
# CONFIG_LOG_BACKEND_UART_OUTPUT_DICTIONARY is not set
# CONFIG_LOG_BACKEND_UART_OUTPUT_CUSTOM is not set
CONFIG_LOG_BACKEND_UART_OUTPUT_DEFAULT=0
# CONFIG_LOG_BACKEND_IPC_SERVICE is not set
# end of Backends

#
# Misc
#
CONFIG_LOG_DOMAIN_ID=0
CONFIG_LOG_USE_VLA=y
# CONFIG_LOG_ALWAYS_RUNTIME is not set
# CONFIG_LOG_FMT_SECTION is not set
# CONFIG_LOG_USE_TAGGED_ARGUMENTS is not set
# CONFIG_LOG_MEM_UTILIZATION is not set
CONFIG_LOG_FAILURE_REPORT_PERIOD=1000
# end of Misc

CONFIG_LOG_OUTPUT=y
# end of Logging

Expected behavior
Messages should be dropped as necessary but the logging system should remain operational.

Impact
minor, patch works for me.

Environment

  • OS: Linux
  • Toolchain: Zephyr sdk 0.16.3
  • Zephyr 3.5.0
@thales-nascimento thales-nascimento added the bug The issue is a bug, or the PR is fixing a bug label Jul 10, 2024
Copy link

Hi @thales-nascimento! We appreciate you submitting your first issue for our open-source project. 🌟

Even though I'm a bot, I can assure you that the whole community is genuinely grateful for your time and effort. 🤖💙

@nashif nashif added the priority: medium Medium impact/importance bug label Jul 11, 2024
@pdgendt
Copy link
Collaborator

pdgendt commented Jul 12, 2024

Can you submit your fix as a PR?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: Logging bug The issue is a bug, or the PR is fixing a bug priority: medium Medium impact/importance bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants