Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix rsyslogd start failed cause by rsyslog.conf is emtpy. #13669

Merged
merged 4 commits into from
Feb 6, 2023

Conversation

lerry-lee
Copy link
Contributor

@lerry-lee lerry-lee commented Feb 6, 2023

Signed-off-by: Chun'ang Li chunangli@microsoft.com

Why I did it

In to-sonic and multi-asic KVM-test, pretest sometimes failed. Reason is rsyslogd process can not start in teamd container. Because rsyslog.conf is empty caused by sonic-cfggen execute failed, and failed log is:

image

How I did it

If sonic-cfggen -d execute failed, execute without -d because the template file has the default value.

How to verify it

Build image and test it over 40 times, all passed pretest.

Which release branch to backport (provide reason below if selected)

  • 201811
  • 201911
  • 202006
  • 202012
  • 202106
  • 202111
  • 202205
  • 202211

Description for the changelog

Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

Signed-off-by: Chun'ang Li <chunangli@microsoft.com>
@lerry-lee lerry-lee requested a review from lguohan as a code owner February 6, 2023 02:51
Copy link
Contributor

@yejianquan yejianquan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@saiarcot895
Copy link
Contributor

saiarcot895 commented Feb 6, 2023

Is it known why sonic-cfggen fails?

Edit: never mind, I just saw the screenshot.

@Junchao-Mellanox
Copy link
Collaborator

Should we fix sonic-cfggen? It should not generate an empty file while exception occurs?

@yejianquan
Copy link
Contributor

yejianquan commented Feb 6, 2023

Is it known why sonic-cfggen fails?

Edit: never mind, I just saw the screenshot.

@saiarcot895
Because sonic-cfggen can't connect to the redis.
I pulled you into an email thread that contains more technical details.

@lerry-lee
Copy link
Contributor Author

Should we fix sonic-cfggen? It should not generate an empty file while exception occurs?

@Junchao-Mellanox
Thanks for the suggestion! But it's a hotfix, we think should resolve the problem first, because it has affected some PR already. For deep reason and sonic-cffgen improvement, we will follow up continuously.

docker cp $TMP_FILE ${DOCKERNAME}:/etc/rsyslog.conf
if [ "$failed" == "false" ]; then
docker cp $TMP_FILE ${DOCKERNAME}:/etc/rsyslog.conf
fi
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a risk that teamd container started without proper rsyslog.conf file. Then probably the log of teamd container can't be received by the host rsyslog service. The consequence is that teamd syslog could be missing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the reminder! Will discuss and improve again.

…ue to generate conf.

Signed-off-by: Chun'ang Li <chunangli@microsoft.com>
…ue to generate conf.

Signed-off-by: Chun'ang Li <chunangli@microsoft.com>
@lerry-lee
Copy link
Contributor Author

#13674 will trace this issue.

@liat-grozovik liat-grozovik merged commit eea5471 into sonic-net:master Feb 6, 2023
mssonicbld pushed a commit to mssonicbld/sonic-buildimage that referenced this pull request Feb 17, 2023
…13669)

- Why I did it
In to-sonic and multi-asic KVM-test, pretest sometimes failed. Reason is rsyslogd process can not start in teamd container. Because rsyslog.conf is empty caused by sonic-cfggen execute failed

- How I did it
If sonic-cfggen -d execute failed, execute without -d because the template file has the default value.

- How to verify it
Build image and test it over 40 times, all passed pretest.

Signed-off-by: Chun'ang Li <chunangli@microsoft.com>
@mssonicbld
Copy link
Collaborator

Cherry-pick PR to 202211: #13862

@StormLiangMS
Copy link
Contributor

@yejianquan @wangxin do we need this for 202205?

mssonicbld pushed a commit that referenced this pull request Feb 17, 2023
- Why I did it
In to-sonic and multi-asic KVM-test, pretest sometimes failed. Reason is rsyslogd process can not start in teamd container. Because rsyslog.conf is empty caused by sonic-cfggen execute failed

- How I did it
If sonic-cfggen -d execute failed, execute without -d because the template file has the default value.

- How to verify it
Build image and test it over 40 times, all passed pretest.

Signed-off-by: Chun'ang Li <chunangli@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants