[202012] Fix vs check install login timeout issue (#11727) #12011
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Why I did it Fix a build not stable issue: sonic-net#11620 The vs vm has started successfully, but failed to wait for the message "sonic login:". There were 55 builds failed caused by the issue in the last 30 days. AzurePipelineBuildLogs | where startTime > ago(30d) | where type =~ "task" | where result =~ "failed" | where name =~ "Build sonic image" | where content contains "Timeout exceeded" | where content contains "re.compile('sonic login:')" | project-away content | extend branchName=case(reason=~"pullRequest", tostring(todynamic(parameters)['system.pullRequest.targetBranch']), replace("refs/heads/", "", sourceBranch)) | summarize FailedCount=dcount(buildId) by branchName branchName FailedCount master 37 202012 9 202106 4 202111 2 202205 1 201911 1 It is caused by the login message mixed with the output message of the /etc/rc.local, one of the examples as below: (see the message rc.local[307]: sonic+ onie_disco_subnet=255.255.255.0 login: ) The check_install.py was waiting for the message "sonic login:", and Linux console was waiting for the username input (the login message has already printed in the console). https://dev.azure.com/mssonic/build/_build/results?buildId=123294&view=logs&j=cef3d8a9-152e-5193-620b-567dc18af272&t=359769c4-8b5e-5976-a793-85da132e0a6f 2022-07-17T15:00:58.9198877Z [ 25.493855] rc.local[307]: + onie_disco_opt53=05 2022-07-17T15:00:58.9199330Z [ 25.595054] rc.local[307]: + onie_disco_router=10.0.2.2 2022-07-17T15:00:58.9199781Z [ 25.699409] rc.local[307]: + onie_disco_serverid=10.0.2.2 2022-07-17T15:00:58.9200252Z [ 25.789891] rc.local[307]: + onie_disco_siaddr=10.0.2.2 2022-07-17T15:00:58.9200622Z [ 25.880920] 2022-07-17T15:00:58.9200745Z 2022-07-17T15:00:58.9201019Z Debian GNU/Linux 10 sonic ttyS0 2022-07-17T15:00:58.9201201Z 2022-07-17T15:00:58.9201542Z rc.local[307]: sonic+ onie_disco_subnet=255.255.255.0 login: 2022-07-17T15:00:58.9202309Z [ 26.079767] rc.local[307]: + onie_exec_url=file://dev/vdb/onie-installer.bin How I did it Input a newline when finished to run the script /etc/rc.local. If entering a newline, the message "sonic login:" will prompt again.
lguohan
approved these changes
Sep 8, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why I did it
Cherry-pick from #11727 and fix conflict.
Fix a build not stable issue: #11620
The vs vm has started successfully, but failed to wait for the message "sonic login:".
There were 55 builds failed caused by the issue in the last 30 days.
AzurePipelineBuildLogs
| where startTime > ago(30d)
| where type =~ "task"
| where result =~ "failed"
| where name =~ "Build sonic image"
| where content contains "Timeout exceeded"
| where content contains "re.compile('sonic login:')" | project-away content
| extend branchName=case(reason=~"pullRequest", tostring(todynamic(parameters)['system.pullRequest.targetBranch']),
replace("refs/heads/", "", sourceBranch))
| summarize FailedCount=dcount(buildId) by branchName
branchName FailedCount
master 37
202012 9
202106 4
202111 2
202205 1
201911 1
It is caused by the login message mixed with the output message of the /etc/rc.local, one of the examples as below: (see the message rc.local[307]: sonic+ onie_disco_subnet=255.255.255.0 login: ) The check_install.py was waiting for the message "sonic login:", and Linux console was waiting for the username input (the login message has already printed in the console). https://dev.azure.com/mssonic/build/_build/results?buildId=123294&view=logs&j=cef3d8a9-152e-5193-620b-567dc18af272&t=359769c4-8b5e-5976-a793-85da132e0a6f
2022-07-17T15:00:58.9198877Z [ 25.493855] rc.local[307]: + onie_disco_opt53=05
2022-07-17T15:00:58.9199330Z [ 25.595054] rc.local[307]: + onie_disco_router=10.0.2.2
2022-07-17T15:00:58.9199781Z [ 25.699409] rc.local[307]: + onie_disco_serverid=10.0.2.2
2022-07-17T15:00:58.9200252Z [ 25.789891] rc.local[307]: + onie_disco_siaddr=10.0.2.2
2022-07-17T15:00:58.9200622Z [ 25.880920]
2022-07-17T15:00:58.9200745Z
2022-07-17T15:00:58.9201019Z Debian GNU/Linux 10 sonic ttyS0
2022-07-17T15:00:58.9201201Z
2022-07-17T15:00:58.9201542Z rc.local[307]: sonic+ onie_disco_subnet=255.255.255.0 login:
2022-07-17T15:00:58.9202309Z [ 26.079767] rc.local[307]: + onie_exec_url=file://dev/vdb/onie-installer.bin
How I did it
Input a newline when finished to run the script /etc/rc.local. If entering a newline, the message "sonic login:" will prompt again.
A picture of a cute animal (not mandatory but encouraged)