Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fast/warm reboot] ignore errors after shutting down critical service(s) #761

Merged
merged 1 commit into from
Dec 10, 2019
Merged

[fast/warm reboot] ignore errors after shutting down critical service(s) #761

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 10 additions & 7 deletions scripts/fast-reboot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,6 @@ function request_pre_shutdown()
debug "Requesting pre-shutdown ..."
/usr/bin/docker exec -i syncd /usr/bin/syncd_request_shutdown --pre &> /dev/null || {
error "Failed to request pre-shutdown"
exit "${EXIT_SYNCD_SHUTDOWN}"
}
}

Expand Down Expand Up @@ -177,9 +176,9 @@ function wait_for_pre_shutdown_complete_or_fail()

if [[ x"${STATE}" != x"pre-shutdown-succeeded" ]]; then
debug "Syncd pre-shutdown failed: ${STATE} ..."
exit "${EXIT_SYNCD_SHUTDOWN}"
else
debug "Pre-shutdown succeeded ..."
fi
debug "Pre-shutdown succeeded ..."
}

function backup_database()
Expand Down Expand Up @@ -399,6 +398,10 @@ if [[ "$REBOOT_TYPE" = "warm-reboot" || "$REBOOT_TYPE" = "fastfast-reboot" ]]; t
fi
fi

# We are fully committed to reboot from this point on becasue critical
# service will go down and we cannot recover from it.
set +e

# Kill radv before stopping BGP service to prevent annoucing our departure.
debug "Stopping radv ..."
docker kill radv &>/dev/null || [ $? == 1 ]
Expand Down Expand Up @@ -471,7 +474,7 @@ if [[ "$REBOOT_TYPE" = "warm-reboot" || "$REBOOT_TYPE" = "fastfast-reboot" ]]; t
fi

debug "Stopping syncd ..."
systemctl stop syncd
systemctl stop syncd || debug "Ignore stopping syncd service error $?"
debug "Stopped syncd ..."

# Kill other containers to make the reboot faster
Expand All @@ -482,20 +485,20 @@ debug "Stopping all remaining containers ..."
for CONTAINER_NAME in $(docker ps --format '{{.Names}}'); do
CONTAINER_STOP_RC=0
docker kill $CONTAINER_NAME &> /dev/null || CONTAINER_STOP_RC=$?
systemctl stop $CONTAINER_NAME
systemctl stop $CONTAINER_NAME || debug "Ignore stopping $CONTAINER_NAME error $?"
if [[ CONTAINER_STOP_RC -ne 0 ]]; then
debug "Failed killing container $CONTAINER_NAME RC $CONTAINER_STOP_RC ."
fi
done
debug "Stopped all remaining containers ..."

# Stop the docker container engine. Otherwise we will have a broken docker storage
systemctl stop docker.service
systemctl stop docker.service || debug "Ignore stopping docker service error $?"

# Stop kernel modules for Nephos platform
if [[ "$sonic_asic_type" = 'nephos' ]];
then
systemctl stop nps-modules-`uname -r`.service
systemctl stop nps-modules-`uname -r`.service || debug "Ignore stopping nps service error $?"
fi

# Update the reboot cause file to reflect that user issued this script
Expand Down