Skip to content

Commit

Permalink
[warm/fast reboot] some service docker might have been stopped already (
Browse files Browse the repository at this point in the history
sonic-net#668)

* [warm/fast reboot] some service docker might have been stopped already

Some services are depending on swss and might have been stopped at the
time the loop of docker killing was executed.

Signed-off-by: Ying Xie <ying.xie@microsoft.com>

* log and ignore all errors while killing containers

* add debug message

* put redirect at the right place
  • Loading branch information
yxieca committed Sep 25, 2019
1 parent b19b125 commit bf79c95
Showing 1 changed file with 10 additions and 2 deletions.
12 changes: 10 additions & 2 deletions scripts/fast-reboot
Original file line number Diff line number Diff line change
Expand Up @@ -423,12 +423,14 @@ if [[ "$REBOOT_TYPE" = "fast-reboot" ]]; then
# then immediately call `systemctl stop teamd` to prevent the service from
# restarting the container automatically.
# Note: teamd must be killed before syncd, because it will send the last packet through CPU port
debug "Stopping teamd ..."
docker exec -i teamd pkill -USR2 teamd || [ $? == 1 ]
while docker exec -i teamd pgrep teamd > /dev/null; do
sleep 0.05
done
docker kill teamd > /dev/null
systemctl stop teamd
debug "Stopped teamd ..."
fi
# Kill swss Docker container
Expand Down Expand Up @@ -464,7 +466,7 @@ if [[ "$REBOOT_TYPE" = "warm-reboot" || "$REBOOT_TYPE" = "fastfast-reboot" ]]; t
# Send USR1 signal to all teamd instances to stop them
# It will prepare teamd for warm-reboot
# Note: We must send USR1 signal before syncd, because it will send the last packet through CPU port
docker exec -i teamd pkill -USR1 teamd || [ $? == 1 ] > /dev/null
docker exec -i teamd pkill -USR1 teamd > /dev/null || [ $? == 1 ]
debug "Stopped teamd ..."
fi
Expand All @@ -476,10 +478,16 @@ debug "Stopped syncd ..."
# We call `docker kill ...` to ensure the container stops as quickly as possible,
# then immediately call `systemctl stop ...` to prevent the service from
# restarting the container automatically.
debug "Stopping all remaining containers ..."
for CONTAINER_NAME in $(docker ps --format '{{.Names}}'); do
docker kill $CONTAINER_NAME > /dev/null
CONTAINER_STOP_RC=0
docker kill $CONTAINER_NAME &> /dev/null || CONTAINER_STOP_RC=$?
systemctl stop $CONTAINER_NAME
if [[ CONTAINER_STOP_RC -ne 0 ]]; then
debug "Failed killing container $CONTAINER_NAME RC $CONTAINER_STOP_RC ."
fi
done
debug "Stopped all remaining containers ..."
# Stop the docker container engine. Otherwise we will have a broken docker storage
systemctl stop docker.service
Expand Down

0 comments on commit bf79c95

Please sign in to comment.