Skip to content

Commit

Permalink
chore(ci): rollback earthly prune (AztecProtocol#6208)
Browse files Browse the repository at this point in the history
We're getting hit by an unfortunately earthly bug, but the workaround
here took down other jobs
  • Loading branch information
ludamad authored May 3, 2024
1 parent 4d8b51c commit 3ccc6ac
Showing 1 changed file with 5 additions and 7 deletions.
12 changes: 5 additions & 7 deletions scripts/earthly-ci
Original file line number Diff line number Diff line change
Expand Up @@ -20,16 +20,14 @@ while [ $ATTEMPT_COUNT -lt $MAX_ATTEMPTS ]; do
echo "Attempt #$ATTEMPT_COUNT failed."

# Check the output for specific errors
if grep 'failed to get edge: inconsistent graph state' $OUTPUT_FILE >/dev/null; then
if grep 'failed to get edge: inconsistent graph state' $OUTPUT_FILE >/dev/null || grep 'failed to get state for index' $OUTPUT_FILE >/dev/null ; then
INCONSISTENT_GRAPH_STATE_COUNT=$((INCONSISTENT_GRAPH_STATE_COUNT + 1))
echo "Got 'inconsistent graph state'."
if [ "$INCONSISTENT_GRAPH_STATE_COUNT" -eq 2 ]; then
echo "Performing 'earthly prune' due to repeated 'inconsistent graph state' errors."
earthly prune
if earthly $@ 2>&1 | tee $OUTPUT_FILE >&2 ; then
exit 0 # Post-prune success
fi
echo "Unable to recover from 'inconsistent graph state' or 'failed to get state for index'. Connect to spot runner and run 'earthly prune'."
exit 1
fi
echo "Got 'inconsistent graph state' or 'failed to get state for index'. Sleeping for 20 seconds and retrying."
sleep 20
elif grep 'Error: pull ping error: pull ping response' $OUTPUT_FILE >/dev/null; then
echo "Got 'Error: pull ping error: pull ping response', intermittent failure when writing out images to docker"
elif grep '================================= System Info ==================================' $OUTPUT_FILE >/dev/null; then
Expand Down

0 comments on commit 3ccc6ac

Please sign in to comment.