race: remote,rootless: podman network create followed by run

[edsantiago]

I've seen this one twice so far:

$ podman-remote network create --subnet 172.16.206.0/24 testnet-TrgvHhKQAK
/etc/cni/net.d/testnet-TrgvHhKQAK.conflist
$ podman-remote run --rm --network testnet-TrgvHhKQAK quay.io/libpod/testimage:20200917 ip a
Error: error preparing container 8efdbe7260f5e082c2a256ee0781a23fecbcd0776d5ac1b43d924d39cfae5086 for attach: error configuring network namespace for container 8efdbe7260f5e082c2a256ee0781a23fecbcd0776d5ac1b43d924d39cfae5086: CNI network "testnet-TrgvHhKQAK" not found
[ rc=127 (** EXPECTED 0 **) ]

Log: from PR 7803, ubuntu remote

Again, I'm not going to spend any time right now trying to reproduce (acting on the assumption that this will be an easy one to track down). I will try to keep track of flake occurrences and log them here.

[edsantiago]

I wonder if the race is in CNI? And I wonder if this is related? It's a very frequent flake in podman network rm, the classic symptom being:

Running: podman [options] network ls --quiet
podman-integrationtest
Running: podman [options] network rm podman-integrationtest
podman-integrationtest
Running: podman [options] network ls --quiet
podman-integrationtest
remove /home/some18764dude/.config/cni/net.d/eaac2f093ac23f24b699d68d1118f986b8620b03e3d5974e14c12147e06dfdab.conflist: no such file or directory

Recent instances (all one-offs, no triplets which cause a complete CI abort):

Podman network [It] podman network rm

• fedora : in-podman fedora
  • PR Make Go builds more consistent #7676
    • 09-17 13:05
• fedora : special_testing_rootless
  • PR Bump github.com/containers/common from 0.22.0 to 0.23.0 #7777
    • 09-25 04:55
• fedora : test fedora
  • PR Fix podman pod create --infra-command and --infra-image #7621
    • 09-16 08:23
• prior-fedora : test prior-fedora
  • PR Include cgroup manager in podman info output #7729
    • 09-22 13:56
• ubuntu : test ubuntu
  • PR set interactive mode with compat create endpoint #7761
    • 09-23 17:38
• ubuntu-19 : test ubuntu-19
  • PR Update vendor of buildah to latest code #7335
    • 08-25 08:56

Apologies in advance if this is unrelated.

[Luap99]

@edsantiago The tests are run in parallel, right?

[edsantiago]

@Luap99 I cannot answer that with confidence, but yes, I believe so.

[baude]

correct. in parallel depending on the run itself. i think remote tests are not in parallel.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@edsantiago @Luap99 Is this still an issue?

[edsantiago]

The last two incidents reported by cirrus-flake-summarize are on 09-29:

• prior-ubuntu : test prior-ubuntu
  • PR Refactor IdleTracker to handle StateIdle transitions #7833
    • 09-29 15:21
• ubuntu : test ubuntu
  • PR Fix handling of remove of bogus volumes, networks and Pods #7825
    • 09-29 16:45

[rhatdan]

Ok I will close this issue, with a hope and a prayer that it has been fixed, reopen if it happens again.