[3.4] linux-amd64-e2e job takes more than 30min #17241
Assignees
Labels
Comments
|
Recently we see lots of workflow timeout. Could anyone take a look at this? |
|
It seems that it always fails when running the following two cases. |
fuweid
added a commit
to fuweid/etcd
that referenced
this issue
Jan 23, 2024
Assume etcd-io#16916 as baseline. The E2E takes `1395.082s`. * etcd-io#16988 It introduced `TestAuthority` which takes `18.39s`. And after etcd-io#16997, it takes `50.05s`. * etcd-io#16995 It introduced `TestInPlaceRecovery` which takes `17.37s`. * etcd-io#17144 - New `TestHTTPHealthHandler` takes `29.9s` - New `TestHTTPLivezReadyzHandler` takes `35.20s` * etcd-io#17173 - New `TestMemberReplace` takes `7.55s`. Ideally, it should increase `140.07s`. It's not larger than `1800s` timeout value. However, we run E2E cases 3 times. By default, we run E2E cases with `-cpu 1,2,4`. That means that we run 3 times. ```bash $ go help testflag -count n Run each test, benchmark, and fuzz seed n times (default 1). If -cpu is set, run n times for each GOMAXPROCS value. Examples are always run once. -count does not apply to fuzz tests matched by -fuzz. ``` I don't think we should run E2E with different GOMAXPROCS value. All the `TestXYZ` are used to control etcd process and we don't set GOMAXPROCS env to etcd process. So, we don't need `-cpu` setting for E2E. Closes: etcd-io#17241 Signed-off-by: Wei Fu <fuweid89@gmail.com>
fuweid
added a commit
to fuweid/etcd
that referenced
this issue
Jan 23, 2024
Assume etcd-io#16916 as baseline. The E2E takes `1395.082s`. * etcd-io#16988 It introduced `TestAuthority` which takes `18.39s`. And after etcd-io#16997, it takes `50.05s`. * etcd-io#16995 It introduced `TestInPlaceRecovery` which takes `17.37s`. * etcd-io#17144 - New `TestHTTPHealthHandler` takes `29.9s` - New `TestHTTPLivezReadyzHandler` takes `35.20s` * etcd-io#17173 - New `TestMemberReplace` takes `7.55s`. Ideally, it should increase `140.07s`. It's not larger than `1800s` timeout value. However, we run E2E cases 3 times. By default, we run E2E cases with `-cpu 1,2,4`. That means that we run 3 times. `1395.082s` + `140.07s * 3` = `1815.292s` > `1800s` ```bash $ go help testflag -count n Run each test, benchmark, and fuzz seed n times (default 1). If -cpu is set, run n times for each GOMAXPROCS value. Examples are always run once. -count does not apply to fuzz tests matched by -fuzz. ``` I don't think we should run E2E with different GOMAXPROCS value. All the `TestXYZ` are used to control etcd process and we don't set GOMAXPROCS env to etcd process. So, we don't need `-cpu` setting for E2E. Closes: etcd-io#17241 Signed-off-by: Wei Fu <fuweid89@gmail.com>
fuweid
added a commit
to fuweid/etcd
that referenced
this issue
Jan 23, 2024
Assume etcd-io#16916 as baseline. The E2E takes `1395.082s`. * etcd-io#16988 It introduced `TestAuthority` which takes `18.39s`. And after etcd-io#16997, it takes `50.05s`. * etcd-io#16995 It introduced `TestInPlaceRecovery` which takes `17.37s`. * etcd-io#17144 - New `TestHTTPHealthHandler` takes `29.9s` - New `TestHTTPLivezReadyzHandler` takes `35.20s` * etcd-io#17173 - New `TestMemberReplace` takes `7.55s`. Ideally, it should increase `140.07s`. It's not larger than `1800s` timeout value. However, we run E2E cases 3 times. By default, we run E2E cases with `-cpu 1,2,4`. That means that we run 3 times. `1395.082s` + `140.07s * 3` = `1815.292s` > `1800s` ```bash $ go help testflag -count n Run each test, benchmark, and fuzz seed n times (default 1). If -cpu is set, run n times for each GOMAXPROCS value. Examples are always run once. -count does not apply to fuzz tests matched by -fuzz. ``` I don't think we should run E2E with different GOMAXPROCS value. All the `TestXYZ` are used to control etcd process and we don't set GOMAXPROCS env to etcd process. Set `CPU=4` to align with main and release/3.5. Closes: etcd-io#17241 Signed-off-by: Wei Fu <fuweid89@gmail.com>
fuweid
added a commit
to fuweid/etcd
that referenced
this issue
Jan 23, 2024
Assume etcd-io#16916 as baseline. The E2E takes `1395.082s`. * etcd-io#16988 It introduced `TestAuthority` which takes `18.39s`. And after etcd-io#16997, it takes `50.05s`. * etcd-io#16995 It introduced `TestInPlaceRecovery` which takes `17.37s`. * etcd-io#17144 - New `TestHTTPHealthHandler` takes `29.9s` - New `TestHTTPLivezReadyzHandler` takes `35.20s` * etcd-io#17173 - New `TestMemberReplace` takes `7.55s`. Ideally, it should increase `140.07s`. It's not larger than `1800s` timeout value. However, we run E2E cases 3 times. By default, we run E2E cases with `-cpu 1,2,4`. That means that we run 3 times. `1395.082s` + `140.07s * 3` = `1815.292s` > `1800s` ```bash $ go help testflag -count n Run each test, benchmark, and fuzz seed n times (default 1). If -cpu is set, run n times for each GOMAXPROCS value. Examples are always run once. -count does not apply to fuzz tests matched by -fuzz. ``` I don't think we should run E2E with different GOMAXPROCS value. All the `TestXYZ` are used to control etcd process and we don't set GOMAXPROCS env to etcd process. Set `CPU=4` to align with main and release/3.5. Closes: etcd-io#17241 Signed-off-by: Wei Fu <fuweid89@gmail.com>
|
Resolved in #17304 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Which github workflows are flaking?
linux-amd64-e2e
Which tests are flaking?
The e2e testcases need more than 30min but workflow timeout is 30min
Github Action link
https://github.com/etcd-io/etcd/actions/runs/7505424685/job/20453159962?pr=17238
Reason for failure (if possible)
Need to investigate e2e runtime.
Anything else we need to know?
No response
The text was updated successfully, but these errors were encountered: