Investigate why increasing worker count/QPS causes E2E tests to fail

[jkowalski]

Got some weird test failures in #497, mostly centered around allocations. Those could be due test flakes due to increased parallelism or a real issue. Lowering worker count back to 2 did not help.

Example failure:

Step #7: {"error":"Internal error occurred: admission webhook \"mutations.stable.agones.dev\" denied the request: NotFound","message":"Allocation ended","severity":"info","time":"2019-01-23T19:12:27.277443739Z"}
Step #7: --- FAIL: TestGameServerAllocationDuringGameServerDeletion (0.00s)
Step #7:     --- PASS: TestGameServerAllocationDuringGameServerDeletion/scale_down (28.39s)
Step #7:     --- FAIL: TestGameServerAllocationDuringGameServerDeletion/recreate_update (39.20s)
Step #7:         <autogenerated>:1: 
Step #7:             	Error Trace:	fleet_test.go:512
Step #7:             	            				fleet_test.go:469
Step #7:             	            				asm_amd64.s:1333
Step #7:             	Error:      	Expected nil, but got: &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:""}, Status:"Failure", Message:"Operation cannot be fulfilled on fleets.stable.agones.dev \"simple-fleet-rk8k7\": the object has been modified; please apply your changes to the latest version and try again", Reason:"Conflict", Details:(*v1.StatusDetails)(0xc0003924e0), Code:409}}
Step #7:             	Test:       	TestGameServerAllocationDuringGameServerDeletion/recreate_update
Step #7:     --- FAIL: TestGameServerAllocationDuringGameServerDeletion/rolling_update (41.16s)
Step #7:         <autogenerated>:1: 
Step #7:             	Error Trace:	fleet_test.go:534
Step #7:             	            				fleet_test.go:469
Step #7:             	            				asm_amd64.s:1333
Step #7:             	Error:      	Expected nil, but got: &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:""}, Status:"Failure", Message:"Operation cannot be fulfilled on fleets.stable.agones.dev \"simple-fleet-tcm2q\": the object has been modified; please apply your changes to the latest version and try again", Reason:"Conflict", Details:(*v1.StatusDetails)(0xc0003e8600), Code:409}}
Step #7:             	Test:       	TestGameServerAllocationDuringGameServerDeletion/rolling_update

See #497 for more failure logs.

[markmandel]

Can we close this one now? Or are we still getting some flaky tests?

[jkowalski]

Yes, can be closed.