Add Fuzz Tests

[roberthbailey]

We should consider adding fuzz testing to Agones.

Kubernetes uses https://github.com/google/gofuzz (originally was in the Kubernetes repo but was split out).

https://github.com/google/oss-fuzz is Google’s free fuzzing-as-a-service platform and has recently added support for Go fuzzers.

There are probably other options as well that I'm not aware of.

[aLekSer]

Question here: What parts of Agones should be covered with Fuzz testing?
Found a recent thread in Kubernetes slack about fuzz testing of CRDs:
https://kubernetes.slack.com/archives/CAR30FCJZ/p1573755986055900
I have tried to make a CRD fuzz test using this tool:
https://github.com/munnerz/crd-schema-fuzz
But we are not using fresh dependencies so I was not able to build a simple test.
Also I could not find https://github.com/kubernetes/apimachinery/tree/master/pkg/api/apitesting this functionality in the vendored apimachinery.

[roberthbailey]

Question here: What parts of Agones should be covered with Fuzz testing?

I would imagine that our api surface would make the most sense. I think looking at how k8s does fuzz testing and doing something similar (which it looks like you were trying to do) would be a great starting point.

[aLekSer]

Thanks for your answer.
Two more projects which is worth to consider:
https://github.com/fuzzitdev/fuzzit
https://github.com/dvyukov/go-fuzz

Fuzzit is free for open source, prometheus is using it, it adds a new fuzzit: pass label on repo main page, and could be exucuted on every pull request.

Currently investigating if we can add fuzz tests also for applyBufferPolicy() func of a FleetAutoScaler.

[aLekSer]

We should somehow use fuzzer to make round-trip test of API:
https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md#update-the-fuzzer

[aLekSer]

Followed an example from the book "Programming Kubernetes" https://github.com/programming-kubernetes/pizza-apiserver/tree/master/pkg/apis/restaurant/fuzzer
Was able to run a simple roundtrip fuzz test for agones.dev/v1.
It seems that we need to have additional addKnownTypes() call for the __internal GroupVersion:

var SchemeGroupVersion = schema.GroupVersion{Group: agones.GroupName, Version: k8sruntime.APIVersionInternal}
//with next line commented out
// metav1.AddToGroupVersion(scheme, SchemeGroupVersion)

[aLekSer]

Picked this ticket back. Now I am using this code as an example:
k8s.io/api/roundtrip_test.go
The point which is failing is:

    roundtrip.go:385: GameServerAllocationPolicyList: fuzzing a copy altered the original, diff:
        object.Items[0].Spec.ConnectionInfo.AllocationEndpoints[0]:
          a: "昭鴁Ō\u00a0袺\"祋ī·#x"
          b: "昭鴁Ō\u00a0袺\"祋ī·#"
    roundtrip.go:166: round tripping /v1, Kind=Status
    roundtrip.go:254:   round tripping external type /v1, Kind=Status
    roundtrip.go:166: round tripping autoscaling.agones.dev/v1, Kind=FleetAutoscalerList
    roundtrip.go:254:   round tripping external type autoscaling.agones.dev/v1, Kind=FleetAutoscalerList
    roundtrip.go:166: round tripping allocation.agones.dev/v1, Kind=CreateOptions
    roundtrip.go:254:   round tripping external type allocation.agones.dev/v1, Kind=CreateOptions
    roundtrip.go:166: round tripping allocation.agones.dev/v1, Kind=UpdateOptions
    roundtrip.go:254:   round tripping external type allocation.agones.dev/v1, Kind=UpdateOptions
    roundtrip.go:166: round tripping multicluster.agones.dev/v1alpha1, Kind=GameServerAllocationPolicy
    roundtrip.go:254:   round tripping external type multicluster.agones.dev/v1alpha1, Kind=GameServerAllocationPolicy
    roundtrip.go:166: round tripping agones.dev/v1, Kind=GameServer
    roundtrip.go:254:   round tripping external type agones.dev/v1, Kind=GameServer
    roundtrip.go:166: round tripping agones.dev/v1, Kind=Fleet
    roundtrip.go:254:   round tripping external type agones.dev/v1, Kind=Fleet
    roundtrip.go:166: round tripping /v1, Kind=APIResourceList
    roundtrip.go:254:   round tripping external type /v1, Kind=APIResourceList
    roundtrip.go:166: round tripping allocation.agones.dev/v1, Kind=GameServerAllocationList
    roundtrip.go:254:   round tripping external type allocation.agones.dev/v1, Kind=GameServerAllocationList
FAIL

[roberthbailey]

It looks like that field is in the v1alpha1 api surface:

agones/pkg/apis/multicluster/v1alpha1/gameserverallocationpolicy.go

Line 40 in 3caee5c

AllocationEndpoints []string `json:"allocationEndpoints,omitempty"`

which means that we can fix it without it being a breaking change.

/cc @pooneh-m (for when she's back in the office)

[roberthbailey]

Is there a way to only fuzz the v1 apis to start so that we can get your PR merged while we decide how to fix this?

[aLekSer]

Hello, will do, I think we can skip alpha for now. Nice solution.