Remove null -> [] slice hack

[liggitt]

Closes #44593

When 1.6 added protobuf storage, the storage layer lost the ability to persist slice fields with empty but non-null values.

As a workaround, we tried to convert empty slice fields to [], rather than null. Compressing null -> [] was just as much of an API breakage as [] -> null, but was hoped to cause fewer problems in clients that don't do null checks.

Because of conversion optimizations around converting lists of objects, the null -> [] hack was discovered to only apply to individual get requests, not to a list of objects. 1.6 and 1.7 was released with this behavior, and the world didn't explode. 1.7 documented the breaking API change that null and [] should be considered equivalent, unless otherwise noted on a particular field.

This PR:

• Reverts the earlier attempt (Ensure slices are serialized as zero-length, not null #43422) at ensuring non-null json slice output in conversion
• Makes results of get consistent with the results of list (which helps naive clients that do deepequal comparisons of objects obtained via list/watch and get), and allows empty slice fields to be returned as null

Protobuf serialization does not distinguish between `[]` and `null`.
API fields previously capable of storing and returning either `[]` and `null` via JSON API requests (for example, the Endpoints `subsets` field) can now store only `null` when created using the protobuf content-type or stored in etcd using protobuf serialization (the default in 1.6+). JSON API clients should tolerate `null` values for such fields, and treat `null` and `[]` as equivalent in meaning unless specifically documented otherwise for a particular field.

[k8s-reviewable]

This change is 

[dims]

/unassign @dims

[liggitt]

@smarterclayton the type aliasing needed to get custom json encoding is throwing off the protobuf encoder:

-	Subsets []EndpointSubset `json:"subsets" protobuf:"bytes,2,rep,name=subsets"`
+	Subsets EndpointSubsetList `json:"subsets" protobuf:"bytes,2,rep,name=subsets"`
 }
 
+// EndpointSubsetList represents a list of EndpointSubset objects.
+type EndpointSubsetList []EndpointSubset

is resulting in generated protobuf code like this:

-                       m.Subsets = append(m.Subsets, EndpointSubset{})
+                       m.Subsets = append(m.Subsets, EndpointSubsetList{})

any ideas?

[smarterclayton]

I wonder if defining a method on the type (or using a pointer??) confuses the generated code. There's a rewrite step in go-to-protobuf that uses AST traversal - it's possible that the change in traversal means the matchers don't match anymore

…

On Wed, May 3, 2017 at 3:35 PM, Jordan Liggitt ***@***.***> wrote: @smarterclayton <https://github.com/smarterclayton> the type aliasing needed to get custom json encoding is throwing off the protobuf encoder: - Subsets []EndpointSubset `json:"subsets" protobuf:"bytes,2,rep,name=subsets"`+ Subsets EndpointSubsetList `json:"subsets" protobuf:"bytes,2,rep,name=subsets"` } +// EndpointSubsetList represents a list of EndpointSubset objects.+type EndpointSubsetList []EndpointSubset is resulting in generated protobuf code like this: - m.Subsets = append(m.Subsets, EndpointSubset{}) + m.Subsets = append(m.Subsets, EndpointSubsetList{}) any ideas? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#45294 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p42euwdMkHMGaqKLb6FcZzbrYdzMks5r2NcIgaJpZM4NPuTK> .

[liggitt]

@smarterclayton - got this working, PTAL

[liggitt]

@wojtek-t PTAL at toUnstructured changes in the "Fix unstructured marshaler to handle all JSON types" commit

[wojtek-t]

@wojtek-t PTAL at toUnstructured changes in the "Fix unstructured marshaler to handle all JSON types" commit

Those changes (last commit) LGTM.

[smarterclayton]

LGTM. I have no objections and follow Jordan's reasoning on this.

[smarterclayton]

/lgtm

[k8s-github-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt, smarterclayton, thockin

Associated issue: 44593

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

• cmd/kubeadm/OWNERS [smarterclayton,thockin]
• federation/OWNERS [smarterclayton,thockin]
• pkg/api/OWNERS [liggitt,smarterclayton,thockin]
• pkg/kubelet/apis/kubeletconfig/OWNERS [smarterclayton,thockin]
• plugin/pkg/admission/resourcequota/apis/resourcequota/OWNERS [smarterclayton,thockin]
• plugin/pkg/auth/OWNERS [liggitt,smarterclayton,thockin]
• staging/OWNERS [smarterclayton,thockin]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

[liggitt]

/retest

[liggitt]

/retest

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[k8s-ci-robot]

@liggitt: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
Jenkins verification	7d9f089ad1dd65f6f708cff548b21652d3a596a2	link	@k8s-bot verify test this

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[k8s-github-robot]

/test all [submit-queue is verifying that this PR is safe to merge]

[k8s-github-robot]

Automatic merge from submit-queue