Starting test plan section of Windows KEP #685

PatrickLang · 2019-01-12T01:47:38Z

I added the test plan into the outline and added the current tests that have been merged and are running in testgrid. This isn't a complete list (see the TODOs), but it's a start.

@astrieanna and @benmoss - any feedback?

Can you take a look @adelina-t and @BCLAU ? Do you think this format makes sense for the tests?

/sig windows

This is continuing to add content to the draft merged in #676

astrieanna · 2019-01-14T15:53:09Z

@PatrickLang It seems like a very long list of tests. Is it useful to list all of them in the KEP when presumably tagging in the code will be the source of truth on that?

PatrickLang · 2019-01-14T18:07:45Z

@astrieanna yeah, I know the list is long but I was getting questions and feedback on "does X work" last time. Let me see if I can get someone from SIG-Testing to clarify what level of detail they'd like to see here.

PatrickLang · 2019-01-14T18:16:43Z

/milestone v1.14

benmoss · 2019-01-14T22:20:59Z

@PatrickLang do we have any plan for keeping track of & making progress on non-conformance e2e tests? We were just looking at the tests for port-forwarding, and with our change some of them pass, and the images were already part of the set ported to Windows, but they aren't being run in Testgrid right now.

jdumars · 2019-01-14T22:37:56Z

/hold

jdumars · 2019-01-14T22:38:45Z

/lgtm
SIG stakeholders, feel free to remove hold at your discretion.

spiffxp · 2019-01-14T23:41:22Z

So, I'm torn since I'm coming from multiple perspectives.

A) This does feel like a lot to enumerate and I’m not sure it’s sig-testing’s job to validate the exhaustive enumeration, but I suspect we have a say in whether the plan seems reasonable. It does seem reasonable to me.

B) As a consumer of this information from the Release Team, I think I'm just asking for a dashboard, or list of dashboards to check. So I would be fine if your test plan was just: "we're going to call this done when all tests at [sig-windows/the-blocking-tests] are green", or "[sig-windows/existing-tests], [sig-windows/substitute-tests], [sig-windows/windows-only-tests]" or something.

To me that then begs the question of whether getting to green by PR'ing failing tests out of those dashboards is OK, or whether the list of tests in those dashboards is OK.

C) As someone who watched SIG Arch flail during 1.13, I feel like there is a desire for exhaustive enumeration of some sort, to figure out whether these tests, or whether the grouping of tests is sufficient to meet user expectations for "how a kubernetes behaves". Reviewing that is something that, to me, seems like it's in SIG Arch's purview (I am super open to suggestions on which SIGs should be involved/federated to instead)

In the interest of an initial pass at this, I'd be in favor of A) & B) where you just describe the test plan and dashboards. I would run by sig-arch whether they need more, or would be comfortable using the dashboards and evaluating at some checkpoint.

PatrickLang · 2019-01-14T23:51:25Z

Thanks @spiffxp . I'll change it to a summary level of detail now for A/B, and if other reviewers want to review it test case by test case - git to the rescue!

mattfarina · 2019-01-15T15:45:38Z

Something I'm considering is perspective of those looking in. When we reviewed the tests in the SIG Arch conversation on Windows GA we looked at a dashboard of failing tests. I would suggest having a dashboard of windows tests and that they be generally as green as those for Linux nodes.

I like the plan of updating tests cases to handle multi-os, having windows specific tests, and having substitute tests cases. With my SIG Apps hat on, seeing the workloads APIs passing on Windows would be fantastic.

michmike · 2019-01-18T03:04:39Z

keps/sig-windows/0000-20190103-windows-node-support.md

+1. [Adapting](#Adapting-existing-tests) some of the existing conformance tests to be able to pass on multiple node OS's
+2. Adding [substitute](#Substitute-test-cases) test cases where the first approach isn't feasible or would change the tests in a way is not approved by the owner. These will be tagged with `[SIG-Windows]`
+3. Last, gaps will be filled with [Windows specific tests](#Windows-specific-tests). These will also be tagged with `[SIG-Windows]`
+


I think it is important to note that any test that is tagged with [SIG-Windows] should always pass on windows. we should specify this in the text.

In addition, is it possible to tag any of the tests that are not applicable to windows with any tag? that way it is easy to differentiate which of the existing conformance tests are not applicable to windows (beyond the ones that were substituted)

michmike

approving this PR to get it merged with the rest of the KEP so that we can continue iterating over it.

michmike · 2019-01-18T03:06:53Z

/approve
/lgtm

michmike · 2019-01-18T03:11:13Z

So, I'm torn since I'm coming from multiple perspectives.

A) This does feel like a lot to enumerate and I’m not sure it’s sig-testing’s job to validate the exhaustive enumeration, but I suspect we have a say in whether the plan seems reasonable. It does seem reasonable to me.

B) As a consumer of this information from the Release Team, I think I'm just asking for a dashboard, or list of dashboards to check. So I would be fine if your test plan was just: "we're going to call this done when all tests at [sig-windows/the-blocking-tests] are green", or "[sig-windows/existing-tests], [sig-windows/substitute-tests], [sig-windows/windows-only-tests]" or something.

To me that then begs the question of whether getting to green by PR'ing failing tests out of those dashboards is OK, or whether the list of tests in those dashboards is OK.

C) As someone who watched SIG Arch flail during 1.13, I feel like there is a desire for exhaustive enumeration of some sort, to figure out whether these tests, or whether the grouping of tests is sufficient to meet user expectations for "how a kubernetes behaves". Reviewing that is something that, to me, seems like it's in SIG Arch's purview (I am super open to suggestions on which SIGs should be involved/federated to instead)

In the interest of an initial pass at this, I'd be in favor of A) & B) where you just describe the test plan and dashboards. I would run by sig-arch whether they need more, or would be comfortable using the dashboards and evaluating at some checkpoint.

i think for this initial iteration of the KEP, we do need (a), (b), and (c) to satisfy curious minds. Ultimately the dashboard with green checkboxes across all tests applicable to windows is what most folks will be looking for, but the approvers of the KEP will likely need to dig in.

PatrickLang · 2019-01-22T04:30:49Z

@justaugustus removed KEP number & renamed

…t cases

PatrickLang · 2019-01-23T01:24:58Z

@benmoss, @astrieanna, @michmike, @adelina-t or @yujuhong - can you review and LGTM? I'd like to get this next update merged. It has a list of the remaining test work that I'm aware of.

More updates coming:

@craiglpeters is working on a PR updating the release criteria and final approver list after some discussions this week
I'm going to do a separate PR bringing in an API support list

claudiubelu · 2019-01-23T13:57:43Z

keps/sig-windows/20190103-windows-node-support.md

+
+- [ ] [sig-network] [sig-windows] Networking Granular Checks: Pods should function for intra-pod communication: http - [PR#71468](https://github.com/kubernetes/kubernetes/pull/71468)
+- [ ] [sig-network] [sig-windows] Networking Granular Checks: Pods should function for intra-pod communication: udp - [PR#71468](https://github.com/kubernetes/kubernetes/pull/71468)
+- [ ] [sig-network] [sig-windows] Networking Granular Checks: Pods should function for node-pod communication: udp - [PR#71468](https://github.com/kubernetes/kubernetes/pull/71468)


For the sake of completeness, you should also include: [sig-network] [sig-windows] Networking Granular Checks: Pods should function for node-pod communication: http, same PR.

Oops my mistake. Adding now

claudiubelu · 2019-01-23T13:59:08Z

As far the tests and the required patches on the testing side, other than the comment above, LGTM.

adelina-t · 2019-01-23T16:13:17Z

LGTM

michmike · 2019-01-24T07:01:00Z

/lgtm
/approve

k8s-ci-robot · 2019-01-24T07:01:09Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jdumars, michmike, PatrickLang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~keps/sig-windows/OWNERS~~ [PatrickLang,jdumars,michmike]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

michmike · 2019-01-24T07:02:56Z

/hold cancel

yujuhong · 2019-01-24T00:16:11Z

keps/sig-windows/20190103-windows-node-support.md

+
+### Non-Goals
+
+- Adding Windows support to all projects in the Kubernetes ecosystem (Cluster Lifecycle, etc)


It's probably pretty obvious to some, but for clarity, I'd suggest adding

- Running Kubernetes master control-plane on Windows nodes.

PR 729 already had this change as Enable the Kubernetes master components to run on Windows

yujuhong · 2019-01-24T00:21:48Z

keps/sig-windows/20190103-windows-node-support.md

+
+The testing for Windows nodes will include multiple approaches:
+
+1. [Adapting](#Adapting-existing-tests) some of the existing conformance tests to be able to pass on multiple node OS's. Tests that won't work will be [excluded](https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes-sigs/sig-windows/sig-windows-config.yaml#L69).


Can we create a file in the "windows-testing" listing the tests excluded? The config is hard to parse, and also the line is likely to change unless you get the permanent link.

I believe those tests are all being tagged [LinuxOnly].
kubernetes/kubernetes#73204
kubernetes/community#3140

yujuhong · 2019-01-24T01:12:57Z

keps/sig-windows/20190103-windows-node-support.md

+
+#### Adapting existing tests
+
+Over the course of v1.12/13, many conformance tests were adapted to be able to pass on either Linux or Windows nodes as long as matching OS containers are run. This was done by creating Windows equivalent containers from [kubernetes/test/images](https://github.com/kubernetes/kubernetes/tree/master/test/images). An additional parameter is needed for e2e.test/kubetest to change the container repos to the one containing Windows versions since they're not part of the Kubernetes build process yet.


We'd need an intermediate solution for managing images soon. IIUC, you can only substitute registry/repos for the test images. It'd be hard to keep the "version/tag" of the windows images in sync with the linux ones. Maybe we can extend the e2e framework to consume images (registry, repository, tag/version) directly from a yaml file? I'd suggest opening an issue in k8s.io/kubernetes and carry on the discussion over there. This KEP can link to that issue.

bgrant0607 · 2019-01-26T01:23:39Z

In the future, please do such file renames as separate commits or PRs. It's hard to see what changed through the UI

bgrant0607 · 2019-01-26T02:16:16Z

keps/sig-windows/20190103-windows-node-support.md

+Additional dashboard pages will be added over time as we run the same test cases with additional CRI, CNI and cloud providers. They reflect work that may be stabilized in v1.15 or later and is not strictly required for v1.14.
+
+- Windows Server 2019 on GCP - this is [in progress](https://k8s-testgrid.appspot.com/google-windows#windows-prototype), but not required for v1.14
+- Windows Server 2019 with OVN+OVS & Dockershim


Are these additional dashboards? Are they all required other than GCP? What configuration will be shown on the main dashboard?

Will clarify in the next update on #732

bgrant0607 · 2019-01-26T05:21:43Z

keps/sig-windows/20190103-windows-node-support.md

+
+All of the test cases will be maintained within the kubernetes/kubernetes repo. SIG-Windows specific tests for 2/3 will be in [test/e2e/windows](https://github.com/kubernetes/kubernetes/tree/master/test/e2e/windows)
+
+Additional Windows test setup scripts, container image source code, and documentation will be kept in the [kubernetes-sigs/windows-testing](https://github.com/kubernetes-sigs/windows-testing) repo. One example is that the prow jobs need a list of repos to use for the test containers, and that will be maintained here - see [windows-testing#1](https://github.com/kubernetes-sigs/windows-testing/issues/1).


Which registry will these images be stored in?

https://github.com/kubernetes/community/blob/master/contributors/guide/coding-conventions.md#testing-conventions

Right now they're in https://hub.docker.com/u/e2eteam but ideally we can consolidate these in the same registry as the other Kubernetes images

These could be pushed to GCR if someone gives us the right access.

We can't build Windows containers in the main build today, but that's the eventual goal. Building these at the same cadence will require adding Windows nodes into the build pool, or moving all images to languages that can be cross-compiled and put into containers with only offline operations (no RUN commands in Dockerfile).

Patrick Lang added 3 commits January 11, 2019 17:00

Adding testing plan outline

b219537

I'm Patrick, I use vim

abf9f58

Adding first portion of test cases

223cbe6

k8s-ci-robot added sig/windows Categorizes an issue or PR as relevant to SIG Windows. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jan 12, 2019

k8s-ci-robot requested review from jdumars and michmike January 12, 2019 01:47

k8s-ci-robot added this to the v1.14 milestone Jan 14, 2019

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 14, 2019

k8s-ci-robot assigned jdumars Jan 14, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 14, 2019

michmike reviewed Jan 18, 2019

View reviewed changes

michmike approved these changes Jan 18, 2019

View reviewed changes

k8s-ci-robot assigned michmike Jan 18, 2019

Patrick Lang added 3 commits January 21, 2019 20:11

Merge from master

1392727

Removing NEXT_KEP_NUMBER references

46da808

Updating with sig list & approver as of 17 Jan SIG-Architecture meeting

218239a

Patrick Lang added 5 commits January 22, 2019 15:53

Fix bad merge

ed93cb6

Re-remove .gitignore added in merge

8c8b7b7

Reference dashboard for test list instead of enumerating existing tes…

9d407f6

…t cases

Adding some in-progress test PRs

c8dd84d

Minor test list cleanup

e25db5b

claudiubelu reviewed Jan 23, 2019

View reviewed changes

Adding missing test case in PR 71468

9787ffc

michmike approved these changes Jan 24, 2019

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 24, 2019

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 24, 2019

k8s-ci-robot merged commit 77ae66a into kubernetes:master Jan 24, 2019

yujuhong reviewed Jan 25, 2019

View reviewed changes

bgrant0607 reviewed Jan 26, 2019

View reviewed changes

bgrant0607 mentioned this pull request Jan 26, 2019

Adding API section to Windows KEP #732

Merged

PatrickLang mentioned this pull request Jan 28, 2019

Rename SIG-Windows conformance to release on TestGrid kubernetes/test-infra#10989

Closed

bgrant0607 mentioned this pull request Jan 28, 2019

Create tooling to generate / manage KEPs #619

Closed

astrieanna mentioned this pull request Feb 15, 2019

REQUEST: New membership for astrieanna kubernetes/org#492

Closed

6 tasks

justaugustus removed the sig/pm label Apr 19, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Starting test plan section of Windows KEP #685

Starting test plan section of Windows KEP #685

PatrickLang commented Jan 12, 2019 •

edited

Loading

astrieanna commented Jan 14, 2019

PatrickLang commented Jan 14, 2019

PatrickLang commented Jan 14, 2019

benmoss commented Jan 14, 2019 •

edited

Loading

jdumars commented Jan 14, 2019

jdumars commented Jan 14, 2019

spiffxp commented Jan 14, 2019

PatrickLang commented Jan 14, 2019

mattfarina commented Jan 15, 2019

michmike Jan 18, 2019

michmike left a comment

michmike commented Jan 18, 2019

michmike commented Jan 18, 2019

PatrickLang commented Jan 22, 2019

PatrickLang commented Jan 23, 2019

claudiubelu Jan 23, 2019

PatrickLang Jan 23, 2019

claudiubelu commented Jan 23, 2019

adelina-t commented Jan 23, 2019

michmike commented Jan 24, 2019

k8s-ci-robot commented Jan 24, 2019

michmike commented Jan 24, 2019

yujuhong Jan 24, 2019

michmike Jan 25, 2019

yujuhong Jan 24, 2019

bgrant0607 Jan 26, 2019

yujuhong Jan 24, 2019

bgrant0607 commented Jan 26, 2019

bgrant0607 Jan 26, 2019

PatrickLang Jan 28, 2019

bgrant0607 Jan 26, 2019

benmoss Jan 27, 2019

PatrickLang Jan 28, 2019


		### Non-Goals

		- Adding Windows support to all projects in the Kubernetes ecosystem (Cluster Lifecycle, etc)


		The testing for Windows nodes will include multiple approaches:

		1. [Adapting](#Adapting-existing-tests) some of the existing conformance tests to be able to pass on multiple node OS's. Tests that won't work will be [excluded](https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes-sigs/sig-windows/sig-windows-config.yaml#L69).


		#### Adapting existing tests

		Over the course of v1.12/13, many conformance tests were adapted to be able to pass on either Linux or Windows nodes as long as matching OS containers are run. This was done by creating Windows equivalent containers from [kubernetes/test/images](https://github.com/kubernetes/kubernetes/tree/master/test/images). An additional parameter is needed for e2e.test/kubetest to change the container repos to the one containing Windows versions since they're not part of the Kubernetes build process yet.


		All of the test cases will be maintained within the kubernetes/kubernetes repo. SIG-Windows specific tests for 2/3 will be in [test/e2e/windows](https://github.com/kubernetes/kubernetes/tree/master/test/e2e/windows)

		Additional Windows test setup scripts, container image source code, and documentation will be kept in the [kubernetes-sigs/windows-testing](https://github.com/kubernetes-sigs/windows-testing) repo. One example is that the prow jobs need a list of repos to use for the test containers, and that will be maintained here - see [windows-testing#1](https://github.com/kubernetes-sigs/windows-testing/issues/1).

Starting test plan section of Windows KEP #685

Starting test plan section of Windows KEP #685

Conversation

PatrickLang commented Jan 12, 2019 • edited Loading

astrieanna commented Jan 14, 2019

PatrickLang commented Jan 14, 2019

PatrickLang commented Jan 14, 2019

benmoss commented Jan 14, 2019 • edited Loading

jdumars commented Jan 14, 2019

jdumars commented Jan 14, 2019

spiffxp commented Jan 14, 2019

PatrickLang commented Jan 14, 2019

mattfarina commented Jan 15, 2019

Choose a reason for hiding this comment

michmike left a comment

Choose a reason for hiding this comment

michmike commented Jan 18, 2019

michmike commented Jan 18, 2019

PatrickLang commented Jan 22, 2019

PatrickLang commented Jan 23, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

claudiubelu commented Jan 23, 2019

adelina-t commented Jan 23, 2019

michmike commented Jan 24, 2019

k8s-ci-robot commented Jan 24, 2019

michmike commented Jan 24, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bgrant0607 commented Jan 26, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PatrickLang commented Jan 12, 2019 •

edited

Loading

benmoss commented Jan 14, 2019 •

edited

Loading