e2e testcase to automatically test k8s nodeport feature

[liwenwu-amazon]

In PR #130, we have manually tested K8S nodeport feature using following steps. We should write a e2e testcase which can automatically execute these steps. Here are the detail steps:

• create a cluster with 3 nodes (e.g. t2.medium)
• kubectl create -f hello.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
spec:
  selector:
    matchLabels:
      app: hello
      tier: backend
      track: stable
  replicas: 15 <-- so that IP of primary ENI are used by these Pods
  template:
    metadata:
      labels:
        app: hello
        tier: backend
        track: stable
    spec:
      containers:
        - name: hello
          image: "gcr.io/google-samples/hello-go-gke:1.0"
          ports:
            - name: http
              containerPort: 80

• kubectl apply -f hello-service.yaml

kind: Service
apiVersion: v1
metadata:
  name: hello
spec:
  selector:
    app: hello
    tier: backend
  ports:
  - protocol: TCP
    port: 80
    targetPort: http

• kubectl apply -f frontend.yaml

apiVersion: v1
kind: Service
metadata:
  name: frontend
spec:
  selector:
    app: hello
    tier: frontend
  ports:
  - protocol: "TCP"
    port: 80
    targetPort: 80
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
spec:
  selector:
    matchLabels:
      app: hello
      tier: frontend
      track: stable
  replicas: 1
  template:
    metadata:
      labels:
        app: hello
        tier: frontend
        track: stable
    spec:
      containers:
      - name: nginx
        image: "gcr.io/google-samples/hello-frontend:1.0"
        lifecycle:
          preStop:
            exec:
              command: ["/usr/sbin/nginx","-s","quit"]

• verify frontend pod is NOT using primary eni

# find out node which runs frontend pod
kubectl get pod -o wide| grep frontend

# ssh into node where frontend pod runs

# ip rule show  to verify frontend pod are using secondary ENIs

• verify node port

# find out port for frontend service
kubectl get svc frontend

# log into one of hello pod
kubectl exec  -ti <hello-pod-xxx> sh

# for each node in the cluster
wget http://<node-ip>:<frontend-port>  should return index.html that contains {"message":"Hello"}

• enable calico policy engine add-on

kubectl apply -f config/v1.1/calico.yaml

• apply Network Policy allow connections from other hosts in the VPC (including K8S nodes) to access node port kubectl apply -f allow-policy.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-hosts
  namespace: default
spec:
  podSelector:
    matchLabels:
      tier: frontend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - ipBlock:
        cidr: 192.168.0.0/16 <-- VPC CIDR

• repeat node port tests step earlier. It should succeed

• delete allow-policy.yaml kubectl delete -f allow-policy.yaml

• apply Network Policy which disallow other hosts in VPC to access node port kubectl apply -f deny-policy.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: default
spec:
  podSelector:
    matchLabels:
      tier: frontend
  policyTypes:
  - Ingress
  ingress: []

• repeat node port tests step earlier. It should failed.

[infa-rasrinivasan]

@liwenwu-amazon - Is this still needed ?

[mogren]

We are currently working on an e2e test suite for the CNI where this could be added. @tiffanyfay

[jayanthvn]

Nodeport tests are added to the integration tests. Adding calico policy is pending.

[jayanthvn]

#1906 calico tests are added. Closing this issue.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.