IPPool is empty error #5599
Comments
|
Hello @yuraxdrumz ! As I can see your apps are based on Feb 26 changes (see at commit 8ddd0be6e5d1 in SDK) The problems could be resolved in the main version of nsm Could you check this with latest version |
|
Hello @denis-tingaikin alpine-composition.txt Strangly enough, I dont see any logs in passthroughs or nse and upon running the nsc for the first time, I get to the request timeout with the Because once in a while it does work, I ruled out the security-group issue. Another thing I tried was putting all components on a single node and I saw that if I restart the manager, I see the nsc trying to reach an old instance of the nse, which should have been deleted from the registry already. The one line that stood out was Apr 28 09:14:14.495 [TRAC] [type:registry] (14.1) recv-response={"network_service_endpoint":{"name":"nse-kernel-69dccd667b-5prpc","network_service_names":["composition"],"network_service_labels":{"composition":{"labels":{"app":"destination-app","nodeName":"ip-192-168-189-67.eu-west-1.compute.internal"}}},"url":"tcp://192.168.189.67:5001","expiration_time":{"seconds":1651072641,"nanos":98046844},"initial_registration_time":{"seconds":1651072144,"nanos":997370982}}} span=40199ab3a2a865e8After checking the expiration of the above EDIT: my guess is the old nse gets selected and it does not exist, so I get to timeout, but for some reason afterwards, I see the IP Pool error |
|
The problems networkservicemesh/sdk#1020 and networkservicemesh/sdk#1007 are not actual. The errors what you're seeing is related to the first incorrect request from the client (nsc). As I can see by some reason the forwarder is selecting @yuraxdrumz Could you please attach your And also it would be nice to get logs from the nse-kernel and from |
|
I tried deleting and switching the apiVersion: networkservicemesh.io/v1
kind: NetworkService
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: >
{"apiVersion":"networkservicemesh.io/v1","kind":"NetworkService","metadata":{"annotations":{},"name":"composition","namespace":"nsm-system"},"spec":{"matches":[{"routes":[{"destination_selector":{"app":"destination-app"}}],"source_selector":{"app":"passthrough-1"}},{"routes":[{"destination_selector":{"app":"passthrough-1"}}]}],"payload":"ETHERNET"}}
creationTimestamp: '2022-04-28T09:06:10Z'
generation: 1
managedFields:
- apiVersion: networkservicemesh.io/v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:spec: {}
manager: kubectl-client-side-apply
operation: Update
time: '2022-04-28T09:06:10Z'
name: composition
namespace: nsm-system
resourceVersion: '5735744'
uid: e4935cd7-4339-42e0-976b-5685dba72d1b
selfLink: >-
/apis/networkservicemesh.io/v1/namespaces/nsm-system/networkservices/composition
spec:
matches:
- routes:
- destination_selector:
app: destination-app
source_selector:
app: passthrough-1
- routes:
- destination_selector:
app: passthrough-1
payload: ETHERNET
Regarding the all The nsm annotation used is annotations:
networkservicemesh.io: kernel://composition/nsm-1full nsc-config ---
apiVersion: v1
kind: Pod
metadata:
name: alpine-composition
labels:
app: alpine-composition
"spiffe.io/spiffe-id": "true"
annotations:
networkservicemesh.io: kernel://composition/nsm-1
spec:
nodeName: ip-192-168-189-67.eu-west-1.compute.internal
containers:
- name: alpine
image: alpine:3.15.0
imagePullPolicy: IfNotPresent
stdin: true
tty: truelogs from cmd-nsc-init alpine-composition.txt |
|
@yuraxdrumz Thanks! Looking Question: Did you try to run this example https://github.com/networkservicemesh/deployments-k8s/tree/main/examples/features/nse-composition without changes? UPD: I'd recommend to delete all non forwarders nses |
|
@yuraxdrumz Im not sure but probabbly you got something like this:
Could you try this: Steps:
Actual: TODO |
|
@yuraxdrumz We have AWS integration testing, but we're using 2 nodes setup. I'll try to reproduce your problem at Monday with your AWS setup. Let me know if you get any updates with ☝️ |
|
Hey @denis-tingaikin.
Same error as before, adding the nsc-cmd-init log - alpine.txt Expected: NSM is working |
|
Anything new? |
|
Related to: networkservicemesh/sdk#1297 |
|
@glazychev-art This is really good catch. Im a bit poked with a code, and I found the real root cause - networkservicemesh/cmd-exclude-prefixes-k8s#144 |
|
@yuraxdrumz Should be fixed for the latest master. Could you re-test ? |
Hello.
During my tests with NSM, things seem to work fine until, sometimes, I reach a point where I get
IPPool is emptyerror and I can't even run 1 nsc.I am doing lots of tests and I delete and add nsc's and nse's all the time.
My guess is something is not getting released with the IP Pool.
I tried running a different CIDR, tried recreating the
NetworkService, tried creating newNetworkService, tried restarting all NSM control plane and data plane components, all of which do not help.I am running on EKS with 4 nodes of m5.large with cilium CNI and vpp as forwarder.
nsc error log - cmd-nsc-init-err.txt
Passthrough example - passthrough.zip
I would love to help on this if needed, thanks!
The text was updated successfully, but these errors were encountered: