Multiple services per one ExitNode?

[insanity54]

Hi thanks for chisel-operator.

I'm looking to use it as a self-hosted ngrok alternative so I can have SSL in my development environment (kind cluster with Tilt.)

I'm trying out chisel-operator and I'm hoping to add two services to one ExitNode. In my case the two services are frontend (next.js) and backend (strapi.js.)

---
apiVersion: chisel-operator.io/v1
kind: ExitNode
metadata:
  name: uwu-exit-node
  namespace: futurenet
spec:
  host: "[REDACTED]"
  port: 9090
  auth: uwu-auth

---
apiVersion: v1
kind: Service
metadata:
  name: strapi
  namespace: futurenet
  # create a tunnel to uwu-exit-node (chisel server)
  # this allows us to have SSL in development
  annotations:
    chisel-operator.io/exit-node-name: "uwu-exit-node"
spec:
  selector:
    app.kubernetes.io/name: strapi
  ports:
    - name: http
      port: 1339
      targetPort: http
      protocol: TCP
  type: LoadBalancer

This setup is working. I have my chisel server running on a VPS and I'm seeing the strapi tunnel session established.

A problem arises when I add a second service, the frontend.

apiVersion: v1
kind: Service
metadata:
  name: next
  namespace: futurenet
  annotations:
    chisel-operator.io/exit-node-name: "uwu-exit-node"
spec:
  selector:
    app.kubernetes.io/name: next
  ports:
    - name: web
      port: 3000
      targetPort: web
      protocol: TCP
  type: LoadBalancer

With this second service, I don't see any tunnel session started on my chisel server logs. Looking in k9s at chisel-operator logs, I see an error.

│  load_balancer_class: None, load_balancer_ip: None, load_balancer_source_ranges: None, ports: Some([ServicePort { app_protocol: None, name: Some(\"web\"), │
│  node_port: Some(30159), port: 3000, protocol: Some(\"TCP\"), target_port: Some(String(\"web\")) }]), publish_not_ready_addresses: None, selector: Some({\ │
│ "app.kubernetes.io/name\": \"next\"}), session_affinity: Some(\"None\"), session_affinity_config: None, type_: Some(\"LoadBalancer\") }), status: Some(Ser │
│ viceStatus { conditions: None, load_balancer: Some(LoadBalancerStatus { ingress: None }) }) }"                                                             │
│ ts=2024-05-29T19:54:40.649276251Z level=error target=chisel_operator::daemon span=error_policy span_path="run>reconciling object>error_policy" err=NoAvail │
│ ableExitNodes object.ref=Service.v1./next.futurenet object.reason="error policy requested retry" 

The part that sticks out is, "NoAvailableExitNodes".

This led me to look through the docs some more. I wonder if this feature is not implemented? I found the TODO section in README.md which shows what I think is the feature I'm looking for. "Multiple tunnel services per exit node (so you don't have to pay for multiple VMs)."

I'm looking for some clarification of that feature. It looks like that feature has a checkmark as if it's completed, but it's still under the TODO heading so I'm unsure.

Is there a recommended way to do two services per one ExitNode?

[korewaChino]

If your services use HTTP, I recommend just running it behind a reverse proxy ingress like Traefik or NGINX.

[insanity54]

Thank you.

Just for my own understanding, I think what you're saying is to run the chisel server behind a reverse proxy ingress. That way, there can be one VPS handling traffic for two different kubernetes services.

I had intuitively already set up something like that using caddyserver.

# Caddyfile
strapi.chisel.example.com {
        reverse_proxy localhost:1339
}
next.chisel.example.com {
        reverse_proxy localhost:3000
}

I manually set up both DNS A records for (example) strapi.chisel.example.com and next.chisel.example.com to point to the VPS. This works great.

This isn't the problem I was bringing up in this issue, though.

The issue I meant to ask about is on the Kubernetes side. In my case it's a cluster running on localhost on a residential internet with a dynamic IP address. With only one ExitNode kubernetes object, uwu-exit-node, I think this means only one chisel tunnel can be created between the chisel client and chisel server.

I was wondering if there was a way to reference the exit node from two different kubernetes services. Illustrated example as follows.

  # next-svc.yaml
  # ...
  annotations:
    chisel-operator.io/exit-node-name: "uwu-exit-node"

  # strapi-svc.yaml
  # ...
  annotations:
    chisel-operator.io/exit-node-name: "uwu-exit-node"

With this setup where two services are referencing the same ExitNode, I was seeing that only the first one created was able to establish a chisel tunnel.

I might be doing something wrong though. I'll keep fiddling with it.

[korewaChino]

Just for my own understanding, I think what you're saying is to run the chisel server behind a reverse proxy ingress. That way, there can be one VPS handling traffic for two different kubernetes services.

Yes.

Anyway, the multiple services feature isn't implemented yet, I'm still figuring out whether if Chisel actually supports forwarding multiple processes to the server or not. I also need to change the CRD schema completely, so the new ExitNode CRD would be a breaking change sadly enough.

Once I get everything else in order, I'll try to work on chisel-operator and get this going.

[insanity54]

I realized I could use two ExitNodes instead of one, and they each connect to my singular chisel server. Thanks again, I'm getting a lot of value out of this project so I'll be sponsoring.

[korewaChino]

I will be fixing this in 0.4.

[korewaChino]

anyway I just realized this issue is a dupe of #9, closing this. Let's talk there instead.

[xyhhx]

anyway I just realized this issue is a dupe of #9, closing this. Let's talk there instead.

isn't that the opposite? they want to expose one service to multiple exit nodes; this wants to use one exit node to expose multiple services

[korewaChino]

Anyway, should be fixed in 0.4