Route based on ports

[skyrocknroll]

Hi ,
We have multiple micro services , Each microservice is running in multiple nodes. Is it possible to configure fabio to listen on one port per microservice ? Then whatever request comes to that port will be sent the corresponding microservice in a loadbalanced manner.

Ex:

port 9999 : authservice running on 10.0.0.1,10.0.0.2
port 7888 : usermanagement running on 10.0.1.1,10.0.1.2

[magiconair]

Not right now. You can configure multiple listeners but the routing table is shared among them and routing is currently done only via host/path. To fully support this fabio would need to be able to dynamically create and shutdown listeners. I need this for the TCP server support but right now that is not possible.

[skyrocknroll]

@magiconair Thanks :)

[mazhack]

@skyrocknroll do not use ports for micro-services, is a nightmare with proxies and firewalls, better use paths or subdomains.

if prefer subdomains using a wildcard domain and wilcard ssl.

[chy168]

+1 for this idea.

In my case, I might separate HTTP and HTTPS for different backend, for example:

• HTTP for nginx
• HTTPS for API server

For API server, i don't want it exposed via HTTP.

in mean time to workaround this, i create TWO fabio containers and use different prefix
registry.consul.tagprefix = urlprefix- to serve nginx and API server independently.

[mazhack]

see #90

[magiconair]

@chy168 with the fix for #90 you can announce a urlprefix-foo.com:443/api which will only be accessible via https assuming your HTTPS listener is configured for port 443. To route based on protocol I need to fix something else first but that should help you for now.

[magiconair]

@skyrocknroll what you could do is to run one fabio per port and use different tag prefixes but that is pushing fabio a bit beyond of what it is currently designed to do. You certainly lose the ability to just route to a new service without configuration since you have to spin up a fabio instances for that new port. I'd go with @mazhack that using static ports for incoming requests is not a good idea. Use different URLs instead but that might not be possible for you.

[stevenscg]

Fantastic work on the project @magiconair!

I wanted to see if there were any new considerations for this request (routing based on ports) after HashiConf EU 2016.

Kelsey mentioned running Fabio as a system service on all nodes of a Nomad cluster to stabilize the ports registered with upstream load balancers (ELB, etc).

Host and/or path-based routing makes total sense when the requested host names are stable.

However, some of our edge applications use the requested subdomain for multi-tenancy purposes (possibly via CNAME in the future), so port-based routing into a particular application seems like the safest way to go.

Right now, we run EC2 instances behind ELBs for edge applications on static ports (8080 and up) and there is an ELB for each application.

The rate of change on the applications and their ports is extremely low, so I was looking at consul-template for fabio.properties listeners or updating the nomad job whenever a change is needed.

I'll detail a scenerio I tried below in case I misunderstood something.

---

A web application needs to respond to *.example.com where * may be one of thousands of possible customer subdomains (too many to keep in consul, etc).

An ELB for this application is listening on port 80 and sending all traffic to registered EC2 instances on 8085 where Fabio is listening:

proxy.addr = :8085 # <= port per edge application (8080 and up), comma-separated

The application is running on port 30000 on one of the instances and would register with consul with the following:

{
  "ID": "web",
  "Name": "web",
  "Tags": [
    "urlprefix-:8085/"
  ],
  "Address": "127.0.0.1",
  "Port": 30000,
  "Check": {
    "HTTP": "http://127.0.0.1:30000/health_check",
    "Interval": "10s"
  }
}

The fabio UI shows host = :8085 and path = / as expected.

However, routing doesn't work as I think we are discussing in this issue:

2016/07/18 20:56:51 [WARN] No route for localhost:8085/

[magiconair]

@stevenscg How is the link between xyz.example.com and port 8085 or port 30000 established in this scenario? If all your requests come in on port 80 you need a way to determine the target. That isn't clear to me from this description.

[magiconair]

Also, I'm starting to think that a different matcher implementation (with some small refactoring) might be a solution to this instead of waiting for the full tag prefix refactor. The lookup function gets the full request and has the full routing table available so any two pieces which allow a matcher to link request and target should work. Right now I'm linking incoming request uri to service uri (or prefix). But one could also build a matcher that matches destination port to some advertised host:port. However, the part that I don't get in the described scenario is how the decision is made that xyz.example.com should end up on that instance running on port 30000. There is nothing that links the domain to that port (or 8085) unless the client is requesting the resource on that port instead of 80.

[stevenscg]

The ELB makes the request to port 8085 on the instance(s). So, I believe that Fabio would see the request as coming in on port 8085 and be able to use it in decision making.

I picked 30000 just as an example of a random unprivileged port that Nomad or raw docker might select for the app.

• Chris

On Jul 18, 2016, at 4:32 PM, Frank Schröder notifications@github.com wrote:

Also, I'm starting to think that a different matcher implementation (with some small refactoring) might be a solution to this instead of waiting for the full tag prefix refactor. The lookup function gets the full request and has the full routing table available so any two pieces which allow a matcher to link request and target should work. Right now I'm linking incoming request uri to service uri (or prefix). But one could also build a matcher that matches destination port to some advertised host:port. However, the part that I don't get in the described scenario is how the decision is made that xyz.example.com should end up on that instance running on port 30000. There is nothing that links the domain to that port (or 8085) unless the client is requesting the resource on that port instead of 80.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[magiconair]

@stevenscg So the ELB has the domain -> port mapping and fabio acts as a second level dynamic router, right? If that's the case then this should be doable at least in step one without dynamic listeners. This assumes that you have either automated the listener configuration and handle fabio restarts or that you don't add new ports that often.

[stevenscg]

Yes, exactly!

I haven't worked through it yet, but should be able to do rolling updates with nomad or Ansible on the rare occasions when the apps or listener ports need to change. TBH, we have always assigned a 8000-series port for internal/dev use when we create a new service and haven't had to make any changes to those edge applications.

Happy to test and/or write up more examples with/without Fabio in the architecture if that helps.

• Chris

On Jul 18, 2016, at 5:35 PM, Frank Schröder notifications@github.com wrote:

@stevenscg So the ELB has the domain -> port mapping and fabio acts as a second level dynamic router, right? If that's the case then this should be doable at least in step one without dynamic listeners. This assumes that you have either automated the listener configuration and handle fabio restarts or that you don't add new ports that often.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[cjdxlgb]

@magiconair
my english is so poor, and i need some help from you, can you help me?
I regist a service in consul like this:

curl -X PUT -d '{"ID": "redis31","Name": "redis31","Tags": ["primary31","v31"],"Address": "172.168.11.16","Port": 9190,"EnableTagOverride": false, "Check":{"id": "Health31","name": "Health31","http": "http://172.168.11.16:9190/website/login","interval": "5s","timeout": "2s"}}' http://127.0.0.1:8500/v1/agent/service/register

curl localhost:8500/v1/health/state/any?pretty
{
"Node": "localhost.localdomain",
"CheckID": "service:redis31",
"Name": "Service 'redis31' check",
"Status": "passing",
"Notes": "",
"Output": "HTTP GET http://172.168.11.16:9190/website/login: 200 OK ",
"ServiceID": "redis31",
"ServiceName": "redis31",
"CreateIndex": 13,
"ModifyIndex": 14
}

and i start fabio by default:

2017/01/10 19:23:05 [INFO] Version 1.3.5 starting
2017/01/10 19:23:05 [INFO] Go runtime is go1.7.3
2017/01/10 19:23:05 [INFO] Using routing strategy "rnd"
2017/01/10 19:23:05 [INFO] Using routing matching "prefix"
2017/01/10 19:23:05 [INFO] Setting GOGC=800
2017/01/10 19:23:05 [INFO] Setting GOMAXPROCS=4
2017/01/10 19:23:05 [INFO] Metrics disabled
2017/01/10 19:23:05 [INFO] consul: Connecting to "localhost:8500" in datacenter "dc1"
2017/01/10 19:23:05 [INFO] Admin server listening on ":9998"
2017/01/10 19:23:05 [INFO] consul: Using dynamic routes
2017/01/10 19:23:05 [INFO] HTTP proxy listening on :9999
2017/01/10 19:23:05 [INFO] consul: Using tag prefix "urlprefix-"
2017/01/10 19:23:05 [INFO] consul: Watching KV path "/fabio/config"
2017/01/10 19:23:05 [INFO] consul: Health changed to #16
2017/01/10 19:23:05 [INFO] consul: Manual config changed to #1
2017/01/10 19:23:05 [INFO] Updated config to
2017/01/10 19:23:05 [INFO] consul: Registered fabio with id "fabio-localhost.localdomain-9998"
2017/01/10 19:23:05 [INFO] consul: Registered fabio with address "10.1.22.12"
2017/01/10 19:23:05 [INFO] consul: Registered fabio with tags ""
2017/01/10 19:23:05 [INFO] consul: Registered fabio with health check to "http://[10.1.22.12]:9998/health"
2017/01/10 19:23:05 [INFO] consul: Health changed to #18
2017/01/10 19:23:58 [WARN] No route for 10.1.22.12:9999/urlprefix/website/login
2017/01/10 19:24:02 [WARN] No route for 10.1.22.12:9999/urlprefix-/website/login
2017/01/10 19:24:05 [WARN] No route for 10.1.22.12:9999/urlprefix-website/login
2017/01/10 19:28:14 [INFO] consul: Health changed to #18
2017/01/10 19:33:19 [INFO] consul: Health changed to #18
2017/01/10 19:37:17 [WARN] No route for 10.1.22.12:9999/urlprefix-/website/login
2017/01/10 19:38:35 [INFO] consul: Health changed to #18
2017/01/10 19:41:28 [WARN] No route for 10.1.22.12:9999/redis31/website/login
2017/01/10 19:41:42 [WARN] No route for 10.1.22.12:9999/urlprefix-/website/login
2017/01/10 19:41:43 [WARN] No route for 10.1.22.12:9999/urlprefix-redis31/website/login
2017/01/10 19:43:35 [INFO] consul: Health changed to #18

and now I do not how to access my service by fibio proxy, the origin service url is: http://172.168.11.16:9190/website/login

I read the doc of fibio github, but i can not understand what the means, what is urlprefix? how to add to my url.

thank you very much!

[sriyer]

hi @magiconair is the feature to be able to create listeners dynamically available yet ?

We use fabio as an ingress router for our services that require tcp based ingress, its kind of implemented in a way that for every new service that needs tcp ingress, an instance of fabio is brought up, the tcp requests are routed via the fabio instance.
The fabio instances are brought up on special hosts that act as ingress router hosts, now we se quite a few containers running on these router hosts and each of them are bound to only a singe back end service in terms of the listener.
It would help if we could run only one instance of fabio and dynamically bring up listeners as new services are brought up.

There is also the concern of the cpu that the fabio instances are using, with the number of instances increase, the cpu shares although set to 0.1 cores, the usage seems to be at an average of 1.4% when idle ( no request activity). This seems to be overloading the host and the consul process running on it. Although this is a separate ticket that we would create for the cpu usage concern, it'd help if we could run a single instance and just bring up multiple listeners.

CC: @samart

[magiconair]

@sriyer No, this hasn't been implemented yet and I don't have the bandwidth to do the refactoring. My guess is that the activity comes from fabio checking consul in the background. If you configure fabio manually anyway you might want to disable that to see whether that helps.

[murphymj25]

@sriyer #626 might be what you are looking for.