Enable TLS for TCP Routing by default, disable unproxied container po…

…rts, and provide ops-files to disable (#1195)

* Disable unproxied container ports and enable TLS for TCP Routing by default, and provide ops-files to disable.

Enables TLS encryption for TCP Routes on the hop between tcp-router +
app containeris. This behaves the same as the TLS encryption between
gorouter and app containers, and relies on the same settings regarding
route integrity.

This allows us to finally disable the unproxied container ports, and
prevent network connections to app ports from hosts other than router
or tcp-router.

If operators wish to disable TLS for TCP routes, they must first
re-enable the unproxied ports by deploying with these ops files (as appropriate):

- operations/disable-tls-tcp-routing-stage-1-unproxied-ports.yml
- operations/disable-tls-tcp-routing-isolation-segment-stage-1-unproxied-ports.yml
- operations/experimental/disable-tls-tcp-routing-windows-stage-1-unproxied-ports.yml

Once the unproxied ports are re-enabled, a second deploy can be
performed to disable TLS for TCP Routes via these ops files (as appropriate):

- operations/disable-tls-tcp-routing-stage-2-tcp-router-and-route-emitter.yml
- operations/disable-tls-tcp-routing-isolation-segment-stage-2-route-emitter.yml
- operations/experimental/disable-tls-tcp-routing-windows-stage-2-route-emitter.yml

Failing to disable this in a two-deploy fashion will result in downtime
for TCP Routes.

* update readmes for new tls for tcp route disabling ops files

* Add unit tests for new opsfiles

cf-deployment.yml

@@ -1465,6 +1465,11 @@ instance_groups:
         router_group: default-tcp
         tls_health_check_cert: ((tcp_router_lb_health_tls.certificate))
         tls_health_check_key:  ((tcp_router_lb_health_tls.private_key))
+        backend_tls:
+          enabled: true
+          client_cert: ((tcp_router_backend_tls.certificate))
+          client_key: ((tcp_router_backend_tls.private_key))
+          ca_cert: ((diego_instance_identity_ca.ca))
       uaa:
         ca_cert: "((uaa_ssl.ca))"
         tls_port: 8443
@@ -1639,13 +1644,16 @@ instance_groups:
       containers:
         proxy:
           enabled: true
+          enable_unproxied_port_mappings: false
           require_and_verify_client_certificates: true
           trusted_ca_certificates:
           - ((gorouter_backend_tls.ca))
           - ((ssh_proxy_backends_tls.ca))
+          - ((tcp_router_backend_tls.ca))
           verify_subject_alt_name:
           - gorouter.service.cf.internal
           - ssh-proxy.service.cf.internal
+          - tcp-router.service.cf.internal
         trusted_ca_certificates:
         - ((diego_instance_identity_ca.ca))
         - ((credhub_tls.ca))
@@ -1684,6 +1692,7 @@ instance_groups:
               client_key: "((nats_client_cert.private_key))"
       tcp:
         enabled: true
+        enable_tls: true
       uaa:
         ca_cert: "((uaa_ssl.ca))"
         client_secret: "((uaa_clients_tcp_emitter_secret))"
@@ -2488,6 +2497,15 @@ variables:
     common_name: gorouter_lb_health_tls
     alternative_names:
     - gorouter.service.cf.internal
+- name: tcp_router_backend_tls
+  type: certificate
+  options:
+    ca: service_cf_internal_ca
+    common_name: tcp-router_backend_tls
+    alternative_names:
+    - tcp-router.service.cf.internal
+    extended_key_usage:
+    - client_auth
 - name: tcp_router_lb_health_tls
   type: certificate
   options:

operations/README.md

@@ -39,6 +39,10 @@ This is the README for Ops-files. To learn more about `cf-deployment`, go to the
 | [`disable-router-tls-termination.yml`](disable-router-tls-termination.yml) | Eliminates keys related to performing TLS termination within the gorouter job. | Useful for deployments where TLS termination is performed prior to the gorouter - for instance, on AWS, such termination is commonly done at the ELB. This also eliminates the need to specify `((router_ssl.certificate))` and `((router_ssl.private_key))` in the var files. | **NO** |
 | [`disable-http2.yml`](disable-http2.yml) | Prevent gorouter from accepting and forwarding HTTP/2 requests. | | **NO** |
 | [`disable-dynamic-asgs.yml`](disable-dynamic-asgs.yml) | Disable dynamic updates for security groups. | | **NO** |
+| [`disable-tls-tcp-routing-stage-1-unproxied-ports.yml`](disable-tls-tcp-routing-stage-1-unproxied-ports.yml) | Stage 1 deployment for disabling TLS for TCP Routes on. See [configuring TCP routes](https://docs.cloudfoundry.org/adminguide/enabling-tcp-routing.html#tls-tcp-routes) for more info. | | **NO ** |
+| [`disable-tls-tcp-routing-stage-2-tcp-router-and-route-emitter.yml`](disable-tls-tcp-routing-stage-2-tcp-router-and-route-emitter.yml) | Stage 2 deployment for disabling TLS for TCP Routes on. See [configuring TCP routes](https://docs.cloudfoundry.org/adminguide/enabling-tcp-routing.html#tls-tcp-routes) for more info. | | **NO ** |
+| [`disable-tls-tcp-routing-isolation-segment-stage-1-unproxied-ports.yml`](disable-tls-tcp-routing-isolation-segment-stage-1-unproxied-ports.yml) | Stage 1 deployment for disabling TLS for TCP Routes on isolation segments. See [configuring TCP routes](https://docs.cloudfoundry.org/adminguide/enabling-tcp-routing.html#tls-tcp-routes) for more info. | | **NO ** |
+| [`disable-tls-tcp-routing-isolation-segment-stage-2-route-emitter.yml`](disable-tls-tcp-routing-isolation-segment-stage-2-route-emitter.yml) | Stage 2 deployment for disabling TLS for TCP Routes on isolation segments. See [configuring TCP routes](https://docs.cloudfoundry.org/adminguide/enabling-tcp-routing.html#tls-tcp-routes) for more info. | | **NO ** |
 | [`enable-cc-rate-limiting.yml`](enable-cc-rate-limiting.yml) | Enable rate limiting for UAA-authenticated endpoints. | Introduces variables `cc_rate_limiter_general_limit` and `cc_rate_limiter_unauthenticated_limit` | **NO** |
 | [`enable-cc-v2-rate-limiting.yml`](enable-cc-rate-limiting.yml) | Enable V2 API rate limiting for UAA-authenticated endpoints. | Introduces variables `cc_v2_rate_limiter_general_limit`, `cc_v2_rate_limiter_admin_limit` and `cc_v2_rate_limiter_reset_interval_in_minutes` | **NO** |
 | [`enable-cpu-throttling.yml`](enable-cpu-throttling.yml) | Configure Garden containers with CPU entitlement. | This ops file requires `set-cpu-weight.yml`. | **YES** |

operations/add-persistent-isolation-segment-diego-cell.yml

@@ -70,13 +70,16 @@
         containers:
           proxy:
             enabled: true
+            enable_unproxied_port_mappings: false
             require_and_verify_client_certificates: true
             trusted_ca_certificates:
             - ((gorouter_backend_tls.ca))
             - ((ssh_proxy_backends_tls.ca))
+            - ((tcp_router_backend_tls.ca))
             verify_subject_alt_name:
             - gorouter.service.cf.internal
             - ssh-proxy.service.cf.internal
+            - tcp-router.service.cf.internal
           trusted_ca_certificates:
           - ((diego_instance_identity_ca.ca))
           - ((credhub_tls.ca))
@@ -134,6 +137,7 @@
             timestamp: "rfc3339"
         tcp:
           enabled: true
+          enable_tls: true
         uaa:
           ca_cert: "((uaa_ssl.ca))"
           client_secret: "((uaa_clients_tcp_emitter_secret))"

operations/disable-tls-tcp-routing-isolation-segment-stage-1-unproxied-ports.yml

@@ -0,0 +1,4 @@
+---
+- type: replace
+  path: /instance_groups/name=isolated-diego-cell/jobs/name=rep/properties/containers/proxy/enable_unproxied_port_mappings?
+  value: true

operations/disable-tls-tcp-routing-isolation-segment-stage-2-route-emitter.yml

@@ -0,0 +1,4 @@
+---
+- type: replace
+  path: /instance_groups/name=isolated-diego-cell/jobs/name=route_emitter/properties/tcp/enable_tls?
+  value: false

operations/disable-tls-tcp-routing-stage-1-unproxied-ports.yml

@@ -0,0 +1,4 @@
+---
+- type: replace
+  path: /instance_groups/name=diego-cell/jobs/name=rep/properties/containers/proxy/enable_unproxied_port_mappings?
+  value: true

operations/disable-tls-tcp-routing-stage-2-tcp-router-and-route-emitter.yml

@@ -0,0 +1,8 @@
+---
+- type: replace
+  path: /instance_groups/name=tcp-router/jobs/name=tcp_router/properties/tcp_router/backend_tls?/enabled
+  value: false
+
+- type: replace
+  path: /instance_groups/name=diego-cell/jobs/name=route_emitter/properties/tcp/enable_tls?
+  value: false

operations/experimental/README.md

@@ -22,6 +22,8 @@ This is the README for Experimental Ops-files. To learn more about `cf-deploymen
 | [`colocate-smoke-tests-on-cc-worker.yml`](colocate-smoke-tests-on-cc-worker.yml) | Colocate the smoke_tests job on the cc-worker instance | A number of other operations files reference this instance group and may be incompatible with this operations file.  Use `find ./operations/ -name "*.yml" | xargs grep "/instance_groups/name=smoke-tests"` to locate said files. | **YES** |
 | [`disable-interpolate-service-bindings.yml`](disable-interpolate-service-bindings.yml) | Disables the interpolation of CredHub service credentials by Cloud Controller. | | **YES** |
 | [`disable-cf-credhub.yml`](disable-cf-credhub.yml) | Completely removes the CF CredHub instances, UAA clients, credentials and certificates. Can be used to save cost if you don't use CredHub to store service credentials. | | **YES** |
+| [`disable-tls-tcp-routing-windows-stage-1-unproxied-ports.yml`](disable-tls-tcp-routing-windows-stage-1-unproxied-ports.yml) | Stage 1 deployment for disabling TLS for TCP Routes on Windows Diego Cells. See [configuring TCP routes](https://docs.cloudfoundry.org/adminguide/enabling-tcp-routing.html#tls-tcp-routes) for more info. | | **NO ** |
+| [`disable-tls-tcp-routing-windows-stage-2-route-emitter.yml`](disable-tls-tcp-routing-windows-stage-2-route-emitter.yml) | Stage 2 deployment for disabling TLS for TCP Routes on Windows Diego Cells. See [configuring TCP routes](https://docs.cloudfoundry.org/adminguide/enabling-tcp-routing.html#tls-tcp-routes) for more info. | | **NO ** |
 | [`enable-app-log-rate-limiting.yml`](enable-app-log-rate-limiting.yml) | Enable rate limiting for number of logs generated by the application.  | Introduces variable `app_log_rate_limit`.  | **NO** |
 | [`enable-app-log-rate-limiting-windows2019.yml`](enable-app-log-rate-limiting-windows2019.yml) | Enable rate limiting for number of logs generated by the application.  | Introduces variable `app_log_rate_limit`. Requires `../windows2019-cell.yml`  | **NO** |
 | [`enable-bpm-garden.yml`](enable-bpm-garden.yml) | Enables the [BOSH Process Manager](https://github.com/cloudfoundry-incubator/bpm-release) for Garden. | This ops file **cannot** be deployed in conjunction with `enable-oci-phase-1.yml`. | **NO** |

operations/experimental/disable-tls-tcp-routing-windows-stage-1-unproxied-ports.yml

@@ -0,0 +1,4 @@
+---
+- type: replace
+  path: /instance_groups/name=windows2019-cell/jobs/name=rep_windows/properties/containers/proxy/enable_unproxied_port_mappings?
+  value: true

operations/experimental/disable-tls-tcp-routing-windows-stage-2-route-emitter.yml

@@ -0,0 +1,4 @@
+---
+- type: replace
+  path: /instance_groups/name=windows2019-cell/jobs/name=route_emitter_windows/properties/tcp/enable_tls?
+  value: false

operations/experimental/enable-nginx-routing-integrity-windows2019.yml

@@ -1,6 +1,9 @@
 - type: replace
   path: /instance_groups/name=windows2019-cell/jobs/name=rep_windows/properties/containers?/proxy/enabled
   value: true
+- type: replace
+  path: /instance_groups/name=windows2019-cell/jobs/name=rep_windows/properties/containers?/proxy/enable_unproxied_port_mappings
+  value: false
 - type: replace
   path: /instance_groups/name=windows2019-cell/jobs/name=rep_windows/properties/containers?/proxy/require_and_verify_client_certificates
   value: true
@@ -9,11 +12,16 @@
   value:
   - ((gorouter_backend_tls.ca))
   - ((ssh_proxy_backends_tls.ca))
+    ((tcp_router_backend_tls.ca))
 - type: replace
   path: /instance_groups/name=windows2019-cell/jobs/name=rep_windows/properties/containers?/proxy/verify_subject_alt_name
   value:
   - gorouter.service.cf.internal
   - ssh-proxy.service.cf.internal
+  - tcp-router.service.cf.internal
+- type: replace
+  path: /instance_groups/name=windows2019-cell/jobs/name=route_emitter_windows/properties/tcp?/enable_tls
+  value: true
 - type: replace
   path: /instance_groups/name=windows2019-cell/jobs/-
   value:

operations/windows2019-cell.yml

@@ -86,6 +86,13 @@
                 client_cert: ((nats_client_cert.certificate))
                 client_key: ((nats_client_cert.private_key))
                 enabled: true
+        internal_routes:
+          enabled: true
+        uaa:
+          ca_cert: "((uaa_ssl.ca))"
+          client_secret: "((uaa_clients_tcp_emitter_secret))"
+        tcp:
+          enabled: true
         logging:
           format:
             timestamp: rfc3339

units/tests/experimental_test/operations.yml

@@ -76,3 +76,12 @@ use-mysql-version-8.0.yml:
   pathvalidator:
     path: /instance_groups/name=database/jobs/name=pxc-mysql/properties/mysql_version?
     expectedvalue: "8.0"
+disable-tls-tcp-routing-windows-stage-1-unproxied-ports.yml:
+  ops:
+  - ../windows2019-cell.yml
+  - enable-nginx-routing-integrity-windows2019.yml
+disable-tls-tcp-routing-windows-stage-2-route-emitter.yml:
+  ops:
+  - ../windows2019-cell.yml
+  - enable-nginx-routing-integrity-windows2019.yml
+  - disable-tls-tcp-routing-windows-stage-1-unproxied-ports.yml

units/tests/standard_test/operations.yml

@@ -154,4 +154,15 @@ use-trusted-ca-cert-for-apps.yml:
 windows2019-cell.yml: {}
 use-cflinuxfs4-compat.yml:
   ops:
-    - use-cflinuxfs4-compat.yml
+    - use-cflinuxfs4-compat.yml
+disable-tls-tcp-routing-isolation-segment-stage-1-unproxied-ports.yml:
+  ops:
+  - add-persistent-isolation-segment-diego-cell.yml
+disable-tls-tcp-routing-isolation-segment-stage-2-route-emitter.yml:
+  ops:
+  - add-persistent-isolation-segment-diego-cell.yml
+  - disable-tls-tcp-routing-isolation-segment-stage-1-unproxied-ports.yml
+disable-tls-tcp-routing-stage-1-unproxied-ports.yml: {}
+disable-tls-tcp-routing-stage-2-tcp-router-and-route-emitter.yml:
+  ops:
+  - disable-tls-tcp-routing-stage-1-unproxied-ports.yml