[R&D] Istio over vl3

[ThetaDR]

We have to add WorkloadEntries for the workloads. One of the ways is adding GRPC service to do so automatically.
Another question is how many istio clusters should be supported - just one or multiple?

Here's a link for diagram with the setup: https://drive.google.com/open?id=1k0hWlgdapko50vI3BQ2rOA0FgQiWxQa6

[edwarnicke]

@ThetaDR Question: is istiod in this picture using the "Istio+NSM cluster"'s k8s-api-server?

[ThetaDR]

@edwarnicke Yes, the istiod on the first cluster uses k8s-api-server of “Istio+NSM cluster”. Workload-2 will add WorkloadEntry to it via grpc service.

[ThetaDR]

Current status: Reproducing the setup for VM of istio. Have faced the problem - istio-sidecar doesn't start on ubuntu container (systemctl), as possible solutions - finding the binary for a sidecar and manually starting it and using the docker image, but tweaking it's entrypoint.

[d-uzlov]

I was able to create a proof-of-concept example:
https://github.com/d-uzlov/deployments-k8s/blob/ed5c317ac82658e3f1d59cee459f95199d08307f/examples/interdomain/nsm_istio_vl3/clean/README.md

I used manual istio sidecar injection and replaced some of the configs it added.
I was able to run this modified deployment on a cluster without istiod and verify that connections between pods in nsm vl3 network go through istio, and are encrypted when istio is configured to do so.
istiod and istio-proxy sidecar communicate through nsm interface.

A sample of input and output of running commands in the example can be found here:
https://github.com/d-uzlov/deployments-k8s/blob/ed5c317ac82658e3f1d59cee459f95199d08307f/examples/interdomain/nsm_istio_vl3/clean/readme-run-sample.md
I also uploaded a cluster-info dump logs nearby:
https://github.com/d-uzlov/deployments-k8s/tree/ed5c317ac82658e3f1d59cee459f95199d08307f/examples/interdomain/nsm_istio_vl3/clean/run-sample

Now we plan to automate it, to avoid all of the manual configuration that I added to this example.

[d-uzlov]

Updated automation proposal:
https://docs.google.com/document/d/1m8x3hIX_4JBjqwx0bDh-QpgooHp4Z-oflD6dM0ZUqfg/edit?usp=sharing

Tasks with rough time estimations:

• Update and simplify proposal for Istio-over-vl3
• Add diagrams to proposal
• Convert services inside vl3 network into Istio ServiceEntries
• • [option 1] Use k8s services with annotations
• • • Create repo for "vl3 service manager" 1h
• • • Set up ServiceEntry creation via k8s API 3h
• • • Create webhook to trigger ServiceEntry updates automatically 3h
• • • Add docker tests for webhook and SE creation 3h
• • • Create deployemnt config 1h
• • • Test deployment in kind 2h
• • [option 2] Use custom registry for services in vl3 network
• • • Create repo for "vl3 service manager" 1h
• • • Create k8s CRD for vl3 services 3h
• • • Set up CRD monitoring in the app 3h
• • • Add docker tests for CRD monitoring 2h
• • • [optional] Add element for vl3 NSE to export endpoints in the network
• • • • Create chain element 4h
• • • • Add unit/sandbox tests for export element 3h
• • • • Maybe we can re-use existing DNS-related elements instead of this
• • • Connect vl3 service manager to vl3 NSE to monitor endpoints 4h
• • • Add docker tests for endpoint monitoring 2h
• • • Set up ServiceEntry creation via k8s API 4h
• • • Add docker tests for automatic ServiceEntry creation 2h
• • • Test webhook in kind 3h
• Create helper applications to generate side-car config for remote cluster
• • Investigate istio source code for config generation 7h
• • Create repo for "config helper" 1h
• • Implement config generation 3h
• • Implement gRPC access to configs 3h
• • Add docker tests 2h
• • Create deployment config 2h
• Automate Istio side-car config modification
• • Create init application that will fetch configs
• • • Connect application to gRPC interfaces of the "config provider
• • • Add Docker tests 2h
• • • Create deployment config 2h
• • • Test app in kind (also test "config helper") 4h
• • [option 1] Try to use Istio's built-in template feature
• • • Try to implement required templates 7h
• • [option 2] Create a special webhook to modify istio-proxy config
• • • Create repo for "istio remote webhook" 1h
• • • Create webhook to watch pods 3h
• • • Implement required patches 4h
• • • Add docker tests 3h
• • • Create deployment config 2h
• • • Test webhook in kind 3h
• Automate side-car injection
• • Proxy-webhook
• • • Create webhook to watch pods and retranslate the webhook request 3h
• • • Test app in kind 3h
• • • Move modifications to "istio remote webhook" 3h
• • • Test modified "istio remote webhook" in kind 3h
• Create final example for deployments-k8s 4h

Time for risks is not included here.

[denis-tingaikin]

• Invesrigate our webhhok ~ 3h
• Invesrigate istio webhhok ~ 7h
• Invesrigate spire + istio ~ 7h + 7h
• update proposal ~ 3h