feat: Add TLS support to the Operator

[tchughesiv]

What this PR does / why we need it:

With this PR, an Operator user will be able to configure feast services with TLS. If the operator detects it's running in an OpenShift cluster, TLS is enabled by default through the use of the service serving certificates feature.

• adds option to enable TLS on a per-service basis. k8s Secret required. TLS is not required across all services. the operator will configured them to work together on a per-service basis.
• adds option to configure client TLS for remote registries. k8s ConfigMap required.
• adds OpenShift cluster detection
• when running in an OpenShift cluster, automatically configures TLS, for all services... unless disabled or overridden by the user
• similarly, if a remote registry (in the same OpenShift cluster) is using auto-TLS, the operator will automatically configure client TLS... unless overridden by the user
• changes OnlineStore liveness & readiness probes to httpGet type. checks /health endpoint
• adds grpcurl initContainer to the online & offlineStore deployments, which will check the registry's grpc.health.v1.Health/Check endpoint before attempting to start those feast services.

Which issue(s) this PR fixes:

Fixes #4770

Misc

Example of an OpenShift deployment and TLS being automatically configured. In a k8s cluster, TLS would need to be manually configured in the FeatureStore.spec. Any required certs, keys, and k8s secrets/configmaps would also have to be created by the user -

apiVersion: feast.dev/v1alpha1
kind: FeatureStore
metadata:
  name: example
  namespace: default
spec:
  feastProject: my_project
  services:
    offlineStore: {}
    onlineStore: {}
status:
  applied:
    feastProject: my_project
    services:
      offlineStore:
        image: 'feastdev/feature-server:0.41.0'
        persistence:
          file:
            type: dask
        tls:
          secretKeyNames:
            tlsCrt: tls.crt
            tlsKey: tls.key
          secretRef:
            name: feast-example-registry-tls
      onlineStore:
        image: 'feastdev/feature-server:0.41.0'
        persistence:
          file:
            path: /tmp/online_store.db
        tls:
          secretKeyNames:
            tlsCrt: tls.crt
            tlsKey: tls.key
          secretRef:
            name: feast-example-registry-tls
      registry:
        local:
          image: 'feastdev/feature-server:0.41.0'
          persistence:
            file:
              path: /tmp/registry.db
          tls:
            secretKeyNames:
              tlsCrt: tls.crt
              tlsKey: tls.key
            secretRef:
              name: feast-example-registry-tls
  clientConfigMap: feast-example-client
  conditions:
    - lastTransitionTime: '2024-11-26T19:04:50Z'
      message: Offline Store installation complete
      reason: Ready
      status: 'True'
      type: OfflineStore
    - lastTransitionTime: '2024-11-25T21:43:28Z'
      message: Online Store installation complete
      reason: Ready
      status: 'True'
      type: OnlineStore
    - lastTransitionTime: '2024-11-25T19:59:23Z'
      message: Client installation complete
      reason: Ready
      status: 'True'
      type: Client
    - lastTransitionTime: '2024-11-26T19:04:50Z'
      message: FeatureStore installation complete
      reason: Ready
      status: 'True'
      type: FeatureStore
    - lastTransitionTime: '2024-11-25T22:26:19Z'
      message: Registry installation complete
      reason: Ready
      status: 'True'
      type: Registry
  feastVersion: 0.41.0
  phase: Ready
  serviceHostnames:
    offlineStore: 'feast-example-offline.default.svc.cluster.local:443'
    onlineStore: 'feast-example-online.default.svc.cluster.local:443'
    registry: 'feast-example-registry.default.svc.cluster.local:443'

Example of a remote registry reference -

apiVersion: feast.dev/v1alpha1
kind: FeatureStore
metadata:
  name: example
  namespace: default
spec:
  feastProject: my_project
  services:
    offlineStore: {}
    onlineStore: {}
    registry:
      remote:
        feastRef:
          name: central
          namespace: test
status:
  applied:
    feastProject: my_project
    services:
      offlineStore:
        image: 'feastdev/feature-server:0.41.0'
        persistence:
          file:
            type: dask
        tls:
          secretKeyNames:
            tlsCrt: tls.crt
            tlsKey: tls.key
          secretRef:
            name: feast-example-online-tls
      onlineStore:
        image: 'feastdev/feature-server:0.41.0'
        persistence:
          file:
            path: /tmp/online_store.db
        tls:
          secretKeyNames:
            tlsCrt: tls.crt
            tlsKey: tls.key
          secretRef:
            name: feast-example-online-tls
      registry:
        remote:
          feastRef:
            name: central
            namespace: test
          tls:
            certName: service-ca.crt
            configMapRef:
              name: feast-example-client-ca
  clientConfigMap: feast-example-client
  conditions:
    - lastTransitionTime: '2024-11-27T15:47:57Z'
      message: Offline Store installation complete
      reason: Ready
      status: 'True'
      type: OfflineStore
    - lastTransitionTime: '2024-11-25T21:43:28Z'
      message: Online Store installation complete
      reason: Ready
      status: 'True'
      type: OnlineStore
    - lastTransitionTime: '2024-11-25T19:59:23Z'
      message: Client installation complete
      reason: Ready
      status: 'True'
      type: Client
    - lastTransitionTime: '2024-11-27T15:53:42Z'
      message: FeatureStore installation complete
      reason: Ready
      status: 'True'
      type: FeatureStore
  feastVersion: 0.41.0
  phase: Ready
  serviceHostnames:
    offlineStore: 'feast-example-offline.default.svc.cluster.local:443'
    onlineStore: 'feast-example-online.default.svc.cluster.local:443'
    registry: 'feast-central-registry.test.svc.cluster.local:443'

Which results in the following client configMap & feature_store.yaml -

kind: ConfigMap
apiVersion: v1
metadata:
  name: feast-example-client
  namespace: default
  labels:
    feast.dev/name: example
    feast.dev/service-type: client
  ownerReferences:
    - apiVersion: feast.dev/v1alpha1
      kind: FeatureStore
      name: example
      uid: e4f030dc-23b1-45bc-ba86-6ef66c491701
      controller: true
      blockOwnerDeletion: true
data:
  feature_store.yaml: |
    project: my_project
    provider: local
    offline_store:
        host: feast-example-offline.default.svc.cluster.local
        type: remote
        port: 443
        scheme: https
        cert: /tls/offline/tls.crt
    online_store:
        path: https://feast-example-online.default.svc.cluster.local:443
        type: remote
        cert: /tls/online/tls.crt
    registry:
        path: feast-central-registry.test.svc.cluster.local:443
        registry_type: remote
        cert: /tls/registry/service-ca.crt
    auth:
        type: no_auth
    entity_key_serialization_version: 3

[dmartinol]

lgtm

[dmartinol]

why for offline store we need a different TLS config?

[tchughesiv]

offline offers that additional TLS related flag for some reason, whereas the others don't.

feast/sdk/python/feast/cli.py

Line 1136 in e198b17

"--verify_client",

that said, its not mentioned in the docs, so maybe its unnecessary? i guess i just felt like we should offer the user all the options. it's certainly added a layer of complexity to the code.

[tchughesiv]

@lokeshrangineni thoughts?

[lokeshrangineni]

Regarding verify_client flag - flight server is very strict with the client validation when we pass the TLS self signed certificates which made it difficult for the integration tests so I had to add this flag for the integration tests. By default verify_client flag is enabled and in the production environment it is supposed to be enabled as well. for the integration tests i marked verify_client as false. I will add about this flag to the documentation as well.

[dmartinol]

can we add the verify_client options to all the servers @lokeshrangineni ?
this way we can keep a consistent CR definition and offer the same set of security functions to all of them

[tchughesiv]

i think this needs the ok-to-test label