istio-gateway charm does not expose an option for internal LB when deploying on EKS

[gustavosr98]

Bug Description

Deploying istio gateway as a LoadBalancer on AWS EKS will by default create an AWS Load Balancer that is exposed to the Internet by default

There are multiple cases where because of security constraints this needs to be only exposed to internal networks and not to the Internet.
This also applies when deploying a private EKS that is only available on private VPCs and not publicly available VPCs.

I would like to have exposed the option to add the required annotation on the charm to deploy the LoadBalancer as a internal LoadBalancer rather than a private one

See https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/guide/service/annotations/#lb-internal

To Reproduce

juju deploy --trust --debug istio-gateway
--channel=1.16/stable
--config kind=ingress
--config gateway_service_type=LoadBalancer

Environment

AWS EKS
Juju 2.9.45
Charm channel: 1.16/stable
Charm revision: 663

Relevant log output

n/a

Additional context

Some additional external reference
https://anshul-prakash17.medium.com/creating-an-internal-load-balancer-in-aws-eks-using-istio-3defcb51c6a2

Workaround

1. Create the service as a NodePort initially, so the LoadBalancer doesn't get created on AWS

juju deploy --trust --debug istio-gateway \
--channel=1.16/stable \
--config kind=ingress \
--config gateway_service_type=NodePort

2. Edit the manifest.yaml located on the charm source code to add the service annotation for internal lb

# [..]
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
  labels:
    app: istio-{{ kind }}gateway
    install.operator.istio.io/owning-resource: unknown
    istio: {{ kind }}gateway
    istio.io/rev: default
    operator.istio.io/component: "{{ kind|capitalize }}Gateways"
    release: istio
  name: istio-{{ kind }}gateway-workload
  namespace: {{ namespace }}
spec:
  ports:
    - name: http2
      port: 80
      protocol: TCP
      targetPort: 8080
    - name: https
      port: 443
      protocol: TCP
      targetPort: 8443
  selector:
    istio: {{ kind }}gateway
  type: {{ 'ClusterIP' if kind == 'egress' else gateway_service_type }}

3. Push the new manifest.yaml into the deployed charm

juju scp new-manifest.yaml istio-gateway/0:/var/lib/juju/agents/unit-istio-gateway-0/charm/src/manifest.yaml

4. Change the service type to LoadBalancer

juju config istio-gateway gateway_service_type=LoadBalancer

[gustavosr98]

Since there are going to be multiple annotations per cloud (AWS, GCP, Azure, etc) I believe an approach could be to expose on the charm an options like extra-annotations to solve any additional customization required per cloud

For this other too as well:

• istio-gateway is not exposing option to select AZs for EKS  #344
• https://chat.charmhub.io/charmhub/pl/gnuzzk1otbdjxmj5ozgqx43zza

[orfeas-k]

This is related to #66.

[orfeas-k]

Thank you @gustavosr98 for the detailed report. I will bring this up in one of the team's meeting in order to evaluate this feature, since this is something that was also requested some time ago in #66.

[gustavosr98]

I have not tested but found these kubernetes-service-annotations, within those there is a kubernetes-service-type option worth testing during deployment type to avoid hacking into the charm code

# juju config istio-ingressgateway | grep kubernetes
  kubernetes-ingress-allow-http:
  kubernetes-ingress-class:
  kubernetes-ingress-ssl-passthrough:
  kubernetes-ingress-ssl-redirect:
  kubernetes-service-annotations:
  kubernetes-service-external-ips:
  kubernetes-service-externalname:
  kubernetes-service-loadbalancer-ip:
  kubernetes-service-loadbalancer-sourceranges:
  kubernetes-service-target-port:
  kubernetes-service-type: