Unable to make requests to an ISVC from outside the cluster

[NohaIhab]

Bug Description

Currently, users are not able to make a request to an ISVC using the istio ingress gateway external IP. The workaround atm is to set the oidc cookie in the requset, this is not an ideal UX and it should ideally work without the cookie.

To Reproduce

• Deploy latest/edge bundle
  juju deploy kubeflow --trust --channel=latest/edge

• Follow the guide to create an ISVC

• to curl the ISVC, from the guide select the tab From Ingress gateway with HOST Header

Environment

kubeflow latest/edge
microk8s 1.25-strict/stable
juju 3.1/stable

Relevant Log Output

curl -v -H "Host: ${SERVICE_HOSTNAME}" -H "Content-Type: application/json" "http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/sklearn-iris:predict" -d @./iris-input.json
*   Trying 10.64.140.43:80...
* TCP_NODELAY set
* Connected to 10.64.140.43 (10.64.140.43) port 80 (#0)
> POST /v1/models/sklearn-iris:predict HTTP/1.1
> Host: sklearn-iris.kserve-test.10.64.140.43.nip.io
> User-Agent: curl/7.68.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 76
> 
* upload completely sent off: 76 out of 76 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< location: http://10.64.140.43.nip.io/dex/auth?client_id=authservice-oidc&redirect_uri=%2Fauthservice%2Foidc%2Fcallback&response_type=code&scope=openid+profile+email+groups&state=MTcwNzIxMzQ3MHxOd3dBTkZwUVZrWktUMVpPTmpaRlZrOU9URFZVTmxJelJVcE5TVlUzVlVSRFNqSTFXbEZhTmpSWFJEVXpWalJRUWpjeldVRktRVkU9fAVJj5wAOmefS9o8GCUkdqbmQ5FYEXpG-ARfXZf-ZT3m
< set-cookie: oidc_state_csrf=MTcwNzIxMzQ3MHxOd3dBTkZwUVZrWktUMVpPTmpaRlZrOU9URFZVTmxJelJVcE5TVlUzVlVSRFNqSTFXbEZhTmpSWFJEVXpWalJRUWpjeldVRktRVkU9fAVJj5wAOmefS9o8GCUkdqbmQ5FYEXpG-ARfXZf-ZT3m; Path=/; Expires=Tue, 28 Jul 2054 10:51:19 GMT; Max-Age=1200000000000
< date: Tue, 06 Feb 2024 09:57:50 GMT
< x-envoy-upstream-service-time: 3
< server: istio-envoy
< content-length: 0
< 
* Connection #0 to host 10.64.140.43 left intact

Additional Context

No response