Can't set up milvus cluster - RootCoordClient mess key not exist

[zeionara]

Hey, I am trying to set up a milvus instance in single-node kubernetes cluster, but only minio and etcd services are ready, the other pods are not starting. That's what I've done:

Install nvidia device plugin

sudo helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
sudo helm repo update
sudo helm search repo nvdp --devel  # remember the chart version
sudo helm upgrade -i nvdp nvdp/nvidia-device-plugin --namespace nvidia-device-plugin --create-namespace --version 0.17.0  # substitute chart version here from the result of previous command execution
sudo kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.17.0/deployments/static/nvidia-device-plugin.yml

Install milvus

sudo helm repo add milvus https://zilliztech.github.io/milvus-helm/
sudo helm install playground milvus/milvus -f custom-values.yaml

Configuration

The content of custom-values.yaml:

indexNode:
  resources:
    requests:
      nvidia.com/gpu: "1"
    limits:
      nvidia.com/gpu: "1"
queryNode:
  resources:
    requests:
      nvidia.com/gpu: "1"
    limits:
      nvidia.com/gpu: "1"

The node description:

Name:               k3d-milvus-server-0
Roles:              control-plane,master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=k3s
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=k3d-milvus-server-0
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=true
                    node-role.kubernetes.io/master=true
                    node.kubernetes.io/instance-type=k3s
Annotations:        alpha.kubernetes.io/provided-node-ip: 172.20.0.2
                    flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"02:6f:db:c1:1e:f6"}
                    flannel.alpha.coreos.com/backend-type: vxlan
                    flannel.alpha.coreos.com/kube-subnet-manager: true
                    flannel.alpha.coreos.com/public-ip: 172.20.0.2
                    k3s.io/hostname: k3d-milvus-server-0
                    k3s.io/internal-ip: 172.20.0.2
                    k3s.io/node-args: ["server","--tls-san","0.0.0.0","--tls-san","k3d-milvus-serverlb"]
                    k3s.io/node-config-hash: D6S6K6U2QTGBIHYTK6DJU7CJYEYKOQY6JWYNORMVSPH5D6UFNU7Q====
                    k3s.io/node-env: {"K3S_KUBECONFIG_OUTPUT":"/output/kubeconfig.yaml","K3S_TOKEN":"********"}
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Thu, 21 Nov 2024 16:54:55 +0300
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  k3d-milvus-server-0
  AcquireTime:     <unset>
  RenewTime:       Fri, 22 Nov 2024 13:32:28 +0300
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Fri, 22 Nov 2024 13:28:23 +0300   Thu, 21 Nov 2024 16:54:55 +0300   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Fri, 22 Nov 2024 13:28:23 +0300   Thu, 21 Nov 2024 16:54:55 +0300   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Fri, 22 Nov 2024 13:28:23 +0300   Thu, 21 Nov 2024 16:54:55 +0300   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Fri, 22 Nov 2024 13:28:23 +0300   Thu, 21 Nov 2024 16:54:55 +0300   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  172.20.0.2
  Hostname:    k3d-milvus-server-0
Capacity:
  cpu:                32
  ephemeral-storage:  1920641288Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             131743276Ki
  nvidia.com/gpu:     1
  pods:               110
Allocatable:
  cpu:                32
  ephemeral-storage:  1868399843502
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             131743276Ki
  nvidia.com/gpu:     1
  pods:               110
System Info:
  Machine ID:
  System UUID:                cbeddc58-6586-ac17-a6fe-d843ae9e92a7
  Boot ID:                    02a24c10-0252-4833-97ee-9609d04ecae9
  Kernel Version:             6.8.0-47-generic
  OS Image:                   K3s v1.31.2+k3s1
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.7.22-k3s1
  Kubelet Version:            v1.31.2+k3s1
  Kube-Proxy Version:         v1.31.2+k3s1
PodCIDR:                      10.42.0.0/24
PodCIDRs:                     10.42.0.0/24
ProviderID:                   k3s://k3d-milvus-server-0
Non-terminated Pods:          (19 in total)
  Namespace                   Name                                            CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                            ------------  ----------  ---------------  -------------  ---
  default                     playground-etcd-0                               0 (0%)        0 (0%)      0 (0%)           0 (0%)         4m35s
  default                     playground-etcd-1                               0 (0%)        0 (0%)      0 (0%)           0 (0%)         4m16s
  default                     playground-etcd-2                               0 (0%)        0 (0%)      0 (0%)           0 (0%)         4m36s
  default                     playground-milvus-datanode-5cfb6cf5dc-znbcc     0 (0%)        0 (0%)      0 (0%)           0 (0%)         4m36s
  default                     playground-milvus-datanode-7d874ddfc6-kwqhc     0 (0%)        0 (0%)      0 (0%)           0 (0%)         4m36s
  default                     playground-milvus-indexnode-5b6988cd54-5rkmq    0 (0%)        0 (0%)      0 (0%)           0 (0%)         4m36s
  default                     playground-milvus-mixcoord-85c4888ff8-7ztw4     0 (0%)        0 (0%)      0 (0%)           0 (0%)         4m36s
  default                     playground-milvus-proxy-68478c4f79-qj5nm        0 (0%)        0 (0%)      0 (0%)           0 (0%)         4m36s
  default                     playground-milvus-proxy-6f67b4bb6b-zqmqh        0 (0%)        0 (0%)      0 (0%)           0 (0%)         4m36s
  default                     playground-minio-0                              0 (0%)        0 (0%)      2Gi (1%)         0 (0%)         4m35s
  default                     playground-minio-1                              0 (0%)        0 (0%)      2Gi (1%)         0 (0%)         4m35s
  default                     playground-minio-2                              0 (0%)        0 (0%)      2Gi (1%)         0 (0%)         4m35s
  default                     playground-minio-3                              0 (0%)        0 (0%)      2Gi (1%)         0 (0%)         4m34s
  kube-system                 coredns-56f6fc8fd7-wrjsb                        100m (0%)     0 (0%)      70Mi (0%)        170Mi (0%)     4m35s
  kube-system                 local-path-provisioner-5cf85fd84d-4hvct         0 (0%)        0 (0%)      0 (0%)           0 (0%)         4m35s
  kube-system                 metrics-server-5985cbc9d7-ck9h2                 100m (0%)     0 (0%)      70Mi (0%)        0 (0%)         4m35s
  kube-system                 nvidia-device-plugin-daemonset-b4rt2            0 (0%)        0 (0%)      0 (0%)           0 (0%)         20h
  kube-system                 svclb-traefik-f9b9f340-9q8lw                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         20h
  kube-system                 traefik-57b79cf995-dwvnf                        0 (0%)        0 (0%)      0 (0%)           0 (0%)         4m35s
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests     Limits
  --------           --------     ------
  cpu                200m (0%)    0 (0%)
  memory             8332Mi (6%)  170Mi (0%)
  ephemeral-storage  0 (0%)       0 (0%)
  hugepages-1Gi      0 (0%)       0 (0%)
  hugepages-2Mi      0 (0%)       0 (0%)
  nvidia.com/gpu     1            1

Error description from mixcoord pod, the same error is printed in other pods' logs which are not starting:

[2024/11/22 10:34:30.985 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2024/11/22 10:34:30.985 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2024/11/22 10:34:30.985 +00:00] [WARN] [grpcclient/client.go:460] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2024/11/22 10:34:31.786 +00:00] [WARN] [grpcclient/client.go:474] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).ListDatabases\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:639\n  | github.com/milvus-io/milvus/internal/datacoord/broker.(*coordinatorBroker).ListDatabases\n  | \t/workspace/source/internal/datacoord/broker/coordinator_broker.go:123\n  | github.com/milvus-io/milvus/internal/datacoord.(*meta).reloadCollectionsFromRootcoord\n  | \t/workspace/source/internal/datacoord/meta.go:234\n  | github.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1.1.1\n  | \t/workspace/source/internal/datacoord/server.go:671\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1.1\n  | \t/workspace/source/internal/datacoord/server.go:670\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]

Can you please help me to find what what's wrong and how to fix it?

[zeionara]

After I've replaced custom-values.yaml with this file, in which I've disabled mixCoord and enabled rootCoord, dataCoord, queryCoord, indexCoord, the following error is started to be printed in the datanode:

[2024/11/22 11:07:58.086 +00:00] [INFO] [sessionutil/session_util.go:201] ["Session try to connect to etcd"]
[2024/11/22 11:07:58.642 +00:00] [DEBUG] [datanode/data_node.go:567] ["DataNode current state"] [State=Abnormal]
[2024/11/22 11:07:58.788 +00:00] [DEBUG] [datanode/data_node.go:567] ["DataNode current state"] [State=Abnormal]
{"level":"warn","ts":"2024-11-22T11:08:03.087Z","logger":"etcd-client","caller":"v3@v3.5.5/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000698700/localhost:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp [::1]:2379: connect: connection refused\""}
[2024/11/22 11:08:06.088 +00:00] [INFO] [sessionutil/session_util.go:201] ["Session try to connect to etcd"]

Similar error is printed in the proxy pod:

[2024/11/22 11:12:40.145 +00:00] [INFO] [sessionutil/session_util.go:201] ["Session try to connect to etcd"]
{"level":"warn","ts":"2024-11-22T11:12:45.146Z","logger":"etcd-client","caller":"v3@v3.5.5/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000316a80/localhost:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp [::1]:2379: connect: connection refused\""}

And RootCoord error is printed in the rootcoord pod:

[2024/11/22 11:14:35.189 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2024/11/22 11:14:35.189 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2024/11/22 11:14:35.189 +00:00] [WARN] [grpcclient/client.go:460] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2024/11/22 11:14:35.189 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=4] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n  | \t/workspace/source/internal/distributed/querycoord/service.go:181\n  | github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n  | \t/workspace/source/internal/distributed/querycoord/service.go:113\n  | github.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n  | \t/workspace/source/cmd/components/query_coord.go:59\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/11/22 11:14:38.390 +00:00] [WARN] [grpcclient/client.go:474] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n  | \t/workspace/source/internal/distributed/querycoord/service.go:181\n  | github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n  | \t/workspace/source/internal/distributed/querycoord/service.go:113\n  | github.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n  | \t/workspace/source/cmd/components/query_coord.go:59\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]

Rootcoord pod is not starting. Events from the pod description:

Events:
  Type     Reason            Age                    From               Message
  ----     ------            ----                   ----               -------
  Warning  FailedScheduling  27m                    default-scheduler  0/1 nodes are available: 1 node(s) were unschedulable. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  27m                    default-scheduler  0/1 nodes are available: 1 node(s) were unschedulable. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
  Normal   Scheduled         27m                    default-scheduler  Successfully assigned default/playground-milvus-rootcoord-5796cd4bb-m9sr2 to k3d-milvus-server-0
  Normal   Pulled            27m (x2 over 27m)      kubelet            Container image "milvusdb/milvus:v2.4.17" already present on machine
  Normal   Created           27m (x2 over 27m)      kubelet            Created container rootcoord
  Normal   Started           27m (x2 over 27m)      kubelet            Started container rootcoord
  Warning  Unhealthy         21m (x6 over 25m)      kubelet            Liveness probe failed: HTTP probe failed with statuscode: 500
  Warning  Unhealthy         7m17s (x85 over 25m)   kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500
  Warning  BackOff           2m19s (x6 over 2m55s)  kubelet            Back-off restarting failed container rootcoord in pod playground-milvus-rootcoord-5796cd4bb-m9sr2_default(697b0a2f-b6a4-4b09-9891-57a85ea16d5b)

Rootcoord logs:

open pid file: /run/milvus/rootcoord.pid
lock pid file: /run/milvus/rootcoord.pid
[2024/11/22 11:15:14.582 +00:00] [INFO] [roles/roles.go:333] ["starting running Milvus components"]
[2024/11/22 11:15:14.582 +00:00] [INFO] [roles/roles.go:182] ["Enable Jemalloc"] ["Jemalloc Path"=/milvus/lib/:/milvus/lib/libjemalloc.so]
[2024/11/22 11:15:14.587 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource]
[2024/11/22 11:15:14.588 +00:00] [DEBUG] [config/etcd_source.go:52] ["init etcd source"] [etcdInfo="{\"UseEmbed\":false,\"EnableAuth\":false,\"UserName\":\"\",\"PassWord\":\"\",\"UseSSL\":false,\"Endpoints\":[\"playground-etcd-0.playground-etcd-headless.default.svc.cluster.local:2379\",\"playground-etcd-1.playground-etcd-headless.default.svc.cluster.local:2379\",\"playground-etcd-2.playground-etcd-headless.default.svc.cluster.local:2379\"],\"KeyPrefix\":\"by-dev\",\"CertFile\":\"/path/to/etcd-client.pem\",\"KeyFile\":\"/path/to/etcd-client-key.pem\",\"CaCertFile\":\"/path/to/ca.pem\",\"MinVersion\":\"1.3\",\"RefreshInterval\":5000000000}"]
[2024/11/22 11:15:14.588 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[playground-etcd-0.playground-etcd-headless.default.svc.cluster.local:2379,playground-etcd-1.playground-etcd-headless.default.svc.cluster.local:2379,playground-etcd-2.playground-etcd-headless.default.svc.cluster.local:2379]"] [minVersion=1.3]
[2024/11/22 11:15:14.589 +00:00] [DEBUG] [config/etcd_source.go:92] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[playground-etcd-0.playground-etcd-headless.default.svc.cluster.local:2379,playground-etcd-1.playground-etcd-headless.default.svc.cluster.local:2379,playground-etcd-2.playground-etcd-headless.default.svc.cluster.local:2379]"]
[2024/11/22 11:15:14.589 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=EtcdSource]
[2024/11/22 11:15:14.590 +00:00] [INFO] [paramtable/component_param.go:4286] ["DeployModeEnv is not set, use default"] [default=0.5]
[2024/11/22 11:15:14.591 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}]
[2024/11/22 11:15:14.591 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource]
[2024/11/22 11:15:14.591 +00:00] [INFO] [logutil/logutil.go:163] ["Log directory"] [configDir=]
[2024/11/22 11:15:14.591 +00:00] [INFO] [logutil/logutil.go:164] ["Set log file to "] [path=]
[2024/11/22 11:15:14.591 +00:00] [INFO] [roles/roles.go:282] [setupPrometheusHTTPServer]
[2024/11/22 11:15:14.591 +00:00] [INFO] [http/server.go:160] ["management listen"] [addr=:9091]
[2024/11/22 11:15:14.591 +00:00] [INFO] [gc/gc_tuner.go:137] ["GC Helper initialized."] ["Initial GoGC"=100] [minimumGOGC=30] [maximumGOGC=200] [memoryThreshold=121414603161]
[2024/11/22 11:15:14.591 +00:00] [INFO] [rootcoord/root_coord.go:154] ["update rootcoord state"] [state=Abnormal]
[2024/11/22 11:15:14.591 +00:00] [INFO] [rootcoord/service.go:154] ["RootCoord listen on"] [address="[::]:53100"] [port=53100]
[2024/11/22 11:15:14.591 +00:00] [INFO] [rootcoord/service.go:196] ["init params done.."]
[2024/11/22 11:15:14.591 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[playground-etcd-0.playground-etcd-headless.default.svc.cluster.local:2379,playground-etcd-1.playground-etcd-headless.default.svc.cluster.local:2379,playground-etcd-2.playground-etcd-headless.default.svc.cluster.local:2379]"] [minVersion=1.3]
[2024/11/22 11:15:14.592 +00:00] [INFO] [rootcoord/service.go:216] ["etcd connect done ..."]
[2024/11/22 11:15:14.592 +00:00] [INFO] [rootcoord/service.go:276] ["start grpc "] [port=53100]
[2024/11/22 11:15:14.693 +00:00] [INFO] [rootcoord/service.go:233] ["grpc init done ..."]
[2024/11/22 11:15:14.693 +00:00] [INFO] [rootcoord/service.go:236] ["RootCoord start to create DataCoord client"]
[2024/11/22 11:15:14.693 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[playground-etcd-0.playground-etcd-headless.default.svc.cluster.local:2379,playground-etcd-1.playground-etcd-headless.default.svc.cluster.local:2379,playground-etcd-2.playground-etcd-headless.default.svc.cluster.local:2379]"] [minVersion=1.3]
[2024/11/22 11:15:14.699 +00:00] [INFO] [rootcoord/service.go:245] ["RootCoord start to create QueryCoord client"]
[2024/11/22 11:15:14.701 +00:00] [INFO] [dependency/factory.go:85] ["try to init mq"] [standalone=false] [mqType=pulsar]
[2024/11/22 11:15:14.709 +00:00] [INFO] [sessionutil/session_util.go:296] ["start server"] [name=rootcoord] [address=10.42.0.170:53100] [id=194] [server_labels={}]
[2024/11/22 11:15:14.709 +00:00] [INFO] [rootcoord/root_coord.go:154] ["update rootcoord state"] [state=Initializing]
[2024/11/22 11:15:14.715 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=2024/11/22 11:11:17.699 +00:00] [save=2024/11/22 11:15:17.710 +00:00] [next=2024/11/22 11:15:14.710 +00:00]
[2024/11/22 11:15:14.715 +00:00] [INFO] [rootcoord/root_coord.go:398] ["id allocator initialized"] [root_path=by-dev/kv] [sub_path=gid] [key=idTimestamp]
[2024/11/22 11:15:14.718 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=2024/11/22 11:11:17.702 +00:00] [save=2024/11/22 11:15:17.716 +00:00] [next=2024/11/22 11:15:14.716 +00:00]
[2024/11/22 11:15:14.718 +00:00] [INFO] [rootcoord/root_coord.go:422] ["tso allocator initialized"] [root_path=by-dev/kv] [sub_path=gid] [key=idTimestamp]
[2024/11/22 11:15:14.718 +00:00] [INFO] [rootcoord/root_coord.go:341] ["Using etcd as meta storage."]
[2024/11/22 11:15:14.720 +00:00] [INFO] [rootcoord/meta_table.go:157] ["recover databases"] ["num of dbs"=1]
[2024/11/22 11:15:14.721 +00:00] [INFO] [rootcoord/kv_catalog.go:741] ["unmarshal all collection details cost"] [db=0] [cost=826ns]
[2024/11/22 11:15:14.721 +00:00] [INFO] [rootcoord/kv_catalog.go:745] ["append partition and fields info cost"] [db=0] [cost=639ns]
[2024/11/22 11:15:14.724 +00:00] [INFO] [rootcoord/kv_catalog.go:741] ["unmarshal all collection details cost"] [db=1] [cost=633ns]
[2024/11/22 11:15:14.724 +00:00] [INFO] [rootcoord/kv_catalog.go:745] ["append partition and fields info cost"] [db=1] [cost=200ns]
[2024/11/22 11:15:14.724 +00:00] [INFO] [rootcoord/meta_table.go:200] ["collections recovered from db"] [db_name=default] [collection_num=0] [partition_num=0]
[2024/11/22 11:15:14.726 +00:00] [INFO] [rootcoord/meta_table.go:216] ["RootCoord meta table reload done"] [duration=7.600315ms]
[2024/11/22 11:15:14.727 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/22 11:17:06.184 +00:00] [WARN] [retry/retry.go:46] ["retry func failed"] [retried=0] [error="connection error"]
[2024/11/22 11:18:44.242 +00:00] [WARN] [roles/roles.go:312] ["Get signal to exit"] [signal=terminated]