Description
We have a weaviate cluster with 3 nodes.
Average request latency (crud/vsearch+QUORUM) 10-70ms.
When one node fails (or a pod restarts), eg: “weaviate-2”, the requests latency increases to 10sec for all requests directed to “weaviate-1” (visible in the log). But all requests directed to “weaviate-0” remain fast (10-70ms). Regardless of which node is down: one remaining node is “slow” and the other is “fast”. This both valid for single requests or high load…
Server Setup Information
- Weaviate Server Version: 1.25.6
- Deployment Method: k8s
- Multi Node? Number of Running Nodes: 3 nodes (repl.factor=3)
- Client Language and Version: Python3, PythonClient3
- Multitenancy?: No
Any additional Information
“replicationConfig”: {
“factor”: 3
},
env:
- name: RAFT_JOIN
value: weaviate-0,weaviate-1,weaviate-2
- name: RAFT_BOOTSTRAP_EXPECT
value: ‘3’
resources:
limits:
cpu: ‘50’
memory: 500Gi
requests:
cpu: ‘50’
memory: 500Gi