Hi, Weaviate community!
Weaviate version: 1.25.25 (and 1.26.13 for other instance)
Index Type: HNSW
Named vectors: 2
Lengths of vectors: 768 and 1024
ef = 320 and 480
efConstruction = 320 and 480
maxConnections = 100 and 120
Load Test (for measuring QPS and Recall) using .pkl files.
Limit: 10
Max CPU: 40 cores
Max RAM: 443 GB (on tests weaviate takes up to 60%)
Env vars:
LIMIT_RESOURCES: ‘true’
PERSISTENCE_HNSW_MAX_LOG_SIZE: 4GiB
ASYNC_INDEXING: ‘true’
GOGC: 90
on number of vectors 5 million - average search speed - 200 obj/s
on number of vectors 20 million - average search speed - 60-70 obj/s on start and goes up to 160-170 obj/s.
On every starting search queries Weaviate increases RAM consumption (sometimes decreases) and performance on start is poor, what impacts on final results. Why is that?