Hi, we still observe constant storage drain in one of our 3 replicas. This replica fails with condensing with the following error:
{"action":"hnsw_condensing","level":"info","msg":"start hnsw condensing","time":"2024-07-22T07:32:26Z"}
{"action":"hnsw_condensing_complete","level":"info","msg":"completed hnsw condensing","time":"2024-07-22T07:32:28Z"}
{"action":"hnsw_commit_logger_combine_condensed_logs","id":"main","input_first":"/var/lib/weaviate/aiskillv2/5SQtIKAXTNQ6/main.hnsw.commitlog.d/1721630712.condensed","input_second":"/var/lib/weaviate/aiskillv2/5SQtIKAXTNQ6/main.hnsw.commitlog.d/1721633482.condensed","level":"info","msg":"successfully combined previously condensed commit log files","output":"/var/lib/weaviate/aiskillv2/5SQtIKAXTNQ6/main.hnsw.commitlog.d/1721630712","time":"2024-07-22T07:32:28Z"}
{"action":"hnsw_condensing","level":"info","msg":"start hnsw condensing","time":"2024-07-22T07:32:28Z"}
{"action":"hnsw_condensing_complete","level":"info","msg":"completed hnsw condensing","time":"2024-07-22T07:32:30Z"}
{"action":"commit_log_file_switched","id":"main","level":"info","msg":"commit log size crossed threshold, switching to new file","new_file_name":"1721633579","old_file_name":"1721633525","old_file_size":105250816,"time":"2024-07-22T07:32:59Z"}
{"action":"hnsw_condensing","level":"info","msg":"start hnsw condensing","time":"2024-07-22T07:32:59Z"}
{"action":"hnsw_condensing_complete","level":"info","msg":"completed hnsw condensing","time":"2024-07-22T07:33:00Z"}
{"action":"hnsw_commit_logger_combine_condensed_logs","id":"main","input_first":"/var/lib/weaviate/aiskillv2/1efo8r43N3tC/main.hnsw.commitlog.d/1721611794.condensed","input_second":"/var/lib/weaviate/aiskillv2/1efo8r43N3tC/main.hnsw.commitlog.d/1721633525.condensed","level":"info","msg":"successfully combined previously condensed commit log files","output":"/var/lib/weaviate/aiskillv2/1efo8r43N3tC/main.hnsw.commitlog.d/1721611794","time":"2024-07-22T07:33:02Z"}
{"action":"hnsw_condensing","level":"info","msg":"start hnsw condensing","time":"2024-07-22T07:33:02Z"}
{"action":"hnsw_condensing_complete","level":"info","msg":"completed hnsw condensing","time":"2024-07-22T07:33:11Z"}
Is there something we could do to improve that?
Can we somehow reduce replicas from 3 to 1 and forget about those failing?