Description
We are ingesting and chunking data from files, and are creating tens of thousands of tenants under a single weaviate class. Our process runs fine for a short time, but weaviate eventually starts throwing disk quota exceeded errors. AWS is telling me that Weaviate is creating locks on EFS files and not releasing them.
{"action":"hnsw_commit_log_condensing","error":"open commit log to be condensed: open /var/lib/weaviate/catalogsearch/CatalogItem_BERT_MultiLingual_e5/main.hnsw.commitlog.d/1736210947: disk quota exceeded","level":"error","msg":"hnsw commit log maintenance (condensing) failed","time":"2025-01-07T03:38:27Z"}
{"action":"hnsw_commit_log_maintenance","error":"stat /var/lib/weaviate/catalogsearch/CatalogItem_BERT_MultiLingual_e5/main.hnsw.commitlog.d/1736212133: use of closed file","level":"error","msg":"hnsw commit log maintenance failed","time":"2025-01-07T03:38:28Z"}
UnexpectedStatusCodeError: Creating object! Unexpected status code: 500, with response body: {'error': [{'message': 'put object: import into index contractexcellence: put local object: shard="e0fc9bf8b418ce94ce0614609813b123f84dd9d4fc2c4a11c6b2214c6a0333ff": flush prop length tracker to disk: open /var/lib/weaviate/contractexcellence/e0fc9bf8b418ce94ce0614609813b123f84dd9d4fc2c4a11c6b2214c6a0333ff/proplengths.tmp: disk quota exceeded'}]}.
Traceback (most recent call last):
File "/var/task/lib/python3.9/site-packages/gpai_document_ragification/handlers/rag_creation_event_lambda_handler.py", line 10, in handle_s3_creation_event
to_return = HandlerModule.get_document_create_handler().handle_s3_event(event)
File "/var/task/lib/python3.9/site-packages/gpai_document_ragification/handlers/document_handler.py", line 32, in handle_s3_event
return self._handle_event(state_machine_input, document_rag_record)
File "/var/task/lib/python3.9/site-packages/gpai_document_ragification/handlers/document_create_handler.py", line 53, in _handle_event
self.save_chunks_to_vector_db(
File "/var/task/lib/python3.9/site-packages/gpai_document_ragification/handlers/document_create_handler.py", line 120, in save_chunks_to_vector_db
self.weaviate_proxy.save_object(
File "/var/task/lib/python3.9/site-packages/gpai_document_ragification/proxy/weaviate_proxy.py", line 26, in save_object
self.weaviate_client.data_object.create(
File "/var/task/lib/python3.9/site-packages/weaviate/data/crud_data.py", line 160, in create
raise UnexpectedStatusCodeException("Creating object", response)
{"action":"hnsw_commit_log_maintenance","error":"stat /var/lib/weaviate/catalogsearch/CatalogItem_BERT_MultiLingual_e5/main.hnsw.commitlog.d/1736212133: use of closed file","level":"error","msg":"hnsw commit log maintenance failed","time":"2025-01-07T03:36:51Z"}
{"action":"lsm_compaction","class":"Contractexcellence","error":"open /var/lib/weaviate/contractexcellence/9994450152d1268c6f6e6c64c10c31168b4d880906a69ce2bc6c410deeb3b22d/lsm/property_text_chunk/segment-1736207677691312573_1736210936799682015.db.tmp: disk quota exceeded","index":"contractexcellence","level":"error","msg":"compaction failed","path":"/var/lib/weaviate/contractexcellence/9994450152d1268c6f6e6c64c10c31168b4d880906a69ce2bc6c410deeb3b22d/lsm/property_text_chunk","shard":"9994450152d1268c6f6e6c64c10c31168b4d880906a69ce2bc6c410deeb3b22d","time":"2025-01-07T03:36:51Z"}
Server Setup Information
- Weaviate Server Version: 1.24.8
- Deployment Method: EKS on AWS
- Multi Node? Number of Running Nodes: 1 ECS Node
- Client Language and Version: PythonV3
- Multitenancy?: Yes