Description
I am connecting with these timeout settings:
client = weaviate.connect_to_local(
headers=headers,
additional_config=AdditionalConfig(
timeout=Timeout(init=30, query=60, insert=120) # Values in seconds
)
)
I am using the dynamic batch functionality which I specifically thought would rate limit itself to avoid any timeout issues, but I am seeing these types of errors at the end of my batch:
Failed to import object with error: %s "ErrorObject(message=‘WeaviateBatchError(\‘Query call with protocol GRPC batch failed with message <AioRpcError of RPC that terminated with:\\n\\tstatus = StatusCode.DEADLINE_EXCEEDED\\n\\tdetails = "Deadline Exceeded"\\n\\tdebug_error_string = "UNKNOWN:Error received from peer {created_time:"2024-07-25T15:35:02.96801-04:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"\\n>.\’)’, object_=_BatchObject(collection=‘Product’, vector=None, uuid=‘190f46bd-9a54-5a77-8463-b4845e0e4384’, properties=
…
I can see that the count of objects doesn’t match the number that went into the batch processor (it’s off by almost half), so I’m guessing there is a timeout going into the vectorizer and so the object fails to insert?
Is there a different timeout I should be setting (infinite?) or some other way I should be throttling or retrying these things?
Do you advise people to infinitely loop over the failed batch objects and retry them until the failed batch array is empty?