Duplicate data detection in weaviate

hi @Rohini_vaidya !!

data.insert will fail when the uuid already exists. So if you have a unique content to pass to generate_uuid5, you can always generate the same uuid.

Here an example:

from weaviate.util import generate_uuid5

print(["123", "456"], generate_uuid5(["123", "456"]))
print(["456", "123"], generate_uuid5(["456", "123"]))
print("123456", generate_uuid5("123456"))
print(["123456"], generate_uuid5(["123456"]))
print("456123", generate_uuid5("456123"))

this will print

['123', '456'] 9f5bdeb4-dc32-5f94-9689-177bf744c134
['456', '123'] d98d6139-cead-5428-8897-6cf46e496aef
123456 a52b2702-9bcf-5701-852a-2f4edc640fe1
['123456'] 14f87c42-4614-504a-88a2-6a10ff4fa6e7
456123 3b8fcdc1-2b41-514a-9a5c-2562dd5813ae