Multimodal search with Bring your own vector

hi @Krishna_C !!

Welcome back to our forums

There are some problems with this schema, and some couple of things you must understand on how our multi2vec_clip works so you can make the most out of it.

First, I see you may be storing the image embeddings as a property. This is far from optimal, as the inverted index will create keyword indexes for your vectors

That will lead to unnecessary memory usage, as you don’t want to perform keyword searches on your vector dimensions

If you really want to do that, you need to make sure to specify that this property shouldn’t be searchable nor filterable, like so:

        wvc.config.Property(
            name="image_embeddings", 
            data_type=wc.DataType.NUMBER_ARRAY, 
            index_filterable=False, 
            index_searchable=False
        ),

The second issue I see is this:

            image_fields=[
                wc.Multi2VecField(name="image_embeddings")
            ],

And the problem here is that the image field must be a blob, and not the image vector, as we state in our docs:

Now, the way that our multi2vec_clip works is that Weaviate will generate a vector for all texts fields and image fields, and combine those vectors.

So you may not need to have one named vector for each of the properties isolated. If you have one vector for all text and image, that vector will ve a representation of those content.

Let me know if this clarifies and how I can help you further.

Thanks!