Locally running RAG pipeline with Verba and Llama3 with Ollama

Hi @bam !!

Welcome to our community

I have just built and run Verba using Ollama to test the newer version.

I noticed that while we have recently implemented a public docker image, it doesn’t have the arm platform images yet.

So first I cloned the repo, built the image, and used this docker compose to spin it that same image up.

Considering the error message you pasted, it looks like the OLLAMA_URL is pointing to localhost. I tried to reproduce this error, but got a different one:

verba-1     | ✘ No documents imported 0 of 1 succesful tasks
verba-1     | ℹ FileStatus.ERROR | netherlands-wikipedia-article-text.pdf | Import
verba-1     | for netherlands-wikipedia-article-text.pdf failed: Import for
verba-1     | netherlands-wikipedia-article-text.pdf failed: Batch vectorization failed:
verba-1     | Vectorization failed for some batches: Cannot connect to host localhost:11434
verba-1     | ssl:default [Connection refused] | 0

Are you running ollama locally? If that’s the case, you should set the OLLAMA_URL accordingly:

---

services:
  verba:
    image: verba-verba
    ports:
      - 8000:8000
    environment:
      - WEAVIATE_URL_VERBA=http://weaviate:8080
      - OPENAI_API_KEY=$OPENAI_API_KEY
      - COHERE_API_KEY=$COHERE_API_KEY
      - OLLAMA_URL=http://host.docker.internal:11434
      - OLLAMA_MODEL=llama3.1
      - OLLAMA_EMBED_MODEL=mxbai-embed-large
      - UNSTRUCTURED_API_KEY=$UNSTRUCTURED_API_KEY
      - UNSTRUCTURED_API_URL=https://api.unstructured.io/general/v0/general
      #- GITHUB_TOKEN=$GITHUB_TOKEN

    volumes:
      - ./data:/data/
    depends_on:
      weaviate:
        condition: service_healthy
    healthcheck:
      test: wget --no-verbose --tries=3 --spider http://localhost:8000 || exit 1
      interval: 5s
      timeout: 10s
      retries: 5
      start_period: 10s

  weaviate:
    command:
      - --host
      - 0.0.0.0
      - --port
      - '8080'
      - --scheme
      - http
    image: semitechnologies/weaviate:1.25.10
    ports:
      - 8080:8080
      - 3000:8080
    volumes:
      - weaviate_data:/var/lib/weaviate
    restart: on-failure:0
    healthcheck:
      test: wget --no-verbose --tries=3 --spider http://localhost:8080/v1/.well-known/ready || exit 1
      interval: 5s
      timeout: 10s
      retries: 5
      start_period: 10s
    environment:
      OPENAI_APIKEY: $OPENAI_API_KEY
      COHERE_APIKEY: $COHERE_API_KEY
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      ENABLE_MODULES: 'e'
      CLUSTER_HOSTNAME: 'node1'

volumes:
  weaviate_data: {}
...

Let me know if that helps!

Thanks!