Quantcast
Channel: Weaviate Community Forum - Latest posts
Viewing all articles
Browse latest Browse all 3605

Please clarify restoring filesystem backup on another instance

$
0
0

Description

I have backed up my weaviate collection on the local filesystem of my ‘finland’ weaviate instance. Here is the resulting tree view:

mema@newisa:/sata/backup/20240722060001_mema_wv_backup$ tree
.
├── backup_config.json
└── finland
    ├── Articles_intfloat_multilingual_e5_large
    │   └── chunk-1
    └── backup.json

I would now like to restore this collection on another machine, that is hosting another database instance (with the CLUSTER_HOSTNAME set to mema3 and not finland).

Of course I will be able to transfer this backup tree on the machine hosting the new database.

Server Setup Information

Original server:
docker compose based server - definition:

  weaviate:
    image: cr.weaviate.io/semitechnologies/weaviate:1.25.4
    command:
      - "--host=0.0.0.0"
      - "--port=8080"
      - "--scheme=http"
    restart: unless-stopped
    environment:
      LOG_LEVEL: info 
      ENABLE_MODULES: 'backup-filesystem'
      BACKUP_FILESYSTEM_PATH: /tmp/backup
      ENABLE_CUDA: 0
      LIMIT_RESOURCES: true
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: true
      PERSISTENCE_DATA_PATH: /var/lib/weaviate
      CLUSTER_HOSTNAME: finland
      DISABLE_TELEMETRY: true
      GOMAXPROCS: 4
    networks:
      - weaviate_net
      - mema_network
    ports:
      - "8080:8080"
      - "50051:50051"
    volumes:
      - weaviate_data:/var/lib/weaviate
      - /sata/backup:/tmp/backup
    logging: *default-logging

destination server - also docker compose based. service definition follows:

  weaviate:
    image: cr.weaviate.io/semitechnologies/weaviate:1.25.8
    command:
      - "--host=0.0.0.0"
      - "--port=8080"
      - "--scheme=http"
    restart: unless-stopped
    environment:
      LOG_LEVEL: info 
      ENABLE_MODULES: 'backup-filesystem'
      BACKUP_FILESYSTEM_PATH: /tmp/backup
      ENABLE_CUDA: 0
      LIMIT_RESOURCES: true
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: true
      PERSISTENCE_DATA_PATH: /var/lib/weaviate
      CLUSTER_HOSTNAME: mema3
      DISABLE_TELEMETRY: true
      GOMAXPROCS: 24
    networks:
      - weaviate_net
      - mema_network
    ports:
      - "8080:8080"
      - "50051:50051"
    volumes:
      - weaviate_data:/var/lib/weaviate
      - /backups:/tmp/backup
    logging: *default-logging

you may notice the cluster hostname id is different, the host voume name for the backups is different too, the minor versions differ a bit, the GOMAXPROCS too, the rest is quite identical.

I am trying with the following script on the new “mema3” machine:

#!/bin/bash
curl -X POST -H "Content-Type: application/json" -d '{"id": "20240722060001_mema_wv_backup"}' "http://localhost:8080/v1/backups/filesystem/20240722060001_mema_wv_backup/restore"

after copying the backup directory from the old machine to the following tree on the new mema3 machine:

mema@mema3:/backups$ tree
.
├── 20240722060001_mema_wv_backup
│   ├── backup_config.json
│   └── finland
│       ├── Articles_intfloat_multilingual_e5_large
│       │   └── chunk-1
│       └── backup.json
└── backup_weaviate.sh

So to recap weaviate on the new machine is started as a container with the following docker compose service declaration:

  weaviate:
    image: cr.weaviate.io/semitechnologies/weaviate:1.25.8
    command:
      - "--host=0.0.0.0"
      - "--port=8080"
      - "--scheme=http"
    restart: unless-stopped
    environment:
      LOG_LEVEL: info 
      ENABLE_MODULES: 'backup-filesystem'
      BACKUP_FILESYSTEM_PATH: /tmp/backup
      ENABLE_CUDA: 0
      LIMIT_RESOURCES: true
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: true
      PERSISTENCE_DATA_PATH: /var/lib/weaviate
      CLUSTER_HOSTNAME: mema3
      DISABLE_TELEMETRY: true
      GOMAXPROCS: 24
    networks:
      - weaviate_net
      - mema_network
    ports:
      - "8080:8080"
      - "50051:50051"
    volumes:
      - weaviate_data:/var/lib/weaviate
      - /backups:/tmp/backup
    logging: *default-logging

which means the /backups host directory is “mounted” in the container as /tmp/backup

but the curl command fails with a reference to the CLUSTER_HOSTNAME of the old machine (“finland”):

mema@mema3:~/code/aisearch/resources$ curl -X POST -H "Content-Type: application/json" -d '{"id": "20240722060001_mema_wv_backup"}' "http://localhost:8080/v1/backups/filesystem/20240722060001_mema_wv_backup/restore"
{"error":[{"message":"cannot resolve hostname for \"finland\""}]}

how can I fix this? Thanks


Viewing all articles
Browse latest Browse all 3605

Trending Articles