Spark connector with Weaviate

Description

I am integrating Glue Streaming with Weaviate using Spark Connector. I have uploaded Spark connector jar in S3, and added S3 path in Glue job. After adding, I have below code in my Glue Job.

@DudaNogueira

try:
    spark = SparkSession.builder.config(
        "spark.jars",
        "spark-connector-assembly-1.3.2.jar", 
    ).appName("KafkaToS3") \
        .getOrCreate()
except Exception as e:
    logger.error(f"Failed to initialize SparkSession: {str(e)}")
    traceback.print_exc()
    sys.exit(1)

@hsm207 - I’m referring your post for the above.

Server Setup Information

• Weaviate Server Version: 1.26
• Deployment Method: Docker
• Multi Node? Number of Running Nodes: 1
• Client Language and Version: Python, 3
• Multitenancy?: Yes