I have since resolved this by deleting all the persistent volumes & claims attached to my replicas and starting fresh, so I don’t think I can further debug anything.
But at the time, I know it was not a networking issue on the clusters because I verified connectivity between the pods themselves before posting here. I also deleted the pods several times but hit the same error on each pod each time.
If you have a cluster of nodes, should the cluster still come up even if all 3 nodes are down? Again, none of the replicas were alive, but based on the logs it seems like each replica assumed that at least 1 other node would be running (or at least that another node would decide to be the leader, which none of them did).
There were no noteworthy K8 events.
Are there resource constraints in order to spin up a node? I was under the impression that they would max out whatever node is available to them. My replicas are running on nodes with 32 GiB of RAM, and I certainly did not have even remotely close to that amount of data.