Reiterating this from Telegram so people can see it.

--

The site went down last night because I added a new database to the cluster which I was hoping to set up quietly without any downtime. Unfortunately, it caused other databases to crash, which I'm having trouble recovering, because when multiple nodes go down restarting everything is a massive pain in the ass.

The new node is going to replace this more complicated setup. Over the last month, I've found more sure footing for our physical hardware I have not been able to put into use for over 6 months now. This transfer is to put our hardware back in use so I can stop juggling servers on multiple continents.

We will still retain redundancy so that if the Kiwi Farms is ever disconnected from the Internet again, there will not be extended downtime. However, the overall topology will be simplified and (hopefully) random crashes like this will stop happening so frequently.

TL;DR: We're basically right back to where we started in August and the last thing I have to do is get Ol' Faithful running again, which unfortunately means some downtime.