@mint@p The adaptation is mainly "get some new hardware, this computer is busted". While I am waiting to be able to do that, I'll just resurrect FSE as a Revolver instance.
@mint@p This is a good idea, but I don't think the datacenter would let me do that, and it's worse than you might imagine: resetting leaves the thing wedged, it has to do a full power-cycle in order to come back to life.
@p I think that is probably the way to go. New box is expensive and will not be ready for a while (read: a couple of paychecks from now, probably longer than it would take to kick Revolver out the door). Keeping this machine running is exhausting and untenable. Additionally, I swore repeatedly that I would not hand FSE off to someone so getting someone else and even if I hadn't promised, that I will not hand it off. For the trouble of a move to new infra, it'd be better to just move the data to Revolver.
I understand that this sucks for people using FSE. My apologies. I think the best way to bring FSE back online is to replace it with the thing that can't be killed, and restore functionality as quickly as possible. If you used FSE and have words/thoughts/suggestions, feel free to let me know (if you have a different username wherever you are now, please mention your username on FSE; I am tired and distracted and may not be able to tell).
I think the best course of action at present is finishing the thing that solves this problem permanently, so that's what I'm going to do unless I hear a better idea from someone with a stake in it. I've got an obligation and I will do my damnedest: the point of FSE was to provide a place for you, I said I would, and it evaporated. So if there's anything I can do (e.g., your follower/block lists for wherever you are staying right now, or just a ping so I know where to let you know when it's back up and running), or if you have any ideas or concerns about how I'm rebuilding the place, or if you want help setting up your own instance, whatever the case, just let me know.
If you had a VM on that box and you want the data out, let me know over DMs and we'll figure something out.
As much as I appreciate :moon: letting me crash on his couch while my house is on fire, so as to avoid putting him out, I might hop to a temporary single-user something or another. The drama club blew through last time I was here and I don't want to shit this place up telling them to fuck off or waste my own time dealing with them, but I also don't want to miss messages from actual people. Maybe I'll just be on IRC more and refuse to talk about anything here besides FSE/Revolver.
@p Time and effort I spend keeping FSE alive, maintaining this piece of shit box and throwing cash at the datacenter and replacement parts, and fixing shit in the middle of the night and waking up and the box has failed and I just drove down there and popped that stick of RAM and thought, "All right, that's the last problem solved. It'll be stable and I can spend my time half on making money and half on finishing Revolver." and here we are again. Box has locked up. Nothing in the IPMI, I expect. I just drove there in the middle of the night to fix the Last Problem and now I gotta file a ticket to get the thing restarted.
Back to job-work in about four hours and I gotta stay up and deal with this box.
I should just take the last backup I have and load that into Revolver, point everything there, and spend the time cramming ActivityPub features and UI endpoints into Revolver instead of maintaining this machine. The problem with that plan is that I committed to keeping it running until Revolver was ready, but if it's a couple of weeks of downtime to get to parity, maybe that's worth the couple of weeks not having to deal with that box. The cost there is all the DMs go away, I'm not putting those into Revolver, plus some media, like there's the old backups but once Revolver's data storage was stable enough, I stopped backing up the uploads and let media.freespeechextremist.com populate the old stuff organically, so there's no guarantee about the size of the gap there, but the missing media would be stuff that no one looked at between the last backup and now, and that's in the off-chance that I can't actually get the old stuff out. Once there's enough stuff implemented that's operable (there are a lot of parts that really aren't ready, so there would be hacks), I could rig enough shit up to probably let people log in with their old passwords and get their accounts live again. Maybe it'd not even require a login, like I have the old DB with tokens and everything, I could generate the keys, map them to local user IDs, then people that are logged in would get greeted with, until the UI endpoints are done, probably just "Here, download your private key, you'll need it later." That actually seems doable. Shitty "user experience", but the box is up and down, it's already a shitty situation.
There are a lot of parts of Revolver that just aren't finished, there are parts that need to be ripped out and replaced, there are parts that are currently *half*-replaced (like the REPL), and there are parts that only work on Plan 9 for right now (all trivial to turn portable, I think, just I haven't done it), but as evidenced by the media server exceeding expectations for handling the load of all the uploads, the base is pretty stable and reliable. (I mean, that's the media server, though. That is a simpler task than handling *all* of the stuff Pleroma was doing. But the only time that thing crashed was when IPFS choked all the RAM out of the Frantech VM it was running on--not hard at 4GB. It hasn't fallen over since I just killed IPFS on that box.) Having to hack fast to get some semblance of parity would destabilize it, sure, but I think I could keep up just fine and the upside is that any errors happening would be bugs to fix in Revolver rather than bailing water trying to keep that machine on life support, which would definitely improve my morale.
On the other hand the service abruptly disappearing would be an inconvenience for most of the people on FSE. We get a phoenix rising from the ashes, but it's a half-retarded phoenix for a while and it takes a couple of weeks.
@p All right, plan before replacing the fucked PSU and the MCE generator impersonating a stick of RAM was to ditch the VMs and just move everything onto the host machine.
That's my last idea short of replacing the entire box.
> Last successful contact: 2023-10-22 (3.4 hours ago)
When I bring it back up, I'm gonna try moving back to an earlier kernel. Similar machine here (much smaller but same CPU) had the same problem *once* with KVM and I downgraded the kernel and it hasn't had that problem since. The one-time problem here might have been a fluke so maybe this doesn't do anything, or it solves the problem with this machine. Impossible to tell.
Alt of a @p@freespeechextremist.com , if you even believe that.If I'm posting here, it's usually because FSE is down.I am working on Revolver: https://liberapay.com/Revolver .