Conversation

Notices

1. sj_zero (sj_zero@social.fbxl.net)'s status on Tuesday, 11-Jun-2024 10:27:13 JST sj_zero
   [admin mode] completed some maintenance tasks on the database(hence the inaccessibility today), it shrank the database a shocking amount so I'd call that a win. Should improve performance, I've still got more to do later but it's fine for now.
   • † top dog :pedomustdie: likes this.
   • † top dog :pedomustdie: (dcc@annihilation.social)'s status on Tuesday, 11-Jun-2024 10:27:35 JST † top dog :pedomustdie:

     in reply to

     • Soul Dessin
     @sj_zero @souldessin Did you use repack? (if not you should)
   • sj_zero (sj_zero@social.fbxl.net)'s status on Tuesday, 11-Jun-2024 10:27:36 JST sj_zero

     in reply to

     • Soul Dessin
     The database cleanup was just running a couple of built in pleroma scripts that clean up old data (cleaning out 25GB of remote posts!), but the real issue seems to have been a number of faulty packets with invalid addresses being aimed at fbxl social in particular (once I stopped accepting packets for fbxl domains the server came back and once I started again)
     
     My hypothesis so far is that either intentionally due to malicious attack or unintentionally due to a misconfigured server a bunch of these malformed packets were sent my way, and filled up connection slots in the kernel, locking up not just http but telnet and icmp. Once I added some configuration changes to increase the number of connection slots and also to filter any packets with bad addresses (sysctl calls them martian packets since they come from "alien" address spaces) the connection issues ceased.
   • Soul Dessin (souldessin@noauthority.social)'s status on Tuesday, 11-Jun-2024 10:27:37 JST Soul Dessin

     in reply to

     @sj_zero
     I'm curious, what was the core of the issue?

   • sj_zero (sj_zero@social.fbxl.net)'s status on Tuesday, 11-Jun-2024 10:29:06 JST sj_zero

     in reply to

     • † top dog :pedomustdie:
     • Soul Dessin
     I've had an automated weekly repack for a year or two now, it has a big impact.
     † top dog :pedomustdie: likes this.