Conversation

Notices

1. ♡ miss olivia ♡ (olivia@pl.voltrina.net)'s status on Tuesday, 15-Oct-2024 01:16:58 JST ♡ miss olivia ♡
   >pleroma database has grown by 7GB in a span of 4 days
   what
   •  (mint@ryona.agency)'s status on Tuesday, 15-Oct-2024 01:16:54 JST 

     in reply to

     • :blank:
     @i @olivia Bet a good half of that would happen to be a bunch of those mystery misskey.io posts.
     † top dog :pedomustdie: likes this.
   • :blank: (i@declin.eu)'s status on Tuesday, 15-Oct-2024 01:16:55 JST :blank:

     in reply to
     @olivia how do you have so many stuck jobs lol, normal oban should hover at a couple of k jobs, if you don't mind losing the unfederated backlog, you can just delete from the table freely
     
     mind showing what pleroma=> select state, queue, attempt, count(*) from oban_jobs group by state, queue, worker, attempt order by count(*) desc; prints?
   • ♡ miss olivia ♡ (olivia@pl.voltrina.net)'s status on Tuesday, 15-Oct-2024 01:16:56 JST ♡ miss olivia ♡

     in reply to

     • :blank:
     oban_jobs seems to be the culprit
   • :blank: (i@declin.eu)'s status on Tuesday, 15-Oct-2024 01:16:57 JST :blank:

     in reply to
     @olivia which tables? https://pl.voltrina.net/phoenix/live_dashboard/ecto_stats
     
     some people keep seeing massive instance killing database growth but never share the details
   •  (mint@ryona.agency)'s status on Tuesday, 15-Oct-2024 01:32:18 JST 

     in reply to

     • :blank:
     @i @olivia Here's another report of the same shit happening: https://git.pleroma.social/pleroma/pleroma/-/issues/3335
     20 retries might indeed be too much (I have around 1500 jobs right now, most of which are completed, a good half of those that aren't are retries for DRC since verita's :cloudflare: rules keep blocking me or something), but I'm more interested in how the fuck do they manage to pile up so hard. Forcefetched the mentioned post without any issues.
     † top dog :pedomustdie: likes this.
   • :blank: (i@declin.eu)'s status on Tuesday, 15-Oct-2024 01:32:19 JST :blank:

     in reply to
     @olivia forgot feld made most things a background when removing worker from select but not group, probably as mint says, remote fetch workers stuck with 20! retries, working through spam
     
     run watch 'sudo -Hu postgres psql pleroma -c "delete from oban_jobs where attempt > 3;"' for some hours and it should clear up
     † top dog :pedomustdie: likes this.
   • ♡ miss olivia ♡ (olivia@pl.voltrina.net)'s status on Tuesday, 15-Oct-2024 01:32:20 JST ♡ miss olivia ♡

     in reply to

     • :blank:

     there you go

     state | queue | attempt | count -----------+----------------------+---------+--------- available | background | 0 | 3773277 available | background | 2 | 1863333 available | background | 1 | 1022895 available | background | 3 | 494206 available | background | 4 | 266730 available | background | 5 | 130975 available | background | 6 | 67975 available | background | 7 | 34023 available | background | 0 | 28042 available | background | 8 | 16993 available | background | 9 | 8558 available | notifications | 0 | 6916 available | background | 10 | 4201 available | background | 11 | 2264 available | background | 12 | 988 available | background | 13 | 480 available | background | 14 | 205 completed | federator_outgoing | 1 | 150 completed | federator_incoming | 1 | 130 completed | search_indexing | 1 | 127 available | background | 15 | 117 cancelled | federator_incoming | 1 | 85 scheduled | background | 0 | 77 retryable | background | 10 | 67 retryable | background | 2 | 56 available | background | 0 | 55 retryable | background | 3 | 34 retryable | slow | 19 | 28 cancelled | slow | 1 | 25 executing | federator_incoming | 1 | 23 available | check_domain_resolve | 0 | 20 available | mailer | 0 | 20 retryable | slow | 18 | 17 retryable | background | 13 | 16 discarded | federator_incoming | 5 | 16 executing | background | 3 | 12 executing | background | 2 | 12 executing | federator_incoming | 4 | 11 retryable | background | 4 | 11 executing | federator_incoming | 2 | 10 retryable | background | 6 | 9 retryable | federator_outgoing | 4 | 9 executing | federator_incoming | 3 | 9 retryable | background | 19 | 8 available | background | 19 | 8 retryable | slow | 17 | 8 retryable | background | 12 | 6 retryable | background | 9 | 6 executing | federator_outgoing | 1 | 6 retryable | background | 5 | 6 available | background | 0 | 5 executing | federator_incoming | 5 | 5 available | background | 0 | 5 discarded | slow | 5 | 5 retryable | background | 8 | 5 executing | background | 4 | 4 retryable | background | 11 | 4 retryable | background | 7 | 4 available | background | 18 | 3 retryable | slow | 15 | 3 completed | federator_incoming | 2 | 2 executing | federator_outgoing | 5 | 1 retryable | background | 14 | 1 retryable | background | 15 | 1 retryable | slow | 11 | 1 retryable | slow | 16 | 1 executing | background | 1 | 1 completed | slow | 1 | 1 completed | web_push | 1 | 1 available | background | 17 | 1 available | background | 0 | 1 (71 rows)
   •  (mint@ryona.agency)'s status on Tuesday, 15-Oct-2024 01:36:22 JST 

     in reply to

     • 
     • :blank:
     @i @olivia Maybe it's related to pinned posts; I have patched my pleromer to not fetch them when seeing a new actor.
     † top dog :pedomustdie: likes this.
   • feld (feld@friedcheese.us)'s status on Tuesday, 15-Oct-2024 01:51:12 JST feld

     in reply to

     • 
     • :blank:
     @i @olivia @mint was there a spam campaign of mostly same text, but no links and the existing MRFs don't detect it?
   • :blank: (i@declin.eu)'s status on Tuesday, 15-Oct-2024 01:51:13 JST :blank:

     in reply to

     • 
     • feld
     @feld @olivia @mint the PASTA WITH EXTRA SPAM, like almost all the previous nuisances would have been discarded if 99% matches of the exact same text posts were ignored
   • feld (feld@friedcheese.us)'s status on Tuesday, 15-Oct-2024 01:51:14 JST feld

     in reply to

     • 
     • :blank:
     @i @olivia @mint dedup for what exactly?
   • :blank: (i@declin.eu)'s status on Tuesday, 15-Oct-2024 01:51:15 JST :blank:

     in reply to

     • 
     • feld
     @mint @olivia i wonder if @feld would ever consider a dedup MRF into pleroma via simhash_ex or ex_lsh, since it would also require a cachex table, and those have to be defined ahead of time, unlike whenever we eventually switch to nebulex