Conversation

Notices

1.  (mint@ryona.agency)'s status on Tuesday, 02-Jul-2024 17:02:23 JST 
   >no federated posts in 4 hours
   The fuck's happening.
   • † top dog :pedomustdie: likes this.
   •  (mint@ryona.agency)'s status on Tuesday, 02-Jul-2024 17:04:40 JST 

     in reply to
     Okay, it's the same shit as before Restarted, and the backlog is now supposedly being processed from the queue table. Definely a bug.
   • † top dog :pedomustdie: (dcc@annihilation.social)'s status on Tuesday, 02-Jul-2024 17:05:04 JST † top dog :pedomustdie:

     in reply to
     @mint Dam
   •  (mint@ryona.agency)'s status on Tuesday, 02-Jul-2024 17:21:14 JST 

     in reply to

     • feld
     @feld Don't know if it's because of what you did to oban, but lately (after updating from 03024318 to 9953b0da) some shit's been happening when the instance essentially gets constipated and stops properly receiving remote activities (and likely also delivering local ones) until you restart it, then immediately after that they begin to come through from the queue somewhere in DB. Any idea what could be happening?
   •  (mint@ryona.agency)'s status on Tuesday, 02-Jul-2024 17:37:10 JST 

     in reply to
     Truncated oban_jobs just in case. 41591 rows seems somewhat excessive, so we'll see how fast it grows.
   • † top dog :pedomustdie: (dcc@annihilation.social)'s status on Tuesday, 02-Jul-2024 18:47:07 JST † top dog :pedomustdie:

     in reply to

     • feld
     • 
     • pistolero
     @mint @feld @mint @p Its because of a few posts in the federation tab (you need to delete some of them) to get it to restart (i have had this issuse two time)
      likes this.
   •  (mint@ryona.agency)'s status on Tuesday, 02-Jul-2024 18:56:43 JST 

     in reply to

     • feld
     • † top dog :pedomustdie:
     • 
     • pistolero
     @dcc @feld @p @mint Not sure what tab you're talking about.
   • † top dog :pedomustdie: (dcc@annihilation.social)'s status on Tuesday, 02-Jul-2024 18:56:47 JST † top dog :pedomustdie:

     in reply to

     • feld
     • 
     • pistolero
     @mint @feld @p @mint twkn
   •  (mint@ryona.agency)'s status on Tuesday, 02-Jul-2024 19:00:15 JST 

     in reply to

     • feld
     • † top dog :pedomustdie:
     • 
     • pistolero
     @dcc @feld @p @mint I don't see how this is related. If there was some post preventing the rest of the posts from coming through, I imagine it'll be in the queue itself instead of the table with already processed posts.
   •  (mint@ryona.agency)'s status on Tuesday, 02-Jul-2024 19:10:13 JST 

     in reply to

     • jesu
     @jesu As lain said, posts from your subscriptions are pushed by remote instances to yours, not fetched from them. Local accounts take as much resources as remote ones, that being a single row in users table, so if an account lays dormant, it doesn't take up any resources aside from whatever his subscriptions push in.
   • jesu (jesu@pl.kotobank.ch)'s status on Tuesday, 02-Jul-2024 19:10:15 JST jesu

     in reply to
     @mint thoughts on this mint? https://pl.kotobank.ch/notice/AjVtK0w4WGQsavFVh2
   •  (mint@ryona.agency)'s status on Tuesday, 02-Jul-2024 19:24:37 JST 

     in reply to

     • jesu
     @jesu What you've been proposing might be doable, possibly even with an MRF. Activities that are shared to followers have actor's /followers address either in `to` (unlisted, lockposts) or `cc` (public). When receiving such an activity, one could check the list of local followers this actor has, then check each follower's last active time and if none were recently active, reject it. The problem? This would break thread contexts, since when fetching a post someone replied to, Pleroma would generate the Create activity with the same structure, and then promptly reject it since it fits the same criteria. Essentially turning your instance to a whitelist-federated one with extra steps.
   • jesu (jesu@pl.kotobank.ch)'s status on Tuesday, 02-Jul-2024 19:24:38 JST jesu

     in reply to
     @mint > aside from
     
     those are the ones I'm referring to and seeing oban clog up for many people gave me this inclination. thank you for your reply
   •  (mint@ryona.agency)'s status on Tuesday, 02-Jul-2024 19:25:19 JST 

     in reply to

     • jesu
     @jesu I'm just a tinkerer.
   • jesu (jesu@pl.kotobank.ch)'s status on Tuesday, 02-Jul-2024 19:25:20 JST jesu

     in reply to

     • jesu
     @mint I don't know why you aren't on pleroma dev team with how much you know but something tells me it has to do with time and/or money and/or don't wanna deal with that
   • jesu (jesu@pl.kotobank.ch)'s status on Tuesday, 02-Jul-2024 19:25:21 JST jesu

     in reply to
     @mint I guess I'm saying "they still hit the federation queue and sit in the db and use ram/cpu, etc"
   •  (mint@ryona.agency)'s status on Tuesday, 02-Jul-2024 19:33:44 JST 

     in reply to

     • jesu
     @jesu This also seems to be the case where non-shared user-specific inboxes would've been something more than gnusoc-era rudiment (since you could filter by recipient more easily), but all modern fedi software I've seen sends stuff to the shared inbox exclusively aside from a few edge cases like DMs (and even then it's inconsistent, Pleroma sends a first DM to user inbox but if you reply to your own DM, it gets sent to a shared one, not like it matters since scopes/recipients have always been defined in an activity itself).
   •  (mint@ryona.agency)'s status on Tuesday, 02-Jul-2024 20:34:49 JST 

     in reply to

     • jesu
     @jesu >how to stop those inactive accounts from, not only receiving the activities, but also how to stop the sending servers from sending them
     Nothing much can be done aside from deleting accounts then, which should send out unfollow activities to each followed user, but that often doesn't work as intended (as stated below). Making some mix task that would mass-unfollow on demand is entirely possible, though.
   • jesu (jesu@pl.kotobank.ch)'s status on Tuesday, 02-Jul-2024 20:34:51 JST jesu

     in reply to
     @mint hmm.. I feel like we are both misunderstanding each other.. I'm not 100% sure about both of us but I'm 100% sure I'm misunderstanding... I also have a tendecy to practically speak in riddles because so much ends up assumed that I don't speak clearly
     
     I feel like I'm misspeaking because "When receiving such an activity" is already not what I mean. I didn't realize activities are sent not received. I will try my best to explain, sorry to repeat myself.
     
     If your server had 1000 inactive accounts, and those accounts had an average of 50 accounts per server, that's 50k accounts that are sending activities to your server, right? So, my idea was kind of a question: how to stop those inactive accounts from, not only receiving the activities, but also how to stop the sending servers from sending them. It's simply wasted resources on both ends.
     
     You propose mrf but that would probably ddos the sending server since the queue for the sending server would build up and those activities would keep being sent and never received (I suppose the receiving server could lie?)
   • jesu (jesu@pl.kotobank.ch)'s status on Tuesday, 02-Jul-2024 20:35:37 JST jesu

     in reply to

     • :blank:
     • Phantasm
     @i @phnt @mint hmm... I notice deactivating an account preserves follows but I'm unsure what happens with the sending servers in this scenario...
   • :blank: (i@declin.eu)'s status on Tuesday, 02-Jul-2024 20:35:38 JST :blank:

     in reply to

     • Phantasm
     • jesu
     @jesu @phnt @mint only if you send a block/undo activity from that user to all the follows, and then refollow everyone once they log in
   • jesu (jesu@pl.kotobank.ch)'s status on Tuesday, 02-Jul-2024 20:35:40 JST jesu

     in reply to

     • Phantasm
     • jesu
     @mint @phnt second question: "could I tell those sending servers which users are inactive so they can stop sending?"
   • jesu (jesu@pl.kotobank.ch)'s status on Tuesday, 02-Jul-2024 20:35:46 JST jesu

     in reply to

     • Phantasm
     • jesu
     @phnt @mint In short "is there any way to stop servers from sending activities to inactive accounts?"
   • jesu (jesu@pl.kotobank.ch)'s status on Tuesday, 02-Jul-2024 20:35:47 JST jesu

     in reply to

     • Phantasm
     @phnt @mint On pleroma follows and followers are retained for a deactivated account, which is good I think
   • Phantasm (phnt@fluffytail.org)'s status on Tuesday, 02-Jul-2024 20:35:48 JST Phantasm

     in reply to

     • Phantasm
     • jesu
     @jesu @mint To clarify. On your local Pleroma instance, it would seem that I no longer follow you, but you still follow me (the remote deactivated account).
   • Phantasm (phnt@fluffytail.org)'s status on Tuesday, 02-Jul-2024 20:35:49 JST Phantasm

     in reply to

     • jesu
     @jesu @mint No, Pleroma does not send any deactivation activities (it's local only). Misskey unfollows everyone before deactivating the account, but the followers still remain.
     
     Imagine we have two separate instances and we both follow each other. You are on Pleroma and I use Misskey. If I delete my own account, Misskey would unfollow you from my perspective, but your follow would still remain even though the account behind it is deactivated even on your local instance.
     
     Mastodon might do something similar with follows, but nobody that follows me from Mastodon deleted their account yet. The same remote account deactivation still happens though. Both Mastodon and Misskey also send Delete objects for all posts of that account before deactivating it.
   • Phantasm (phnt@fluffytail.org)'s status on Tuesday, 02-Jul-2024 20:35:50 JST Phantasm

     in reply to

     • jesu
     @jesu @mint Both Misskey and Mastodon send account deactivation activities to remote instances. Misskey and maybe Mastodon also unfollow every user the deactivated account followed.
   • jesu (jesu@pl.kotobank.ch)'s status on Tuesday, 02-Jul-2024 20:35:50 JST jesu

     in reply to

     • Phantasm
     @phnt @mint > Misskey and maybe Mastodon also unfollow every user the deactivated account followed.
     
     So if I deactivated this account other people would be force-unfollowed on misskey and masto?
   • jesu (jesu@pl.kotobank.ch)'s status on Tuesday, 02-Jul-2024 20:35:51 JST jesu

     in reply to

     • jesu
     @mint my idea was "tell other servers that the account is deactivated" but I have no idea how masto or misskey respond to that
   • jesu (jesu@pl.kotobank.ch)'s status on Tuesday, 02-Jul-2024 20:36:52 JST jesu

     in reply to

     • :blank:
     • Phantasm
     • jesu
     @i @mint @phnt that's why I'm thinking "can I just lie and say this inactive account that has been inactive is deactivated when it's not? would *actually* deactivating it work?
   •  (mint@ryona.agency)'s status on Tuesday, 02-Jul-2024 20:40:42 JST 

     in reply to

     • jesu
     @jesu It already wastes its own resources by mass spamming delete activities for each post of deleted user to the entire network it's aware of. But yeah, like everything else in AP this isn't enforceable and is likely a request on the level of restricting who can reply to the post.
   • jesu (jesu@pl.kotobank.ch)'s status on Tuesday, 02-Jul-2024 20:40:43 JST jesu

     in reply to
     @mint well, now I know this is a pipe dream, because mastodon will never do anything good and this heavily relies on cooperation.
     
     It'd be very very very nice to say "hey, this user has been inactive for 3 months, stop sending statuses to them." I can't imagine how many resources are wasted on this with mastodon.social alone
   •  (mint@ryona.agency)'s status on Tuesday, 02-Jul-2024 20:43:43 JST 

     in reply to

     • Phantasm
     • jesu
     @jesu @phnt The queue issue cropped up only recently after the update and I haven't experienced anything like that in all two years of hosting the instance, so I'm fairly confident it's a bug. Most of my inbound federation comes from relays anyway.
   • jesu (jesu@pl.kotobank.ch)'s status on Tuesday, 02-Jul-2024 20:43:44 JST jesu

     in reply to

     • Phantasm
     @phnt @mint I suppose my next question would be "would this be beneficial?" and mint and lain thought not, but I keep seeing queues getting bogged up and I'm curious what it could be
   • Phantasm (phnt@fluffytail.org)'s status on Tuesday, 02-Jul-2024 20:43:45 JST Phantasm

     in reply to

     • jesu
     @jesu @mint None that I can think of currently in Pleroma. It probably ignores some of them though (DMs and maybe some others).
   •  (mint@ryona.agency)'s status on Tuesday, 02-Jul-2024 20:48:59 JST 

     in reply to

     • Phantasm
     • jesu
     @phnt @jesu Not sure, it happened once on salon as well, where I did no maintenance. And there it offected outbound federation as well, pernia's posts were coming hours late after the restart.
   • Phantasm (phnt@fluffytail.org)'s status on Tuesday, 02-Jul-2024 20:49:00 JST Phantasm

     in reply to

     • jesu
     @mint @jesu Also if it's the same thing that occurred during the DB pruning, then only incoming federation was affected. I saw your posts almost in real time it was happening on Ryona Agency.
   • feld (feld@bikeshed.party)'s status on Tuesday, 02-Jul-2024 22:31:29 JST feld

     in reply to
     @mint looking through the logs:
     
     git log 03024318..9953b0da
     
     
     not seeing anything that seems relevant right now 😭
      likes this.
   • :blank: (i@declin.eu)'s status on Tuesday, 02-Jul-2024 22:43:45 JST :blank:

     in reply to

     • feld
     @mint @feld see if Oban.Plugins.Lifeline doesn't help
      likes this.
   •  (mint@ryona.agency)'s status on Tuesday, 02-Jul-2024 22:45:27 JST 

     in reply to

     • feld
     @feld I've truncated the table just in case after experiencing it. It was around 50k when I woke up and saw no new posts in TWKN, and 41k after I restarted pleromer and the federation queue caught up.
     Screenshot_20240702_164321.png
   • feld (feld@bikeshed.party)'s status on Tuesday, 02-Jul-2024 22:45:28 JST feld

     in reply to
     @mint
     
     SELECT worker, count(worker) FROM oban_jobs GROUP BY worker;
     
     let's see how many jobs you have, period.
     
     SELECT worker, count(worker) FROM oban_jobs WHERE state = 'executing' GROUP BY worker;
     
     and how many are executing
   •  (mint@ryona.agency)'s status on Tuesday, 02-Jul-2024 22:46:47 JST 

     in reply to

     • feld
     @feld Something similar, however, happened on salon and I did not flush its queue after restart. There's a shitload of user refresh workers.
     Screenshot_20240702_164551.png
   • feld (feld@bikeshed.party)'s status on Tuesday, 02-Jul-2024 22:51:15 JST feld

     in reply to

     • :blank:
     @i @mint I think it could be reasonably common for there to be stale "executing" jobs in the table that are left indefinitely and should be recycled due to them remaining from a crash/failure/unclean shutdown. I check mine occasionally but so far I haven't found any.
      likes this.
   • feld (feld@bikeshed.party)'s status on Tuesday, 02-Jul-2024 22:51:41 JST feld

     in reply to
     @mint aha! Delete those
     
     DELETE FROM oban_jobs WHERE worker = 'Pleroma.Workers.UserRefreshWorker';
     
     I guess they pulled develop branch between my change and the fix for that 😭 I set those jobs to be unique and the unique setting wasn't being honored
      likes this.
   •  (mint@ryona.agency)'s status on Tuesday, 02-Jul-2024 22:53:01 JST 

     in reply to

     • feld
     @feld Yup, checked newer commits.
     https://git.pleroma.social/pleroma/pleroma/-/merge_requests/4164
     I'll merge the changes a little later and see if anything similar happens again.
   •  (mint@ryona.agency)'s status on Monday, 08-Jul-2024 23:30:33 JST 

     in reply to

     • feld
     @feld A-and it happened again, on 9e363350 without user refresh workers.
     Screenshot_20240708_172704.png
     † top dog :pedomustdie: likes this.
   • † top dog :pedomustdie: (dcc@annihilation.social)'s status on Monday, 08-Jul-2024 23:31:22 JST † top dog :pedomustdie:

     in reply to

     • feld
     @mint @feld Man do i have shit to do when i get home :alex_lol:
      likes this.
   • feld (feld@bikeshed.party)'s status on Monday, 08-Jul-2024 23:42:46 JST feld

     in reply to
     @mint you mean you're seeing high binary memory usage?
   •  (mint@ryona.agency)'s status on Monday, 08-Jul-2024 23:42:46 JST 

     in reply to

     • feld
     @feld Don't think it's related to memory usage, htop shows only ~2 gigs used out of 24.
   • feld (feld@bikeshed.party)'s status on Tuesday, 09-Jul-2024 00:19:13 JST feld

     in reply to
     @mint I'm misunderstanding the problem; what's happening with the server?
     
     edit: ReceiverWorker high job count meaning you're seeing federation halt?
      likes this.
   • feld (feld@bikeshed.party)'s status on Tuesday, 09-Jul-2024 00:19:48 JST feld

     in reply to

     • feld
     @mint would be nice to see what the errors are with something like...
     
     
     SELECT errors FROM oban_jobs
     WHERE worker = 'Pleroma.Workers.ReceiverWorker'
     AND state = 'executing'
     AND attempt > 1;
      likes this.
   •  (mint@ryona.agency)'s status on Tuesday, 09-Jul-2024 00:21:12 JST 

     in reply to

     • feld
     @feld I think so, yes. Until you restart Pleroma, then the workers proceed to get processed from where they left off prior to halt.
   •  (mint@ryona.agency)'s status on Tuesday, 09-Jul-2024 00:22:12 JST 

     in reply to

     • feld
     @feld Nothing but a couple timeouts at different points in time.
     Screenshot_20240708_182000.png
   • feld (feld@bikeshed.party)'s status on Tuesday, 09-Jul-2024 00:46:49 JST feld

     in reply to
     @mint I'd look through the args column of these that are mysteriously timing out, grab an activity/object URL, and manually try to run:
     
     Pleroma.Object.Fetcher.fetch_object_from_id("URL_HERE")
      likes this.
   •  (mint@ryona.agency)'s status on Tuesday, 09-Jul-2024 01:32:05 JST 

     in reply to

     • feld
     @feld All from regular servers (FSE, DRC, varis, etc.), fetching a couple either timed out due to remote instance being down or brought the already fetched object.
   •  (mint@ryona.agency)'s status on Tuesday, 09-Jul-2024 01:59:29 JST 

     in reply to

     • feld
     @feld I run my own fork with a few minor changes (MRFs, DMs about registration requests, stuff like that), but none of them touch Oban or HTTP adapters except for a workaround for Hackney not being able to follow redirects when using HTTP proxy.
   • feld (feld@bikeshed.party)'s status on Tuesday, 09-Jul-2024 01:59:30 JST feld

     in reply to
     @mint you're running latest-ish develop branch and using the default Tesla HTTP client (Hackney) I presume?
   • feld (feld@bikeshed.party)'s status on Tuesday, 09-Jul-2024 02:09:28 JST feld

     in reply to
     @mint
     
     > Hackney not being able to follow redirects when using HTTP proxy.
     
     i know that's unrelated, but can you send me a diff of that ?
     
     also very curious if this problem exists (the main issue, not redirects) when you use Gun instead of Hackney
      likes this.
   •  (mint@ryona.agency)'s status on Tuesday, 09-Jul-2024 02:12:19 JST 

     in reply to

     • feld
     @feld I've tried Gun at one point, and HTTP proxy is extremely unstable with it, failing to connect to random instances at all.
     Here's related commits for Hackney redirects:
     https://gitgud.io/ryonagency/pleroma/-/commit/08c8abc053d2b1b36f4cc21402a8df254a6c7795
     https://gitgud.io/ryonagency/pleroma/-/commit/ed31bad2c481633520cc593912586b403c1ac6d4
     https://gitgud.io/ryonagency/pleroma/-/commit/8e8b3be31345ce18150b38ebbe84a3fc23f13e0e
   •  (mint@ryona.agency)'s status on Friday, 12-Jul-2024 09:13:06 JST 

     in reply to

     • feld
     @feld Got notified by a friend that it happened to his instance as well (though I have SSH access to it and run the same fork with the same proxy setup there), checked the issues and someone else with supposedly vanilla setup reported the issue, so it isn't just me.
     https://git.pleroma.social/pleroma/pleroma/-/issues/3286
     I think HTTP adapters are unrelated, since the incoming activities get stuck in queue before ever going through the pipeline that would fetch context, apply MRFs, etc. Plus forcefetching works fine in this condition.
   •  (mint@ryona.agency)'s status on Friday, 12-Jul-2024 16:59:29 JST 

     in reply to

     • feld
     @feld I have NP other explanation for what is happening, yes.
   • feld (feld@bikeshed.party)'s status on Friday, 12-Jul-2024 16:59:30 JST feld

     in reply to
     @mint In that issue you said it's the ReceiverWorker queue getting stuck and just piling up with jobs, right?
   • feld (feld@bikeshed.party)'s status on Friday, 12-Jul-2024 23:30:08 JST feld

     in reply to
     @mint the other queues were reasonably sized ?
   •  (mint@ryona.agency)'s status on Friday, 12-Jul-2024 23:30:08 JST 

     in reply to

     • feld
     @feld Don't know how to check that, but I think yes.
   • feld (feld@bikeshed.party)'s status on Saturday, 13-Jul-2024 03:36:14 JST feld

     in reply to
     @mint I am actively investigating this, trying to find any possible reason this is happening.
     
     My best guess so far is orphaned jobs making Oban think it can't run more jobs because they're dead / stuck in "executing" state.
     
     This should really never happen because Oban itself doesn't crash, but I guess if you restarted Pleroma and it didn't clean itself up gracefully this could happen.
     
     Any chance some of these are Docker deployments or the service could have crashed and restarted automatically due to low resources (OOM, etc)?
      likes this.
   •  (mint@ryona.agency)'s status on Saturday, 13-Jul-2024 03:39:00 JST 

     in reply to

     • feld
     @feld >Any chance some of these are Docker deployments or the service could have crashed and restarted automatically due to low resources (OOM, etc)?
     No, two instances are in LXC container with postgres in separate container being connected to via virtual bridge; I haven't set up any quotas. Third one is running without any containers on a cheapish VPS.
     † top dog :pedomustdie: likes this.
   • feld (feld@bikeshed.party)'s status on Tuesday, 16-Jul-2024 02:26:31 JST feld

     in reply to
     @mint I found it's possible for background queue to get stuck because of super long timeout (15 mins) and some other jobs which were missing timeouts (defaults to infinity), so I've fixed these issues. Some other tweaks in here too.
     
     These changes do not have anything directly to do with the ReceiverWorker, but it may be possible that Oban is not scheduling those jobs because of existing running jobs being stuck. This is unclear to me and doesn't feel like it should work that way in the BEAM, so it could be an Oban-specific behavior with how it is choosing to execute available work.
     
     Investigation is still ongoing until I am certain nothing else could be causing this.
     
     https://git.pleroma.social/pleroma/pleroma/-/merge_requests/4176
      likes this.
   •  (mint@ryona.agency)'s status on Tuesday, 16-Jul-2024 02:27:27 JST 

     in reply to

     • feld
     @feld Much obliged, I'll update a little later and see if it happens again.
   • feld (feld@bikeshed.party)'s status on Wednesday, 17-Jul-2024 00:20:50 JST feld

     in reply to
     @mint if it's interesting to you, I forked the Oban Lifeline plugin and made a new one called Lazarus which is configured to revive a stuck/dead/orphaned job even if it was at its last attempt due to max_attempts: 1
     
     if the job has failed multiple times though, it lets it go
     
     the original Lifeline plugin would throw away a bunch of our jobs just because they're max_attempts: 1 and it's not super necessary to have them tried again, just dumb that if they failed hard for a totally unhandled reason it wouldn't try again
     
     https://git.pleroma.social/pleroma/elixir-libraries/oban_plugins_lazarus
      likes this.
   •  (mint@ryona.agency)'s status on Wednesday, 17-Jul-2024 08:52:58 JST 

     in reply to

     • feld
     @feld Apparently it happened again on boymoder.biz running 683c4f08. Nothing on instances hosted by myself yet.
   •  (mint@ryona.agency)'s status on Wednesday, 17-Jul-2024 09:01:37 JST 

     in reply to

     • feld
     @feld Yeah, it's subbed to 63 Pleroma's internal relays. It also has this thingy enabled that fetches announced notes separately while rejecting the announce activity itself to reduct DB bloat (like more than 30% of activities on agency were those announces when I decided to write it).
     https://gitgud.io/ryonagency/pleroma/-/blob/ryona-dev/lib/pleroma/web/activity_pub/mrf/relay_optimization.ex
     I don't think it should have any effect on federation since it basically just skips transmogrifier and other steps which eventually still call the same fetch_object_from_id.
   • feld (feld@bikeshed.party)'s status on Wednesday, 17-Jul-2024 09:01:38 JST feld

     in reply to
     @mint relays being used too?
   •  (mint@ryona.agency)'s status on Wednesday, 17-Jul-2024 22:40:58 JST 

     in reply to

     • feld
     • :blank:
     @feld Rewrote it to use RemoteFetcherWorker in case calling fetch_object_from_id directly happens to be a blocking operation. Background tasks (including RemoteFetcherWorker) have lower priority than direct federation, so maybe this'll improve performance for non-side channel federation (e.g. direct user-to-user subscriptions).
     https://gitgud.io/ryonagency/pleroma/-/commit/4fa4b233c133e5a220a29d9dfd2d5346e9f74209
     Also forgot to mention that the task of fetching announces/likes/reacts is located earlier in the pipeline than MRF filtering, which makes it impossible to, say, reject likes to a post of some usual suspect without getting that post to federate. I patched that before realizing this behavior applies to relay MRF as well.
     https://ryona.agency/notice/AiQRnbdzTphcHc3f96
     https://gitgud.io/ryonagency/pleroma/-/commit/a78c8aeb0bee0287c368a7b145de79a23cb6e351
     cc @i
     
     Either way, now that the draft for 2.7.0 is in works, we'll soon see if the frozen federation issue happens on larger scale than just me and the other guy.
     † top dog :pedomustdie: likes this.
   • :blank: (i@declin.eu)'s status on Wednesday, 17-Jul-2024 22:49:24 JST :blank:

     in reply to

     • feld
     @mint @feld nice, mine was already piping everything through RemoteFetcherWorker, hopefully that's the rest of the oban issues
      likes this.
   • feld (feld@bikeshed.party)'s status on Wednesday, 17-Jul-2024 23:01:38 JST feld

     in reply to

     • :blank:
     @i @mint I'm running out of ideas, but I did merge in the Oban Live Dashboard so you can go to /phoenix/live_dashboard on your instance as an admin and watch the Oban jobs live (just change page refresh time to like 2 seconds)
     
     at least this way you should be able to spot jobs stuck executing, click on them, and see exactly what it was working on
     
     https://git.pleroma.social/pleroma/pleroma/-/merge_requests/4180
      likes this.
   • :blank: (i@declin.eu)'s status on Wednesday, 17-Jul-2024 23:05:47 JST :blank:

     in reply to

     • feld
     @feld @mint that's what i did in march after getting tired of pulling up psql, really handy even if it's a bit lackluster in that you can only order the table instead of filtering on the state/queue/worker you'd want to focus on
      likes this.
   •  (mint@ryona.agency)'s status on Wednesday, 17-Jul-2024 23:11:23 JST 

     in reply to

     • feld
     • :blank:
     @feld @i Looks like receiver workers keep being retried upon MRF reject. I thought it was patched already, but apparently it was about remote fetches instead.
     https://git.pleroma.social/pleroma/pleroma/-/merge_requests/4174
     Screenshot_20240717_170754.png
   •  (mint@ryona.agency)'s status on Wednesday, 17-Jul-2024 23:20:52 JST 

     in reply to

     • feld
     • :blank:
     @feld @i I don't see a reason why we shouldn't discard them immediately.
     https://git.pleroma.social/pleroma/pleroma/-/merge_requests/4181
     † top dog :pedomustdie: likes this.
   • feld (feld@bikeshed.party)'s status on Wednesday, 17-Jul-2024 23:22:20 JST feld

     in reply to

     • :blank:
     @mint @i I literally left myself a note about that last night :)
      likes this.
   • feld (feld@bikeshed.party)'s status on Wednesday, 17-Jul-2024 23:24:46 JST feld

     in reply to

     • :blank:
     @mint @i oooh actually :cancel is the right thing, not :discard
     
     These used to be :error so they retry, not :discard. I fixed that a few days ago.
     
     https://git.pleroma.social/pleroma/pleroma/-/merge_requests/4175
     
     :discard is meant for internal use and is being removed from Oban in 3.0
      likes this.
   • feld (feld@bikeshed.party)'s status on Wednesday, 17-Jul-2024 23:26:32 JST feld

     in reply to

     • feld
     • :blank:
     @mint @i I left myself that note because I swear I saw some returning {:reject, _} without being wrapped by {:error, _} and wanted to investigate further
     
     still waking up over here, need more #cofe
      likes this.
   •  (mint@ryona.agency)'s status on Wednesday, 17-Jul-2024 23:35:18 JST 

     in reply to

     • feld
     • :blank:
     @feld @i The docs mention that cancelled job should be prevented from running.
     https://hexdocs.pm/oban/Oban.html#cancel_job/2
     So why doesn't it cancel the job?
   •  (mint@ryona.agency)'s status on Wednesday, 17-Jul-2024 23:36:52 JST 

     in reply to

     • feld
     • :blank:
     @feld @i Ah, I see, it's a nested :error that's being unhandled by the worker.
     Screenshot_20240717_173528.png
   • :blank: (i@declin.eu)'s status on Wednesday, 17-Jul-2024 23:39:01 JST :blank:

     in reply to

     • feld
     @mint @feld yeah big pain of values as errors in the land of untyped pattern matching, too many things accidentally shaped wrong in the spaghetti of case fallthroughts
      likes this.
   • feld (feld@bikeshed.party)'s status on Wednesday, 17-Jul-2024 23:39:22 JST feld

     in reply to

     • :blank:
     @mint @i that's the docs for running a function to cancel an specific job, not return :cancel as the result of the job
     
     Check this here
     
     https://hexdocs.pm/oban/Oban.Worker.html#t:result/0
      likes this.
   • feld (feld@bikeshed.party)'s status on Wednesday, 17-Jul-2024 23:40:44 JST feld

     in reply to

     • :blank:
     @mint @i ahh yes, this annoying thing. It can be solved with more @specs being used as this will get noticed by dialyzer
      likes this.
   •  (mint@ryona.agency)'s status on Wednesday, 17-Jul-2024 23:52:10 JST 

     in reply to

     • feld
     • :blank:
     @feld @i I just added a line to handle those nested rejects instead.
     https://gitgud.io/ryonagency/pleroma/-/commit/9ecc46730873ff2168e17da4b3744fe7aa18b15e
     Still, I feel it's caused by my transmogrifier hack (activities that aren't handled by it, like deletes, don't seem to end up as cancelled despite having delete rejections on), so there's likely no need to upstream it.
     Screenshot_20240717_174622.png
   • feld (feld@bikeshed.party)'s status on Thursday, 18-Jul-2024 02:11:16 JST feld

     in reply to

     • :blank:
     @mint @i I wanted to make better decisions on the Rich Media jobs so we don't retry jobs that will just fail again as I identified some deficiencies there, so I've got some refactoring of the helper/parser/backfill functions to help with this. About to start testing it live.
     
     I noticed a couple harmless bugs in the process, but making computers do less work is important :)
     
     https://git.pleroma.social/pleroma/pleroma/-/merge_requests/4182
      likes this.