Conversation

Notices

1. Terry (terry@bae.st)'s status on Thursday, 04-Jan-2024 20:25:03 JST Terry
   
   video_2024-01-03_22-32-15.mp4
   • pistolero :thispersondoesnotexist: (p@freespeechextremist.com)'s status on Thursday, 04-Jan-2024 20:24:57 JST pistolero :thispersondoesnotexist:

     in reply to

     • Your New Kemono Waifu :verified: :cornbread_the_cat:
     • J上校🎄🎄🇺🇦
     @colonelj @Terry Something weird going on with bae.st, started a few weeks ago but there were some weird incidents before that with them sending double posts.
     
     It's been hitting both the public inbox endpoint and the user-specific inboxes. I don't know why it's doing that, but it's even doing it for users that no longer exist.
     
     I don't think it's that, though, I think it's probably that it takes us longer to do the insert than bae.st's local timeout. (cc @sjw ) Delivering a post is several steps¹, most of which have overlapping timers on both ends, so something can get delivered but might not be viewed as a success on the other end and they try to deliver it again.
     
     There should probably just be a uniqueness constraint on (user_id,type,activity_id) for notifications. That would slow things down (explanation in the footnote), though, and it's already slow.
     
     ¹ Abridged and simplified:
     
     1. Worker receives job to deliver post to FSE. Baest DB connection pool checkout timer starts.
     2. Worker does DNS lookup. This is fast usually but should be locally cached because a lot of them happen and it is holding a DB connection from the pool.
     3. Worker eventually sends the signed request to FSE to deliver the post. Web request timer starts.
     4. FSE receives the post by HTTP. nginx and Pleroma both start incoming request timers.
     5. FSE starts processing the post. The post gets verified, goes through the MRF chain, etc. FSE's DB connection pool checkout timer starts.
     6. FSE completes processing, including inserting the notification into the notifications table, which is huge.
     7. FSE finishes processing the request. FSE's DB connection pool timer stops.
     8. FSE sends the request back out through nginx. Pleroma and nginx request timers stop.
     9. Baest receives the response. Baest's web request timer stops.
     10. Baest worker processes the request, noting that it is a success, and recording that to the DB. Baest DB connection pool checkout timer stops.
     11. The job queue records the job's success and the job stops getting retried.
     
     If any of the timers expires after step 6 is over but before step 11 is over, double deliveries could be attempted and, although the post would not get recorded twice, double notifications might be created. A number of these steps are slower than they might be, like step 6: FSE has received 9,679,872 notifications so far; several purges have occurred, it currently holds 3,385,727 rows, but is frequently and repeatedly accessed by reads/writes and is indexed all to hell (which makes writes to the table more expensive). Like Poast, Baest's outgoing requests go through proxies and some of these proxies (e.g., M247) have been responsible for spam accounts so FSE doesn't treat all IP ranges the same. (There are a lot of weird things going on, because bae.st and FSE are both old servers that have accumulated quirks and scar tissue, plus both servers are kind of, like, outliers in terms of use.)
     
     Typically, the way you solve this kind of thing is a two-phase commit, but ActivityPub doesn't have a provision for that kind of thing. It's just "toss the activity at the other server and let's all hope for the best". Granted, the network is not designed to be bulletproof and it's difficult to design a decentralized protocol that *is* bulletproof, and even if it's designed well, it is more complicated to implement that sort of thing than the naïve version, so a two-phase commit might have seemed like overkill. (Something tells me that The Mastadan Netwark's German dictator was closer to "didn't know what a two-phase commit is" than "prudently decided not to complicate the protocol". In any case, if you want lots of implementations and broad adoption, it's better not to require more complicated implementations than you have to.)
     
     So you end up having to deduplicate on both ends. (Content-addressed storage makes deduplication trivial. :revolvertan:)
     † top dog :pedomustdie: likes this.
   •  (mint@ryona.agency)'s status on Thursday, 04-Jan-2024 20:24:57 JST 

     in reply to

     • Your New Kemono Waifu :verified: :cornbread_the_cat:
     • Yukkuri
     • pistolero :thispersondoesnotexist:
     • J上校🎄🎄🇺🇦
     @p @Terry @sjw @colonelj Worth noting that Pleroma also sends out notifications to user's websocket transitiently, before they even land into DB. I've seen it happen with @iamtakingiteasy's reportbot when there was an upstream bug with random reports not federating.
     Machismo repeated this.
   • J上校🎄🎄🇺🇦 (colonelj@freespeechextremist.com)'s status on Thursday, 04-Jan-2024 20:24:58 JST J上校🎄🎄🇺🇦

     in reply to

     • pistolero :thispersondoesnotexist:
     @Terry i keep getting double pinged @p
   • Terry (terry@bae.st)'s status on Thursday, 04-Jan-2024 20:24:59 JST Terry

     in reply to

     • J上校🎄🎄🇺🇦
     @colonelj so you saying..
     my source is that I made it the fuck up.mp4
   • Terry (terry@bae.st)'s status on Thursday, 04-Jan-2024 20:25:00 JST Terry

     in reply to

     • J上校🎄🎄🇺🇦
     @colonelj who did CNN source?
   • J上校🎄🎄🇺🇦 (colonelj@freespeechextremist.com)'s status on Thursday, 04-Jan-2024 20:25:00 JST J上校🎄🎄🇺🇦

     in reply to
     @Terry no one because sources are fir relying on another person's word
   • J上校🎄🎄🇺🇦 (colonelj@freespeechextremist.com)'s status on Thursday, 04-Jan-2024 20:25:01 JST J上校🎄🎄🇺🇦

     in reply to
     @Terry I'm not gonna pretend politifact sourcing CNN would somehow convince you
   • J上校🎄🎄🇺🇦 (colonelj@freespeechextremist.com)'s status on Thursday, 04-Jan-2024 20:25:02 JST J上校🎄🎄🇺🇦

     in reply to
     @Terry "NEWS ORGS CAN'T USE THEIR OWN NEWS ARTICLES AS SOURCES!" 🤦
   • Terry (terry@bae.st)'s status on Thursday, 04-Jan-2024 20:25:02 JST Terry

     in reply to

     • J上校🎄🎄🇺🇦
     @colonelj its called circular referencing you buffoon.
     
     Seriously why do you choose this opportunity to engage with me in order to defend pedophiles?
   • pistolero :thispersondoesnotexist: (p@freespeechextremist.com)'s status on Thursday, 04-Jan-2024 20:54:44 JST pistolero :thispersondoesnotexist:

     in reply to

     • Your New Kemono Waifu :verified: :cornbread_the_cat:
     • 
     • Yukkuri
     • J上校🎄🎄🇺🇦
     @mint @Terry @colonelj @iamtakingiteasy @sjw
     
     > Worth noting that Pleroma also sends out notifications to user's websocket transitiently,
     
     If I've seen it happen, then it can't be that, because I'm using bloat. Now that I think about it, for all I know, though, there could be an off-by-one error in the paging of notifications: muted threads always create moderate flakiness. I haven't seen this in the ssh client, which ignores thread-muting.
     
     Next time it happens, I will have a look at the DB.
      likes this.
   • pistolero :thispersondoesnotexist: (p@freespeechextremist.com)'s status on Friday, 05-Jan-2024 01:57:34 JST pistolero :thispersondoesnotexist:

     in reply to

     • Your New Kemono Waifu :verified: :cornbread_the_cat:
     • 
     • Yukkuri
     • pistolero :thispersondoesnotexist:
     • J上校🎄🎄🇺🇦
     @mint @Terry @colonelj @iamtakingiteasy @sjw
     
     > Next time it happens, I will have a look at the DB.
     
     Okay, it is two rows in the DB. Activity AdVKpA5RgC3jKDypGa occupied rows 9680118 and 9680119 in the DB, both with my user_id and type "mention". So it's definitely a backend problem. It coincided with a burst of traffic. (The traffic was Applebot and archive.org, and apparently some dipshits are hitting an RSS feed on the blog every 30 seconds so I should probably trim it instead of just putting everything in there.)
      and † top dog :pedomustdie: like this.
   • pistolero :thispersondoesnotexist: (p@freespeechextremist.com)'s status on Saturday, 06-Jan-2024 00:30:51 JST pistolero :thispersondoesnotexist:

     in reply to

     • Your New Kemono Waifu :verified: :cornbread_the_cat:
     • 
     • Yukkuri
     • J上校🎄🎄🇺🇦
     @mint @Terry @colonelj @iamtakingiteasy @sjw Well, I have verified that there is double-delivery going on after dumping some of the traffic. I do not know why. FSE indicates a successful result and then bae.st keeps trying to re-deliver and then it gets 5xx responses starting with the second time.
     
     I wish @sjw would take an interest in this because bae.st today has been responsible for 6.8% of *all* requests hitting FSE, and 63.6% of the 5xx errors, and yesterday it was 10.7% of all requests and 80.3% of the 5xx errors and the server keeps getting flooded because everything on his server is getting re-sent repeatedly. bae.st is responsible for 53.54% of the POST reqs. It is enough traffic to affect the ping times. I am going to start just knocking /16s off the wall until my pipe is no longer saturated because I have to work.
     screenshitter.png
      likes this.
   • Machismo (zerglingman@freespeechextremist.com)'s status on Saturday, 06-Jan-2024 00:31:23 JST Machismo

     in reply to

     • Your New Kemono Waifu :verified: :cornbread_the_cat:
     • 
     • Yukkuri
     • pistolero :thispersondoesnotexist:
     • J上校🎄🎄🇺🇦
     @p @sjw @iamtakingiteasy @colonelj @Terry @mint DESPITE
   • Yukkuri (iamtakingiteasy@eientei.org)'s status on Saturday, 06-Jan-2024 04:17:38 JST Yukkuri

     in reply to

     • Your New Kemono Waifu :verified: :cornbread_the_cat:
     • 
     • pistolero :thispersondoesnotexist:
     • J上校🎄🎄🇺🇦
     @sjw @Terry @colonelj @p @mint Most likely entry isn't removed from oban federator_outgoing queue. Purely software errors might get recorded in errors column, that should be easiest to check first, if not, enabling debug logs, triggering/waiting for delivery attempt and then sifting through many megabytes of output would be required. Postgres logs at the same moment might also be helpful. Could be conflicting lock, transaction getting rolled back due to software error or some database error. Either way, combined output is likely to contain some hints.
     
     Further speculations include runtime cache(s) not getting evicted, lingering oban workers from restarts/reloads without terminating main process. If issue stops reproducing after clean instance restart, could be the case, but locating such issues is ought to be involved and interactive process.
     
     Could also be synchronization issue, if multiple workers somehow are getting the same task, but thats nearing improbability, it's not that difficult to implement row-locking postgres queue correctly to doubt oban implementation before other options are exhausted.
      likes this.
   • Your New Kemono Waifu :verified: :cornbread_the_cat: (sjw@bae.st)'s status on Saturday, 06-Jan-2024 04:17:39 JST Your New Kemono Waifu :verified: :cornbread_the_cat:

     in reply to

     • 
     • Yukkuri
     • pistolero :thispersondoesnotexist:
     • J上校🎄🎄🇺🇦
     @p @Terry @iamtakingiteasy @colonelj @mint any idea why it'd be doing that?
   •  (mint@ryona.agency)'s status on Saturday, 06-Jan-2024 04:44:08 JST 

     in reply to

     • Your New Kemono Waifu :verified: :cornbread_the_cat:
     • Yukkuri
     • pistolero :thispersondoesnotexist:
     • J上校🎄🎄🇺🇦
     @iamtakingiteasy @Terry @sjw @colonelj @p Wonder if it's related to recently introduced priority queue.
     https://git.pleroma.social/pleroma/pleroma/-/merge_requests/4004
     None of that would've been an issue if sjw didn't shit his instance up with hundreds of bots reposting images from Danbooru.
   • pistolero :thispersondoesnotexist: (p@freespeechextremist.com)'s status on Saturday, 06-Jan-2024 21:11:22 JST pistolero :thispersondoesnotexist:

     in reply to

     • Your New Kemono Waifu :verified: :cornbread_the_cat:
     • 
     • Haelwenn /элвэн/ :triskell:
     • Yukkuri
     • J上校🎄🎄🇺🇦
     @mint @iamtakingiteasy @Terry @colonelj @sjw
     
     > Wonder if it's related to recently introduced priority queue.
     
     Sounds likely. I had a look at the diff (https://git.pleroma.social/pleroma/pleroma/-/merge_requests/4004/diffs) and it looks like it just uses the "cc" attribute and doesn't distinguish by type, which matches what I'm seeing: a big flood of "Like" activities. publisher.ex:141 and :217 and there's some discussion around the problem. When I did the dump, it was Likes getting delivered to specific people and to followers addresses. (e.g., Terry smashed that like on a post on some other server, but https://bae.st/users/Terry/followers was in the CC field.)
     
     I was just talking to @lanodan (who did the merge, according to https://git.pleroma.social/pleroma/pleroma/-/merge_requests/4004 ) about some unrelated stuff (periodic "PL complaints" thread :terrysmoking:), but I'm not sure there's enough information even to file an issue. Code seems a little shaky (I have only had an initial look at it, I could be wrong), it does seem like it matches the behavior I'm seeing here. The dedup is trivial string match on line 258, maybe should dedup by host. I'm not sure that I have read the bit that handles shared/private inboxes correctly.
     
     > None of that would've been an issue if sjw didn't shit his instance up with hundreds of bots reposting images from Danbooru.
     
     I think they're responsible for a small amount of the actual traffic, probably just a lot of people following each other on both servers.
      likes this.
   • Your New Kemono Waifu :verified: :cornbread_the_cat: (sjw@bae.st)'s status on Saturday, 06-Jan-2024 22:43:03 JST Your New Kemono Waifu :verified: :cornbread_the_cat:

     in reply to

     • 
     • Haelwenn /элвэн/ :triskell:
     • Yukkuri
     • pistolero :thispersondoesnotexist:
     • J上校🎄🎄🇺🇦
     @p @Terry @iamtakingiteasy @colonelj @lanodan @mint all of my bots have been down for a few days
      likes this.
   • pistolero :thispersondoesnotexist: (p@freespeechextremist.com)'s status on Saturday, 06-Jan-2024 22:59:21 JST pistolero :thispersondoesnotexist:

     in reply to

     • Your New Kemono Waifu :verified: :cornbread_the_cat:
     • 
     • Haelwenn /элвэн/ :triskell:
     • Yukkuri
     • pistolero :thispersondoesnotexist:
     • J上校🎄🎄🇺🇦
     @sjw @Terry @colonelj @iamtakingiteasy @lanodan @mint It should be easy to tell if it fixes it because the vast majority of incoming POSTs (including interactive use) are from bae.st, and another ~15% are from ryona.agency, so I wonder if that patch is on there. 80+15=95, so 19 out of 20 requests are from two instances that I am fairly certain do not, when the software is working correctly, send enough activities to swamp the server.
     everything_is_baest.png
      likes this.
   • pistolero :thispersondoesnotexist: (p@freespeechextremist.com)'s status on Saturday, 06-Jan-2024 22:59:22 JST pistolero :thispersondoesnotexist:

     in reply to

     • Your New Kemono Waifu :verified: :cornbread_the_cat:
     • 
     • Haelwenn /элвэн/ :triskell:
     • Yukkuri
     • J上校🎄🎄🇺🇦
     @sjw @Terry @colonelj @iamtakingiteasy @lanodan @mint Yeah, I don't think it's got anything to do with the bots. Can you try reverting that change and seeing if it still happens?
   •  (mint@ryona.agency)'s status on Saturday, 06-Jan-2024 23:03:32 JST 

     in reply to

     • Your New Kemono Waifu :verified: :cornbread_the_cat:
     • Haelwenn /элвэн/ :triskell:
     • Yukkuri
     • pistolero :thispersondoesnotexist:
     • J上校🎄🎄🇺🇦
     @p @Terry @sjw @iamtakingiteasy @colonelj @lanodan Yeah, it's been merged to upstream and I've synced the sources a while ago since there's a long-standing problem with delayed federation on cum.salon, hence I though partially mitigating it might be good.
   •  (mint@ryona.agency)'s status on Saturday, 06-Jan-2024 23:07:19 JST 

     in reply to

     • Your New Kemono Waifu :verified: :cornbread_the_cat:
     • 
     • Haelwenn /элвэн/ :triskell:
     • Yukkuri
     • pistolero :thispersondoesnotexist:
     • J上校🎄🎄🇺🇦
     @p @Terry @colonelj @iamtakingiteasy @lanodan @sjw Salon is on the same server as ageny and runs the same patch, so I'd guess most of those 15% come from there.