Notices by feld (feld@friedcheese.us), page 2
-
feld (feld@friedcheese.us)'s status on Saturday, 26-Oct-2024 23:12:09 JST 
feld
@mint @p after first load it really should be served by nginx or varnish cache tho. Ideally. -
feld (feld@friedcheese.us)'s status on Saturday, 26-Oct-2024 17:16:44 JST
feld
@p
> If it's targeted, that's great, but a meg of stuff, something useful should have been in there.
Not really. The only good stuff in there would be the Ecto stats but they're not granular enough to be useful. Someone sharing the raw Postgres stats from pg_exporter would have been better.
> but "the directory listing is 187MB" is a real problem (even if it's not a pointer-chase, you're still reading 187MB from the disk and you're still copying 187MB of dirents into userspace and `ls` takes 45s), and that gets marked as a dup of a "nice to have" S3 bug, but this is the default Pleroma configuration. It's stuff you can't write off, you know? You hit real constraints.
Where is the "ls" equivalent happening for Pleroma? Fetching a file by name is not slow even when there are millions in the same directory.
> Yeah, BEAM using CPU isn't the bottleneck, though. Like I said in the post you're replying to, it's I/O.
BEAM intentionally holds the CPU in a spinlock when it's done doing work in the hopes that it will get more work. That's what causes the bottleneck. It might not look like high CPU usage percentage-wise, but it's preventing the kernel from context switching to another process.
And what IO? Can someone please send dtrace, systemtap, some useful tracing output showing that BEAM is doing excessive unecessary IO? BEAM should be doing almost zero IO; we don't read and write to files except when people upload attachments. Even if you're not using S3 your media files should be served by your webserver, not Pleroma/Phoenix.
> That is cool, but if you could do that for fetching objects from the DB, you'd have a bigger bump.
patches welcome, but I don't have time to dig into this in the very near future.
> Anyway, I am very familiar with fedi's n*(n-1)/2 problems. (Some time in the future, look for an object proxy patch.)
plz plz send
> But you know, back-pressure, like lowering the number of retries based on the size of the table, that could make a big difference when a system gets stressed.
patches welcome. You can write custom backoff algorithms for Oban. It's supported.
> You could ping graf; it's easier to just ask stressed instances than to come up with a good way to do stress-testing.
Everyone I've asked to get access to their servers which were stuggling has refused except mint. Either everyone's paranoid over nothing or far too many people have illegal shit on their servers. I don't know what to think. It's not exactly a motivator to solve their problems.
> Oh, yeah, so 403s? What counts as permanent?
Depends on what it is. If you get a 403 on an object fetch or profile refresh they're blocking you, so no point in retrying. If it was deleted you get a 404 or a 410, so no point in retrying that either... (when a Delete for an activity you didn't even have came in, it would put in the job to fetch the activity it was referencing... and kept trying to fetch it over and over and over...)
> You think you might end up with a cascade for those? Like, if rendering TWKN requires reading 10MB...
No, I mean it was hanging to fetch latest data from remote server before rendering the activity, which was completely unnecessary. Same with rich media previews -- if it wasn't in cache, the entire activity wouldn't render until it tried to fetch it. Stupid when it could be fetched async and pushed out over websocket like we do now.
> The schedule doesn't bug me. ... the following bug is a big problem, that's a basic thing that was broken in a release. Some kind of release engineer/QA situation could have caught it.
Again, it wasn't broken in a release. There were two different bugs: one was that we used the wrong source of truth for whether or not you were successfully following someone. The other bug became more prominent because more servers started federating Follow requests without any cc field and for some reason our validator was expecting at least an empty cc field when it doesn't even make sense to have one on a Follow request.
You seem to have a lot of opinions and ideas on how to improve things but nobody else on the team seems to give a shit about any of this stuff right now. So send patches. I'll merge them. Join me. -
feld (feld@friedcheese.us)'s status on Saturday, 26-Oct-2024 02:12:49 JST
feld
@p @cvnt @phnt @NonPlayableClown @Owl @dj @ins0mniak @transgrammaractivist Nobody proved there was an *Oban* bottleneck and still haven't.
I'm always running my changes live on my instances. They were massively overpowered. Now I have a severely underpowered server and it's still fine.
If I could reproduce reported issues it would be much easier to solve them but things generally just work for me.
A ton of work has been put into correctness (hundreds of Dialyzer fixes) and tracking down elusive bugs and looking for optimizations like reducing JSON encode/decode work when we don't need to, avoiding excess queries, etc.
I'm halfway done with an entire logging rewrite and telemetry integration which will make it even easier to identify bottlenecks.
It's actually been going really great :shrug: -
feld (feld@friedcheese.us)'s status on Saturday, 26-Oct-2024 02:12:48 JST
feld
@p
> I mean, like I mentioned, the Prometheus endpoints were public at the time.
Problem is that this data is useful for monitoring overall health of an instance but doesn't give enough granular information to track down a lot of issues. With the metrics/telemetry work I have in progress we'll be able to export more granular Pleroma-specific metrics that will help a lot.
> The main bottleneck is the DB
So often it's just badly configured Postgres. If your server has 4 cores and 4 GB of RAM you can't go use pgtune and tell it you want to run Postgres with 4 cores and 4GB. There's nothing leftover for the BEAM. You want at least 500MB-1GB dedicated to BEAM, more if your server has a lot of local users so it can handle memory allocation spikes.
And then what else is running on your OS? That needs resources too. There isn't a good way to predict the right values for everyone. 😭 Like I said, it's running *great* on my little shitty thin client PC with old slow Intel J5005 cores and 4GB RAM. But I have an SSD for the storage and almost nothing else runs on the OS (FreeBSD). I'm counting a total of 65 processes before Pleroma, Postgres, and Nginx are running. Most Linux servers have way more services running by default. That really sucks when trying to make things run well on lower specced hardware.
You also have to remember that BEAM is greedy and will intentionally hold the CPU longer than it needs because it wants to produce soft-realtime performance results. This needs to be tuned down on lower resource servers because BEAM itself will be preventing Postgres from doing productive work. It's just punching itself in the face then. Set these vm.args on any server that isn't massively overpowered:
+sbwt none
+sbwtdcpu none
+sbwtdio none
> using an entire URL for an index is costing a lot in disk I/O
For the new Rich Media cache (link previews stored in the db so they're not constantly refetched) I hashed the URLs for the index for that same reason. Research showed a hash and the chosen index type were super optimal.
Another thing I did was I noticed we were storing *way* too much data in Oban jobs. Like when you federated an activity we were taking the entire activity's JSON and storing it in the jobs. Imagine making a post with 100KB of content that needs to go to 1000 servers? Each delivery job in the table was HUGE. Now it's just the ID of the post and we do the JSON serialization at delivery time. Much better, lower resource usage overall, lower IO.
Even better would be if we could serialize the JSON *once* for all deliveries but it's tricky because we gotta change the addressing for each delivery. Jason library has some features we might be able to leverage for this but it doesn't seem important to chase yet. Even easier might be to put placeholders in the JSON text, store it in memory, and then just use regex or cheaper string replacement to fill those fields at delivery time. Saves all that repeat JSON serialization work.
Other things I've been doing:
- making sure Oban jobs that have an error we should really treat as permanent are caught and don't allow the job to repeat. It's wasteful for us, rude to remote servers when we're fetching things
- finding every possible blocker for rendering activities/timelines and making those things asynchronous. One of the most recent ones I found was with polls. They could stall rendering a page of the timeline if the poll wasn't refreshed in the last 5 mins or whatever. (and also... I'm pretty sure polls were still being refreshed AFTER the poll was closed 🤬)
I want Pleroma to be the most polite Fedi server on the network. There are still some situations where it's far too chatty and sends requests to other servers that could be avoided, so I'm trying to plug them all. Each of these improvements lowers the resource usage on each server. Just gotta keep striving to make Pleroma do *less* work.
I do have my own complaints about the whole Pleroma releases situation. I wish we were cutting releases like ... every couple weeks if not every month. But I don't make that call. -
feld (feld@friedcheese.us)'s status on Tuesday, 22-Oct-2024 03:59:51 JST
feld
@SlicerDicer @tinker as soon as you start getting offshoots on the wheat your crop is pretty much spoiled because you'll have a mix of unripe and ripened wheat berries so now you gotta figure out how to sort them all
or you just feed it all to cattle I guess -
feld (feld@friedcheese.us)'s status on Tuesday, 22-Oct-2024 03:59:37 JST
feld
@tinker why did you post with such bold optimism that this is something you could figure out without any prior knowledge?
The amount of time it takes to go from planting to harvest is LONG which means you only get to learn from your failures maybe a couple times a year, that's it. -
feld (feld@friedcheese.us)'s status on Tuesday, 22-Oct-2024 03:59:36 JST
feld
@tinker if you want real advice on what's possible you should talk to to @SlicerDicer -
feld (feld@friedcheese.us)'s status on Tuesday, 15-Oct-2024 02:19:48 JST
feld
@mint @i @olivia you don't want to raise an exception on a duplicate job in Oban; that would break a lot of stuff needlessly. It just drops the job silently. It's not an error scenario that needs to raise / cause the process to abort. -
feld (feld@friedcheese.us)'s status on Tuesday, 15-Oct-2024 02:12:20 JST
feld
@mint @i @olivia
> 5. pleromer inserts RemoteFetcherWorker jobs for those posts
these inserts were not set to be unique, but they are now -
feld (feld@friedcheese.us)'s status on Tuesday, 15-Oct-2024 02:10:08 JST
feld
@mint @i @olivia that redirects me to an account named beer_bastard. Same? -
feld (feld@friedcheese.us)'s status on Tuesday, 15-Oct-2024 01:59:42 JST
feld
@mint @i @olivia weird, why would it keep fetching them? can you confirm for me the profile so I can take a closer look?
also the dupes shouldn't happen with latest develop branch, at least if it tried it would cancel inserting the job every time because a duplicate one existed (until pruner kicks in and clears up old Oban jobs) -
feld (feld@friedcheese.us)'s status on Tuesday, 15-Oct-2024 01:51:14 JST
feld
@i @olivia @mint dedup for what exactly? -
feld (feld@friedcheese.us)'s status on Tuesday, 15-Oct-2024 01:51:12 JST
feld
@i @olivia @mint was there a spam campaign of mostly same text, but no links and the existing MRFs don't detect it? -
feld (feld@friedcheese.us)'s status on Saturday, 12-Oct-2024 06:07:05 JST
feld
@mint @pete_wright @karinjiri yeah that's one way to do it, but I'd like to be able to farm out that work automagically to connected nodes with BEAM's clustering.
I've got some plans to refactor the caching to be cluster-aware too so as long as you have S3 or an NFS mount for media and a network reachable database you can have a highly-available cluster running Pleroma across various devices, like a handful of RPis or something :) -
feld (feld@friedcheese.us)'s status on Saturday, 12-Oct-2024 06:03:13 JST
feld
@pete_wright @karinjiri I'm currently focused on trying to make Pleroma run better on low resource servers. This has been going well.
I'm currently running on a 4-core 4GB RAM "server" of a Wyse 5070 thin client with an SSD. FreeBSD, but UFS because of lower resources. Just runs Nginx, Postgres, Pleroma.
last pid: 51675; load averages: 0.17, 0.31, 0.38 up 15+01:25:41 20:51:29
19 processes: 1 running, 18 sleeping
CPU: 1.9% user, 0.0% nice, 0.3% system, 0.0% interrupt, 97.8% idle
Mem: 908M Active, 1283M Inact, 571M Laundry, 765M Wired, 378M Buf, 288M Free
I'm connected to some large relays to pull as many posts in as possible and it continues to run great.
cheese_prod=# select count(*) from objects;
-[ RECORD 1 ]--
count | 1431463
So I'm pretty happy with it right now. I only wish it had a bit more power for MediaPreview Proxy so I can have images/video thumbnails enabled. I'm actually investigating refactoring that so it can run as an independent service on another computer. It wouldn't be hard then to simply start up another Pleroma Elixir instance in a limited mode that does nothing but the transcoding/thumbnailing. I could run it on my laptop or another server, and if they go offline it just reverts back to not doing the transcoding/thumbnailing work. -
feld (feld@friedcheese.us)'s status on Saturday, 12-Oct-2024 03:04:20 JST
feld
Mastodon has an In Memoriam feature now, but in Pleroma we're going to make their profile show this for the profile pages of dead people you blocked -
feld (feld@friedcheese.us)'s status on Tuesday, 08-Oct-2024 00:55:43 JST
feld
@matrix @SuperDicq my wisdom teeth were like that -
feld (feld@friedcheese.us)'s status on Tuesday, 01-Oct-2024 04:19:56 JST
feld
@mint @sneeden @prettygood I'd also appreciate if the Mute Domain feature wasn't buried in the settings and was visible on a post or user profile too -
feld (feld@friedcheese.us)'s status on Saturday, 28-Sep-2024 02:34:40 JST
feld
BREAKING: Gargron admits Mastodon uses algorithms
RT: https://mastodon.social/users/Gargron/statuses/113209415544805404 -
feld (feld@friedcheese.us)'s status on Wednesday, 25-Sep-2024 00:16:15 JST
feld
@r000t @p @NEETzsche @lanodan Debian removed support for installing any Python packages system-wide with pip. It needs to be in the Debian package repo or go make a virtual env. Python has a PEP for this and Fedora is going to do it too.
https://peps.python.org/pep-0668/
Why?
Because on Linux so many things including the package manager use Python, and pip can literally break them.
All ０７６萌ＳＮＳ content and data are available under the