You can see the periodic pauses in the Apotheosis Stream, those are caused by flushing the IDIOT (which should possibly be done more frequently, as at present it actually makes a couple of the child processes time out when it does this).
Visible in the Prometheus stats is the count of blocks and the count of activities, from which we can get the number of blocks required per activity. Since it's content-addressed, the number of blocks per activity gradually goes down. (It'll go down faster when I finish up that stupid JSON thing I was threatening to do.)
Since screamshitter.club handles FSE's media uploads, its average block size stays near the maximum (8192 bytes): videos/images/mp3s are already compressed so they don't usually get deflated, and because the files are bigger than activities in JSON, you've got, like, a thousand full blocks and one at the end, dragging the average towards the maximum. When more posts arrive there than uploads on FSE, then it pulls the average down. storage_accounting.png ingestion_process.png block_accounting.png
> EST: 8.1 days from now, 0.9705% done (1563286 of 161084319)
You can see why I was a little frustrated with the import speed. FSE's big but it's not so big that it should take a week to convert. Dropping all of the tables except users/activities/objects and then all of the indexes except object.data->>'id' (that is, removing everything the import script doesn't need) shaved off almost half the database's size, though. (~500GB to 284GB.)
Anyway, I've gotta restart it in a little while, but this time it'll be resuming from where it left off rather than starting from zero again.
If you were watching and saw that most recent restart, that wasn't Gargron's fault, that was because I'm a fucking idiot and just did ^C in the wrong window followed by doing ^P^J in a panic.
*Actually*, I just spotted a bug. (There are a lot of comments in the code cursing Mastodon.) So, it's restarting.
I spotted the bug thanks to the stream, actually: I was trying to get it to show a reasonable-ish timeline on the top (which it now does) and a bunch of Japanese posts were blank. The import bit only looks for stuff in "contentMap" if "content" is empty and apparently, at some point, Mastodon put "<p></p>" in "content" while populating "contentMap". Why put an empty paragraph in? Who the fuck knows? Fuck Mastodon. mastodon-for-no-good-reason.png
Are you worried the world won't be ready? What if Revolver is the Great Filter? The thing that throws our species into chaos and ruin? Like a bunch of cavemen being presented with alien technology far too advanced and powerful for them to ever hope to wield responsibly.