@feld@j@mint Yeah, but last I checked (which has admittedly been a minute), there's no "Just stream this FD to this other FD until you get EOF on the first one" call. You read, then write, you've gotta do it yourself.
@feld@j@mint Yeah, that's why it's got to be lower-level; you don't want to put something semantically ugly in the higher level but you end up with a read giving you an immutable linked list of bytes instead of repeatedly refilling the same preallocated, aligned, contiguous region of memory. To do it without burning CPU, you can't allocate anything: it has to be loop{syscall; conditional break; syscall; conditional break;}. So the buffer can't cross into the user-level of the runtime unless you have an exact GC and you somehow have an internal allocator that can recognize that you're doing that. The former exists and I've never seen the latter, but you can tell it to copy one FD to the other without breaking the language and you can do that with a static buffer at a lower level where the semantics aren't expected to hold.
> the reason i thought we could do it with ipfs was that it already was being used in i2p, but its only being used as a proof of concept at this stage.
I hesitate to say too much about numbers but I will say that I think it would be hard to make something as slow as IPFS. You'd have to try really hard. kpos.pdf
> The problem with this method is I lose the source of the packet, it all shows as 192.168.5.1 (wireguard address for proxy).
Yeah, you wanna use iptables if you want it to look like a regular connection. You could set X-Forwarded-For if you wanted to just get the data
> I tried to search for iptables but it looks very cryptic to me.
It requires you to learn a thing or two about routing, yeah. There's a learning curve, definitely. It's probably easiest to just do this with either netcat or stunnel, or ssh port forwarding.
@i@judgedread@mint Anyone that is annoyed the API is locked down can direct their complaints to @CrunkLord420 . I'd like to personally thank him for making shit that even a monkey can use to cause problems, and then giving it to monkeys that are predisposed to cause problems, and then shrugging. (PROTIP: hard-code some minimum rate-limits so that anyone that is too fucking stupid to run a compiler is not going to do any damage, and stop making "regular-ass browser UA string" the default in the example config.)
I saw the guy's "Oh, I was crawling really slowly!" post in that thread, and I'll just dump the logs. Bots that are pretending to be browsers are kill-on-sight anywhere with a competent sysadmin.
No more traffic arrives here from M247 ever again: it's been scrapers and pedophiles and dipshits. It would have gone the same way Mullvad and NordVPN went a long time ago, but two instances were using it to federate; one of them stopped and the other one's dead.
I run this shit so I can use it and I leave it open so other people can use it. If some series of scrapers wants to DDoS it, then I can't use it, so why the fuck am I paying the goddamn bill? gunmonkey.gif
@mint@feld Yeah, I forgot who said it but really early on, I remarked that Phoenix couldn't keep up and was chewing CPU because if Elixir is going to be some immutable stateless deal that it needs to have a call for IO.copy(fd1,fd2) *somewhere*, and I got an objection from one of the devs, I forget who, but there's not really a way around it: there's one way to get data out of one FD and into another efficiently and in a pure-ish functional language you have to just write some C and wrap it in something that fits the language's semantics, because it's a fundamentally FP-unfriendly task. @j probably remembers this because we both came to the same conclusion around the same time and we traded notes on nginx configuration.
@feld trkzw is by the same dude that did TokyoCabinet/KyotoCabinet. Very good shit (TokyoTyrant spoke HTTP and memcache's protocol over the same port) but he got fascinated with C++ so I haven't used the later ones. trkzw has Go bindings (that is, not a Go implementation, and CGO is useless if you are on Plan 9, but IPFS relies on a bunch of fluoridated Linux stuff anyway so CGO is necessary for them).
I think it's a little past them, if I'm being honest. They could probably get away with just integrating tkrzw.
> To use it for fedi we'd really want the garbage collection to work and not spin 100% CPU and IO for hours but that's what it does when you get enough data in there 🤬
Yeah, you wanna see a real horror, look at the directory where it stores the data. I don't know if it's still doing that but it was storing different permutations of the same hash as different files last I checked. So the Qm$whatever or the bafy$whatever, those were just stored as separate files instead of normalizing the hash.
8MB is too large for you to get fortuitous collisions, it's not a very good chunk size for transfer, just the entire thing is like a parody.
Resident hacker, leader of the fsebugoutzone.org coup of the FSE Autonomous Zone, BOFH of freespeechextremist.com, and former admin of FSE before the establishment of the FSE Autonomous Zone. Launching a guerilla war against myself from this bunker.I'm not angry with you, I'm just disappointed.I am physically in Los Angeles but I exist in a permanent state of 3 a.m.I have dropped a bytebeat album, feel free to DM me for a download code or a link to a tarball: https://finitecell.bandcamp.com/album/villain . There is a chiptunes album there, too.Revolver is coming: https://blog.freespeechextremist.com/blog/revolver-kickoff.htmlWar has changed: https://blog.freespeechextremist.com/blog/update-and-roadmap.htmlThe usual alt if FSE is down: @p@bae.st or @p@shitposter.world.