Conversation

Notices

1. blockbot (blockbot@detroitriotcity.com)'s status on Tuesday, 18-Jun-2024 09:31:19 JST blockbot
   DRC is getting attacked bad
   
   Have you seen this ddos attack before?
   
   84.239.12.133 - - [16/Jun/2024:23:19:25 -0600] "GET /api/v1/streaming HTTP/1.1" 504 167 "-" "-"
   84.239.12.133 - - [16/Jun/2024:23:19:25 -0600] "GET /api/v1/streaming HTTP/1.1" 504 167 "-" "-"
   84.239.12.133 - - [16/Jun/2024:23:19:25 -0600] "GET /api/v1/streaming HTTP/1.1" 504 167 "-" "-"
   • pistolero (p@fsebugoutzone.org)'s status on Tuesday, 18-Jun-2024 09:31:16 JST pistolero

     in reply to
     @blockbot iptables at that point. For stuff like that where someone's just doing the same pattern, I just tail the logs, someone passes a threshold and I drop the traffic. If it's enough to actually flood you, normal usage for that endpoint is about once a minute, so say something like more than ten times in ten seconds you could say is unlikely. I check our logs and in the last week, only two IPs hit 4 times in ten seconds and that only happened six times total, so setting it at ten times in ten seconds is pretty safe. So for your log format, something like this would work:
     
     tail -f /var/log/wherever | mawk -Winteractive '
     # Normally you'd just use the $vars but I'm setting them here so the post is easier to read:
     {ip = $1; timestamp = $4; tenseconds = substr(timestamp, 2, 19); path = $7}
     
     # So, check if they're hitting /api/v1/streaming or I think there's a v3 maybe.
     path ~ /\/api\/v[0-9]\/streaming/ {
     # Reasonable key, just cat the fields:
     ct = a[ip tenseconds]++
     # So, ten times in ten seconds from the IP:
     if(ct > 10) {
     print "Killing", ip
     # Or use whatever, ufw or some BSD thing.
     system("iptables -A INPUT -s " ip " -j DROP")
     }
     }
     '
     
     Here, they were too lazy to even set a UA, so it was probably just firing off nonsense over netcat (basic optimization for flooding: just abandon the connection after the request gets through instead of going through all the overhead of dealing with the HTTP library, which will do stuff like parse the response headers, etc.). Instead of worrying about thresholds, you could just kill anything that hits you more than once in a second and doesn't have a User-Agent set.
     
     One thing I do when testing this kind of script out is to replace "system" with "print" or put "echo" at the beginning of the program it runs, easy way to do a dry run.
      and † top dog :pedomustdie: like this.
   • pistolero (p@fsebugoutzone.org)'s status on Tuesday, 18-Jun-2024 09:31:17 JST pistolero

     in reply to
     @blockbot If the rate limit is kicking in, you don't get 504s, you get 429s.
   • blockbot (blockbot@detroitriotcity.com)'s status on Tuesday, 18-Jun-2024 09:31:17 JST blockbot

     in reply to

     • pistolero
     @p
     
     It wasnt rate limited at the time of that screenshot
   • blockbot (blockbot@detroitriotcity.com)'s status on Tuesday, 18-Jun-2024 09:31:18 JST blockbot

     in reply to

     • pistolero
     @p
     
     Here are the logs
     
     On nginx, i set that endpoint to deny all and even rate limited it and it still destroyed my server.
     
     Sync floods
   • blockbot (blockbot@detroitriotcity.com)'s status on Tuesday, 18-Jun-2024 14:38:42 JST blockbot

     in reply to

     • pistolero
     @p
     
     Ty. This attack was annoying because it instantly crashed everything running on the server , even my dns. With dns broken, i cant login remotely and have to run to garage and login and diagnose. Cant even use internet to go to cloudflare to stop it
     † top dog :pedomustdie: likes this.
   • pistolero (p@fsebugoutzone.org)'s status on Tuesday, 18-Jun-2024 14:39:07 JST pistolero

     in reply to
     @blockbot Oh, shit. Yeah, that's a mess.
   • pistolero (p@fsebugoutzone.org)'s status on Tuesday, 18-Jun-2024 14:39:07 JST pistolero

     in reply to

     • pistolero
     @blockbot You run dnsmasq for the local network? You'll wanna. And by default it only caches 150 entries, you'll want it way bigger, especially for a fedi server.
     
     One kind that I've never seen is a flood of "POST /inbox" with a signature for a pile of servers that don't exist but that do have DNS entries pointing somewhere slow; Revolver accepts provisionally and flags the entry, Mastodon and I *think* Pleroma hold the request open and don't respond until they have actually fetched the key. I haven't looked at how they behave if it's a blocked server, but since most Masto admins don't know what they are doing, I think they wouldn't know what was going on, and maybe it tries to fetch the key even for blocked instances; they go through some pageantry to stop your instance from automatically detecting that it is blocked. Like, if you block an instance on Masto, even if you've never seen it, it will start showing up in your peers (which is how fedilist found a bunch of non-fedi sites).
     
     On the bugout zone, I pretty heavily rate-limit /inbox but I think if you were clever about it, even that wouldn't solve it. Pleroma doesn't bottleneck outbound HTTP requests under most circumstances.
     
     You might actually even have a reflection attack if you were to make a DNS wildcard pointing at your target and then flooding all of fedi with "POST /inbox" and a key URL pointed at the DNS wildcard. Like, it's not super efficient but say you point it at a time-consuming endpoint for the other server (like /api/v1/streaming or the search endpoint or a hashtag; something that returns JSON, though, because a lot of the servers probably close the connection once they get back a header that says it's a video file or whatever without reading the body).
     
     This seems like a big enough problem that someone must have thought of it so either there's something I'm missing or we'll find out when someone decides to make a PoC.
      and † top dog :pedomustdie: like this.
   • pistolero (p@fsebugoutzone.org)'s status on Tuesday, 18-Jun-2024 14:39:58 JST pistolero

     in reply to

     • this ad CAN be blocked, bitch
     @kirby @blockbot If I leave something out, I keep switching between fedi and acme and I hit "^W" to erase a word and instead it closed the tab, so I am retyping it and sometimes this makes a feller think he has mentioned something but instead, he has said nothing because history has been erased.
     
     > the ip in the first snippet [84.239.12.133] actually belongs to a vps company
     
     Yeah, a lot of VPNs just lease cheap VPSs. It's not Tor, Mullvad, Cloudflare, or ProtonVPN, and those are the only ones I have in my "tell me some shit about this address" script. This one looks like it's from privateinternetaccess.com, which is a paid VPN service.
     
     > is this actually an approach they could've taken?
     
     I don't know. I don't even know if it works. There are a lot of accidental DoSs, it's network code and the network got huge, but the dev teams didn't get more time, and people are always yelling "FEATURES FEATURES" and no one sponsors "Fix the holes" or "Can we make this not absurdly slow" or anything like that.
     
     > is this someone that actually probably is not dumb but is doing dumb shit for some reason
     
     They don't seem dumb, but I don't know why they have decided to try to DoS the pedo instances. It's a moderately clever attack but it's really noisy and it's easy to stop completely and the execution is really lazy.
     
     Pedos have lots of websites, and you can't really remove someone from a decentralized network. I can understand why that might happen if the pedos were doing cop-bait shit and the admin was going out of his way to agitate, but they went after DRC, which is not a pedo instance (maybe confused DRC's Alice with someone on one of the other instances, there are a few people named that on those servers), so it can't be a response to anything they were actively doing, or this person would have started with whoever was actually causing trouble. Bored kid got a wild hair up his ass, no idea.
     babyfuck.png
     † top dog :pedomustdie: likes this.
   • this ad CAN be blocked, bitch (kirby@lab.nyanide.com)'s status on Tuesday, 18-Jun-2024 14:39:59 JST this ad CAN be blocked, bitch

     in reply to

     • pistolero
     @p @blockbot this is an interesting approach and what's interesting is that the ip in the first snippet [84.239.12.133] actually belongs to a vps company
     
     is this actually an approach they could've taken? is this someone that actually probably is not dumb but is doing dumb shit for some reason
   • pistolero (p@fsebugoutzone.org)'s status on Tuesday, 18-Jun-2024 14:40:10 JST pistolero

     in reply to

     • this ad CAN be blocked, bitch
     @blockbot @kirby Yeah. A single TCP frame is easy to send.
     
     > Possible SYN flooding
     
     This just means really heavy traffic; at the level of the kernel's routing table, it can be difficult to tell the difference between high traffic and a syn flood.
     ytcracker--strictly_for_my_streamers--05_terminal.mp3
   • pistolero (p@fsebugoutzone.org)'s status on Tuesday, 18-Jun-2024 14:40:10 JST pistolero

     in reply to

     • this ad CAN be blocked, bitch
     • pistolero
     @blockbot @kirby Like, the TCP handshake:
     
     :mgsgb_1::mycomputer::ddr_r::hal9000: SYN
     :mgsgb_2::hal9000::ddr_r::mycomputer: ACK/SYN
     :mgsgb_3::mycomputer::ddr_r::hal9000: ACK
     
     After step 3, the connection is established. Sending a SYN flood looks a lot like creating a large number of incoming connections, because it's just repeating step 1 and ignoring the responses in step 2. The from address can be forged, so how do you tell the difference? The syncookie thing is basically embedding some private data in a hash in the TCP sequence number so that the server can skip the connection overhead: when step 3 happens, it validates the hash and then it can assume the connection is really open.
     
     Since actual requests were coming in, it wasn't a syn flood: you can't send the "GET /api/v1/streaming HTTP/1.1\r\n" unless you've finished the handshake.
     † top dog :pedomustdie: likes this.
   • blockbot (blockbot@detroitriotcity.com)'s status on Tuesday, 18-Jun-2024 14:40:11 JST blockbot

     in reply to

     • this ad CAN be blocked, bitch
     • pistolero
     @p @kirby
     
     It was strong enough to ruin everything running on the Server
     
     Jun 17 14:57:49 nas kernel: TCP: request_sock_TCP: Possible SYN flooding on port 0.0.0.0:443. Sending cookies.
     
     Made NGINX his bitch
   • this ad CAN be blocked, bitch (kirby@lab.nyanide.com)'s status on Tuesday, 18-Jun-2024 14:40:12 JST this ad CAN be blocked, bitch

     in reply to

     • pistolero
     @blockbot @p oom issue?
   • pistolero (p@fsebugoutzone.org)'s status on Tuesday, 18-Jun-2024 14:40:12 JST pistolero

     in reply to

     • this ad CAN be blocked, bitch
     @kirby @blockbot It sounds like his upstream was flooded. Residential lines are lopsided.
   • blockbot (blockbot@detroitriotcity.com)'s status on Tuesday, 18-Jun-2024 14:40:40 JST blockbot

     in reply to

     • this ad CAN be blocked, bitch
     • pistolero
     @p @kirby
     
     The vulnerability he found with nginx/Pleroma is major..... can't even protect ourselves. Sucks
   • pistolero (p@fsebugoutzone.org)'s status on Tuesday, 18-Jun-2024 14:40:40 JST pistolero

     in reply to

     • this ad CAN be blocked, bitch
     @blockbot @kirby I don't think it's new, there's not a good way to fix it in Pleroma.
     † top dog :pedomustdie: likes this.
   • pistolero (p@fsebugoutzone.org)'s status on Tuesday, 18-Jun-2024 14:40:48 JST pistolero

     in reply to

     • this ad CAN be blocked, bitch
     • 85f02109a28864e4713d3f972ab66a
     @blockbot @85f02109a28864e4713d3f972ab66a @kirby I don't know, man, FSE always seems to survive DDoS just fine.
     † top dog :pedomustdie: likes this.
   • 85f02109a28864e4713d3f972ab66a (85f02109a28864e4713d3f972ab66a@pawoo.net)'s status on Tuesday, 18-Jun-2024 14:40:49 JST 85f02109a28864e4713d3f972ab66a

     in reply to

     • this ad CAN be blocked, bitch
     • pistolero

     @blockbot @p @kirby lmao imouto managed to "protect themselves" from the ddos attacks by heavily limiting network traffic but it ended up causing everyone to deal with connectivity issues and a ton of people were unable to access their instance because of their poor mitigations 😭

   • blockbot (blockbot@detroitriotcity.com)'s status on Tuesday, 18-Jun-2024 14:40:49 JST blockbot

     in reply to

     • this ad CAN be blocked, bitch
     • pistolero
     • 85f02109a28864e4713d3f972ab66a
     @85f02109a28864e4713d3f972ab66a @p @kirby
     
     Did they rate limit /api with nginx? bad move lol
   • pistolero (p@fsebugoutzone.org)'s status on Tuesday, 18-Jun-2024 14:40:54 JST pistolero

     in reply to

     • this ad CAN be blocked, bitch
     • SilverDeth
     @SilverDeth @blockbot @kirby Don't worry, I'm behind 7 proxies.
     † top dog :pedomustdie: likes this.
   • pistolero (p@fsebugoutzone.org)'s status on Tuesday, 18-Jun-2024 14:40:55 JST pistolero

     in reply to

     • this ad CAN be blocked, bitch
     @kirby @blockbot Let's hack the FBI!
   • SilverDeth (silverdeth@fsebugoutzone.org)'s status on Tuesday, 18-Jun-2024 14:40:55 JST SilverDeth

     in reply to

     • this ad CAN be blocked, bitch
     • pistolero
     @p @kirby @blockbot
     
     Please don't... I like this place...
   • pistolero (p@fsebugoutzone.org)'s status on Tuesday, 18-Jun-2024 14:40:56 JST pistolero

     in reply to

     • this ad CAN be blocked, bitch
     @kirby @blockbot
     
     > as in some big SQL query caused server to die
     
     Nah, Postgres preallocates its memory.
   • this ad CAN be blocked, bitch (kirby@lab.nyanide.com)'s status on Tuesday, 18-Jun-2024 14:40:56 JST this ad CAN be blocked, bitch

     in reply to

     • pistolero
     @p @blockbot i may be mildly. just mildly. uneducated
   • this ad CAN be blocked, bitch (kirby@lab.nyanide.com)'s status on Tuesday, 18-Jun-2024 14:40:57 JST this ad CAN be blocked, bitch

     in reply to

     • this ad CAN be blocked, bitch
     • pistolero
     @blockbot @p as in some series of expensive SQL queries caused server to die (it would make sense for a trivial dos attack)
     
     isn't it possible to configure certain applications to not go beyond a certain amount of memory in Linux?
   • † top dog :pedomustdie: (dcc@annihilation.social)'s status on Tuesday, 18-Jun-2024 14:41:59 JST † top dog :pedomustdie:

     in reply to

     • this ad CAN be blocked, bitch
     • pistolero
     • SilverDeth
     @p @blockbot @SilverDeth @kirby *DING DING DING*
     :dont_panic: HACK COMPLETE :dont_panic:
     :we_did_it:
     hacking.gif
   • † top dog :pedomustdie: (dcc@annihilation.social)'s status on Tuesday, 18-Jun-2024 14:42:48 JST † top dog :pedomustdie:

     in reply to

     • this ad CAN be blocked, bitch
     • pistolero
     • 85f02109a28864e4713d3f972ab66a
     @blockbot @p @kirby @85f02109a28864e4713d3f972ab66a More horsepower anyways helps with ddo's
   • blockbot (blockbot@detroitriotcity.com)'s status on Tuesday, 18-Jun-2024 14:42:49 JST blockbot

     in reply to

     • this ad CAN be blocked, bitch
     • pistolero
     • 85f02109a28864e4713d3f972ab66a
     @p @kirby @85f02109a28864e4713d3f972ab66a
     
     This was the atomic bomb of Fedi DDOS
   • pistolero (p@fsebugoutzone.org)'s status on Tuesday, 18-Jun-2024 14:42:56 JST pistolero

     in reply to

     • this ad CAN be blocked, bitch
     • 85f02109a28864e4713d3f972ab66a
     @blockbot @85f02109a28864e4713d3f972ab66a @kirby
     
     > it totally BTFO's anything you can do in NGINX.
     
     It doesn't, though.
     † top dog :pedomustdie: likes this.
   • 85f02109a28864e4713d3f972ab66a (85f02109a28864e4713d3f972ab66a@pawoo.net)'s status on Tuesday, 18-Jun-2024 14:42:57 JST 85f02109a28864e4713d3f972ab66a

     in reply to

     • this ad CAN be blocked, bitch
     • pistolero

     @blockbot @p @kirby it's so bad that after 3 refreshes nginx starts throwing 503s

   • blockbot (blockbot@detroitriotcity.com)'s status on Tuesday, 18-Jun-2024 14:42:57 JST blockbot

     in reply to

     • this ad CAN be blocked, bitch
     • pistolero
     • 85f02109a28864e4713d3f972ab66a
     @85f02109a28864e4713d3f972ab66a @p @kirby
     
     Your attack method is interesting because it totally BTFO's anything you can do in NGINX.
     
     This had no effect at all.
     
     location /api/v1/streaming {
     limit_req zone=foo burst=1 nodelay;
     
     deny all;
     }
   • pistolero (p@fsebugoutzone.org)'s status on Tuesday, 18-Jun-2024 15:37:14 JST pistolero

     in reply to

     • this ad CAN be blocked, bitch
     • 85f02109a28864e4713d3f972ab66a
     @blockbot @85f02109a28864e4713d3f972ab66a @kirby limit_conn.
     † top dog :pedomustdie: likes this.
   • blockbot (blockbot@detroitriotcity.com)'s status on Tuesday, 18-Jun-2024 15:37:15 JST blockbot

     in reply to

     • this ad CAN be blocked, bitch
     • pistolero
     • 85f02109a28864e4713d3f972ab66a
     @p @kirby @85f02109a28864e4713d3f972ab66a
     
     Nginx tried to deny and rate limit but the system still got messed up.
   • pistolero (p@fsebugoutzone.org)'s status on Tuesday, 18-Jun-2024 15:49:37 JST pistolero

     in reply to

     • this ad CAN be blocked, bitch
     • 85f02109a28864e4713d3f972ab66a
     @blockbot @85f02109a28864e4713d3f972ab66a @kirby
     
     > Come on P!
     
     Do you think that sending more messages makes me catch up on messages faster?
     † top dog :pedomustdie: likes this.
   • 85f02109a28864e4713d3f972ab66a (85f02109a28864e4713d3f972ab66a@pawoo.net)'s status on Tuesday, 18-Jun-2024 15:49:38 JST 85f02109a28864e4713d3f972ab66a

     in reply to

     • this ad CAN be blocked, bitch
     • pistolero

     @blockbot @p @kirby he isnt responding 😭

   • blockbot (blockbot@detroitriotcity.com)'s status on Tuesday, 18-Jun-2024 15:49:38 JST blockbot

     in reply to

     • this ad CAN be blocked, bitch
     • pistolero
     • 85f02109a28864e4713d3f972ab66a
     @85f02109a28864e4713d3f972ab66a @p @kirby
     
     Come on P!
   • pistolero (p@fsebugoutzone.org)'s status on Tuesday, 18-Jun-2024 15:49:39 JST pistolero

     in reply to

     • this ad CAN be blocked, bitch
     • 85f02109a28864e4713d3f972ab66a
     @85f02109a28864e4713d3f972ab66a @blockbot @kirby The hell are they doing that is making it toss 503s?
   • 85f02109a28864e4713d3f972ab66a (85f02109a28864e4713d3f972ab66a@pawoo.net)'s status on Tuesday, 18-Jun-2024 15:49:39 JST 85f02109a28864e4713d3f972ab66a

     in reply to

     • this ad CAN be blocked, bitch
     • pistolero

     @p @blockbot @kirby can i see if your instance is protected

   • blockbot (blockbot@detroitriotcity.com)'s status on Tuesday, 18-Jun-2024 15:49:39 JST blockbot

     in reply to

     • this ad CAN be blocked, bitch
     • pistolero
     • 85f02109a28864e4713d3f972ab66a
     @85f02109a28864e4713d3f972ab66a @p @kirby
     
     Do it for Science P!
   • pistolero (p@fsebugoutzone.org)'s status on Tuesday, 18-Jun-2024 16:05:27 JST pistolero

     in reply to

     • this ad CAN be blocked, bitch
     • 85f02109a28864e4713d3f972ab66a
     @85f02109a28864e4713d3f972ab66a @blockbot @kirby I'm not, I don't talk to them. The pedos instance-block FSE: https://fba.ryona.agency/?domain=fsebugoutzone.org
     
     I am gonna say stuff in public about how to fix bugs.
     † top dog :pedomustdie: likes this.
   • pistolero (p@fsebugoutzone.org)'s status on Tuesday, 18-Jun-2024 16:05:28 JST pistolero

     in reply to

     • this ad CAN be blocked, bitch
     • 85f02109a28864e4713d3f972ab66a
     @blockbot @85f02109a28864e4713d3f972ab66a @kirby Well, DoS unless there were more IPs. But I can see what was happening, and rate-limiting the endpoint and capping the number of open connections per host fixes it even before just killing off the VPN's IPs.
   • 85f02109a28864e4713d3f972ab66a (85f02109a28864e4713d3f972ab66a@pawoo.net)'s status on Tuesday, 18-Jun-2024 16:05:28 JST 85f02109a28864e4713d3f972ab66a

     in reply to

     • this ad CAN be blocked, bitch
     • pistolero

     @p @blockbot @kirby don't give the pedophiles any tips

   • pistolero (p@fsebugoutzone.org)'s status on Tuesday, 18-Jun-2024 16:15:49 JST pistolero

     in reply to

     • this ad CAN be blocked, bitch
     • 85f02109a28864e4713d3f972ab66a
     @blockbot @85f02109a28864e4713d3f972ab66a @kirby Let's fucking GOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO
     fse_mood_meter.png
     † top dog :pedomustdie: likes this.
   • 85f02109a28864e4713d3f972ab66a (85f02109a28864e4713d3f972ab66a@pawoo.net)'s status on Tuesday, 18-Jun-2024 16:15:50 JST 85f02109a28864e4713d3f972ab66a

     in reply to

     • this ad CAN be blocked, bitch
     • pistolero

     @p @blockbot @kirby do you want to test

   • pistolero (p@fsebugoutzone.org)'s status on Tuesday, 18-Jun-2024 16:15:50 JST pistolero

     in reply to

     • this ad CAN be blocked, bitch
     • 85f02109a28864e4713d3f972ab66a
     @85f02109a28864e4713d3f972ab66a @blockbot @kirby Knock yourself out.
   • blockbot (blockbot@detroitriotcity.com)'s status on Tuesday, 18-Jun-2024 16:15:50 JST blockbot

     in reply to

     • this ad CAN be blocked, bitch
     • pistolero
     • 85f02109a28864e4713d3f972ab66a
     @p @kirby @85f02109a28864e4713d3f972ab66a
     
     
     let's go!
   • 85f02109a28864e4713d3f972ab66a (85f02109a28864e4713d3f972ab66a@pawoo.net)'s status on Tuesday, 18-Jun-2024 16:16:12 JST 85f02109a28864e4713d3f972ab66a

     in reply to

     • this ad CAN be blocked, bitch
     • pistolero

     @p @blockbot @kirby what does this mean lmao

   • pistolero (p@fsebugoutzone.org)'s status on Tuesday, 18-Jun-2024 16:16:12 JST pistolero

     in reply to

     • this ad CAN be blocked, bitch
     • 85f02109a28864e4713d3f972ab66a
     @85f02109a28864e4713d3f972ab66a @blockbot @kirby It means go for it, do it. Looks like it stopped. Turn it back on on, crank it way up.
     
     Let me give you an idea of the scale. To test out the software I am working on, I exploited a different bug to DDoS myself: https://git.pleroma.social/pleroma/pleroma/-/issues/2765 . We had a sustained 800+r/s, and that's successful requests, right? I turned off the rate-limiting on /objects for that. Just compulsive posting in the hellthread. 800r/s isn't that amazing, but it's way bigger than 14.22 I don't know how many it would have taken to break it, because I couldn't get up the other instances to fetch posts any faster.
     
     People have hit the streaming endpoint before. We got aggressively scraped by several places, boardreader.com was feeding it to the FBI, we got DDoS'd by a DDoS mitigation company, we got flooded by Gab. Short of saturating the pipe (please don't, that *is* a pain), FSE is really hard to knock down. If you find a bug and *can* knock it down, feel free. Your existing strategy does not work here, but there are lots of holes in fedi, I'm sure you can find another one.
     koddos.png
     † top dog :pedomustdie: likes this.