Conversation

Notices

1. Rusty Crab (rustycrab@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 05:37:53 JST Rusty Crab
   does anybody know how I might bypass signed fetch to download post history (I'm making an ebooks bot). I'd rather not set up a whole ass server if I can help it.
   •  (mint@ryona.agency)'s status on Tuesday, 26-Dec-2023 05:37:53 JST 

     in reply to
     @RustyCrab Why not download through mastoapi?
   • Rusty Crab (rustycrab@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 05:45:07 JST Rusty Crab

     in reply to

     • 
     @mint well I am trying to scrape an account on NCD (nigbot) and they seem to be pretty locked down. I am new to this so I am not sure about the various ways you can get post history. I am using the ebooks outbox query Python script currently. I reached out to Matty but he seems busy.
      likes this.
   • Rusty Crab (rustycrab@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 05:46:25 JST Rusty Crab

     in reply to

     • Zero :zt_think: :artix:
     @zero currently trying to use the ebooks script and I am getting blocked as unauthorized. That's signed fetching isn't it?
   •  (mint@ryona.agency)'s status on Tuesday, 26-Dec-2023 05:46:25 JST 

     in reply to

     • Zero :zt_think: :artix:
     @RustyCrab @zero Yes. Shameful display from them.
   • Zero :zt_think: :artix: (zero@strelizia.net)'s status on Tuesday, 26-Dec-2023 05:46:26 JST Zero :zt_think: :artix:

     in reply to
     @RustyCrab why would you need to bypass it or set up a server
   •  (mint@ryona.agency)'s status on Tuesday, 26-Dec-2023 05:48:42 JST 

     in reply to
     @RustyCrab Alex has a readymade library for HTTP signatures, but it's in node and I don't know if anyone made anything similar for Python.
     https://gitlab.com/soapbox-pub/fedisign
   • :blank: (i@declin.eu)'s status on Tuesday, 26-Dec-2023 06:02:20 JST :blank:

     in reply to
     @RustyCrab making an ebooks bot off a fortune fille is :niggainsanest:, if getting it off cyberia's timeline doesn't work out, i'll make a dump later
      likes this.
   • Rusty Crab (rustycrab@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 06:02:21 JST Rusty Crab

     in reply to

     • :blank:
     @i @\nice-nigger@nicecrew.digital
   • :blank: (i@declin.eu)'s status on Tuesday, 26-Dec-2023 06:02:22 JST :blank:

     in reply to
     @RustyCrab what account? could backfill their profile and get you all their json fresh from the outbox
   • cassidyclown (cassidyclown@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 20:31:05 JST cassidyclown

     in reply to

     • :ihavenomouth:
     • Kerosene
     @Kerosene @Inginsub @RustyCrab > What is this thing
     I think you should find the shortcode sufficiently descriptive
      likes this.
   • Rusty Crab (rustycrab@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 20:31:06 JST Rusty Crab

     in reply to

     • :ihavenomouth:
     • cassidyclown
     @Inginsub @cassidyclown I think I see what's wrong :transchiggerniggaseverementaldistress:
   • Kerosene (kerosene@bae.st)'s status on Tuesday, 26-Dec-2023 20:31:06 JST Kerosene

     in reply to

     • :ihavenomouth:
     • cassidyclown
     @RustyCrab @Inginsub @cassidyclown What is this thing man I can't even lmao
     image.png
   • Rusty Crab (rustycrab@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 20:31:07 JST Rusty Crab

     in reply to

     • :ihavenomouth:
     • cassidyclown
     @Inginsub @cassidyclown if I feed the model nothing but niggaposts it actually freezes. What is this ghost in the machine :chiggerniggaseverementaldistress:
   • Rusty Crab (rustycrab@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 20:31:08 JST Rusty Crab

     in reply to

     • :ihavenomouth:
     • cassidyclown
     @cassidyclown @Inginsub markovify with the pleroma ebooks python script
   • Rusty Crab (rustycrab@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 20:31:08 JST Rusty Crab

     in reply to

     • :ihavenomouth:
     • cassidyclown
     @cassidyclown @Inginsub I think I'm going to have to play around with the model because this isn't working at all :grinching:
   • cassidyclown (cassidyclown@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 20:31:09 JST cassidyclown

     in reply to

     • :ihavenomouth:
     @RustyCrab @Inginsub I mean what are you using to do the markov chain stuff
   • Rusty Crab (rustycrab@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 20:31:10 JST Rusty Crab

     in reply to

     • :ihavenomouth:
     • cassidyclown
     @cassidyclown @Inginsub it's a surprise
     
     if I can stop hitting brick walls :niggadementia:
   • cassidyclown (cassidyclown@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 20:31:11 JST cassidyclown

     in reply to

     • :ihavenomouth:
     @RustyCrab @Inginsub how
   • cassidyclown (cassidyclown@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 20:31:11 JST cassidyclown

     in reply to

     • :ihavenomouth:
     • cassidyclown
     @RustyCrab @Inginsub what are you using for it?
   • Rusty Crab (rustycrab@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 20:31:12 JST Rusty Crab

     in reply to

     • :ihavenomouth:
     • cassidyclown
     @cassidyclown @Inginsub this is actually absurd. The model is being fed niggaposts but it's outright refusing to incorporate them into any outputs.
   • Rusty Crab (rustycrab@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 20:31:13 JST Rusty Crab

     in reply to

     • :ihavenomouth:
     @Inginsub the markov chain is very staunchly discriminating against the niggaposts :niggadementia:
   • cassidyclown (cassidyclown@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 20:31:13 JST cassidyclown

     in reply to

     • :ihavenomouth:
     @RustyCrab @Inginsub :nigganigbot:
   • Rusty Crab (rustycrab@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 20:31:14 JST Rusty Crab

     in reply to

     • :ihavenomouth:
     @Inginsub that's okay I don't need that many
   • Rusty Crab (rustycrab@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 20:31:15 JST Rusty Crab

     in reply to

     • :ihavenomouth:
     @Inginsub I have written an iterator and I have obtained niggaposts. 11,000 niggaposts
   • :ihavenomouth: (inginsub@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 20:31:15 JST :ihavenomouth:

     in reply to
     @RustyCrab there’s 20,000 niggaposts total
   • :ihavenomouth: (inginsub@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 20:31:16 JST :ihavenomouth:

     in reply to
     @RustyCrab yeah, pleroma only returns 40 and offset does nothing, so you need to fetch posts older than a specified id
   • Rusty Crab (rustycrab@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 20:31:17 JST Rusty Crab

     in reply to

     • :ihavenomouth:
     @Inginsub I will try but that isn't the way it's built. You have to give it an access token to an account which is following the accounts you want to scrape. It then queries outboxes. I will see if I can get it to query the local user instead.
   • Rusty Crab (rustycrab@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 20:31:17 JST Rusty Crab

     in reply to

     • :ihavenomouth:
     @Inginsub came back from festivities. Figured out how to use the API to query statuses from the cyberian nigbot. However there seems to be a serverside limit of how many it will return so I am going to have to make a loop to pull them down using the last post id gotten. Always something.
   • :ihavenomouth: (inginsub@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 20:31:18 JST :ihavenomouth:

     in reply to
     @RustyCrab I don't know how the ebooks script works, but it probably should be able to pull posts from ANcyiVnc3H7nkCtR2G
   • :ihavenomouth: (inginsub@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 20:31:19 JST :ihavenomouth:

     in reply to
     @RustyCrab why not scrape its local accout, https://clubcyberia.co/users/$ANcyiVnc3H7nkCtR2G
     the posting history will be incomplete, but I don't think you'll lose a lot
   • Rusty Crab (rustycrab@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 20:31:19 JST Rusty Crab

     in reply to

     • :ihavenomouth:
     @Inginsub yes that was my thought and I'm surprised that the ebooks script does not work that way to begin with. I'll need to figure out how to scrape the local instance instead rather than having it query the outbox on the home instance. I'll be honest I thought this was just going to be really simple like it always has been so I wasn't eager to learn a new API/toolset.
   • Rusty Crab (rustycrab@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 20:31:20 JST Rusty Crab

     in reply to

     • :ihavenomouth:
     @Inginsub I'm looking to scrape nigbot's account. Not continually just one time. I try querying the activititypub outbox and I'm getting "unauthorized", which I think is signed fetching.
   • :ihavenomouth: (inginsub@clubcyberia.co)'s status on Tuesday, 26-Dec-2023 20:31:21 JST :ihavenomouth:

     in reply to
     @RustyCrab use an instance that bypasses signed fetch?
   • Zero :zt_think: :artix: (zero@strelizia.net)'s status on Thursday, 28-Dec-2023 00:05:23 JST Zero :zt_think: :artix:

     in reply to

     • 
     @mint @RustyCrab i just read about what that shit does, why does an instance like nicecrew enable it? baffling
      likes this.
   • Rusty Crab (rustycrab@clubcyberia.co)'s status on Thursday, 28-Dec-2023 00:06:00 JST Rusty Crab

     in reply to

     • 
     • Zero :zt_think: :artix:
     @zero @mint I think it prevents some of the more blatant kinds of malicious scraping like MIT has been doing lately. Doesn't stop it fully but a lot of these scrapers are lazy and just won't notice if an instance gets missed
      likes this.