Conversation
Notices
-
@p what is the Fedilist frontend written in demo.fedilist.com/asbestos.cafe
-
@meso
> do you ever scrape HTML with the scraper?
There's no scraper, just a crawler. The mastodongs keep calling it a scraper because that sounds worse, but it doesn't scrape any content. It's just hitting regular-ass API endpoints, not extracting data from pages or pretending to be a browser. It doesn't scrape anything.
It does get HTML from two places: Mastodon makes instance description HTML, and bios are HTML. I scrub the HTML before displaying it, but in the case of bios, those don't even show on the page if the person has the "discoverable" flag set to false.
-
@meso Oh, it's just Ruby. Ruby's easy for string-mangling. I generate the HTML instead of using templates; templates are terrible, I avoid them if I can. (Hard to validate automatically, but it's easier to guarantee that you are *generating* valid markdown. Aside from that, it's a pain in the ass to try to factor templates, and the code parts are always ugly.)
The crawler itself uses libcurl, through the Typhoeus library. That is the part I would *like* to rewrite, but probably will not any time soon.
-
@p huh that's pretty cool. do you ever scrape HTML with the scraper?
-
@p how? what language? what pipeline? very curious about this
-
@meso No, it's not static HTML, I generate it and I throw it down the pipeline.
-
@p huh? what? how? is it fully static?
-
@meso It doesn't have a real frontend, I just spew some HTML at the pipe.
-
@p @meso funny how they screech about language matters when you describe something because it might cause inevitable disaster around the community but then pull these gimmicks
meso
pistolero :thispersondoesnotexist:
God's Silliest Soldier
All ０７６萌ＳＮＳ content and data are available under the