Anubis is not free either, it is a matter of how much it costs to run Anubis vs to let bots in.
When I see LKML using Anubis when the pages it serves are tens of kB, all presumably static, I wonder if they do it out of spite rather than to protect their servers.
As a user/reader/viewer I absolutely hate Anubis and usually turn around when I see it pop up (at least on my phone where it takes ages to compute), but with stats like that, I get why a site operator would resort to using it.
I think the kernel.org post proves the parent point rather than contradicts it.
> At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.
14 CPU cores total for running a website like kernel.org is laughable. This is not worth burning cycles in Anubis on client's devices, this is not worth the time of the engineer who worked on it. Provisioning more hardware would have been literally better for everyone.
> But no, let's in fact choose the stupidest possible way of doing it — by rendering everything as HTML commit by commit and then parsing it.
This drives me crazy with so-called SOTA LLMs that have "achieved AGI".
Fable, Sol, Astra, will start by trying to reverse engineer a binary to figure out how something works when software is open source and one search query away.
You let them know it's open source, and they will start using github API instead of just cloning and grepping.
When I see LKML using Anubis when the pages it serves are tens of kB, all presumably static, I wonder if they do it out of spite rather than to protect their servers.