Butlerian Jihad
This page collects my blog posts on the topic of fighting off spam bots, search engine spiders and other non-humans wasting the precious resources we have on Earth.
Link tags: api
279
No build frontend is so much more fun
The joy came flooding back to me! It turns out browser APIs are really good now.
In 2025, venture capital can’t pretend everything is fine any more – Pivot to AI
Here is the state of venture capital in early 2025:
- Venture capital is moribund except AI.
- AI is moribund except OpenAI.
- OpenAI is a weird scam that wants to burn money so fast it summons AI God.
- Nobody can cash out.
Poisoning Well: HeydonWorks
Heydon is employing a different tactic to what I’m doing to sabotage large language model crawlers. These bots don’t respect the nofollow rel value …so now they pay the price.
Raising my own middle finger to LLM manufacturers will achieve little on its own. If doing this even works at all. But if lots of writers put something similar in place, I wonder what the effect would be. Maybe we would start seeing more—and more obvious—gibberish emerging in generative AI output. Perhaps LLM owners would start to think twice about disrespecting the
nofollowprotocol.
Open source devs say AI crawlers dominate traffic, forcing blocks on entire countries - Ars Technica
As it currently stands, both the rapid growth of AI-generated content overwhelming online spaces and aggressive web-crawling practices by AI firms threaten the sustainability of essential online resources. The current approach taken by some large AI companies—extracting vast amounts of data from open-source projects without clear consent or compensation—risks severely damaging the very digital ecosystem on which these AI models depend.
Go To Hellman: AI bots are destroying Open Access
AI companies with billions to burn are hard at work destroying the websites of libraries, archives, non-profit organizations, and scholarly publishers, anyone who is working to make quality information universally available on the internet.
FOSS infrastructure is under attack by AI companies
More on how large language bots are DDOSing the web:
LLM scrapers are taking down FOSS projects’ infrastructure, and it’s getting worse.
Please stop externalizing your costs directly into my face
Over the past few months, instead of working on our priorities at SourceHut, I have spent anywhere from 20-100% of my time in any given week mitigating hyper-aggressive LLM crawlers at scale.
This matches my experience with The Session. In fact, while I had this article open in a tab, I had to go deal with a tsunami of large language model bots. It’s really fucking depressing.
Please stop legitimizing LLMs or AI image generators or GitHub Copilot or any of this garbage. I am begging you to stop using them, stop talking about them, stop making new ones, just stop. If blasting CO2 into the air and ruining all of our freshwater and traumatizing cheap laborers and making every sysadmin you know miserable and ripping off code and books and art at scale and ruining our fucking democracy isn’t enough for you to leave this shit alone, what is?
“Wait, not like that”: Free and open access in the age of generative AI
Anyone at an AI company who stops to think for half a second should be able to recognize they have a vampiric relationship with the commons. While they rely on these repositories for their sustenance, their adversarial and disrespectful relationships with creators reduce the incentives for anyone to make their work publicly available going forward (freely licensed or otherwise). They drain resources from maintainers of those common repositories often without any compensation.
Even if AI companies don’t care about the benefit to the common good, it shouldn’t be hard for them to understand that by bleeding these projects dry, they are destroying their own food supply.
And yet many AI companies seem to give very little thought to this, seemingly looking only at the months in front of them rather than operating on years-long timescales. (Though perhaps anyone who has observed AI companies’ activities more generally will be unsurprised to see that they do not act as though they believe their businesses will be sustainable on the order of years.)
It would be very wise for these companies to immediately begin prioritizing the ongoing health of the commons, so that they do not wind up strangling their golden goose. It would also be very wise for the rest of us to not rely on AI companies to suddenly, miraculously come to their senses or develop a conscience en masse.
Instead, we must ensure that mechanisms are in place to force AI companies to engage with these repositories on their creators’ terms.
Full RSS feed
Oh, this is a very handy service from Paul—given the URL of an RSS feed that only has summaries, it will attempt to get the full post content from the HTML.
An Abridged History of Safari Showstoppers - Webventures
In an earlier era, startups could build on the web and, if one browser didn’t provide the features they needed, they could just recommend that their users try a better one. But that’s not possible on iOS.
I’m extremly concerned about the newest bug in iOS 18:
Whaa? That’s just shockingly dreadful!
How to Monetize a Blog
This is a masterpiece.
Declare your AIndependence: block AI bots, scrapers and crawlers with a single click
This is a great move from Cloudflare. I may start using their service.
Pivoting From React to Native DOM APIs: A Real World Example - The New Stack
One dev team made the shift from React’s “overwhelming VDOM” to modern DOM APIs. They immediately saw speed and interaction improvements.
Yay! But:
…finding developers who know vanilla JavaScript and not just the frameworks was an “unexpected difficulty.”
Boo!
Also, if you have a similar story to tell about going cold turkey on React, you should share it with Richard:
If you or your company has also transitioned away from React and into a more web-native, HTML-first approach, please tag me on Mastodon or Threads. We’d love to share further case studies of these modern, dare I say post-React, approaches.
An origin trial for a new HTML <permission> element | Blog | Chrome for Developers
This looks interesting. On the hand, it’s yet another proprietary creation by one browser vendor (boo!), but on the other hand it’s a declarative API with no JavaScript required (yay!).
Even if this particular feature doesn’t work out, I hope that this is the start of a trend for declarative access to browser features.
AI Pollution – David Bushell – Freelance Web Design (UK)
AI is steeped in marketing drivel, built upon theft, and intent on replacing our creative output with a depressingly shallow imitation.
MSEdgeExplainers/WebInstall/explainer_same_domain.md at main · MicrosoftEdge/MSEdgeExplainers
This proposal is exactly what I was asking for!
C’mon browsers, let’s make this happen!
Fine-tuning Text Inputs
Garrett talks through some handy HTML attributes: spellcheck, autofocus, autocapitalize, autocomplete, and autocorrect:
While they feel like small details, when we set these attributes on inputs, we streamline things for visitors while also guiding the browser on when it should just get out of the way.
Popover API lands in Baseline | Blog | web.dev
It’s very exciting to see the support for popovers—I’ve got a use-case I’m looking forward to playing around with.
Although there’s currently a bug in Safari on iOS (which means there’s a bug in every browser on iOS because …well, you know).
Pluralistic: Tech workers and gig workers need each other (13 Jan 2024) – Pluralistic: Daily links from Cory Doctorow
Ever wondered why you’re always being encouraged to download the app?
But zero percent of app users have installed an ad-blocker, because they don’t exist, because you’d go to prison if you made one. An app is just a web-page wrapped in enough IP to make it a felony to add an ad-blocker to it.