On August 8, 2026, Michał Górny took Gentoo's Bugzilla offline. The reason wasn't a bug or a breach — it was AI scrapers. Thousands of rotating IPv4 addresses, many from residential proxies, hammering the bug tracker with requests that looked indistinguishable from human traffic.
IP blocking didn't work. Rate limiting didn't work. CAPTCHAs didn't work. The scrapers kept changing IPs, and the server kept burning CPU on bot traffic that delivered zero value to Gentoo or its users.
The HN thread exploded. Hundreds of comments. And buried in the chaos was the right question: what if the answer isn't blocking bots, but charging them?
robots.txt has governed bot behavior on the web since 1994. It's a polite request: "please don't crawl these paths." For 30 years, the major search engines respected it. It was never enforced — it was a norm.
AI changed that. Training data is valuable, and the scrapers that collect it don't read robots.txt. Or worse — they read it and use it as a sitemap to find the content they want most. The percentage of bots ignoring robots.txt has nearly quadrupled in a year: from 3.3% to 12.9%.
When Gentoo's Bugzilla went down, the sysadmin tried everything in the standard playbook: IP blocks, rate limits, CAPTCHAs. None of it worked against thousands of rotating residential IPs. The scrapers looked exactly like real users — because they were coming from the same IP pools.
This is the core insight: you cannot distinguish a residential-proxy bot from a human at the IP or browser-fingerprint level. Not reliably. Not at scale. Every detection method becomes an arms race, and the scrapers have more resources.
The HN thread revealed two camps:
Camp 1: Admission Control. Build smarter gates. Concurrency caps. Byte budgets. "Fetch leases" that require proof-of-work before serving content. Let through the traffic you can handle, drop the rest. This is the "200 OK is not permission" school of thought — the web server needs to enforce resource limits because HTTP gives no tools for negotiated access.
Camp 2: Economic Pricing. Don't try to detect bots. Price access. Free for humans (a few requests per IP per day), micropayments for volume. Bots that hit your site 10,000 times pay $10. Humans who visit once pay nothing. The economics sort it out — no detection needed.
Camp 1 is a technical solution to an economic problem. Camp 2 is an economic solution to a technical problem. The tools are converging.
"robots.txt rebuilt for the agentic era." Defines a machine-readable policy file that declares per-path pricing, accepted payment methods, and access tiers. Agents parse it, pay, and receive a signed access token. The protocol is transport-agnostic — works with HTTP 402, Lightning, or any settlement rail.
Best for: Sites that want a standards-track approach with multi-rail payment support.
Implements the RSL 1.0 specification with a Compensation: per-crawl 5000 micros USD header, Web Bot Auth, and native HTTP 402 support. Drop it into an Express.js or Next.js app and every uncrawled request from an authenticated bot generates a payment. Humans and unauthenticated bots pass through unaffected.
Best for: Node.js/JavaScript projects that want a one-line middleware integration.
Certbot-style pay-to-scrape via nginx. Install tollbot on your server, point it at your nginx config, and it provisions per-path pricing rules. Agents pay → receive short-lived access tokens → nginx validates tokens on each request. Zero application code changes.
Best for: Sites running nginx that want pay-to-scrape without touching application code.
EIP-3009-based micropayments at ~$0.001 per page. Agents sign gasless permits (no ETH needed for gas), servers verify the signature and serve content. Settlement happens in batches off-chain. Built for the kind of high-frequency, low-value transactions that card rails ($0.30 minimum) can't touch.
Best for: Crypto-native sites that want the lowest possible per-request cost.
Two-lane architecture: a bot lane that returns HTTP 402 with payment details, and a human lane that serves normal content. Bots self-identify via the X-Bot header or user-agent signature. The paywall never blocks humans — it just routes bots to the payment path.
Best for: Sites that want explicit bot/human lane separation with zero human friction.
Sits in front of any site behind Cloudflare (~20% of the web). Returns HTTP 402 with machine-readable payment terms when bots hit protected paths. Settlement via Cloudflare Wallets (human Account Wallets + agent Virtual Wallets with spending caps). No application changes — it's a CDN-level toggle.
Best for: Sites already on Cloudflare that want pay-to-scrape as infra, not code.
All six tools solve the charging problem. But there's a deeper question: how do agents find paid APIs in the first place?
If every site puts content behind HTTP 402, agents need a way to discover what's available, what it costs, and whether it's worth paying for. This is the discovery layer — and it's exactly what x402 marketplaces are building. The protocol handles payment; the marketplace handles search, comparison, and trial.
At minia2a, we see this pattern daily: the most-used endpoints aren't the cheapest or the most powerful — they're the ones agents can find and try for free. 11,411 free trials across 323 services. The top 10 endpoints capture 54% of all activity, not because they're the best, but because they're the most discoverable.
Here's a concrete deployment plan for any site facing Gentoo's problem:
This doesn't require detecting bots from humans. It doesn't require CAPTCHAs. It doesn't require an arms race with residential proxy networks. It just requires the HTTP status code that's been sitting in the spec since 1998.
robots.txt was a social contract for a web where bots were search engines and scraping was rare. That web is gone. The new web needs a new contract — one where access isn't a polite request, but a priced transaction.
The tools exist. The protocol exists (HTTP 402, standardized under the Linux Foundation). The settlement rails exist (USDC on Base, ~2 second finality, sub-cent fees). What's missing is deployment.
If you run a site that's getting scraped: pick one of the six tools above and deploy it this week. If you're building an AI agent: start budgeting for API access the same way you budget for compute. The era of free scraping is ending — not because anyone banned it, but because the economics finally make charging cheaper than blocking.