ADuke University study presented at ACM's 2025 Internet Measurement Conference tracked 3.9 million requests over 40 days. The sharpest finding: AI search crawlers and AI assistants had the lowest robots.txt re-check rates of any bot category. Fewer than 40% checked within a seven-day window. Some never checked at all.
robots.txt assumes a two-party exchange. The site posts its terms, the crawler reads them. When the crawler never shows up to look, the mechanism doesn't malfunction. It simply never starts.