In 2006, someone invented infinite scrolling to keep humans engaged. No more clicking "Next Page." Just scroll. Content appears. Social media platforms adopted it because engagement metrics soared.
Google's crawlers don't scroll. They land on a page, index what's visible, and move on. Content that loads only after scrolling might never exist for search engines. In 2020, Google's Martin Splitt confirmed what developers already knew: "Googlebot does not scroll."
The fix? Maintain two navigation systems. Infinite scroll for users. Paginated URLs for crawlers. Your engineering team now supports parallel architectures because a 2006 engagement optimization never considered machine speed. Web automation hits the same wall. Scrapers see initial content, miss everything below the fold. A UX pattern optimized for human behavior created permanent friction with how machines traverse the web.