Google Zero
AI is taking over the internet. It's being shoved into anything and everything. One of the biggest impacts of AI though has been the lack of traffic to websites because of Google's crawlers.
Google and other AI scrapers have been crawling the internet, sucking up as much of the web's content as they can possible grab, and throwing it into their LLMs. Then when people search for things, instead of providing a list of links, Google provides the user with a summary of the content on the search results page. People are no longer clicking through, they're just taking what Google says in their summaries and moving on to the next search.
The deal websites essentially had with Google and other search engines, was they'd scrape your site, and in exchange, they'd send you traffic. Now though, they're still sucking up site's content, but are no longer sending the traffic. Big sites have already talked about how they're seeing a big drop in traffic, and it makes me think about what we're missing when Google Zero becomes a thing.
I try to make posts when I find an issue and solve it. I hope that it might provide value to others who are having the same issue I'm having. Instead of scouring the web trying to find a solution to their problem, the user can click into my site and view it right there. People can see my site, poke around, and see who I am. Maybe they'll find something cool that inspires them. With only the summaries, how many people are still going to click into the websites, even if they're properly cited? How many people are going to miss the amazing sites out there because Google just regurgitates the content. People will end up missing sites that are far more interesting than mine.
Then there's the counter balance in trying to preserve the open internet.
Mike Masnick over at Techdirt wrote a post last year titled " We’re Walling Off The Open Internet To Stop AI—And It May End Up Breaking Everything Else". There are many people who are against AI and scrapers, that have been going and trying to block AI from scraping their sites. The New York Times has even filed a lawsuit against OpenAI and Microsoft over it. The New York Times is probably too large to really notice much of this, but by blocking the AI companies, what second order side effects are there going to be? Will blocking other tools that are beneficial to users, or to preserving the internet?
The power and excitement of an open web was that it was open and accessible to all. The web’s core principle wasn’t “open to everyone except the technologies we don’t like.” It was “open, period.” Once we start making exceptions based on who might benefit or what technology might be used to access content, we’ve abandoned that principle entirely.
I don't know the answer to these questions. I'm not really sure anyone does at the moment. You can't fault people for not wanting some big behemoth of a company making millions off people's hard work. I can't say I'm super happy about OpenAI or others sucking up all my writing and spitting it out on their search results, especially when they might take things out of context or butcher it in other ways that might not necessarily paint me in the best light.
The other sad part, is that the scraping is probably covered under fair use. Like it or not, scraping books, websites, and everything else in itself is fine, and as Mike Masnick again points out, saying it isn't will have some pretty bad side effects.
AI training absolutely is fair use, and a ruling the other way would blow a hole in fair use protections that have nothing to do with AI, including everything from search engines to book scanning to reverse engineering to (most importantly) text and data mining for research. Also, the NY Times’ case against OpenAI is incredibly weak, and (as I’ve discussed) involves bizarre theories of copyright that would put all sorts of companies (including the NY Times itself!) at real risk of liability for doing basic reporting. So, having the federal government weigh in and make good points could actually be helpful.
I write this musing about what we end up missing when Google redirects all traffic to itself. We miss the amazing people out there making amazing websites. Wanting to stop Google, Perplexity, OpenAI and others is understandable. I feel the same way. I don't like the idea of them gatekeeping everything making it harder to discover cool new people and sites. I also don't know what the solution is without creating a closed, multi-tiered internet.