The Internet is Changing. And no this isn't some web3 thing.

You've heard of AI, you've heard of LLMs, and you've seen the magic of what they are capapable of, but depending on who you are, you may or may not understand where this technology can take the future of the internet. Let's dive into where it might be going and tell me what you think.

Changes Today

The two consumer apps right now that have shown us a glimpse into changes coming to the Internet are Perplexity and Arc Search. You can go take a look for yourself at what they do right now, so I'm not going to explain in detail how they work. The important detail is that they market themselves as "answer engines" as opposed to "search engines" because instead of giving you a list of results like Google, they can scrape those search results, feed them to an LLM, and generate an answer to your question every time you ask.

Arc Search even takes this a step further and not only gives you a paragraph response, but it generates a completely custom UI (user interface) on the fly! This means that the user doesn't even have think about the actual websites where the app got the information.

If you're not sure where I'm going with this then, you might be thinking that's really not that cool and barely "groundbreaking" tech. But extrapolate this trend a little further out, and we get to something big.

Where this might lead

At a high level, these apps may simply be answering the equivalent of a Google search in a slightly more efficient way. But what they are working towards is an Internet without websites. Within this new paradigm, there is no need to go directly to the New York Times to read an article, or Reddit to answer a question you had. We should just be able to ask one of these next gen answer engines and have it a) find me a collection of good sources and b) summarize them for me.

But, that's really just the beginning. Given that LLM powered applications can generate their own UI, what is even the point of me looking directly at X.com? Couldn't my personal answer engine/AI agent just get all the latest tweets on my timeline and generate a feed exactly to my liking? Why should the window in which I interact with the Internet be restricted or predetermined by some random software engineer in an office? It doesn't. Not with this new generative power of AI.

And that is exactly what the Internet is changing to. There is no longer a real need for websites to look exactly like they do and for every user to interact in a uniform way. AI agents could potentially be your daily interface on the web.

How does this affect things right now?

This future sounds interesting! However, how could my theoretical AI agent access my X account's information, or NYT subscription etc? Right now, the go to method is by scraping. Just like Perplexity and Arc. But this is incredibly unfair to the content providers. How could this be done fairly?

That is an unanswered question that has led to the NYT and several other publishers suing OpenAI. These publishers (and a lot of other businesses on the web) fundamentally rely on eyeballs on their website that lead to ad revenue. But, they know this is slowly going away, and eventually it may be impossible to distinguish agents from humans on the web.

So how can these businesses thrive given the change?

The value of a pageview is slowly trending towards 0. Some people's solution is to just try and block all non-human traffic using services like Cloudflare, and make it so that only humans can ever access the contents. That may work in the short term, but it's a cat and mouse game that is really only going to end in the scrapers winning.

But the content is still extremely valuable, there must be solution. This is exactly what we are working on at Tollbit. Our approach is to let the floodgates open (1). We are building infastructure for these content providers so that they can publish their first-party data for a small fee. Therefore, LLM applications don't have to pay for expensive scraping 3rd parties like ScrapingBee, and instead can pay directly for the content on a per access basis.

Extra Notes:

1. Why wouldn't bots keep scraping? That's a great question and we'll get to that in the next post. But believe us we have that covered.