On the evening of 20 September 2026, people using Meta’s new personal agent Muse on Amazon were met by a pop-up: “Continued access by an unauthorized AI agent violates Amazon’s Conditions of Use.” Banks, airlines, Yelp and a growing part of the web are now turning AI agents away, and access to search results and news sites is shrinking too.
The web is not closing. It is getting a door policy. Agents will increasingly have to identify themselves, pay or use a separate entrance. That leaves every organisation with two decisions it can no longer leave to chance: how its own agents get access to the outside world, and what its website lets in. This article explains why the doors are closing, what US courts have and have not settled, where European rules come in, and what an agent policy should cover.
A third kind of visitor the web was not built for
The web has long had two kinds of visitor. People get pages, ads and a shopping basket. Bots get robots.txt, firewalls and CAPTCHAs. Personal AI agents fit neither box. They act for one person, but they click, read and compare at machine speed.
Those agents have arrived. Meta launched Muse on 8 September 2026, OpenAI has Dots, Perplexity builds agents into its Comet browser, xAI offers Grok Bot teammates that log into your apps, and Anthropic is testing Claude Duet, which keeps working while you sleep. According to MIT Technology Review, Muse was downloaded 5 million times in its first month and agent traffic on the web tripled from August to September 2026, mostly towards online shops. Over 2025 as a whole, security firm HUMAN measured agent traffic growth of more than 7,800% in its State of AI Traffic report.
The research for this article ran into the same wall. Our own research agent was shown nothing but a bot check that makes the browser solve a computing puzzle on one site, and was refused outright by another. That is the problem in miniature: the agent cannot tell whether it was blocked on purpose or by accident, and the person behind it usually never finds out.
How high the wall already is
For its State of Web Access 2026, Zyte audited 11,100 popular landing pages. Only 18.5% had no barrier at all. A web application firewall sits on 92.4% of pages, a CAPTCHA on 22.5%, and half use two or more barriers at once. In robots.txt, 12.7% of sites name AI crawlers in a disallow rule and 27.2% shut out every bot with a wildcard. Together that is nearly four in ten sites. Just 1.8% explicitly allow AI crawlers.
News media go further. In Zyte’s data, 64% of newspapers block at least one AI crawler by name. Of 70 major publishers that FT Strategies examined in July 2026, 60 restrict at least one AI crawler, while all 70 still let Googlebot in.
Four reasons the doors are closing
The old bargain no longer holds
Search engines and websites had a simple deal: Google reads your pages and sends you visitors in return. AI crawlers read far more than they send back. According to figures from Cloudflare Radar, Anthropic fetched almost 2,000 pages per referral in July 2026 and OpenAI around 250, against roughly 5 for Google. The ratios are rough, because apps do not send a referrer, but the direction is clear.
Cloudflare, which handles an estimated fifth of web traffic, acted on that. Since July 2025 it has blocked AI crawlers by default for new domains. Since July 2026 it sorts bots into three categories, Search, Agent and Training, and since 15 September it blocks Agent and Training bots by default on ad-carrying pages of new domains. Dual-purpose bots such as Googlebot and Bingbot are caught by that block when a site switches off training. Plenty of organisations therefore already have an agent policy they never chose.
Agents do not look at ads
An agent looking for a product skips the sponsored results. In its lawsuit against Perplexity, Amazon argues that agents make it hard to know whether an ad was seen by a human. According to TechCrunch, Yelp allows no non-human traffic unless the agent maker pays through its data licensing programme. For banks the incentive is different: Apollo economist Torsten Slok warns that banks could lose a large share of their cheap deposits if agents automatically move idle cash to the best savings rate. Android Headlines puts the cost at tens of billions in extra interest.
An agent looks like an intruder
An agent that logs in, clicks through pages and fills in forms behaves like a bot taking over accounts. Amazon accuses Muse of not identifying itself, holding on to login credentials and scraping account data. Meta had said at launch that Muse has no visibility into passwords or payment methods, but did not respond to the other claims. JPMorgan Chase, Bank of America and Citigroup keep automated visitors out of customer accounts altogether, Android Headlines reports. Sometimes it is simply a malfunction: Walmart, a Muse partner, called failed purchases unintentional and pointed to a “verify you’re human” check that throws the agent out.
Trust is eroding because not every AI bot follows the rules either. According to TollBit, about 15% of AI page-fetching agents in Europe reached URLs that robots.txt disallowed in the first half of 2026. OpenAI’s documentation says robots.txt rules “may not apply” to ChatGPT-User, the bot that fetches pages when you ask a question, because a person starts the request. Anthropic says all its bots respect robots.txt.
Platforms want their own agents
Amazon blocks Muse while building its own agents, such as Buy for Me, which shops on other brands’ sites and does identify itself there. Search shows the same pattern. Microsoft retired the Bing Search APIs on 11 August 2025 and points developers to Grounding with Bing in Azure, whose results cannot be used directly in other applications. On 20 January 2026 Google announced the end of its Custom Search JSON API. New customers can no longer sign up, and it switches off for existing customers on 1 January 2027. Full-web results then require a partner agreement, according to Parallel. Any organisation building an agent that searches the web will pay more, or depend on a handful of suppliers.
What US courts have and have not settled
The key case is Amazon against Perplexity. A federal court in California barred the Comet agent from Amazon.com in March 2026. On 4 August the Ninth Circuit vacated that injunction, reasoning that Comet is “a tool, not a person”, so the user, not Perplexity, accesses Amazon. The court refused a rehearing on 10 September, and on 21 September Amazon filed an amended complaint. It alleges that the iOS version of Comet copied the user’s session cookie to Perplexity’s servers, and that Comet ran at least 185,712 sessions on Amazon, The Next Web reports.
This was a preliminary ruling. The court stressed that it was not creating a new legal regime for agents, and it left open Amazon’s ability to regulate access through its terms of use. Anti-hacking law looks like a weak tool against agents running on a user’s own device, but the contract between a site and its users is becoming the main battleground, as law firm Pearl Cohen sums it up.
The picture around search results is just as mixed. A judge dismissed Google’s DMCA case against scraping service SerpApi on 20 July 2026. Reddit’s case against SerpApi and Perplexity, which allegedly collected Reddit content through Google results, was allowed to proceed in a ruling of 31 July. As far as we could find, no European court has yet ruled on a personal agent in a comparable case.
Why this matters more than it looks
For organisations that are rolling out agents, the first effect is quiet failure. An agent that cannot open a site does not always say so. It picks another source, returns half an answer or fails to complete a purchase. Users often cannot tell whether a site blocks their agent on purpose or whether an anti-bot system stepped in by accident, TechCrunch notes.
The second effect is on the quality of what agents read. Researchers Nicolas Steinacker-Olsztyn, Devashish Gosain and Ha Dao found that 60% of reputable news sites refuse at least one AI crawler, against 9.1% of misinformation sites. A research or procurement agent that searches the open web reads mostly what is left open. For market research, due diligence or competitor monitoring, that is a bias worth knowing about.
The third effect lands on customers. Stricter bot checks catch real people too: one shopper posted on X that Walmart’s attempt to stop agent purchases locked him out as a human. According to MIT Technology Review, Ethan Mollick of Wharton expects blocking to spread across the web, because agents sent to request refunds or negotiate prices could swamp customer service. He also expects people to grow “very annoyed” when their agents are not allowed to do what they can do themselves.
The European angle
Europe has no law on personal agents yet, but three existing frameworks shape the field.
- The AI Act covers crawling for training. Under Article 53, providers of general-purpose AI models need a policy to respect rights holders’ opt-outs. Signatories of the GPAI Code of Practice commit to respecting robots.txt, and the European Commission is working with stakeholders towards an agreed list of machine-readable opt-out protocols that the AI Office will use to assess compliance. These rules concern training, not an agent shopping on your behalf.
- The DMA watches gatekeepers. In its April 2026 review, the Commission said it monitors AI within designated platform services as a priority, noted that, under competition law, it is pursuing interim measures to reverse Meta’s exclusion of third-party AI assistants from WhatsApp, and will assess whether some AI services should be designated. It sees no need to amend the DMA for now. Whether a gatekeeper may lock out third-party agents is an open question the Commission is clearly watching.
- Open banking is the precedent. Europe has done this once before. Under PSD2, banks must give licensed third parties access to accounts through an interface, and those parties must identify themselves to the bank, as laid down in the regulatory technical standards. The follow-up package, PSD3 and the Payment Services Regulation, reached political agreement in November 2025 and tightens requirements on those interfaces, with application expected around late 2027, Morrison Foerster writes. An agent that wants to move your savings in Europe will most likely have to come through that licensed door, not by clicking through the bank’s website.
The open-banking model is also a hint of where the wider web may go: identified third parties, a dedicated entrance and clear liability. That is roughly what the industry is now building for agents.
What an agent policy should cover
Most organisations sit on both sides of the door. They deploy agents, and they run websites that agents visit. A short policy covers both.
| Side | Question | What to decide |
|---|---|---|
| Your agents | How do they reach outside data? | Prefer official connectors, APIs and partnerships over agents clicking through websites. Budget for paid search grounding now that free search APIs are disappearing. |
| Your agents | Whose credentials do they use? | Never let an agent log in with staff credentials where a site’s terms forbid automated access. Large retailers explicitly ban unauthorised agents. |
| Your agents | What do they not see? | Make agents report which sources they could not open, so blocked sources do not silently skew research. |
| Your website | What is the current setting? | Check your CDN and robots.txt. Behind Cloudflare, agents may already be blocked on ad pages without anyone deciding it. |
| Your website | Training, search or action? | Treat training crawlers, search bots and customer agents as three different visitors, per section of your site. Many publishers block GPTBot but allow OAI-SearchBot. |
| Your website | Do you want agents as customers? | Shops can offer agents a controlled entrance, such as Shopify’s WebMCP checkout or protocols like Google’s Universal Commerce Protocol. |
| Both | Who owns the decision? | Give one owner the call across marketing, security, legal and e-commerce, and review it every quarter while the standards settle. |
Where this is heading
Access for agents is unlikely to end up simply open or closed. Three layers are growing side by side.
An ID for the agent. The core problem is that a site cannot see who is knocking. With Web Bot Auth, an agent signs each request cryptographically so a site can verify its origin. Visa built its Trusted Agent Protocol on it with Cloudflare in October 2025, and Mastercard and American Express are working with Cloudflare on the same technique. In early October 2026, Meta and Sierra announced the Personal Agent Protocol, backed by Walmart, Shopify, Stripe and others. An agent enters as a guest, for instance to check stock, and signs in with OAuth for account tasks. The customer decides what the agent may do, the business which actions it allows. OpenAI and Anthropic are not on the list so far.
A toll per request. The status code HTTP 402, “Payment Required”, existed for decades without real use. Cloudflare said in August 2025 that it already sent over a billion of them a day, and in July 2026 it announced a Monetization Gateway that lets sites, datasets and APIs charge agents per request, Cloudmagazin reports. The payment side is still largely in beta.
A separate entrance. Since 28 September 2026, Shopify lets browser agents read and complete an open checkout through WebMCP once the buyer confirms. In Shopify’s own test, 60 of 60 purchases succeeded this way against 56 with ordinary browser automation, in 10.3 rather than 27.4 seconds on average.
These layers solve the agent’s problem at a price. MIT Technology Review warns of a web that cares less about ordinary visitors, with heavier checks to prove you are human and prices that may differ per agent. Our expectation for the next twelve months: more blocking first, then a wave of agreements. Large platforms will sign partnerships and choose which agents get in. Smaller sites will decide through the default settings of their CDN or shop platform. And because agents increasingly carry company context into these interactions, the point we made in models are rented, context is owned applies here too: who controls the door controls what your agents can learn and do.
Frequently asked questions
Can a website legally block my AI agent?
In practice, yes. A site decides through its terms and its technology who may enter. In the United States, the Ninth Circuit found in a preliminary ruling that Amazon is unlikely to prove Perplexity itself accessed its site, because the agent runs on the user’s device. The court explicitly left open that sites regulate access through their terms. As far as we could find, no comparable European ruling exists yet.
What is the difference between an AI crawler and an AI agent?
A crawler fetches pages in bulk to train a model or fill a search index. An agent works for one user and carries out a task, such as comparing prices, booking a table or buying something. Since July 2026, Cloudflare sorts bots into Search, Agent and Training.
Does the EU AI Act stop AI companies from ignoring robots.txt?
For training, the AI Act requires providers of general-purpose models to respect rights holders’ opt-outs, and the GPAI Code of Practice commits signatories to respecting robots.txt. It does not regulate agents that visit a site on a user’s behalf.
Should my web shop let AI agents in?
It depends on your model. If you live on advertising, agents bring little. If you sell products, an agent may bring a customer. Then give agents a controlled entrance, such as an agent checkout or an official integration, rather than letting them click through your regular pages.
The open web was a default, now it is a decision
On the early web, the winners opened their doors to search engines. On the agent web, the winners will know who is knocking, on whose behalf and with what mandate. That applies to the agents you send out as much as to the ones you let in. An organisation without an agent policy still has one. It just belongs to its CDN.
