| AI crawlers are automated programs that read the pages on your website so an AI system can train on them, index them for search, or quote them in an answer. They arrive without a browser, and most of them name themselves in your server logs. |
A Timaru wholesaler found its website slow every weekday morning.
Nothing had changed on the site. The product catalogue was the same size it had been in March and nobody had added a plugin.
The host said the plan had gone over its resource limit and offered a larger one for another $70 a month. The firm took it and the mornings got better.
Six weeks later the site was slow again.
The office manager finally opened the server log. Of roughly 27,000 requests that month, about 24,000 carried a bot name and 3,000 came from people.
Half the machine traffic was one training crawler working through a 900-page product catalogue every few days. It had been doing that since February.
The figures below are illustrative. That firm employs 31 people and sells to trade customers across the lower South Island.
The larger hosting plan cost $840 a year. A bot-mitigation add-on the host sold it three months later cost another $1,140 a year.
Machine traffic was costing $1,980 a year in hosting before anybody had made a decision about it.
The one-off sat separately. The web agency billed $600 to work out why the site was slow, and it reached the wrong answer twice before the log was opened at all.
The hosting bill was only half of it. The same traffic decides whether that wholesaler turns up when a customer asks an AI assistant to name a supplier in the South Island, and nobody there had considered that at all.
Work your own version out from your hosting control panel and your own hourly rates. It takes half an hour and the answer will be specific to your site.
None of this was the crawlers’ fault. The site had never told them what they could take.
Are AI Crawlers Already Reading Your Website?
Yes. Any page a search engine can reach is being read by AI crawlers, and on many small business sites the machines now outnumber the people.
You will recognise the shape of this if your website has ever been described as fine but slow. The pages load and the forms work, and the whole thing runs a second behind where it used to be.
Every request leaves a line in a log, and each line carries a user agent string describing what made it. A browser names itself Chrome or Safari.
A crawler names itself too. GPTBot, ClaudeBot, PerplexityBot, Bytespider, Amazonbot and CCBot all announce who they are, usually with a URL pointing at a page that explains the bot.
Ask whoever holds your hosting account for last month’s access log, or for the traffic report behind it. Most control panels list the top user agents on a single screen.
This runs in the opposite direction from an AI policy for staff. Our post on what your people may do with AI tools covers information leaving your business. Everything below covers other people’s software arriving at your website.
How do you tell a bot from a person in your logs?
Look for three signals together: a bot name in the user agent field, a request rate no person could produce, and a source address that resolves back to the operator.
Google publishes reverse DNS hostnames for its crawlers and OpenAI publishes the address ranges each of its bots uses, so a name in a log can be verified. Anything claiming to be a named crawler from an address the operator does not publish is something else.
Start with volume. A single user agent making thousands of requests a month is worth naming before anything gets blocked.
Why does bot traffic keep growing on a site that never changes?
Bot traffic grows because new crawlers keep appearing and existing ones keep re-reading pages they have already seen.
A model trained in April holds no memory of your March prices, so the same catalogue gets fetched again. Sites with many similar URLs, such as product filters and paginated listings, get fetched hardest of all.
What Types of AI Bots Crawl a Business Website?
Three kinds, and each deserves a different answer: crawlers that collect pages for model training, crawlers that index pages for an AI search product, and agents that fetch one page live while somebody waits for an answer.
Training crawlers work through a site systematically. GPTBot from OpenAI, ClaudeBot from Anthropic, CCBot from the Common Crawl project and Bytespider from ByteDance all belong in this group, along with the Google-Extended and Applebot-Extended tokens.
Search crawlers build an index that an AI product answers from. OAI-SearchBot, Claude-SearchBot and PerplexityBot belong here, and so does the ordinary Googlebot that has crawled your site for years.
Live retrieval agents are the newest of the three. ChatGPT-User and Claude-User fetch a page because a person has just asked a question your page might answer.
The operators draw the same lines themselves. OpenAI’s crawler documentation separates GPTBot for training from OAI-SearchBot for search, and names ChatGPT-User for pages fetched at a user’s request. Anthropic’s crawler notes split their three bots the same way.
What is a live retrieval agent?
A live retrieval agent is a single fetch made while a person waits for an answer. It reads the page as it stands that minute, including a price you changed yesterday.
The difference matters commercially. A training crawler may help your business appear in an answer in two years, and a retrieval agent can put your phone number in front of somebody this afternoon.
Blocking the retrieval agent in order to stop the training crawler is a common and expensive mistake.
Which crawler names should a business recognise?
Recognising them in a log is enough, because a robots.txt rule has to name the bot exactly:
- GPTBot and OAI-SearchBot, from OpenAI
- ClaudeBot, Claude-SearchBot and Claude-User, from Anthropic
- Google-Extended, a robots.txt token covering Gemini training and grounding, with no crawler name of its own
- PerplexityBot and Perplexity-User, from Perplexity
- Bytespider, from ByteDance
- CCBot, from the Common Crawl Foundation, whose public archive many models are trained on
Anything on that list can be addressed by name in a robots.txt file. Unnamed traffic has to be handled at the network edge, which the section on rate limiting covers.
Should You Block AI Crawlers From Your Website?
No. Blocking everything takes your business out of the AI answers people now get before they ever reach a search results page. The three kinds of crawler carry three different trade-offs and deserve three separate decisions.
The commercial question starts with where your enquiries come from. A trade supplier whose customers ring after reading a specification page has an obvious interest in being quoted accurately.
A firm whose value comes from original written work has the opposite interest, and its case for blocking a training crawler is much stronger.
This is a different question from your own data going into a model. Our post on putting business information into AI tools covers that side, and this one stays on the pages you have already published to the public.
| What visits | What it does with your pages | What blocking it costs you |
|---|---|---|
| Training crawler: GPTBot, ClaudeBot, CCBot | Collects pages that may train a future model | Little today, and possibly less presence in the models people ask in two years |
| Google-Extended | Controls Gemini training and grounding on your content | No effect on Google Search inclusion or ranking |
| AI search crawler: OAI-SearchBot, PerplexityBot | Indexes pages so an AI product can cite them | Your site stops appearing in ChatGPT and Perplexity answers |
| Live retrieval agent: ChatGPT-User, Claude-User | Fetches one page while a person waits for an answer | You lose the visit that was about to become an enquiry |
| Googlebot | Crawls the site for Google Search | Your site drops out of Google altogether |
| Unnamed high-volume traffic | Takes bandwidth and returns nothing | Nothing at all |
What do you lose by blocking an AI search crawler?
You lose the chance to be the business an AI answer names.
Somebody asking a chatbot for a supplier in Christchurch gets a short answer with a couple of sources under it. A site that has disallowed the search crawler cannot be one of them.
That referral costs nothing and it comes from a buyer who has already narrowed the field. Weigh it against whatever you think a training crawler is taking.
When is blocking a training crawler worth it?
When the writing on your site is the product you sell.
A consultancy that publishes detailed technical guidance, a photographer with a portfolio, or a publisher whose archive is licensed all have something to protect. Most trade and service businesses do not.
A blocked training crawler still leaves the search crawlers and the retrieval agents working, so the business keeps appearing in AI answers.
Do AI answers actually send traffic back?
They send some traffic back, and less of it per appearance than a search result sends.
An AI answer resolves the question on the page it is shown on, so fewer readers click through than would from a search listing. The ones who do click arrive further along, because they have already read what you do and decided you look right.
Measure it in enquiries and quotes. A flatter analytics chart alongside the same number of quotes going out is the outcome to expect.
How Do You Block AI Bots With robots.txt?
You block an AI bot by naming it in the robots.txt file at the root of your domain and disallowing the paths it may not read.
The file is plain text and sits at a single address, yourdomain.co.nz/robots.txt. Anybody can read yours right now, including your competitors, so nothing confidential belongs in it.
The format is a published standard. RFC 9309 put the Robots Exclusion Protocol, shortened to REP, on the IETF standards track in September 2022, after nearly thirty years as the informal convention Martijn Koster wrote in 1994. Every serious operator reads the same file, so one set of rules covers all of them.
A robots.txt file has nothing to do with your DNS records. Our post on the registrar and nameservers behind your domain covers that layer, and a crawler rule works above it, inside the website itself.
Most content management systems generate a robots.txt file on their own, and the generated version rarely names an AI crawler. WordPress serves a virtual file until somebody writes a real one, and a plugin can overwrite whatever is there on its next update.
Check yours after any plugin or platform change. Nothing warns you when a rule disappears, so put that check on the same list as the update itself.
What does a robots.txt rule for one bot look like?
Three lines. A user-agent line naming the bot, a disallow line carrying a single forward slash, and a blank line before the next group.
Naming a bot specifically overrides the general rule. A group addressed to GPTBot applies to GPTBot alone, and the wildcard group applies to everything with no group of its own.
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: /
That example opts out of model training and stays eligible to be cited in AI search. Most trade and service businesses want that setting. A bot left out of the file altogether is allowed by default, so the last two groups are there to make the intent obvious to whoever reads the file next.
Anthropic also supports crawl-delay, a non-standard line asking a bot to slow down. Several operators honour it and several ignore it, so set it without depending on it.
How long does a robots.txt change take to work?
About a day for most operators.
RFC 9309 tells crawlers to avoid using a cached copy of a robots.txt file for more than 24 hours. OpenAI puts roughly the same figure on a search opt-out reaching its own systems.
Confirm the file loads before you start waiting. A robots.txt that returns a server error is treated as a full disallow by a compliant crawler, which can drop a site out of search results by accident.
Do All AI Bots Respect the robots.txt File?
No, although most of the named ones do. The file is a request, and the standard says so directly. RFC 9309 states that the rules in a robots.txt file “are not a form of access authorization”.
Two kinds of traffic get past it, and only one of them is misbehaving.
The first is expected. A fetch triggered by a person, such as ChatGPT-User following a link somebody pasted into a chat window, may not be treated as an automatic crawl at all, and OpenAI records that the robots.txt rules may not apply to it.
The second is scraping that never identifies itself. It arrives carrying a browser user agent from a residential address and looks like ordinary traffic in a log.
There is no robots.txt line for a bot that does not read robots.txt.
What actually stops a crawler that ignores the rules?
Rate limiting at the edge, with a firewall rule behind it if that fails.
A content delivery network or web application firewall sits in front of your site and can cap requests per address per minute. Most hosting plans include a basic version and most owners have never turned it on.
Set the rules first and watch what obeys them. Rate-limit whatever keeps going after that.
What Does AI Bot Traffic Cost a Business Website?
AI crawlers cost a business website in hosting resources and in staff time, and on the Timaru example that came to $1,980 a year before anybody made a decision.
Bandwidth and origin load are consumed whether or not a fetch ever produces a visit. A catalogue page costs the same to serve to a machine as it does to a customer.
Shared hosting meters this as a resource limit, and the remedy the host offers is usually a bigger plan.
Staff time is the other half of the bill. Hours go on investigating the slowdown and on approving an upgrade the office manager could not explain, and none of those hours are booked against the website.
The pattern repeats because nothing was fixed. The upgrade removes the symptom. The crawler carries on at the same rate.
Does setting crawler rules lower the hosting bill?
Usually yes, and the size of the drop follows how much of your traffic was machines.
The wholesaler above went back to its original plan and kept a $25 a month edge tier for rate limiting. Recurring cost fell from $1,980 a year to $300.
The recurring saving is $1,680 a year. The one-off sits beside it and does not net off, because setting the rules costs about $600 on top of the $600 already spent on the wrong diagnosis.
Price it against your own hosting invoice. If your plan was upgraded in the last year and customer traffic has not risen, the log will show what it was actually serving.
How much of a business site’s traffic is machine traffic?
Often more than half on a small site with a large catalogue.
AI bot traffic concentrates where there are many URLs. A brochure site of eight pages sees very little of it, and a 900-product catalogue with filters sees a great deal.
Count yours before deciding anything. Your host can pull the figure from last month’s log.
Does Blocking AI Crawlers Affect Your Google Ranking?
No, provided you block the right token.
Google’s crawler documentation states that Google-Extended is a standalone product token covering training and grounding for its Gemini models, and that it does not affect a site’s inclusion in Google Search or work as a ranking signal.
Disallowing Googlebot is a different act with a different result. That rule removes the site from Google Search, so the two tokens are worth keeping straight.
Check yours by opening your robots.txt in a browser and reading every group. A single misplaced disallow under the wildcard group reaches every crawler that has no group of its own.
How Do You Set an AI Crawler Policy for Your Website?
Start with a month of logs, decide one line per bot type, then write the file and watch what changes.
- Get last month’s access log from whoever holds the hosting account, or the top user agent report from the control panel.
- List every user agent making more than 100 requests in the month, and put a name and an owner to each.
- Decide one line for each group. Search crawlers and retrieval agents carry commercial value on most business sites, and the training crawlers are the only real decision on the list.
- Write the robots.txt file with one group per bot, then open yourdomain.co.nz/robots.txt in a browser and confirm it loads.
- Turn on rate limiting at the edge for traffic that will not identify itself, starting with the free tier your host already includes.
- Read the log again in a month and compare the request count against the enquiries that came in.
None of that needs new software and most of it takes an afternoon.
Write down what you decided and why. The next person to open the file will otherwise treat a blocked crawler as a mistake and undo it.
Review it twice a year. New crawlers appear faster than most businesses update a text file, and a policy written in 2024 is missing most of the names above.
Keep the file with the rest of the website paperwork. Whoever holds the hosting login is the person who can change it, and our post on who holds the keys to your website hosting covers that ownership question.
The Timaru wholesaler paid $1,980 a year and $600 in fees for traffic it had never agreed to serve. The rules that fixed it fitted into eleven lines of text.
Do You Know What Your Website Is Serving to Machines?
Exodesk has supported South Island businesses since 1989 and works with clients across Canterbury, Otago and Southland from offices in Christchurch and Dunedin. Our AI solutions team reads a month of your logs and writes the crawler rules with you.
We set the file so your business still turns up in AI answers, and we cap the traffic you are paying to serve for nothing.
Contact us today to discuss how we can help your business or connect with us on LinkedIn to stay updated with more insights.
Frequently Asked Questions
What is an AI crawler?
An AI crawler is software run by an artificial intelligence company to fetch web pages at scale. Some collect material for model training and some build an index for an AI search product, and a third group fetches a single page while a user waits. Every one of them identifies itself by name in the user agent field of the request.
Is it worth blocking AI crawlers on a small business website?
Only some of them, and only for some businesses. A trade supplier or a service firm usually gains from being read, because an AI answer naming your business is a referral you did not pay for. A business whose published writing is the product it sells has a stronger case for blocking the training crawlers. The search crawlers and the retrieval agents are worth keeping in almost every case.
What is a robots.txt file?
A robots.txt file is a plain text file at the root of a website telling automated visitors which parts of the site they may read. It holds one group per crawler, and each group names a user agent and lists the allow and disallow paths that apply to it.
Does a robots.txt file stop every bot?
No. Compliant operators honour it, and the standard behind it describes the rules as a request with no enforcement attached. Traffic that hides behind a browser user agent has to be handled at the network edge with rate limiting.
Will blocking GPTBot remove my business from ChatGPT?
No, although it will reduce what a model learns about you over time. GPTBot collects pages for model training, while OAI-SearchBot is the crawler behind ChatGPT search results and ChatGPT-User is the agent that fetches a page when somebody asks a question. Disallowing all three takes a business out of ChatGPT. Most businesses block the first and keep the other two.
What is Google-Extended?
Google-Extended is a robots.txt token controlling whether Google may use content it has already crawled to train its Gemini models and to ground them, which means supplying pages to the model at the moment somebody asks it a question. Google states that it has no effect on a site’s inclusion in Google Search and does not work as a ranking signal. It has no separate user agent string, so it appears in your robots.txt file and nowhere in your server logs.
Can AI bot traffic push a website over its hosting limit?
Yes, and it is a common reason for an unexplained hosting upgrade. Crawlers consume bandwidth and origin processing whether or not a fetch ever produces a visit, and a large catalogue with filtered URLs multiplies the number of pages available to fetch. Ask your host which resource actually went over before agreeing to a bigger plan.
How do I find out which bots are visiting my site?
Ask your hosting provider for last month’s access log, or open the traffic report in your hosting control panel and sort by user agent. Anything appearing thousands of times in a month with a name ending in Bot is worth identifying. Most operators publish a page explaining what their crawler does and how to address it. Reverse DNS and published address ranges confirm that a bot is who it says it is.
Is llms.txt the same thing as robots.txt?
No. The llms.txt proposal, and the ai.txt proposal alongside it, point an AI system at a plain text summary or a usage policy, and neither has a standards body behind it. No major operator treats it as an access control, so a business wanting to control crawling still writes robots.txt.
Can a business charge AI companies for crawling its website?
Some content delivery networks now offer per-request charging or outright blocking for AI bots, aimed mainly at large publishers with heavy traffic. For a typical New Zealand small business the saving comes from the traffic it stops serving, and any per-request revenue would be too small to chase.
Do AI crawlers read pages behind a login?
No. A crawler reads what an anonymous visitor can reach, so anything behind a password stays out of its way unless credentials leak. Pages left publicly reachable by accident carry the real risk here, because a page with no link pointing at it is still readable to anything that finds the URL.
Does Exodesk set AI crawler rules for New Zealand businesses?
Yes. Exodesk works with businesses across Canterbury, Otago and Southland from offices in Christchurch and Dunedin, and an AI crawler policy is part of the website work we do. We have operated in the South Island since 1989 and work alongside whoever hosts or maintains your site.
NEXT STEP
How much of your bandwidth went to bots last month?
Your robots.txt file, a month of server logs and the rate limiting on your hosting plan decide how much of your site is served to machines, and whether your business turns up when a customer asks an AI assistant for a supplier. An IT assessment reads all three, alongside the rest of your setup.
Or read more about our managed IT services.
