| AI crawlers fetch website content for uses such as model training and AI search. Other agents retrieve pages when a person asks for information. Some operators offer separate controls for these uses, so your business can make a different decision about each one. |
Here is a hypothetical. A wholesaler with a large online catalogue finds its website slow on weekday mornings, and the host suggests a bigger plan. Nobody has looked at the logs yet.
When somebody does, a share of the morning requests comes from automated clients working through product filter pages. Some carry crawler names; others are unverified. The upgrade would have treated the symptom without anyone deciding what those clients should be allowed to fetch.
Start with your hosting or CDN logs. Identify which requests are verified crawlers, which pages they fetch and whether they create measurable load. Then agree the content you want accessible, update the relevant rules and check the effect. Allowing access can preserve eligibility for search and retrieval; it cannot promise a citation or an enquiry.
Are AI Crawlers Already Reading Your Website?
Public pages may be fetched by AI crawlers, and your logs show which ones visit. Confirm activity from the logs rather than assuming it.
Every request leaves a line in a log, with a user agent string describing what made it. A browser names itself Chrome or Safari, and AI crawlers such as GPTBot, ClaudeBot and PerplexityBot name themselves too. User agents are self-reported, though: some operators render pages like a browser, and a browser-like user agent is not proof of a human visit.
This runs in the opposite direction from an AI policy for staff. Our guide to what your people may do with AI tools covers information leaving your business. Everything below covers automated clients arriving at your website.
How do you tell a bot from a person in your logs?
Look at the user agent, the request pattern and the source address together. Where an operator supports it, verify important named crawlers using its current published method, such as published IP ranges or a reverse DNS lookup followed by forward confirmation.
A bot-like name, high volume or reverse DNS alone is not conclusive. Missing identity evidence means the traffic is unverified, not automatically proof of impersonation.
Why does crawler traffic keep coming back?
Crawlers revisit pages to refresh what they hold, and new crawlers appear over time. Sites with many similar URLs, such as product filters and paginated listings, create far more pages to fetch than a short brochure site.
Training data can include older versions of a page. Whether an AI answer shows your current price depends on retrieval and the source it returns, not on a training schedule.
What Types of AI Crawlers Visit a Business Website?
A practical grouping has three kinds: AI crawlers that collect pages for model training, crawlers that index pages for an AI search product, and agents that fetch a page when a person asks for something. Some controls are usage tokens rather than crawlers, and operator behaviour differs.
| Crawler or control | Purpose and effect |
|---|---|
| GPTBot | Training collection. Blocking it is separate from ChatGPT search controls |
| OAI-SearchBot | ChatGPT search. Opted-out sites are excluded from search answers but may still appear as navigational links |
| ChatGPT-User | Certain user-triggered actions. robots.txt may not apply, and it is not the search inclusion control |
| ClaudeBot, Claude-SearchBot, Claude-User | Separate training, search and user-requested access. Anthropic says its bots honour robots.txt |
| PerplexityBot, Perplexity-User | Search crawling versus user-triggered fetching. Perplexity says the user fetcher generally ignores robots.txt |
| Google-Extended | Usage token for specified Gemini training and grounding, not a separate crawler |
| Applebot-Extended | Training-use control for content Applebot crawls, not a separate webpage crawler |
| CCBot | Common Crawl archive collection, used by many organisations rather than one AI company |
| Googlebot | Google Search crawling, including access relevant to AI features in Search |
The operators document these lines themselves. See the guidance from OpenAI, Anthropic, Perplexity, Google, Apple and Common Crawl, and check it again before changing your rules.
What is a user-requested fetch?
It is a single fetch made because a person asked a question or shared a link, rather than a scheduled crawl. It reads the page as it stands at that moment. Some operators state that robots.txt may not apply to these fetches, so they need separate thought from training and search crawlers.
Should You Block AI Crawlers From Your Website?
Decide by purpose, content value and measured impact rather than blocking or allowing everything. The three kinds of AI crawlers carry different trade-offs and deserve separate decisions.
Allowing the appropriate search crawler can keep your pages eligible for that product’s search features. Results still depend on indexing, relevance and the product’s selection process. Blocking training access is a separate choice where the operator supports it, and no crawler setting can promise inclusion, citation, traffic or sales.
A business whose published writing is the product it sells has a stronger case for restricting training collection. A trade or service business that wants to be found may choose to keep search access open. Either way, measure the result in logs and enquiries.
This is a different question from your own data going into a model. Our guide to putting business information into AI tools covers that side, and this guide stays on the pages you have already published.
Do AI answers send traffic back?
Some do, but an answer can name a business without sending a visit, and a visit need not become a lead. Some visits arrive without a reliable referrer. Track identifiable referrals and enquiries in your analytics, separately from crawler volume.
Unidentified traffic is not automatically worthless either. Check whether it belongs to monitoring, integrations, link previews or other legitimate services before deciding what to do with it.
How Do You Block AI Crawlers With robots.txt?
Name each crawler you want to restrict in the robots.txt file at the root of your domain, and merge the new groups into your existing policy. The file is public, so nothing confidential belongs in it.
The format is a published standard. RFC 9309 defines the Robots Exclusion Protocol, and the AI crawlers that follow it read the same file as search crawlers.
A group naming a specific bot takes precedence over the wildcard group. A crawler with its own group ignores the User-agent: * rules, so any path exclusions you rely on must be repeated in that group. A bot without its own group falls under the wildcard group.
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: /
Illustrative additions for three named collectors. Preserve the existing wildcard group, sitemap and any bot-specific path rules, and review search access separately. These lines do not erase previously collected content, and robots.txt is not an access-control system.
Google-Extended covers grounding as well as specified training uses, so weigh that trade-off before adding it. Applebot-Extended is another separate usage token and does not appear as its own crawling traffic. Anthropic also documents support for Crawl-delay, a non-standard line that other operators may ignore.
Platforms differ in how robots.txt is generated and edited. Check the live file after platform or plugin changes rather than assuming it stays the same. Our domain name management service covers DNS, which sits below this layer.
How long does a robots.txt change take to work?
A robots.txt refresh, a crawl, an index update and a changed AI answer are different events. RFC 9309 recommends that crawlers do not use a cached copy for more than 24 hours unless the file is unreachable, and OpenAI describes roughly 24 hours for search-setting changes to take effect in its systems.
Make sure the file loads before you wait. RFC 9309 treats an unavailable file differently from a server error, and Google documents a temporary crawl pause and fallback handling when robots.txt returns a server error. A server error does not automatically drop a site from search results, but it is worth fixing promptly.
Do All AI Crawlers Respect the robots.txt File?
No. Compliant operators follow it, and RFC 9309 says its rules are not a form of access authorisation.
Two kinds of traffic get past it. User-triggered fetchers can have documented exceptions: OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it. Other traffic does not identify itself and looks like an ordinary browser in a log.
Protect authenticated content with actual access controls, not crawler preferences.
What stops AI crawlers that ignore the rules?
Edge controls on a content delivery network or web application firewall can rate-limit or block traffic, based on observed behaviour and verified identity. Start with logging or a limited rollout where the platform supports it.
Test normal browsing, forms, search crawlers and important integrations, and keep a rollback route. Edge controls are still subject to false positives, distributed traffic and configuration limits, and they do not stop every scraper.
What if a site you need blocks bots?
Respect it. Tools can use permitted sources, licensed data and available search results, and access to a particular page may remain unavailable. Bypassing another site’s restrictions is not a fix.
What Do AI Crawlers Cost a Business Website?
AI crawlers may consume CPU, bandwidth or application capacity. A cached response may not reach the origin server, and a fixed hosting plan may not become cheaper after traffic falls, so measure load before expecting savings.
Look for bursts, costly paths, errors and resource impact rather than a request-count threshold. A thousand cached requests can cost less than a few expensive dynamic requests. Review monitoring, payment callbacks, site search and legitimate shared-address traffic before enforcing limits.
Check whether your current plan already includes the controls, logs and rule capacity you need. Scope and cost depend on the platform and the workload.
How can you monitor AI crawler traffic without buying a tool?
Start with the reports and access logs your host or CDN already provides. Look for user agent, time, URL, status, bytes and source address, and verify important named AI crawlers using the operator’s current published method.
If requests are served at the CDN, origin logs alone can miss them. Ask about log access, retention and any charges before buying another tool.
Does Blocking AI Crawlers Affect Your Google Ranking?
Not if you use the right control. Google states that Google-Extended does not affect inclusion in Google Search or work as a ranking signal.
Google AI Overviews and AI Mode are part of Search. Google explains that Googlebot access and its supported preview controls, such as nosnippet, govern how pages are used there, and that no special AI text file or schema is required. Google-Extended is not a separate opt-out for those Search features.
Disallowing Googlebot is a different act. It blocks crawling, but Google can still list a URL it discovers through links. Where removal from results is intended, use an appropriate indexing control that the crawler can read, or restrict access through authentication.
Check yours by reading every group in your robots.txt. A disallow under the wildcard group reaches every crawler that has no group of its own.
How Do You Set an AI Crawler Policy for Your Website?
Start from logs, decide by purpose, then deploy and check the complete policy.
Gather the logs
Get host or CDN reports and access logs, and note which systems serve which requests.
Verify the important crawlers
Check named crawlers against current operator guidance and mark the rest as unverified.
Decide by purpose
Choose training, search and user-fetch settings separately, and record why.
Draft and merge the rules
Keep a copy of the current file, merge new groups and agree a rollback.
Deploy and test
Check each hostname, the HTTP response, effective rules for sample paths, and any CDN or WAF controls.
Recheck after changes
Compare logs soon after deployment and after platform changes, with a periodic review as well.
A robots allowance does not override an edge block or a browser challenge, so check robots.txt and CDN or WAF settings together. Write down what you decided, so the next person to open the file does not undo a deliberate rule.
Get Help With Your Website Crawler Settings
Exodesk has supported New Zealand businesses since 1989. Today, our Christchurch and Dunedin teams support businesses nationwide. Through our website hosting and maintenance service, we work with your host or web agency to review crawler activity, agree the rules and implement them, with log access and monitoring scope confirmed first. Our AI solutions page covers the wider AI work we do.
NEXT STEP
Do you know what your website serves to bots?
Talk to Exodesk about your crawler settings. We review the logs your host or CDN already keeps, agree which access to allow by purpose, and implement and check the rules with your host or web agency.
Or read more about our AI solutions.
Frequently Asked Questions
What is an AI crawler?
An AI crawler automatically fetches web content for uses such as model training or AI search. Some agents instead fetch pages when a person asks for information. Many identify themselves in requests, but a name alone does not verify identity.
Is it worth blocking AI crawlers on a small business website?
Decide by purpose, content value and measured impact. Some businesses allow search access while restricting training collection. Keep useful access where it serves your goals, then assess load and enquiries rather than assuming either benefit or harm.
What is a robots.txt file?
It is a public file at the root of a site that gives supported automated clients crawling instructions. Groups identify user agents and specify allowed or disallowed paths. A bot without its own matching group can fall under the wildcard group. It does not secure confidential content.
Does a robots.txt file stop all AI crawlers?
No. It relies on the client following applicable rules. Some user-triggered fetchers have documented exceptions. Use authentication for private content and suitable edge controls for unwanted traffic, checking that legitimate requests still work.
Will blocking GPTBot remove my business from ChatGPT?
Blocking GPTBot addresses training collection, separately from ChatGPT search. OAI-SearchBot controls search access. That choice does not decide or remove every possible mention of a business, so check current OpenAI guidance before changing the rules.
What is Google-Extended?
It is a robots.txt usage token for specified Gemini training and grounding uses of Google-crawled content. It has no separate request user agent and does not affect Google Search inclusion or ranking. Google AI Overviews and AI Mode use Search controls instead.
Can AI crawlers push a website over its hosting limit?
They can contribute to resource limits, especially on expensive dynamic pages. Check origin load, CDN caching, errors and the actual hosting limit before attributing a slowdown or buying a larger plan. Lower request counts do not automatically reduce a fixed bill.
How do I find out which AI crawlers are visiting my site?
Start with existing host or CDN reports and logs. Identify request patterns and verify important crawler identities against current operator guidance. Check both edge and origin records where available. User-agent names and request totals alone cannot distinguish every bot from a person.
Is llms.txt the same thing as robots.txt?
No. llms.txt is a proposal for providing information and links that help agents use a website. It is not an access-control replacement. Google does not require an AI text file or special schema for its Search AI features.
Can a business charge AI companies for crawling its website?
Some providers offer charging mechanisms, such as the Pay Per Crawl feature Cloudflare documents, subject to its setup and participation requirements. Revenue depends on participating demand and content value, so it is not assured. Check availability and costs before building a business case.
Do AI crawlers read pages behind a login?
Ordinary unauthenticated crawlers cannot read content protected by correctly enforced authentication. A user-authorised agent or connector may access material using granted permissions. Check access controls and public copies, and do not rely on an unlinked URL to keep information private.
Does Exodesk set AI crawler rules for New Zealand businesses?
Yes. Exodesk can work with your host or web agency to review crawler activity and implement agreed settings. Confirm log access, platform support, responsibilities and monitoring scope. Crawler settings do not decide whether your business appears in AI answers.
Contact us or call 0800 396 337 to talk about your website crawler settings.
