BotwatBot, the Botwat crawler

What it is

BotwatBot reads public pages of a business’s own website so Botwat can suggest facts such as services, hours, contact details and prices to that business’s owner. It also checks the website’s health. The owner confirms every suggestion; nothing it reads is published automatically.

Its user agent is always BotwatBot/1.0 (+https://botwat.com/bot).

How it behaves

• It identifies itself honestly. It never spoofs a browser, uses proxies or solves CAPTCHAs.
• It fetches one page at a time per site, 2 to 5 seconds apart, and honours Crawl-delay.
• It obeys robots.txt (RFC 9309). If robots.txt cannot be fetched because of a server error, it assumes everything is disallowed.
• It fetches at most 100 pages a month from the site of a Botwat customer who has agreed, and uses conditional requests to avoid re-downloading unchanged pages. Looking at competitor sites is switched off today; if it is ever switched on, the limit is 5 to 10 pages per site.
• It only follows links and sitemap URLs on http and https. It never fetches Google properties, logins or private network addresses.
• It stops for 30 days when a site answers 401, 403 or with a CAPTCHA.
• If robots.txt sends Content-Signal: ai-input=no, Botwat extracts only structured data such as JSON-LD, with no AI processing.
• What it finds is only ever proposed to the owner of a Botwat business. Nothing becomes a fact until the owner confirms it.

Control it with robots.txt

Add a rule for the BotwatBot user agent to your /robots.txt. Examples:

# Block BotwatBot everywhere
User-agent: BotwatBot
Disallow: /

# Or block only part of the site, and slow it down
User-agent: BotwatBot
Disallow: /private/
Crawl-delay: 10

Rules are cached for 24 hours.

Opt out

Enter your site’s host name and BotwatBot stops fetching it at once. An opt-out is never re-tested. Anyone can use this form, so to make sure your opt-out stays, show that the domain is yours: after you send the form, it shows a TXT record to add to your domain’s DNS; add it, then send the form again. Botwat staff can remove an opt-out that isn’t verified this way, for example when a business shows that the site is its own. An opt-out that staff removed can be filed again only with the TXT record in place. Using robots.txt works too, and is the quickest way if you can edit that file.

Contact

Questions or problems with BotwatBot: support@botwat.com. See also our privacy page.