About our bot

What it fetches

One page, the one a visitor typed in, plus a handful of documents about it: your robots.txt, your sitemap, llms.txt if you have one, one deliberately non-existent address to see how your site answers a missing page, your favicon, and the http and www variants of your address to see where they lead. Nine requests at most, once, and never more than a couple of megabytes. It follows at most five redirects and does not crawl: no links are followed, no images are downloaded, and a sitemap index is counted, not read.

The visitor also asks Google's PageSpeed Insights to load the same page, which arrives from Google's own crawler, not from us.

Why

Every check is started by a person who wants to know how a site is doing, usually their own. The same address is not fetched again for 24 hours however many people ask, and no site is fetched more than a few times a day from here.

How to turn it away

Add these two lines to your robots.txt and the check will stop before it reads anything else, and tell the visitor that the site's owner asked us not to look:

User-agent: HarriHealthCheck
Disallow: /

A general User-agent: * block is not treated as an opt-out, because that is the normal state of a staging site and this is a single, human-started fetch rather than a crawl; it is reported as a finding instead.

Reports

A report is visible only to people who have its link, is marked so that search engines and AI assistants do not index or quote it, and is deleted after 90 days. If a report about your site exists and you would like it removed, or you would like your site excluded altogether, email [email protected] and we will do it by hand.

The tool is run by Harri Digital, a product studio in Plymouth. Back to the health check.