Resources requested
The submitted public page, its redirect destination when applicable, /robots.txt, /sitemap.xml and a favicon fallback. Each check is a lightweight public GET request.
Crawler transparency
About TotalSiteDataBot.
TotalSiteDataBot powers the user-initiated website health check. It requests a small number of public resources only when someone submits a URL; it is not a continuous web crawler.
Identification
Requests identify themselves with this user agent:
TotalSiteDataBot/1.0 (+https://totalsitedata.com/crawler)
Safety limits
The service limits request time, response size and redirects. It accepts standard public HTTP or HTTPS targets and blocks credentials, unusual ports, local hostnames and private, loopback, link-local or reserved network addresses.
Request frequency
Scans are user initiated and rate limited. The service does not schedule recurring crawls from the free checker or follow arbitrary page links.
robots.txt
The scanner reports whether a root robots file is available. Because the check is an on-demand diagnostic requested by a user, that file is not treated as proof that the requester owns the site or as an authentication mechanism.
Block or report requests
A site owner can block the exact user agent at the web server, content delivery network or web application firewall. Returning 403 to TotalSiteDataBot prevents a successful scan.
If you believe the crawler is causing a problem or a URL should not be checked, email taskifiai@gmail.com with the affected domain, approximate request time and relevant server-log details. Do not send passwords or private credentials.
Results and limitations
The crawler analyses only public responses and does not log into websites, submit forms, bypass access controls or access private analytics and Search Console accounts. Read the full scoring and limitation details before relying on a result.
Read the methodology