AdintelBot
If you found this address in your server logs, this page says what AdintelBot is, how often it comes and how to stop it.
What is AdintelBot?
AdintelBot is the crawler Adintel uses to check that a web page still opens. It requests the pages that Google Ads accounts connected to Adintel send people to: the addresses set on their ads, sitelinks, asset groups and keywords. Apart from each site’s robots.txt and any redirect on the same site, it visits no other page, and it follows no link on a page.
How does it identify itself?
Every request carries this user agent:
Mozilla/5.0 (compatible; AdintelBot/1.0; +https://adintel.io/bot)How often does it visit?
Each page at most once a day, usually in the morning UTC. A page it did not reach in the morning may be tried at about 13:00 UTC. On any one site it asks for one page at a time and waits at least five seconds before the next. If the site’s robots.txt sets a longer crawl delay, it waits that long, up to thirty seconds. If a site answers 429, answers 503 with a Retry-After header, or shows a check for bots, AdintelBot asks that site for nothing more until the next day.
Does it respect robots.txt?
Yes. It reads a host’s robots.txt before it asks for any page there, and follows the rules written for AdintelBot, or those for every crawler when none names it. If robots.txt cannot be read, it asks for no page on that host. To stop it visiting your site, add this to your robots.txt:
User-agent: AdintelBot
Disallow: /It still reads robots.txt each day, so if you remove the rule later, it sees that by the next morning.
What does it do with a page?
It records the status code, how long the answer took and, for a redirect that leaves the site or sends a page to the home page, where it led. It follows a redirect within the same site, and stops at one that leaves the site or sends a page to the home page, or after five redirects. It reads at most the first 64 KB of a page, only to tell a page from a check for bots, and keeps none of it. It runs no scripts, loads no images, sends no cookies and keeps none, and it submits no forms.
How do I reach you?
Write to support@adintel.io. Tell us which site, and we will look at it.