Troy Index

Crawler

TroyIndexBot

How Troy Index reads dealer sites, and how to keep it off yours.

TroyIndexBot is the crawler behind troyindex.com. It reads the public product pages of the dealers we track, records the asking price and the conditions attached to it, and leaves. It does not log in or use a cart, and it skips every page a site’s robots.txt asks it to stay off.

How it identifies itself

Every request carries this user agent:

TroyIndexBot/0.1 (+https://troyindex.com/bot; contact: crawl@troyindex.com)

How it behaves

It reads robots.txt before it schedules a single request against a site. A Disallow for TroyIndexBot, or for every crawler, is honoured, and a source whose product pages are disallowed is excluded from crawling. A published Crawl-delay caps how fast we read that site. Within a site, requests go out one at a time, spaced out by that delay. Product pages we treat as busy are re-read every 5 minutes; everything else, every hour.

To keep it off your site

Add this to robots.txt:

User-agent: TroyIndexBot
Disallow: /

The next crawl reads it and stops. If you would rather we did not read your site at all, or you think the crawler is misbehaving, write to crawl@troyindex.com with the site and the times you saw it.