Crawler
TroyIndexBot
How Troy Index reads dealer sites, and how to keep it off yours.
TroyIndexBot is the crawler behind troyindex.com. It reads the public product pages of the dealers we track, records the asking price and the conditions attached to it, and leaves. It never logs in, never adds to a cart, and never reads a page a site’s robots.txt asks it not to.
How it identifies itself
Every request carries this user agent:
TroyIndexBot/0.1 (+https://troyindex.com/bot; contact: crawl@troyindex.com)How it behaves
It reads robots.txt before it schedules a single request against a site, and treats it as policy, not advice: a Disallow for TroyIndexBot, or for every crawler, is honoured, and a source whose product pages are disallowed is excluded from crawling rather than crawled anyway. A published Crawl-delay is a hard ceiling on how fast we read that site, never a floor we crawl up to. Within a site, requests go out one at a time, spaced out by that ceiling. Product pages we treat as busy are re-read every 5 minutes; everything else, every hour.
To keep it off your site
Add this to robots.txt:
User-agent: TroyIndexBot
Disallow: /The next crawl reads it and stops. If you would rather we did not read a site at all, or you think the crawler is misbehaving, write to crawl@troyindex.com with the site and the times you saw it. We stop first and ask questions after.