The BidDog crawler

Who we are

BidDog is a search engine for auction lots. Our crawler identifies itself with this user agent:

The product token to use in robots.txt is BidDog.

What we collect

• Facts about each lot: title, category, location, dates, price, and lot number.
• One thumbnail image per lot.
• The link to the lot's own page, so buyers bid with you.

We do not copy full descriptions or photo galleries unless the auctioneer has given us permission. Every listing credits and links to its source.

How we crawl

• A robots.txt rule that names BidDog is always honored. We check robots.txt before every page we fetch, and we keep a copy for no more than 24 hours before reading it again. If we cannot read it because of a server error, we treat the whole site as off limits until we can.
• One exception, and it is short. Our owner has approved a short list of public listing sites that we may read past a rule written for every bot (User-agent: *). On those sites we use our real name, wait at least 15 seconds between requests, and stop at the first block. A rule that names BidDog still stops us there.
• We go slowly. We honor Crawl-delay. Without one, we wait at least 5 seconds between requests to the same site, and we back off when a site asks us to slow down.
• We prefer the front door. Feeds, sitemaps, and partner submissions come first. Apart from that short list, we crawl public lot pages only where robots.txt allows it.
• We never sneak. We do not log in, solve CAPTCHAs, rotate identities, or try to get around a block.

How to opt out

To stop our crawler, add this to your robots.txt:

It takes effect the next time we read your robots.txt, within 24 hours. To remove lots that are already in our index, or to opt out completely, use our removal form. We act within 2 business days.

Contact

Seeing something you do not expect from our crawler? Email hello@biddog.co. If you would like more buyers instead, send us your feed and we will point them to you.