CRAWLER INFORMATION

Meet OutThereBot.

OutThereBot discovers public event information for HappenHop. It visits an explicit list of event websites, reads robots.txt before crawling, and stays within each configured website origin.

It runs one crawl at a time, waits at least three seconds between pages (longer when Crawl-delay requires it), and requests no more than ten pages per run. Cities rotate through a daily crawl queue, with at most two website jobs per scheduler tick and a shared five-minute website cooldown. It stops on access denials, rate limits, server errors, and bot challenges.

It identifies itself as OutThereBot. It does not use rotating identities, bypass logins, or solve access challenges. Event pages link to their original sources.

Controlling access

Use robots.txt rules for the OutThereBot product token. For example, this stops automatic event-page crawling on the next policy check:

User-agent: OutThereBot
Disallow: /

The crawler also respects noindex and nofollow directives. Temporary server failures prevent crawling rather than granting access.