The Horse's Mouth: fetch policy and contact

This page explains what our crawler does on your site and how to ask us to stop.

What The Horse's Mouth is

The Horse's Mouth is a non-commercial research API. Given a topic, it returns what the people and organizations involved actually said, as verbatim quotes from primary sources: their own transcripts, statements and remarks, not news reports about them.

We never alter the text. Every quote is checked character for character against a stored copy of the page it came from. A quote that does not match exactly is not accepted. Each quote is shown with its speaker, its date where known, and a link back to the source page.

How our fetcher behaves

• User-Agent: HorsesMouthBot/1.0 (+https://thehorsesmouth.org/; contact@thehorsesmouth.org).
• We read robots.txt before fetching from a host and follow it. We apply the rules for the token HorsesMouthBot (any case, with or without a version); if there is no group for it, we follow the * group.
• If robots.txt cannot be read (a server error, a timeout, an unexpected type or size), we fetch nothing from that host. We try again after one hour. A missing robots.txt (404 or 410) means no restrictions.
• We cache robots.txt for up to 24 hours.
• One request at a time per host, and at least 5 seconds between the end of one request and the start of the next. A larger Crawl-delay in robots.txt is honoured. If it asks for more than 60 seconds, we do not fetch from that host at all.
• We honour Retry-After on 429 and 503 responses. If it asks us to wait more than 5 minutes, we stop fetching that host for the rest of the run.
• We fetch only from a fixed list of hostnames that we have reviewed. Every redirect is checked against the same list; we follow at most 5 redirects.
• We fetch only text pages (HTML, plain text, JSON, XML), up to 10 MB each. No images, scripts or other media.
• We do not crawl the web in general or use search engines or site search. We do not collect news reporting, only pages that carry a speaker's own words.

What we store and why

We keep a copy of each page we fetch, unchanged, as evidence. This is how anyone can check that a quote matches its source exactly. Stored copies are never edited.

The API returns a quote, the speaker's passage that contains it and the passage before it as context, and the source page's address. It does not return whole pages. Each result links back to the page on your site.

Contact us, or ask us to stop

Write to contact@thehorsesmouth.org. If you ask us to stop fetching your site, we will stop. We mark the host as blocked in our list, and from then on our fetcher sends it no requests.

You can also block us in robots.txt with a group for HorsesMouthBot.