Guides
The FlockStack-Briefings fetcher
How to recognize it
It sends this user agent on every request:
FlockStack-Briefings/1.0 (+https://flockstack.io/docs/briefings-bot)In robots.txt it follows the group for FlockStack-Briefings, else the one for FlockStack, else the one for every agent (*). It sends no cookies and logs in nowhere, and it never presents itself as a browser or as another program.
What it reads
Only pages that a FlockStack user's briefing found: a result of a search for the briefing's topic, a story in a feed the briefing reads, or a page linked from one of those. It does not crawl a site, follow its links beyond one step, or keep a copy of a page after the briefing's run: it keeps the facts it needs for a short summary, at most 25 words of quotation, and the link to the page.
It reads at most 1 MB of a page, waits at most 8 seconds, makes at most 2 requests to one host at a time and at most 300 requests in one run, and only to public addresses over https.
What it never does
It never works around a paywall, a login or a block: no other user agents, no cookies, no headless browser, no rotating proxies, no cached or archived copies. A page that answers 401, 403 or 429 is not asked again another way, and a page behind a paywall is not used beyond what it shows to everyone.
How to opt out
Disallow it in your robots.txt. The fetcher reads the file before its first request to a host and keeps what it read for up to 24 hours.
User-agent: FlockStack-Briefings
Disallow: /A narrower rule, such as Disallow: /members/, keeps it out of those paths only. When robots.txt can't be read (a server error or no answer), the fetcher stays away from the host for an hour. A newsletter owner can also block a host for their own briefing.
Contact
For a question or a problem with the fetcher, write to hello@flockstack.io with the address it requested and the time, and we will look into it.