OrbitalSEOBot

What it is

OrbitalSEOBot is the in-house SEO tool of Orbital Web Studio. It reads public web pages for two reasons:

• To check the technical health of our own website and the websites of the clients we look after: broken links, redirects, page titles, structured data, speed, accessibility.
• To keep a short list of other businesses’ public websites in view, once a week, so we can see how the market is moving: new pages, changed titles, changed prices that are printed on the page.

What it reads, and what it keeps

It reads robots.txt, your sitemap and public pages. It never logs in, never fills in a form, never touches a cart or a checkout, and never reads anything robots.txt disallows.

It keeps facts about a page, not the page: the title, the headings, the word count, a fingerprint of the text, the kinds of structured data, the number of links, and prices that are written on the page. We do not copy your text and we do not republish it. We use these facts to see what changed from one week to the next.

How politely it behaves

• It reads robots.txt first, at the start of every visit, and follows it, including Crawl-delay. A rule for OrbitalSEOBot wins over the rule for everyone (*).
• One request at a time. On websites that are not ours it waits at least 3 seconds between requests, usually 4, and visits a few hundred pages at most, once a week.
• It asks before it downloads: pages that have not changed since last time answer with a tiny “not modified”, and pages whose sitemap date has not changed since a recent visit are not requested at all.
• If your server answers with “too many requests”, “forbidden” or a challenge page, it stops, does not come back for at least a week, and waits longer if it is turned away again. It does not retry around a block. It does not rotate addresses, pretend to be a browser, or try to solve a CAPTCHA.
• On our own websites it may load a handful of pages in a headless browser to check that they work with JavaScript and are accessible. It uses the same name and does not load images, fonts or video.

Slow it down or stop it

Add this to the robots.txt of your site. It takes effect at the bot’s next visit, because it re-reads robots.txt every time.

User-agent: OrbitalSEOBot
Disallow: /

Or just ask it to go slower, or to skip a folder

Crawl-delay is in seconds. Disallow works for any path.

User-agent: OrbitalSEOBot
Crawl-delay: 30
Disallow: /private/

A block is a block: if you turn the bot away at the firewall instead of robots.txt, it backs off and our team is told.

Other public data we use

To see which websites link to which, we also use the public link graph published by Common Crawl. We download their files; we do not request your pages for it.