Bots

Bot detection is deliberately aggressive: a user agent is a bot as soon as one matcher says so, unless it is a known exception.

The lists
Matchers
Extending the lists
The corpus
Data sources

The lists

319named botsbots.yml, keyed by a lowercase substring of the user agent

6search enginessearch_engines.yml: Googlebot, Bingbot, Baidu, DuckDuckBot, Slurp, Ask Jeeves

4exceptionsbot_exceptions.yml: never a bot, whatever a matcher says

browser.bot.name is the name from bots.yml, or Generic Bot when another matcher caught it. The lists are in the repository.

Matchers

Matcher	A bot when
`Browser::Bot::EmptyUserAgentMatcher`	The user agent is empty.
`Browser::Bot::KnownBotsMatcher`	The user agent contains a key of `bots.yml`.
`Browser::Bot::KeywordMatcher`	The user agent matches `/crawl\|fetch\|search\|monitoring\|spider\|bot/`.

browser.bot.why? returns the matcher that fired. A matcher is any callable taking the lowercased user agent and a browser:

Browser::Bot.matchers << ->(ua, _browser) { ua.match?(/externalhit/i) }

To start from nothing, Browser::Bot.matchers.clear; to get the built-in ones back, Browser::Bot.matchers += Browser::Bot.default_matchers. To restore browser 2’s detection, remove the keyword and empty user agent matchers:

Browser::Bot.matchers.delete(Browser::Bot::KeywordMatcher)
Browser::Bot.matchers.delete(Browser::Bot::EmptyUserAgentMatcher)

Extending the lists

Browser::Bot.bots.merge!(new_bots_hash)
Browser::Bot.bot_exceptions += new_exceptions
Browser::Bot.search_engines.merge!(new_search_engines_hash)

The prefix-trie regexp behind each list is rebuilt when the list grows or is replaced. To add a bot to BrowserX itself, add its user agent to test/ua_bots.yml and its name to bots.yml; a search engine also goes in test/ua_search_engines.yml and search_engines.yml.

A trap. Public crawler lists call “bot” anything that is not a human browser: crawlers, but also test drivers (Playwright, Puppeteer, Selenium), scripts and scanners. Adding them all to bot? would flag the system tests of the apps using it, so BrowserX keeps them out of it.

The corpus

corpus/user_agents.txt holds 20,328 distinct user agents. A change that is not meant to change an answer proves it there: one script writes every answer of a gem as JSON lines, another diffs two snapshots.

ruby -I path/to/upstream/lib script/corpus_snapshot.rb > /tmp/upstream.jsonl
ruby -I lib script/corpus_snapshot.rb > /tmp/browserx.jsonl
ruby script/corpus_diff.rb /tmp/upstream.jsonl /tmp/browserx.jsonl

path/to/upstream is a plain clone of fnando/browser at dd88933. A change meant to change answers, like a fresher bot list, lists the differences it expects, and the diff must show exactly those.

Data sources

The corpus gathers, deduplicated, the user agents of:

the test cases of ua-parser/uap-core, Apache License 2.0;
the instances of monperrus/crawler-user-agents, MIT;
the test fixtures of fnando/browser, MIT.

The bot list is meant to be kept fresh from permissively licensed sources only: crawler-user-agents and ai.robots.txt, both MIT. Details in corpus/SOURCES.md.