Configuring BitFire Bot Control

version 3.5.1+

BitFire Bot Control configuration guide

Cory Marsh
Cory Marsh
Cory Marsh has over 20 years Internet security experience. He is a lead developer on the BitFire project and regularly releases PHP security and programming videos on BitFire's you tube channel.
Controlling Web Robots
Controlling automated web scanners

About Web Scanners and Bots

Automated web scanners, also known as web crawlers or bots, are essential tools for many legitimate online activities, such as search engine indexing and online price comparison. However, these scanners can also be used for malicious purposes, such as scraping sensitive information, launching DDoS attacks, or even spreading malware. As a result, it is important to block automated web scanners in order to protect both personal and organizational information and systems.

In addition to scraping sensitive information, automated web scanners are used for DDoS attacks. These scanners can be used to flood a website with a large number of requests, overwhelming the server and causing the website to go offline. This can disrupt business operations and damage a company's reputation.

Automated web scanners can also be used to spread malware. These scanners can be programmed to search for vulnerabilities in websites and systems, and then exploit them to install malware or launch an attack. This can cause significant damage to both personal and organizational information and systems.

security can be complex
Controlling Automated Bots

What Is BitFire Bot Control?

The Bot Control page in the BitFire dashboard allows you to control exactly how to handle each robot. There are many thousands of automated crawlers, tools, scanners bots and many other automated systems accessing web sites on the Internet. Many of these bots are benign and helpful, some are not.

BitFire's remote API maintains a list of several thousand bots. Each of these bots has network authentication information which is used to securely identify each bot. Secure network identification identifies the source network for the request, this ensures that bots claiming to be "GoogleBot" are only allowed from Google IP addresses, and not Romania or Colombia, etc.

How Does It Work?

By default, BitFire will allow all known benign bots that have secure network authentication. Any time a new bot is seen accessing the website, BitFire authenticates the bot and if the secure auth check passes the bot is allowed.

During the learning phase in the first 5 days of operation, any bot BitFire sees will be added to the allow list from the same origin network. This feature ensures that any unknown web tool or third party service will still be allowed access once the Firewall is in full block mode.

Understanding the Bot Control Page

The bot control page contains a lot of information about each bot accessing your website. You can switch between known bots and unknown bots by selecting the "Show Known Bots" toggle switch. All bots are sorted in the order they were last seen.

Example bot list item

  1. 1: The bot icon if the bot is identified. Your first indicator the bot is not malicious
  2. 2: The bot identified bot name. Clicking on the name will take you to a webpage with additional info.
  3. 3: The bot vendor name.
  4. 4: Complete User Agent sent by the bot.
  5. 5: The first time the bot was seen, and the last time it was seen.
  6. 6: Number of invalid page requests. A high number indicates malicious.
  7. 7: Number of requests that failed network authentication.
  8. 8: Number of requests that passed network authentication.
  9. 9: Green - Validate Network Authentication. Black - blocked. Red - Allow from Anywhere.
  10. 10: The first 30 IP addresses seen using this bot's User-Agent
  11. 11: The domain networks valid for secure authentication
  12. 12: List of origin countries bot was seen from
  13. 13: Internal classification if known
  14. 14: Bot control button. Click to change this bot handling.

Using Bot Control Effectively

To effectively use the bot control provided by BitFire you should review the the known and unknown bot pages after Firewall learning is complete. Take a moment to go through the list. Make sure that any third party tools or services that you use are set to either "Authenticated" or "ANY IP". Most bots and services can be restricted to authenticated IPS and networks, but some services may not have static networking and must be set to "ANY IP".

Be careful when configuring any bot to allow "ANY IP" as this will allow anyone anywhere on the planet to use this User-Agent to bypass your bot filtering. The more well known the bot (think GoogleBot, Bing, etc) the more likely setting this to "ANY IP" could result in abuse. If is very unlikely for an attacker to set their User-Agent to "Bob's Unkown Third Party Tool" in hopes of bypassing Firewall Filtering. The most likely case for abuse is popular search engines.

Allowed Bots Are Still Firewalled

It is important to note, that even if a bot is allowed in Bot Control, the BitFire firewalling features are still run. This includes, XSS, SQLi, CSRF, SSRF, XXE, and other firewall features. Bot Control is intended to block scanners looking for unknown or difficult to detect abuse from completing their scans.

Cory Marsh
Cory Marsh
Cory Marsh has over 20 years Internet security experience. He is a lead developer on the BitFire project and regularly releases PHP security and programming videos on BitFire's you tube channel.