The internet is loud, crowded, and most importantly: constantly under surveillance.
No, not by humans – but by bots. Automated programs that crawl the web 24/7. Some help index sites (e.g., Googlebots), but others are much more aggressive: They scan websites for vulnerabilities, look for login forms, test passwords, or collect email addresses for spam.
If you run a website or server, you might not notice it immediately. But your logs tell you otherwise: Unfamiliar IPs, strange requests, suspicious URLs.
In this article, I'll show you:
- What scanner bots do on the internet
- Why you should protect yourself against them
- And how you can implement protection practically with Fail2Ban to block these bots
What do these bots actually do?
Here are some typical examples of "malicious" bots:
- Directory Scanner – searches your site for /admin, /login, /phpmyadmin, etc.
- Login-Brute-Forcer – attempts passwords on WordPress, Mail, or FTP.
- Scraper – copies content, product data, or customer lists.
- Email Collector – gathers email addresses from legal notices or contact forms.
These bots are often not just annoying but a real security risk. Because they can:
- Generate server load
- Prepare real attack attempts
- Discover vulnerabilities
- Get your emails on spam lists
Protective measures: What can I do?
In addition to firewalls and regular software updates, there is a very effective measure to actively combat these attacks: => Detect – React – Block
You can do this using the tool Fail2Ban.
Tutorial: Blocking scanner bots with Fail2Ban
Fail2Ban is an excellent tool that monitors log files on your server and automatically blocks IP addresses when suspicious patterns are detected. Here, we focus specifically on identifying scanner bots that search for non-existent .php files.
Example: Protecting against suspicious HTTP requests (for dynamic websites)
The following tutorial checks the logfile of your web server and automatically blocks IP addresses that repeatedly search for non-existing .php files (e.g., attacks by scanners or brute force attempts).
1. Adjust NGINX Config
Add the following log format to your NGINX Config
log_format fail2banlog
In the location block:
access_log /
2. Install Fail2Ban (on Debian/Ubuntu)
First, you need to install Fail2Ban on your server. Use the following command:
sudo apt install fail2ban
After installation, ensure that the Fail2Ban daemon is running:
sudo systemctl enable fail2ban sudo systemctl start fail2ban
3. Create a filter file for NGINX Bad Bot Detection
Create a filter file to detect specific attacks, such as when non-existent .php files are requested.
sudo nano /etc/fail2ban/filter.d/nginx-bot-detection.conf
- failregex: This is the key element. It looks for requests where a .php file is requested, ending in an HTTP status 404 (not found) or 403 (forbidden). This means that a bot asks for a non-existent file and receives an error message.
- ignoreregex: Here, you can define additional requests that shouldn't be filtered, but in this case, it remains empty.
- journalmatch: Ensures that only NGINX logs are considered.
4. Create a Jail File (nginx-bot-detection)
Now create a Jail file that instructs Fail2Ban to use the filter file and automatically block IP addresses.
sudo nano /etc/fail2ban/jail.d/nginx-bot-detection.local[nginx-bot-detection] enabled =
- enabled: Activates the rule.
- port: The standard web port for HTTP and HTTPS.
- logpath: The path to the NGINX log file. Make sure to specify the correct path.
- maxretry: The number of attempts allowed within findtime (10 minutes) before the IP address is blocked.
- bantime: The duration for which an IP address is blocked (here, 1 hour).
- action: Defines the actions taken upon violation, such as notifications, IP blocks, and reporting to AbuseIPDB.
5. AbuseIPDB Configuration (optional)
If you want to use AbuseIPDB to report malicious IP addresses, you need to store your API key in the Fail2Ban configuration.
- Register on AbuseIPDB and obtain your API key.
- Edit the action.d/abuseipdb.conf file:
sudo nano /etc/fail2ban/action.d/abuseipdb.conf
6. Restart Fail2Ban
After all configurations are completed, restart Fail2Ban to apply the changes:
sudo systemctl restart fail2ban
You can check the status of Fail2Ban to ensure that everything is working correctly:
sudo fail2ban-client status
Tutorial: Ban Scanner Bots via SSH with Fail2Ban
Fail2Ban offers an excellent way to protect against brute-force attacks and scanner bots that attempt to infiltrate your server via SSH. This guide shows you how to set up Fail2Ban to monitor SSH login attempts and block IP addresses that repeatedly enter incorrect passwords.
- Install Fail2Ban (on Debian/Ubuntu)
If you haven't installed Fail2Ban yet, you can do so with the following commands:
sudo apt update sudo apt install fail2ban
Once the installation is complete, ensure that Fail2Ban starts automatically:
sudo systemctl enable fail2ban sudo systemctl start fail2ban
- Check Default SSH Fail2Ban Configuration
Fail2Ban already includes a default configuration for SSH protection, which you can adjust.
Open the file responsible for SSH monitoring:
sudo nano /etc/fail2ban/jail.local
If this file doesn't exist, create it and add the following lines:
[sshd] enabled =
Explanation of the settings:
- enabled: Activates SSH protection.
- port: Defines the port being monitored (default 22 for SSH).
- logpath: The path to the log file where failed login attempts are recorded. On most Linux servers, this is /var/log/auth.log.
- maxretry: The maximum number of failed attempts allowed before an IP address is blocked (here, 5 attempts).
- bantime: The duration for which an IP address is banned (here, 1 hour).
- findtime: The period during which maxretry failed attempts are counted (here, 10 minutes).
- Restart Fail2Ban
After adjusting the configuration, restart Fail2Ban to make the changes effective:
sudo systemctl restart fail2ban
You can check the status of Fail2Ban to ensure that SSH monitoring runs correctly:
sudo fail2ban-client status sshd
This command shows you the number of currently blocked IPs and provides an overview of the active rules.
- Additional SSH Security Measures (optional)
Besides Fail2Ban monitoring, there are other recommendations you can implement to make SSH even more secure:
a) Change the Default SSH Port
The default SSH port is 22 and is often targeted by bots. To change this, edit the file /etc/ssh/sshd_config:
sudo nano /etc/ssh/sshd_config
Change the line:
<span class="hljs-selector-id">#Port</span>
to a different number, e.g.,:
Port 2222
After saving the file, restart the SSH service:
sudo systemctl restart ssh
b) Use SSH Keys Instead of Passwords
Use SSH keys instead of passwords for logging in. This is a much more secure method and prevents brute-force attacks on your password.
Generate an SSH key:
ssh-keygen -t rsa -b 4096
Add the public key to ~/.ssh/authorized_keys on your server to disable password login.
Edit the file /etc/ssh/sshd_config and set:
PasswordAuthentication no
Restart the SSH service after the change:
sudo systemctl restart ssh
- AbuseIPDB Integration (optional)
If you want to use AbuseIPDB to report malicious IPs, you can integrate it into Fail2Ban.
- Register API Key on AbuseIPDB
- Adjust Fail2Ban Configuration by entering the API key. Edit the file /etc/fail2ban/action.d/abuseipdb.conf and add your API key.
<span class="hljs-comment"># AbuseIPDB config</span>
Add the AbuseIPDB command to the file /etc/fail2ban/jail.local:
action = %(action_mwl)s %(action_abuseipdb)s[abuseipdb_category=
- Monitor Fail2Ban Logs
To ensure Fail2Ban is working, you can monitor the log files. Fail2Ban logs all actions:
sudo tail -f /
This allows you to see in real-time which IPs are being blocked.