Articles in this section

How to block custom bots on a Plesk server?

Taras Ermoshin
Updated
kb: how-to Plesk for Linux

Applicable to:

  • Plesk for Linux

Question

How to block custom bots on Plesk server?

Answer

It is possible to add the required bots into Fail2ban "apache-badbot" jail:

  1. Log in the server via SSH

  2. Create backup of apache-badbots.conf file:

    # cp -p /etc/fail2ban/filter.d/apache-badbots.conf /root/apache-badbots.conf_back

  3. Edit /etc/fail2ban/filter.d/apache-badbots.conf file and add the required bot.
    In the example below ClaudeBot is added:

Fail2ban will start blocking access from both Nginx and Apache. The settings can be checked at Tools & Settings > IP Address Banning (Fail2Ban) > Jails > plesk-apache-badbot:

CONFIG_TEXT: [plesk-apache-badbot]
enabled = true
filter = apache-badbots
action = iptables-multiport[name=BadBots, port="http,https,7080,7081"]
logpath = /var/www/vhosts/system/*/logs/*access*log
/var/log/httpd/*access_log
maxretry = 1

To ensure the jail is working, check Tools & Settings > IP Address Banning (Fail2Ban) > Banned IP addresses or fail2ban log /var/log/fail2ban.log.

 

Some custom bots are not blocked in certain cases using this method, the failregex can be modified for them, see the article https://support.plesk.com/hc/en-us/articles/29089393480343.

Also, the particular User-Agent can be added rather than modifying the failregex as was advised in third-line https://webpros.atlassian.net/browse/PPS-17230?focusedCommentId=1497685:

add the modified user agent to the badbots = section, as it appears in actual requests:

|\(compatible; Bytespider; spider-feedback@bytedance\.com\)|

Was this article helpful?

Comments

5 comments
Date Votes
  • Avatar
    Ron Nieskens

    Hello,

    I entered the above code.

    However I get an error message.

    What am I doing wrong?

    Does anyone know another solution?

    ------

    Error: Unable to add the jail filter: f2bmng failed: ERROR:__main__:File contains no section headers.
    file: '<stdin>', line: 1
    'badbotscustom = SputnikBot|omgili|socialmediascanner|Jooblebot|SeznamBot|Scrapy|CCBot|linkfluence|veoozbot|Leikibot'.

    ------

    Thanx

    0
  • Avatar
    Thomas Jarvis

    Please add a copy and past line for badbotscustom = examplebot|examplebot| to the explanation - Not just a screenshot

    0
  • Avatar
    Anton Kuznetsov

    Hello Thomas,

    This article describes where users can put their own custom values. The ones on the screenshot are just an example and should not be copied.

    0
  • Avatar
    Andy Bird

    this entry already exists in Plesk.. you just need to edit it

     

    0
  • Avatar
    Reinaldo Rodriguez

    Does not work, even the bytedance and bytespider are in the jail filter, still receive massive crawl..

    0

Please sign in to leave a comment.