Symptoms
Starting from May 26, 2023 Plesk domains are not accessible in a browser with either the 403 Forbidden error or with the 123 Apache Test Page.
Cause
With the update of the aum package provided by Atomicorp, the Apache module mod_evasive was installed, and this module false-positively blacklisted the server's IP address - errors like below are found in the log file /var/log/messages or in the Apache log:
CONFIG_TEXT: mod_evasive[177717]: Blacklisting address 192.0.2.2: possible DoS attack
The issue is resolved in the further update of the aum package 6.0.48-29386.
Resolution
-
Connect to the server using SSH.
-
Update
aumto the latest version and remove the Apache modulemod_evasive:# aum -uf ; yum -y remove mod_evasive ; aum -u
As a workaround, apply one of the following:
-
Connect to the server using SSH.
-
Open the configuration file
/etc/httpd/conf.d/mod_evasive.conffor editing. For example, use the vi text editor:# vi /etc/httpd/conf.d/mod_evasive.conf
-
Scroll to the bottom of the file and uncomment the
DOSWhitelistdirectives, for example:
-
Before:
#DOSWhitelist 127.0.0.1
#DOSWhitelist 192.0.2.2 -
After:
DOSWhitelist 127.0.0.1
DOSWhitelist 192.0.2.2Note: Replace the IP address
192.0.2.2with the exact IP address of the server that is present in the error message from/var/log/messages.
-
Save changes and close the file.
-
Restart Apache:
# service httpd restart
-
Connect to the server using SSH.
-
Execute the following command to disable
mod_evasive:# sed -i -e '/LoadModule evasive20_module/s/^/#/g' /etc/httpd/conf.d/mod_evasive.conf
-
Restart Apache:
# service httpd restart
When the issue with the aum package is fixed on the Atomicorp side, to re-enable the module execute the below command
# sed -i '/#LoadModule evasive20_module/s/^#//g' /etc/httpd/conf.d/mod_evasive.conf
And restart Apache to apply changes:
# service httpd restart
Comments
4 comments
Server's external IP also have to be whitelisted (DOSWhitelist xxx.xxx.xxx.xxx) to resolve the issue.
Yes exactly. We also struggled with this until we saw in the httpd service status message that our external IP has been banned by mod_evasive.
Edit: a hint has been added to article now, thank you.
Same here, spent almost two hours trying to find out what was going on. Fortunately this article now exists, great job on posting it so quickly! Although too late for us others will find this very useful. Do not forget to also add the servers IPv6 address if applicable. The external/resolvable servers IP address(es).
No one changed at thing and suddenly this morning customers were facing intermittent downtime. Via mentioned line coming by colored in red (is why it stood out) on a journalctl tail I ended up reading about mod_evasive at https://www.howtogeek.com/devops/how-to-configure-mod_evasive-for-apache-ddos-protection/ and trying to fix it by adding the servers IP addresses to this DOSWhitelist, and that solved the incident.
I purposely have auto updates of both Plesk and OS packages disabled, to have the control myself on a set interval and try to prevent surprises like this from happening. So find this out about the automatic installation of these OS package was quite irritating causing this incident:
...
May 24 03:40:36 Updated: 1:aum-6.0.48-29078.el7.art.x86_64
May 25 03:33:42 Updated: 1:aum-6.0.48-29360.el7.art.x86_64
May 26 03:55:27 Updated: 1:aum-6.0.48-29375.el7.art.x86_64
May 26 03:56:12 Installed: mod_evasive-1.10.1-22.el7.x86_64
May 26 03:56:15 Installed: mod_qos-11.70-1.el7.x86_64
...
Apperently the WAF is resposible for that.
We have the same problem. I have disable the Fail2Ban and atomicorp WAF but the problem still exists.
I have whitelist the IP of the server restart the httpd service and waiting forward to see that is OK.
Please sign in to leave a comment.