Bots cannot access web server: 403 Forbidden

Symptoms

1. Search bots cannot access the web server with 403 Forbidden error. The following could be found in /var/www/vhosts/system/example.com/logs/access_log :

   PLESK_INFO: 162.249.110.67 - - [30/Aug/2017:17:56:55 +0000] "GET /robots.txt HTTP/1.0" 403 1034 "-" "-"
   182.18.161.42 - - [30/Aug/2017:17:57:03 +0000] "GET / HTTP/1.0" 403 5138 "-" "RPT-HTTPClient/0.3-3E"
   182.18.161.42 - - [30/Aug/2017:17:57:03 +0000] "GET / HTTP/1.0" 403 5138 "-" "RPT-HTTPClient/0.3-3E"

2. The following messages could be found in /var/www/vhosts/system/example.com/logs/error_log :

   PLESK_INFO: [Wed Aug 30 17:00:47.547061 2017] [:error] [pid 8462:tid 140078479243008] [client 203.0.113.2] ModSecurity: Access denied with code 403 (phase 2). Operator EQ matched 0 at REQUEST_HEADERS. [file "/etc/httpd/conf/modsecurity.d/rules/modsecurity_crs-plesk/modsecurity_crs_21_protocol_anomalies.conf"] [line "47"] [id "960015"] [rev "1"] [msg "Request Missing an Accept Header"] [severity "NOTICE"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/MISSING_HEADER_ACCEPT"] [tag "WASCTC/WASC-21"] [tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"] [hostname "example.com"] [uri "/robots.txt"] [unique_id "WabvP8f6VUiHLUaNwppktgAAABI"]

   or

   PLESK_INFO: [Wed Aug 30 17:38:51.982853 2017] [:error] [pid 15030:tid 140493958645504] [client 203.0.113.2] ModSecurity: Access denied with code 403 (phase 2). Operator EQ matched 0 at REQUEST_HEADERS. [file "/etc/httpd/conf/modsecurity.d/rules/modsecurity_crs-plesk/modsecurity_crs_21_protocol_anomalies.conf"] [line "66"] [id "960009"] [rev "1"] [msg "Request Missing a User Agent Header"] [severity "NOTICE"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/MISSING_HEADER_UA"] [tag "WASCTC/WASC-21"] [tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"] [hostname "example.com"] [uri "/robots.txt"] [unique_id "Wab4K9Lm6o6udYNyzupTVAAAAIc"]

3. ModSecurity with OWASP rule set is installed in Plesk

Cause

The following OWASP ModSecurity rules are triggered on search bots and block their activity:

• OWASP_CRS/PROTOCOL_VIOLATION/MISSING_HEADER_ACCEPT with ID 960015
• OWASP_CRS/PROTOCOL_VIOLATION/MISSING_HEADER_UA with ID 960009

Resolution

The OWASP rule set is very restrictive and thus might block some functions, such as file sharing, webmail, and some web applications.

As workaround, disable these rules by TAG or ID at Tools & Settings > Web Application Firewall (ModSecurity) > Switch off security rules: