Bots cannot access web server: 403 Forbidden

Symptoms

• Search bots cannot access the web server with 403 Forbidden error. The following could be found in /var/www/vhosts/system/example.com/logs/access_log :

  CONFIG_TEXT: 203.0.113.2 - - [30/Aug/2017:17:56:55 +0000] "GET /robots.txt HTTP/1.0" 403 1034 "-" "-"
  203.0.113.2 - - [30/Aug/2017:17:57:03 +0000] "GET / HTTP/1.0" 403 5138 "-" "RPT-HTTPClient/0.3-3E"
  203.0.113.2 - - [30/Aug/2017:17:57:03 +0000] "GET / HTTP/1.0" 403 5138 "-" "RPT-HTTPClient/0.3-3E"

• The following messages could be found in /var/www/vhosts/system/example.com/logs/error_log :

  CONFIG_TEXT: [:error] [pid 8462:tid 140078479243008] [client 203.0.113.2] ModSecurity: Access denied with code 403 (phase 2). Operator EQ matched 0 at REQUEST_HEADERS. [file "/etc/httpd/conf/modsecurity.d/rules/modsecurity_crs-plesk/modsecurity_crs_21_protocol_anomalies.conf"] [line "47"] [id "960015"] [rev "1"] [msg "Request Missing an Accept Header"] [severity "NOTICE"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/MISSING_HEADER_ACCEPT"] [tag "WASCTC/WASC-21"] [tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"] [hostname "example.com"] [uri "/robots.txt"] [unique_id "WabvP8f6VUiHLUaNwppktgAAABI"]

• ModSecurity with OWASP rule set is installed in Plesk at Tools & Settings > Web Application Firewall (ModSecurity).

Cause

The following OWASP ModSecurity rules are triggered on search bots and block their activity:

• OWASP_CRS/PROTOCOL_VIOLATION/MISSING_HEADER_ACCEPT with ID 960015
• OWASP_CRS/PROTOCOL_VIOLATION/MISSING_HEADER_UA with ID 960009

Resolution

The OWASP rule set is very restrictive and thus might block some functions, such as file sharing, webmail, and some web applications.

As workaround, disable these rules by TAG or ID at Tools & Settings > Web Application Firewall (ModSecurity) > Switch off security rules: