Unable to send mail out to certain domain with Qmail: SSL_routines: SSL3_CHECK_CERT_AND_ALGORITHM: dh_key_too_small

Symptoms

1. Unable to send mail out from Plesk to certain domain with Qmail.
2. Errors in /var/log/maillog :

   CONFIG_TEXT: sendmail[5616]: STARTTLS=client: 5616:error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small:s3_clnt.c:2429:

   CONFIG_TEXT: qmail: delivery deferral: TLS_connect_failed:_error:14082174:SSL_routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh_key_too_small;_connected_to_203.0.113.2/

Cause

Different security settings on the servers: destination mail server has a Diffie-Hellman key with size less than 768 bit, but recent versions of OpenSSL in Plesk consider such keys as insecure.

Resolution

Use either of the following solutions:

• Switch to Postfix
• If Qmail still needed, contact administrators of the destination mail server in order to update the keys to the more secure ones.
• Log in to Plesk server using SSH and allow it to connect to non-secure remote mail servers:

  Warning: this will decrease the server security and might be used only in case of emergency! 

  click here to view the options

  Option I: Add the remote mail server to trusted hosts list in Qmail:

  # mkdir -p /var/qmail/control/notlshosts
  # touch /var/qmail/control/notlshosts/mail.example.com

  Note: Qmail will start sending messages without TLS encryption to such domains.

  Option II: Contact system administrator to downgrade openssl package. 

  Option III: Disable DH keys exchange for Qmail outgoing connections to destination mail servers:

  # echo 'DEFAULT:!DH' > /var/qmail/control/tlsclientciphers

  Note: this solution affects connections to all mail servers and potentially can lead to connection problems to some servers.

  Restart Qmail to apply the changes:

  # service qmail restart