Unable to send e-mail messages from Plesk to a remote server: TLS library problem, Cannot start TLS: handshake failure

Symptoms

Outgoing messages to a specific remote server (e.g.: mx10.example.com) are deferred with errors like these in /var/log/maillog:

postfix/smtp[7555]: SSL_connect error to mx10.example.com[203.0.113.2]:25: -1
postfix/smtp[7555]: warning: TLS library problem: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:769:
postfix/smtp[7555]: 054F92A3E0BD: to=, relay=mx10.example.com[203.0.113.2]:25, delay=0.4, delays=0.24/0/0.16/0, dsn=4.7.5, status=deferred (Cannot start TLS: handshake failure)

Cause

The connection to remote server is closed because local server and remote server have no SSL ciphers in common, which prevents the TLS handshake from happening.

In this example, mail.myserver.com is the Plesk server's hostname and mx10.example.com is the remote server:

# nmap -sV --script ssl-enum-ciphers -p 25 mail.myserver.com -Pn
...
| ssl-enum-ciphers:
| TLSv1.2:
| ciphers:
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 - strong
| TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 - strong
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 - strong
| TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 - strong

# nmap -sV --script ssl-enum-ciphers -p 25 mx10.example.com -Pn
...
| TLSv1.2:
| ciphers:
| TLS_DHE_RSA_WITH_AES_128_CBC_SHA - strong
| TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 - strong | TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 - strong
| TLS_DHE_RSA_WITH_AES_256_CBC_SHA - strong
| TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 - strong
| TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 - strong
| TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA - strong
| TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA - strong
| TLS_RSA_WITH_AES_128_CBC_SHA - strong
| TLS_RSA_WITH_AES_128_CBC_SHA256 - strong
| TLS_RSA_WITH_AES_128_GCM_SHA256 - strong
| TLS_RSA_WITH_AES_256_CBC_SHA - strong
| TLS_RSA_WITH_AES_256_CBC_SHA256 - strong
| TLS_RSA_WITH_AES_256_GCM_SHA384 - strong
| TLS_RSA_WITH_CAMELLIA_128_CBC_SHA - strong
| TLS_RSA_WITH_CAMELLIA_256_CBC_SHA - strong

Resolution

Add at least one of the ciphers supported by the remote server to the local server's Postfix configuration:

1. Connect to the server via SSH

2. Edit the file /etc/postfix/main.cf by adding one of the ciphers supported by the remote server to tls_medium_cipherlist value:

   tls_medium_cipherlist = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES256-SHA256:AES256-GCM-SHA384

   Note: nmap output lists ciphers in IANA format. However, /etc/postfix/main.cf requires ciphers to be entered in OpenSSL format. You can use tools like Cipher Suite Info to find the cipher code in both formats.

3. Restart Postfix:

   Warning: this will cause a temporary downtime for all mail services

   # systemctl restart postfix