Configuring Sendmail 8.13.3 with TLS and SSL

To change the directory to /opt/openssl/misc, enter the following command:

cd /opt/openssl/misc

To copy the CA.pl script to the CA.pl.ORIGINAL script, enter the following command:

cp CA.pl CA.pl.ORIGINAL

Replace the entries marked with numbers in the following CA.pl script:

exit 0;
} elsif (/^-newcert$/) {
 	# create a certificate

system ("$REQ -new -x509 -keyout newkey.pem -out newcert.pem $DAYS"
);1

$RET=$?; 
 print "Certificate is in newcert.pem, private key is in newkey.pem\n"
  } elsif (/^-newreq$/) {
system ("$REQ -new -keyout newkey.pem -out newreq.pem $DAYS");2

$RET=$?;
print "Request is in newreq.pem, private key is in newkey.pem\n";
 } elsif (/^-newreq-nodes$/) 

1	Replace this line with the following: system ("$REQ -new -nodes -x509 -keyout newkey.pem -out newcert.pem $DAYS");
2	Replace this line with the following: system ("$REQ -new -nodes -keyout newkey.pem -out newreq.pem $DAYS");

The only change is the addition of the -nodes option while generating certificates. If you do not include this option, Sendmail 8.13.3 cannot load the encrypted key during startup. As a result, it logs the following error message in the /var/adm/syslog/mail.log file:

“May 4 11:55:20 XXXXX sm-mta[23544]: STARTTLS=server, error: SSL_CTX_use_PrivateKey_file(/etc/mail/certs/servername-key.pem) failed”

	NOTE: You must modify the first line in the CA.pl script to the location of the perl interpreter on your system. Otherwise, the following error message is logged in the /var/adm/syslog/syslog.log file: interpreter “/opt/perl/bin/perl” not found

Follow this procedure to create your own CA, and to create certificates and keys for your Sendmail 8.13.3 server:

To create a subdirectory certs under the /etc/mail directory, enter the following command:

To set the appropriate permissions to the certs subdirectory, enter the following command:

chmod 755 certs

To change the directory location to certs, enter the following command:

To copy the previously created CA certificate, the Sendmail 8.13.3 server certificate, and the key from the /opt/openssl/misc/ directory to the /etc/mail/certs directory, enter the following commands:

To create a hashed symbolic link to the CA certificate, enter the following command:

This command reads the cacert.pem file and creates an 8-character cryptographic hash, which is used as the filename (with '.0' appended) that links to the CA certificate. During a certificate exchange in an SSL handshake, Sendmail 8.13.3 computes the hash of the received public key of the CA certificate, appends '.0' to the hash, and compares the computed hash with its own copy of the public key of the CA certificate.

To verify whether the symbolic link to the CA certificate is created properly, enter the following command:

Ensure that you obtain an output similar to the following:

2197 lrwxrwxrwx 1 root sys 10 Jul 9 09:44 fea4e1bb.0 -> cacert.
pem

	NOTE: The link name fea4e1bb.0 is only an example. The link name must be of the format <characters>.0.

To back up the existing Sendmail 8.13.3 configuration file, enter the following command:

To change the directory to the /usr/newconfig/etc/mail/cf/cf directory, enter the following command:

To run the gen_cf utility, enter the following command:

Select the 2: STARTTLS option under the 4: Security Options option in the main menu.

Select the 5: Generate sendmail.cf option and press Enter to generate the Sendmail 8.13.3 configuration file (sendmail.cf.gen) with the STARTTLS feature enabled.

Repeat Steps 1–4.

Select the 6: Generate submit.cf option and press Enter to generate the configuration file of the Sendmail 8.13.3 client queue runner (submit.cf.gen) with the STARTTLS feature enabled.

Copy the previously created sendmail.cf.gen file and the submit.cf.gen file to the /etc/mail/sendmail.cf file and the /etc/mail/submit.cf file, respectively.

Open the Sendmail 8.13.3 configuration files (/etc/mail/sendmail.cfand /etc/mail/submit.cf) and edit the UseTLS, CACertPath, CACertFile, ServerCertFile, ServerKeyFile, ClientCertFile, ClientKeyFile, and RandFile options, as follows:

# If set, Sendmail enables the TLS feature 
UseTLS=True
# CA directory 
CACertPath=/etc/mail/certs 
# CA file
CACertFile=/etc/mail/certs/cacert.pem
# Server Cert
ServerCertFile=/etc/mail/certs/servername-cert.pem 
# Server private key
ServerKeyFile=/etc/mail/certs/servername-key.pem 
# Client Cert
ClientCertFile=/etc/mail/certs/servername-cert.pem
# Client private key
ClientKeyFile=/etc/mail/certs/servername-key.pem 
# Random data source (required for systems without /dev/urandom under OpenSSL) 
RandFile=egd:/dev/random

Follow this procedure if Mail Submission Program (MSP) is enabled for Sendmail 8.13.3:

Follow this procedure if MSP is disabled in Sendmail 8.13.3:

Examine the output of the mtail command to ensure that Sendmail 8.13.3 does not contain any error or warning after configuring TLS/SSL.

Send a test mail using Sendmail 8.13.3 and verify if the mail is delivered to the destination address.

Ensure that you notice STARTTLS in certain Sendmail 8.13.3 log entries to ascertain the proper configuration of STARTTLS.

Establish a Telnet session to port 25 or *587 of the server configured recently, to ensure that it offers the STARTTLS support in response to the EHLO command. The *587 port is used if MSP is enabled for Sendmail 8.13.3.

Following is a sample Telnet session, which ascertains the STARTTLS support:

$ telnet localhost 25
Trying...
Connected to localhost.<domain_name>
Escape character is '^]'.
220 <hostname>.<domain-name> ESMTP Sendmail @(#)Sendmail 
version 8.13.3 - Revision
 2.005 - 12 January 2007/8.13.3; Fri, 4 May 2007 18:00:30 +
0530 (IST)

where:

Enter the EHLO command, as follows:

$ EHLO localhost

The following output displays:

250-<hostname><domain-name> Hello localhost [127.0.0.1], 
pleased to meet you
250-ENH ANCEDSTATUSCODES
	250-PIPELINING
	250-EXPN
	250-VERB
	250-8BITMIME
	250-SIZE
	250-DSN
	250-ETRN
	250-AUTH DIGEST-MD5 CRAM-MD5
250-STARTTLS
	250-DELIVERBY
	250 HELP

You must ensure that you obtain a response similar to 250-STARTTLS from the Sendmail 8.13.3 server. Additionally, ensure that you get the 250-STARTTLS line in response to the SMTP EHLO command. This indicates that STARTTLS is configured correctly.

Enter the following to indicate that you want to quit the Telnet session:

$ QUIT

The following output displays:

221 2.0.0 <hostname>.<domain-name> closing connection
	Connection closed by foreign host.

Send a mail using the STARTTLS configured Sendmail 8.13.3 server and ensure that you get the (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK) line in the following message header:

From root @<sender_name>.<domain_name> Thu Jul 5 19:19:30 
IST 2007
Received: from <sender_name>.<domain_name> by <hostname>.
<domain_name>
(@(#)Sendmail version 8.13.3 - Revision 2.005 - 12 January 
2007/8.13.3) with ESMTP id l65DnTLe028546
(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 
verify=OK)
for <hostname>.<domain_name>; Thu, 5 Jul 2007 19:19:30 
+0530 (IST)
Received: (from root@localhost) by <sender_name>.<domain_name>
((@(#)Sendmail 
version 8.13.3 - Revision 1.000 - 1st August,2006/8.13.3)id 
l65Dbpdc008315
for root@<hostname> Thu, 5 Jul 2007 19:07:51 +0530 (IST)
Date: Thu, 5 Jul 2007 19:07:51 +0530 (IST)
From: <server_name>.<domain_name>

The verify macro in the message header in the mtail command output contain the result of the verification of the presented certificate.

The verify macro can contain the following values: