Exchange Server 2010 : Ensuring Message Integrity (part 2) - Using TLS and MTLS & Implementing Domain Security

2. Using TLS and MTLS

Server-to-server connections (for example, connections between SMTP servers on an organizational internetwork or the Internet) rely on MTLS for mutual authentication. On an MTLS connection, the server originating a message and the server receiving it exchange certificates from a mutually trusted CA. The certificates prove the identity of each server to the other.

The TLS protocol provides secure web communications on the Internet or intranets. It enables clients to authenticate servers and (optionally) servers to authenticate clients. It provides a secure channel by encrypting communications. However, when TLS is deployed, it typically provides only confidentiality in the form of encryption. Sometimes no authentication occurs between the sender and the receiver, and sometimes only the receiving server is authenticated. For example, the Secure Sockets Layer (SSL) protocol, which is the Hypertext Transfer Protocol (HTTP) implementation of TLS, authenticates only the receiving server.

When using MTLS authentication, on the other hand, each server verifies the identity of the other server by validating a certificate provided by that server. When messages are received from external domains over verified connections in an Exchange Server 2010 environment, Microsoft Outlook displays a Domain Secured icon. MTLS is a manageable technology for implementing the various features required for domain security, such as certificate management, connector functionality, and Outlook client behavior.

In Exchange 2010, Setup creates a self-signed certificate, and TLS is enabled by default. This enables any sending system to encrypt the inbound SMTP session. Exchange Server 2010 also attempts to use TLS for all remote connections by default. All traffic between Edge Transport servers and Hub Transport servers is authenticated and encrypted using MTLS.

Exchange Server 2010 uses direct trust to authenticate the certificates. Active Directory is considered a trusted storage mechanism, and the certificate is validated because it is present in Active Directory or Active Directory Lightweight Directory Services (AD LDS). When direct trust is used, it does not matter whether the certificate is self-signed or signed by a CA. When you subscribe an Edge Transport server to the Exchange organization, the Edge subscription publishes the Edge Transport server certificate in Active Directory for the Hub Transport servers to validate. The Microsoft Exchange EdgeSync service updates AD LDS with the set of Hub Transport server certificates for the Edge Transport server to validate.

TLS and MTLS require a certificate for authentication of inbound connections to a front-end server (for example, an Edge Transport server) and for outbound connections from the Front End Server. The certificate is provided by the server in response to authentication challenges from clients or servers that send messages to this server. Each Edge Transport server must have a certificate for MTLS communication with other servers on the network, in particular Hub Transport servers.

2.1. Inbound Anonymous TLS Certificates

Inbound anonymous TLS certificates can authenticate SMTP sessions between Edge Transport servers and Hub Transport servers. They can also be used to encrypt SMTP sessions between Hub Transport servers. In the latter case, where anonymous TLS and the public keys from certificates are used to encrypt the session between Hub Transport servers, the Kerberos protocol is used for authentication. When an SMTP session is established, the receiving server initiates a certificate selection process to determine which certificate to use in the TLS negotiation. The sending server also performs a certificate selection process.

An inbound STARTTLS certificate is selected whenever SMTP hosts request TLS security when communicating with Edge Transport servers. The requesting host may be any SMTP host other than the Edge Transport server. Note that SMTP hosts other than Edge Transport servers requesting TLS security is a feature of the domain security scenario. Domain security is discussed later in this lesson.

An inbound STARTTLS certificate is also used when SMTP clients, such as Microsoft Outlook Express, request TLS security when communicating with Hub Transport servers and when Internet-facing Hub Transport servers request TLS security with Edge Transport servers. When an SMTP session is established, the receiving server initiates a certificate selection process to determine which certificate to use in the TLS negotiation.

Note:

OUTBOUND CERTIFICATES

When the receiving server requests an inbound STARTTLS certificate, the sending server also performs a certificate selection process and selects an outbound anonymous TLS certificate. The selection of outbound anonymous TLS certificates is discussed next.

2.3. Outbound Anonymous TLS Certificates

An outbound anonymous TLS certificate is selected for authentication during an SMTP session between an Edge Transport server and a Hub Transport server. This type of certificate is also used to encrypt SMTP sessions between Hub Transport servers by using public keys. When an SMTP session is established, the receiving server initiates a certificate selection process to determine the outbound anonymous TLS certificate to use in the TLS negotiation.

3. Implementing Domain Security

Domain security provides a lower-cost alternative to S/MIME or other message-level security solutions. The domain security feature provides a method of managing secured message paths between business partners over the Internet. After secured message paths are configured, messages that have successfully traveled over these paths from authenticated senders are displayed to users as domain secured in the Outlook and OWA interfaces.

Domain security uses MTLS authentication to provide session-based authentication and encryption. MTLS authentication differs from a typical TLS implementation. When TLS is implemented, the client verifies that the connection securely connects to the intended server by validating the server’s certificate. The client authenticates the server before transmitting data. However, the server does not authenticate the session with the client. When MTLS authentication is used, each server verifies the connection with the other server by validating a certificate provided by that other server—in other words, both the message sender and the message recipient are validated.

Exchange Server 2010 provides a set of cmdlets that create, request, and manage TLS certificates. By default, TLS certificates are self-signed. That is, they are signed by their own creator. In Exchange Server 2010, self-signed certificates are created on the Exchange server by using the Microsoft Windows Cryptography Application Programming Interface. Self-signed certificates are considered less trustworthy than certificates generated by PKI or a trusted third-party CA and are typically used for internal mail only. However, you can use self-signed certificates to secure email messages from your organization to another Exchange Server 2010 organization if the receiving organization agrees to install your self-signed certificates in the trusted root certificate store in each of its inbound Edge Transport servers. In this case, the self-signed certificates are trusted explicitly.

To secure email messages that traverse the Internet, you would typically generate TLS certificates with a PKI or obtain them from a third-party CA. Suppose, for example, that you are an Exchange administrator at the Adatum Corporation and you want to configure Adatum’s Exchange Server 2010 organization to exchange domain-secured email with its partner organization, NorthWind Traders. You want to ensure that all email messages sent to and received from NorthWind Traders are protected with MTLS, and you want to configure domain security functionality so that all mail between the Adatum Corporation and NorthWind Traders is rejected if mutual TLS cannot be used.

Adatum has an internal PKI that generates certificates. The PKI’s root certificate has been signed by a trusted third-party CA. NorthWind Traders uses the same third-party CA to generate its certificates. Therefore, both organizations trust the other’s root CA. By default, the public third-party CA is one of the trusted root certificates in the Microsoft Windows certificate store in the adatum.com domain. Therefore, any client that includes the same third-party CA in its trusted root store and that connects to Adatum can authenticate to the certificate presented by Adatum.

The Edge Transport server VAN-EX2.adatum.com requires a certificate. You therefore generate a base64-encoded PKCS#10 certificate request on that server by entering the following commands:

$Data1 = New-ExchangeCertificate -GenerateRequest -FriendlyName "Internet certificate
for VAN-EX2" -SubjectName "DC=com,DC=Adatum,CN=VAN-EX2.adatum.com" -DomainName mail
.adatum.com
Set-Content -Path "C:\Certificates\VAN-EX2-request.req" -Value $Data1

					  

Import-ExchangeCertificate -FileData ([Byte[]]$(Get-Content -Path C:\Certificates\
VAN-EX2-certificate.pfx -Encoding Byte -ReadCount 0)) | Enable-ExchangeCertificate
-Services SMTP

					  

Note that the VAN-EX2-certificate.pfx file must exist in the path specified; otherwise, the command returns an error.

Set-TransportConfig -TLSSendDomainSecureList northwindtraders.com

You then use the Set-SendConnector EMS cmdlet to set the DomainSecureEnabled property on the Send connector that sends email to this domain. For a Send connector named Internet and configured for Internet connection, you enter the following command:

Set-SendConnector Internet -DomainSecureEnabled:$true

Neither of these two commands generates an output. If they complete without error, you have specified a target domain and enabled domain security on the Send connector. Note that an appropriately configured Send connector named Internet must exist for the second command to be successful.

The final step in configuring outbound domain security is to check that the Send connector you are using to send domain-secured email routes mail using the Domain Name System (DNS) and that the fully qualified domain name (FQDN) of the Send connector matches either the Subject Name or the Subject Alternative Name of certificates that you are using for domain security. To verify the Send connector settings, enter the following command:

Get-SendConnector Internet | FL Name,DNSRoutingEnabled,FQDN,DomainSecureEnabled

					  

Figure 4 shows the output of this command. Note that the Fqdn value is shown as blank. The default value of the Fqdn parameter is $null, which indicates that the actual default FQDN value is the FQDN of the Edge Transport server that contains the Send connector.

When you have configured and verified outbound domain security, you next need to configure inbound domain security. To do this, you use the Set-TransportConfig EMS cmdlet to specify the domain from which you want to receive domain-secured email and, on the Edge Transport server, enable domain security on the Receive connector from which you want to receive domain-secured email. Because domain security requires MTLS authentication, you must also enable TLS on the Receive connector. To specify the domain from which you want to receive domain-secured email, you run the following command on an internal Exchange Server 2010 server (for example, a Hub Transport server):

Set-TransportConfig -TLSReceiveDomainSecureList northwindtraders.com

Figure 5 shows this command entered on the VAN-EX1 Hub Transport server.

You need to configure the Receive connector on each Edge Transport server that accepts mail from the domain from which you want to receive domain-secured email—in the example given, the VAN-EX2.adatum.com Edge Transport server. Assuming that the Adatum environment is configured to have a single Internet Receive connector, with an Identity parameter value of Internet on this Edge Transport server, you enable domain security and TLS by running the following command on the Edge Transport server:

Set-ReceiveConnector Internet -DomainSecureEnabled $true -AuthMechanism TLS

This command generates no output. If it completes without error, you have enabled domain security on the specified Receive connector.

The Enable-ServiceEmailChannel cmdlet allows you to enable the .NET service channel for a specific user. It does not enable domain security or use TLS or MTLS to send secure email between domains.

3.2. Testing Domain-Secured Mail Flow

You can review the send and receive protocol logs to determine whether TLS negotiation has been successful. You should set the protocol logging level to Verbose on the connectors that your organization uses to send and receive domain-secured email. You need to enter the following command on all Edge Transport servers involved in domain security:

Set-ReceiveConnector Internet -ProtocolLoggingLevel Verbose

To enable protocol logging on the Send connector, you need to enter the following command on an internal Exchange server, such as a Hub Transport server:

Set-SendConnector Internet -ProtocolLoggingLevel Verbose

Neither of these two commands generates an output. If they complete without error, you have successfully enabled protocol logging on the specified Receive and Send connectors.

You can use the following performance counters under the MSExchange Secure Mail Transport object in Exchange Server Performance Monitor to monitor domain security:

• Domain-Secured Messages Received

• Domain-Secured Messages Sent

• Domain-Secured Outbound Session Failures

Figure 6 shows these counters.

A counter log file for domain-secured mail flow that logs the values of these counters using the Performance Logs and Alerts MMC snap-in helps you monitor the number of messages sent and received and also to monitor failed MTLS sessions.