The first thing to consider is that SSL/TLS is carried over TCP, so all the same DoS attacks that work against regular TCP listening ports work against SSL/TLS. A Cisco IDS sensor is highly effective at detecting these attacks.
Next, there is a DoS mounted against the SSL/TLS handshake. The two levels of handshake attack are those mounted with no special privileges and those with enough privileges to complete the handshake.
By far the most common case is for an SSL/TLS service to allow anonymous connections to complete the handshake. In tight security environments, the SSL/TLS server process is configured to request a client certificate during the handshake, and terminate the connection if the client fails to provide one the server trusts. Sites wanting to limit their exposure to attacks mounted after the handshake is successfully completed should enable this feature.
We'll examine these attacks in a moment. Let us first explore the problems an attacker can cause before finishing the handshake.
The attacker can, with little processing effort, send handshake messages that force the server to perform the compute-intensive private key decrypt operation before it can determine the attacker's message is invalid.
Large e-commerce sites will minimize this threat by offloading the private-key decryption operations to hardware accelerators. But ultimately, even that resource is limited and can be exhausted by a persistent attacker. The defense is to use an SSL/TLS server that provides the ability to monitor the number of "broken handshakes", that is, the number of handshakes started but not completed.
Network-based IDS would be ineffective at detecting this attack because the final handshake messages are encrypted, making it impossible for a NIDS sensor to differentiate between a server successfuly completing the handshake and one that is sending an alert message that typically ends all broken handshakes.
With complete control of the SSL/TLS implementation, one could detect a potential DoS attack by measuring the number of broken handshakes over a recent time period. For example, if the number of broken handshakes in the last 60 seconds is more than 100, a DoS might be in progress.
Now what? One countermeasure would be to adjust the software firewall on the host running the SSL/TLS server to temporarily deny access to the IP addresses from which the broken handshakes originated. This defense is ineffective against an attacker forging the source IP address, or against a distributed DoS attack.
The next defense would be to throttle the rate at which the expensive private key decrypts are allowed to be performed. Say a given server can effectively service 6000 connections per minute. Use a technique called "blinding" to force the private-key decrypt operation to return in exactly 10mS (even though the operation takes a great deal less time than that with a powerful CPU or adequate hardware acceleration). This limits the resources the server will allocate to the establishment of new connections, but leads to yet another DoS. Legitimate connection requests are blocked waiting for their turn in line, along with the attacker's connection requests. There is not enough information available at this layer to differentiate between legitimate and attacker handshake messages.
In network security, it quickly becomes apparent that there is not an adequate defense against DoS attacks. Every defense produces yet another vector for attack.
All we can hope for is a reliable measurement that indicates whether an attack is in progress. Since the last handshake messages are encrypted, the only way to differentiate between completed handshakes and broken handshakes is to utilize SSL/TLS server software that provides a measurement of broken handshakes.
Now suppose the server is configured to allow anonymous connections to complete the handshake. In this case, the server did not require the client to submit a valid certificate. Now all the DoS attacks effective against an unencrypted service (say, HTTP for example) are now available to the attacker.
Often, the resources available on a HTTP over SSL/TLS server require the client to be authenticated. Normally, authentication is carried out using the HTTP basic authentication mechanism. The server must be aware that an unauthenticated connection receives lower priority than authenticated ones. HTTP/1.1 persistent connections in particular should be dropped as soon as a resource is requested that requires a higher access permission level than the client possesses. (The server returns "401" Unauthorized and closes the connection.)
The HTTP service should also limit the number of bytes an unauthenticated client can include in a POST entity-body, the number of bytes in the request URI or in any of the request header lines, timeouts for submitting a request, number of requests allowed on a single persistent connection, etc.
Because of encryption, these security measures must be included in the server software. Host-based IDS is effective at detecting weaknesses in the server software and at detecting signs that an attacker has gained control of the host. However, one should start with server software that has been designed from the start with security in mind. That leaves fewer holes for the Host-based IDS solution to catch.
A number of hardware vendors provide what are called "red boxes". These are SSL/TLS servers that possess the private key associated with the host certificate. They live in the DMZ network and have tunnels to the actual cleartext servers located inside the protected network. To the client it appears that the SSL/TLS connection is with the red box. The payload protocol is available to the network IDS sensor, allowing full stateful analysis of its contents. This configuration provides the highest level of performance and security.
