443 traffic blocked for external network

[randy27-2000]

I have multiple networks that come together into an aggregate.  Each network have its own routers, host and perimeter firewalls and switches.  Each network is mirrored to look exactly the same.  I have a management network that come off the aggregate to monitor and manage these external networks.  One of the networks has become unreachable via 443.  Everything else seems to be working just fine.  I can SSH from clients in the management network, monitoring using SNMP, can connect using http to servers in that network.  Its only devices that use port 443 that I can't connect to.  For example, ASDM on the firewalls or using a Cisco Security Manager (CSM) within the management network.  The management network has its own host firewall to control access.  This firewall, using CSM Event Viewer, shows denies by this host firewall for any 443 traffic coming from the problem network.  I have check the rules on the host firewall and verified the problem network is treated exactly the same as all other networks.  I've also checked the router access-list for any changes that could be blocking port 443 from leaving the network.  I've also done a packet capture on CSM to look at the traffic as a client in the management network tries to connect to ASDM on one of the firewalls.  As far as I can tell, the thee-way handshake takes place but it is failing with SSL.  A client with in the problem network can connect to HTTPS website with no problem.  This issue is very perplexing because I can't find any real reason for 443 traffic to be blocked.  Anyone have any ideas or troubleshooting steps that could help?

[Marvin Rhoads]

Even though your pcap seems to incidacte the traffic is allowed, I'd check on the problem network firewall using the packet-tracer tool from the cli. 

If the SSL is failing to connect to the firewall itself, it could also be certificate or cipher-related. Doublecheck that the 3DES-AES license is present and look at the available SSL ciphers.

show ver | i 3DES

show run ssl

show ssl

[randy27-2000]

Marvin,

Thanks for the quick response.  I did the packet tracer from cli on the trouble network perimeter firewall and saw normal traffic to include three-way handshake, pushing of packets, reset and fin.  So nothing out of the ordinary.  BTW we have ASA 5525 for management and 5515 for other enclaves.

We aren't really running a remote VPN or anything, so we don't have SSL configured on the firewalls.  Encryption-3DES-AES is enabled and perpetual. These networks come together at the aggregate using BGP.  For the most part, they are directly connected via distribution switches for each network enclave.  The demarcation point is the perimeter firewall. It's only one specific network that seems to have its 443 traffic going into the management network.  In other words, clients in the management network can't connect to this network using HTTPS.  All other traffic makes it through fine.  The 443 connectivity worked before about two weeks ago but for some reason is now being blocked even though no significant changes were made.  Any other thoughts or ideas that I can do to either test out or work around this issue?

[Marvin Rhoads]

You're welcome.

Traffic to the firewalls on tcp/443 will use the ssl ciphers and configuration bits independent of there being any remote access SSL VPN configured.

Also, traffic TO the firewall is not generally affected by access-lists (unless you are using control plane keyword).

SSL traffic to the firewall for management is affected by the "http" command and has a prerequisite of a secret key (rsa key), certificate (self-signed or CA-issued) and common ssl cipher suite being negotiated between the client (your management PC or system) and the server (the ASA).

[randy27-2000]

I understand what your saying about SSL traffic traversing the firewalls.  Both the perimeter and host firewall on the problem network had http server enable and allowing the management network through.  Along with that, crypto is configured on both firewalls.  As a test, I reapplied the crypto key so that I can see my management client get the new certificate.  So at this point I have confirmed that everything else is working like SSH.

On both the perimeter and host firewalls, I configure a static rule that permitted all https traffic from the management network.  Also on the management host firewall, I permitted the problem network in using https as well.  These are redundant rules with other rules on their perspective firewall but I wanted to at least confirm the redundancy and eliminate the possibility of any rules causing this problem.  So, like you were saying, access-list does not affected traffic to the firewall like https.  So I am back to trying to determine why SSL is not being properly negotiated between the problem network and the management network. 

One other note, on the problem network I attempted to connect to a monitoring system on the management network using https and it failed as well.  With that said, the client on the problem network can go out to secure website with no issue. So the issue does seem to be bi-directional with the problem network.  Maybe I'm running down the wrong rabbit hole on this one.

[randy27-2000]

I do have a question concerning licensing for the ASA firewalls.  The ASA 5515s have the Security Plus license and Encryption DES and Encryption-3DES-AES enabled.  I also run "show run all ssl" to display current ssl configuration.  It looks like ssl encryption for rc4-sha1, dhe-aes128-sha1, dha-aes256-sha1, aes128-sha1, aes256-sha1, 3des-sha1 is configured.  My question is can the license eventually expire and start blocking ssl traffic?  After doing some investigation online, I have seen others who had to download new license from Cisco in order to get this to work.

[Marvin Rhoads · Accepted solution]

The 3DES-AES licenses are permanent. I have seen people get an RMA unit an fail to fixup the license on the replcement unit but once it is on a given unit it never leaves.

If you can dig into the packet capture from the ASA while attrempting to connect, you should see SSL cipher spec negotiation attempting to take place. If it fails, there will be an SSL handshake failure packet highlighted in Wireshark.

[randy27-2000]

Marvin, Thanks again for your support. Your suggestion for using Wireshark to look at the SSL handshake pointed me to the culprit causing the problem. Turns out it was a McAfee Web Gateway in the network that, for some reason, decide to block the web connection between the two networks. I'm not sure why now it has decided to do so but the problem is now resolved.

[Marvin Rhoads]

You're welcome. Thanks for letting us know it's resolved and for rating.

I was thinking about asking if there was any web proxy in the mix but your earlier post didn't mention it. They can be disruptive to things in unusual ways.

I don't suggest it starighatway since most people posting here aren't comfortable looking at a decode but, when all else fails - looking at the low level detail can be very informative.