Accepted Solutions
Gentlemen,
Thank you all for your responses. Pavel and I are colleagues at the same department and I suggested to Pavel to ask here on CSC for ideas about this pesky problem after we were unable to solve it ourselves after several days of experimenting.
We have eventually solved this and we'd like to share the solution. The problem actually wasn't directly caused by the WLC but rather by a couple of unfortunate coincidences.
To reiterate on the problem, we were faced with a loss of all but ICMP connectivity with the WLC immediately after we configured a dynamic interface on the WLC and placed it into 192.168.10.0/24 network on VLAN 10. This network is our internal departmental network - our idea was to have an SSID for wireless department clients that would be bridged onto the wired VLAN 10 into a single department network, hence the same IP network space. As we were configuring the WLC, we were accessing it under its management IP 192.168.255.9/24 in VLAN 255 from a PC in our 192.168.10.0/24 network. Routing between the 192.168.10.0/24 and 192.168.255.0/24 is done by an ASA box sitting on both these networks (VLANs). The logical topology resembles the following diagram:
The cause of the problem probably now starts to be obvious. The PC 192.168.10.222 was accessing the WLC at 192.168.255.9 while the WLC was configured with both 192.168.255.9/24 and 192.168.10.9/24. The PC was communicating with the WLC via the ASA box as its default gateway while the WLC responded to the PC directly, as it indeed was on the same subnet with the PC. The ASA saw the first TCP SYN from the PC towards the WLC but never saw the TCP SYN/ACK from the WLC back to the PC. When the TCP ACK from the PC towards the WLC arrived at the ASA box, it dropped it, preventing the TCP 3-way handshake from ever completing.
If the ASA was replaced with a common router not performing stateful firewalling, this issue would not have occured despite the asymmetrical routing. I have also verified that an IOS-based router running IP Inspect (CBAC) would cause the same connectivity issue.
It is interesting that if the WLC responds to ICMP ECHO messages in particular, it responds through the same interface through which the ICMP ECHO arrived, regardless of the source. In other words, pinging 192.168.255.9 from 192.168.10.9 worked because the WLC sent the reply via the ASA box 192.168.255.1 and not directly to 192.168.10.9. This fact was quite confusing during the troubleshooting, as it diverted our attention. Actually, we first started to suspect a problem in routing and reachability only after we moved the management PC from VLAN10 to VLAN255 and regained the IP connectivity with the WLC.
The easiest solution appears to be to simply bridge the wireless SSID onto a different VLAN than the one that will be (occassionally) used to manage the WLC, to force the WLC to always respond through the ASA when being managed.
I would like to sincerely thank to everyone that has joined this thread. Your effort is very much appreciated! It's almost a shame that the problem was this silly..
Thank you!
Best regards,
Peter
Gentlemen,
Thank you all for your responses. Pavel and I are colleagues at the same department and I suggested to Pavel to ask here on CSC for ideas about this pesky problem after we were unable to solve it ourselves after several days of experimenting.
We have eventually solved this and we'd like to share the solution. The problem actually wasn't directly caused by the WLC but rather by a couple of unfortunate coincidences.
To reiterate on the problem, we were faced with a loss of all but ICMP connectivity with the WLC immediately after we configured a dynamic interface on the WLC and placed it into 192.168.10.0/24 network on VLAN 10. This network is our internal departmental network - our idea was to have an SSID for wireless department clients that would be bridged onto the wired VLAN 10 into a single department network, hence the same IP network space. As we were configuring the WLC, we were accessing it under its management IP 192.168.255.9/24 in VLAN 255 from a PC in our 192.168.10.0/24 network. Routing between the 192.168.10.0/24 and 192.168.255.0/24 is done by an ASA box sitting on both these networks (VLANs). The logical topology resembles the following diagram:
The cause of the problem probably now starts to be obvious. The PC 192.168.10.222 was accessing the WLC at 192.168.255.9 while the WLC was configured with both 192.168.255.9/24 and 192.168.10.9/24. The PC was communicating with the WLC via the ASA box as its default gateway while the WLC responded to the PC directly, as it indeed was on the same subnet with the PC. The ASA saw the first TCP SYN from the PC towards the WLC but never saw the TCP SYN/ACK from the WLC back to the PC. When the TCP ACK from the PC towards the WLC arrived at the ASA box, it dropped it, preventing the TCP 3-way handshake from ever completing.
If the ASA was replaced with a common router not performing stateful firewalling, this issue would not have occured despite the asymmetrical routing. I have also verified that an IOS-based router running IP Inspect (CBAC) would cause the same connectivity issue.
It is interesting that if the WLC responds to ICMP ECHO messages in particular, it responds through the same interface through which the ICMP ECHO arrived, regardless of the source. In other words, pinging 192.168.255.9 from 192.168.10.9 worked because the WLC sent the reply via the ASA box 192.168.255.1 and not directly to 192.168.10.9. This fact was quite confusing during the troubleshooting, as it diverted our attention. Actually, we first started to suspect a problem in routing and reachability only after we moved the management PC from VLAN10 to VLAN255 and regained the IP connectivity with the WLC.
The easiest solution appears to be to simply bridge the wireless SSID onto a different VLAN than the one that will be (occassionally) used to manage the WLC, to force the WLC to always respond through the ASA when being managed.
I would like to sincerely thank to everyone that has joined this thread. Your effort is very much appreciated! It's almost a shame that the problem was this silly..
Thank you!
Best regards,
Peter
