Asymmetric Traffic Flow/Routing When Connecting to Cisco Router running IPSec Tunnel

[obsanu001]

Hi All,

We have a concerning issue that is affecting 50% of our locations. We are A small service provider connecting 30 locations of a small financial Services company. They are running CISCO Routers at their Data-center Core and at their Branches. We are delivering our services over a larger operators MPLS/SDH Network and we have deployed VPLS Tunnels and VLANS from our remote devices to our Core and hand off to them at the branch and at their core as is indicated below:

ClientCoreRouter=>MWLink=>OurCore=>Provider MPLS/SDH Cloud=>OurRemoteCPE=>MWLink=>ClientBranchRouter

Our client is running IPSec Tunnels from their branch routers to a single interface on their core router which connects to our core router via the network we provide. At about 50% of the locations the network works perfectly and we see proper traffic profiles at our MWLink Ethernet Interface, at their branches. In the other locations we see proper traffic flow coming from their data-center (Tx Traffic) BUT we see NO/VERY LITTLE traffic coming from their branch router destined for their core router.

We have done a myriad of tests and have confirmed that our system is working and carrying traffic properly, capacity and routing wise, we have also taken captures of the traffic from between our MWLink and the branch router and confirmed that traffic is originating from the data-center core router and reaching the branch BUT virtually no traffic is returning down this path. The client has confirmed that their onsite ATM Machines work properly and their internal chat service is working but NONE of their core applications work.

This suggests to us that larger packet sizes seem to be either blocked/dropped or are being routed through some other interface or route. It really appears to us that this Asymmetric traffic flow is the root cause, but why it is happening we have no idea. I have attached a file showing the traffic profiles we see at the LAN interface of our MWLink in the branch if this helps. 

Can anyone help clarify what the issue is and a possible solution? This is quite urgent so all assistance would be appreciated.

Regards

Bola

[Richard Burts]

Bola

It is interesting that some branches work and that some branches do not work. I would suggest that you ask the customer if there are any differences between branches that do work and branches that do not work. It might be a difference in hardware, or in software versions, or difference in config. If they do identify any differences evaluate whether that might cause this symptom.

If they find no differences then there are some other questions to investigate. One question would be whether they send all of their traffic using the IPsec tunnel or perhaps they send some traffic (their core applications) via IPsec but send other traffic (chats, etc) without encryption. If some traffic does not get encrypted then it might explain why some types of traffic do work while others do not.

If all of the branch traffic is sent via the IPsec tunnel then perhaps it is an issue about packet size. When using IPsec tunnel the encryption process does add extra header information to the packet. So if the application sends a large packet and then encryption adds extra bytes it can result in a packet that is too large. So smaller packets will work fine but large packets may have a problem. I have encountered this situation at some customers and have used ip tcp adjust-mss in router interface configuration to control the packet size. Perhaps this might be helpful in your situation.

HTH

Rick