packet loss on IPSEC VPN connection but not on internet connection

mitchen · ‎05-12-2006

We have recently installed a Cisco 837 at a remote office with a VPN connection terminating on our Head Office PIX515E firewall and a site-to-site VPN connection to another remote site with another Cisco 837.

We have a large number of this type of connection all working fine.

However, at this site we were initially having real problems establishing the VPN tunnel.

Part of the problem has been with the line itself - our ISP/telco have been out a number of times to investigate.

We now find ourselves in the situation where we have established the tunnel between the Cisco 837 at this site and the Cisco 837 at another site. However, when we try to pass traffic across it, we see a lot of packet loss - this is rendering user applications virtually unusable.

At the moment the VPN connection between our PIX and the Cisco 837 is down - though at one stage it was up for a couple of hours.

For internet access, the users break out to the internet locally, rather than passing their traffic over the VPN.

This traffic seems largely unaffected by any packet loss.

Since we can establish an internet connection, our ISP/telco have basically washed their hands of the problem.

But why should our IPSEC VPN suffer so badly at this site but not at others, with the exact same equipment and set-up?

Would IPSEC traffic be more affected by a faulty line (e.g. noisy line?)

If so, what can we do to prove this and go back to our ISP/telco?

Our ISP say our noise, attentuation levels etc are all within their limits.

Has anybody got any suggestions!

a.kiprawih · ‎05-15-2006

Hi,

The faulty line will affect both VPN and normal traffic, not only IPSec VPN. Packet loss can be due to ISP link, cable or interface issues.

If you issue command 'show interface' pointing to the interface used by VPN to send out traffic to remote peer, what is the statistic looks like - interface reset, collisions, errors?

Can you ping to your remote C837 through the VPN tunnel and outside the tunnel (ping to C837 serial interface)?

What's the result looks like? Do you see timeout even if you ping directly outside VPN tunnel?

Rgds,

AK

mitchen · ‎05-15-2006

Hi,

I've been able to do some more tests today and it looks like you're right - both the internet connection and the VPN connection are showing packet loss. (at least that makes a bit more sense now - I couldnt understand why there was no packet loss on the internet connection but it looks like that was just coincidence at the time of the original tests last week!)

What makes things a bit more difficult to troubleshoot is that every now and then the connection seems to work for a while with no packet loss but for the majority of the time, there is so much packet loss, it is rendering the ADSL line virtually unusable.

The interface statistics dont show any collisions - there are a couple of resets on the ATM interface and it is also showing a high number of CRC errors.

If I issue a "show dsl interface atm0" command then I can see, what seems to me, to be an extremely high number of errors on the line:

Interleave Fast Interleave Fast

Speed (kbps): 8000 0 448 0

Reed-Solomon EC: 6633 0 5254 6890

CRC Errors: 60324 0 5948 9652

Header Errors: 27201 0 6014 5177

Bit Errors: 0 0

BER Valid sec: 0 0

BER Invalid sec: 0 0

In the router logs, I constantly see the Virtual-Access interface going up and down (though NOT the physical ATM interface)

e.g.

May 15 13:18:59.159: %LINK-3-UPDOWN: Interface Virtual-Access1, changed state to down

May 15 13:20:22.459: %LINK-3-UPDOWN: Interface Virtual-Access1, changed state to up

May 15 13:23:26.359: %LINK-3-UPDOWN: Interface Virtual-Access1, changed state to down

May 15 13:23:52.967: %LINK-3-UPDOWN: Interface Virtual-Access1, changed state to up

May 15 13:26:48.503: %LINK-3-UPDOWN: Interface Virtual-Access1, changed state to down

May 15 13:27:23.179: %LINK-3-UPDOWN: Interface Virtual-Access1, changed state to up

May 15 13:44:23.863: %LINK-3-UPDOWN: Interface Virtual-Access1, changed state to down

May 15 13:44:50.331: %LINK-3-UPDOWN: Interface Virtual-Access1, changed state to up

Am I right in thinking that this evidence tends to point to some sort of line fault with the ISP/telco?

Or is there anything else I can do to eliminate our own equipment (which is what the ISP/telco are trying to blame even though we have this set-up working fine at around 50 other sites!)

We have replaced the router, replaced the microfilter, etc but none of this has made any difference.

Any further suggestions/advice would be welcome!

a.kiprawih · ‎05-15-2006

Hi,

Yes, you're right. Most of the time (and from my own experience), this kind of problem is related to internet link/ISP.

It was quite clear - attach any new box to the ISP line and yet, the errors was still there. This is very straight forward - internet link problem.

Maybe, before you call them in, do your own test - prepare your new boxes, attached them one by one to the line, clear interface statistics, and see the amount of errors generated when VPN traffic pass through. Once this is confirmed, then call the ISP folks, and show them the errors again.

Good luck.

Rgds,

AK

mitchen · ‎05-16-2006

Problem now resolved - as suspected, it was an ISP/telco fault (though they dont actually tell us what it was, just that its been fixed now!)

Thanks for your help.

a.kiprawih · ‎05-16-2006

Glad to hear that.. :)

I guess the ISP (not all, but most of them) response towards their client's complaint about their link is all the same all around the world. Even if they know the fault, they hardly admit it, but at the same time, they quietly tried to fix it and pretend nothing happens..

Rgds,

AK