Packet loss on many remote location - Page 4

Sandeep Choudhary · ‎06-19-2012

Hello Everyone,

I am facing this issue from long time and still coudn't find the core issue.

specially we are using CAD servers and some applications(e.g. outlook...email...ftp) on the remote location

Every remote location is connected via gre tunnels and dial back technic from HQ.

can any body give me some suggestion to try something on remote router or HQ router to eliminate these problems.

Regards

Lee Smitherman · ‎06-26-2012

Hi, finding it odd that your tunnel config mtu is not appear as the actual tunnel config MTU(hightlighted below) where on all the other tunnels it does.

LARCVPN1# sh int tunnel1

Tunnel1 is up, line protocol is up

Hardware is Tunnel

Description: *** Tu1, Telekom ISP#2 10M ***

Internet address is 10.13.22.2/30

MTU 17916 bytes, BW 100 Kbit/sec, DLY 50000 usec,

reliability 255/255, txload 1/255, rxload 221/255

Encapsulation TUNNEL, loopback not set

Keepalive not set

Tunnel source ISP2, destination HqISP1

Tunnel protocol/transport GRE/IP
Key disabled, sequencing disabled
Checksumming of packets disabled
Tunnel TTL 255
Fast tunneling enabled
Tunnel transport MTU 1476 bytes

interface Tunnel1

description *** Tu1, Telekom ISP#2 10M ***

ip address 10.13.22.2 255.255.255.252

ip mtu 1300

ip tcp adjust-mss 1260

tunnel source ISP2

tunnel destination HQISP1

tunnel protection ipsec profile vpn_profile_hasel_aes

Whats the IP address of your PC? and could you on the remote router show us a "show processes cpu"?

Thanks, Lee.

Sandeep Choudhary · ‎06-26-2012

LARCVPN1#sh int tunnel1

Tunnel1 is up, line protocol is up

Hardware is Tunnel

Description: *** Tu1, Telekom ISP#2 10M ***

Internet address is 10.13.22.2/30

MTU 17916 bytes, BW 100 Kbit/sec, DLY 50000 usec,

reliability 255/255, txload 1/255, rxload 162/255

Encapsulation TUNNEL, loopback not set

Keepalive not set

Tunnel source 195.243.201.186, destination 195.243.205.104

Tunnel protocol/transport GRE/IP

Key disabled, sequencing disabled

Checksumming of packets disabled

Tunnel TTL 255

Fast tunneling enabled

Tunnel transport MTU 1476 bytes

LARCVPN1#sh int tunnel2

Tunnel2 is up, line protocol is up

Hardware is Tunnel

Description: *** Tu2, Telekom ISP#2 10M ***

Internet address is 10.13.122.2/30

MTU 17916 bytes, BW 100 Kbit/sec, DLY 50000 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation TUNNEL, loopback not set

Keepalive not set

Tunnel source 195.243.201.186, destination 212.185.41.196

Tunnel protocol/transport GRE/IP

Key disabled, sequencing disabled

Checksumming of packets disabled

Tunnel TTL 255

Fast tunneling enabled

Tunnel transport MTU 1476 bytes

LARCVPN1#sh int tunnel3
Tunnel3 is up, line protocol is up
Hardware is Tunnel
Description: *** Tu3, Telekom ISP#1 ***
Internet address is 10.14.22.2/30
MTU 17916 bytes, BW 100 Kbit/sec, DLY 50000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
Encapsulation TUNNEL, loopback not set
Keepalive not set
Tunnel source REISP1, destination HQISP1
Tunnel protocol/transport GRE/IP
    Key disabled, sequencing disabled
    Checksumming of packets disabled
Tunnel TTL 255
Fast tunneling enabled
Tunnel transport MTU 1276 bytes

#sh int tunnel4
Tunnel4 is up, line protocol is up
Hardware is Tunnel
Description: *** Tu4, Telekom ISP#1 ***
Internet address is 10.14.122.2/30
MTU 17916 bytes, BW 100 Kbit/sec, DLY 50000 usec,
     reliability 255/255, txload 38/255, rxload 1/255
Encapsulation TUNNEL, loopback not set
Keepalive not set
Tunnel source 195.243.193.34, destination 212.185.41.197
Tunnel protocol/transport GRE/IP
    Key disabled, sequencing disabled
    Checksumming of packets disabled
Tunnel TTL 255
Fast tunneling enabled
Tunnel transport MTU 1476 bytes

LARCVPN1#sh process cpu

CPU utilization for five seconds: 17%/14%; one minute: 22%; five minutes: 23% LARCVPN1

Now i changed all the tunnels with 1276 byte mtu size:

have a look:

LARCVPN1#sh int tunnel1
Tunnel1 is up, line protocol is up
Hardware is Tunnel
Description: *** Tu1, Telekom ISP#2 10M ***
Tunnel transport MTU 1276 bytes

LARCVPN1#sh int tunnel2
Tunnel2 is up, line protocol is up
Hardware is Tunnel
Description: *** Tu2, Telekom ISP#2 10M ***
Fast tunneling enabled
Tunnel transport MTU 1276 bytes

LARCVPN1#sh int tunnel3
Tunnel3 is up, line protocol is up
Hardware is Tunnel
Description: *** Tu3, Telekom ISP#1 2M***
Tunnel transport MTU 1276 bytes

Tunnel4 is up, line protocol is up
Hardware is Tunnel
Description: *** Tu4, Telekom ISP#1 2M***
Tunnel transport MTU 1276 bytes

I noticed that when i shutdown the tunnel3 and 4 ...then everthing works normal but this line is only 2Mb.

Lee Smitherman · ‎06-26-2012

Hi,

Im still not quite sure on your routing, it looks as though you have load balancing in some form across the 10mb and 2Mb circuit.

Could you again (sorry) in one place

1)let me know your PC IP

2)show ip route on the primary (Tunnel 122 and 322) router

3)should ip route on 10.22.6.254.

Need to see the paths the packets are taking.

Lee.

nkarpysh · ‎06-26-2012

If problem is service impacting I would recomend to open Service Request with Cisco TAC. You will have dedicated engineer who can run live TS session with you and help to narrow down the issue.

Nik

HTH,
Niko

Sandeep Choudhary · ‎06-27-2012

HI ....Lee,

My pc ip is 10.18.19.7.

sh ip route on remote router :

RemoteCVPN1#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is 10.14.122.1 to network 0.0.0.0

     212.185.41.0/32 is subnetted, 2 subnets
S       HQISP2 [1/0] via REISP2
S       HQISP2 [1/0] via REISP1
     212.185.199.0/32 is subnetted, 1 subnets
S       212.185.199.2 [1/0] via REISP2
                      [1/0] via REISP1
     195.243.193.0/29 is subnetted, 1 subnets
C       REISP1GW is directly connected, FastEthernet0/0
     195.243.201.0/29 is subnetted, 1 subnets
C       REISP2GW is directly connected, FastEthernet0/1
     10.0.0.0/8 is variably subnetted, 8 subnets, 3 masks
C       10.13.22.0/30 is directly connected, Tunnel1
C       10.14.22.0/30 is directly connected, Tunnel3
C       10.22.6.0/24 is directly connected, Vlan6
D       10.18.2.0/24 [90/26905600] via 10.14.122.1, 17:40:24, Tunnel4
S       10.22.0.0/16 [1/0] via 10.22.6.1
C       10.22.3.0/24 is directly connected, Vlan3
C       10.13.122.0/30 is directly connected, Tunnel2
C       10.14.122.0/30 is directly connected, Tunnel4
     62.0.0.0/32 is subnetted, 1 subnets
S       INISP3 [1/0] via REISP2
                      [1/0] via REISP1
     195.243.205.0/32 is subnetted, 2 subnets
S       HQISP1 [1/0] via REISP1
S       HQISP1 [1/0] via REISP2
D*EX 0.0.0.0/0 [170/26905600] via 10.14.122.1, 17:40:24, Tunnel4

Lee Smitherman · ‎06-27-2012

Morning,

Nik is right, if this is production affecting, you need to get an engineer hands on, onto your network.

From looking at the outputs,

From your PC 10.18.19.7 > 10.22.6.254 remote site, you have two paths in use down Tunnel 322 and 122 from your primary core router. Then the Route back to your PC 10.18.19.7 is via Tunnel4! to your secondary router, now this may be by design but isnt Tunnel4 only 2MB??? And also why two tunnels from the same router on your Primary to the remote site.

You really need to have an audit of your routing as Im very worried its not right.

Lee.

Sandeep Choudhary · ‎06-27-2012

Hi Lee,

we have 4 tunnels from HQ to remote:

HQ1Provider 1st ip (Tunnel122)--------------------------------------------tunnel1REMOTE(10Mb)1st provider

HQ2Provider 1st ip (Tunnel222)--------------------------------------------tunnel2REMOTE(10Mb)1st provider

HQ1Provider 2nd ip (Tunnel322)--------------------------------------------tunnel3REMOTE(2Mb)2nd provider-For backup

HQ2Provider 2nd ip (Tunnel422)--------------------------------------------tunnel4REMOTE(2Mb)2ndprovider- For backup

Just for redudency we have 4 tunnels...

Lee Smitherman · ‎06-27-2012

Hi,

Well from your PC`s subnet 10.18.19.0/24 on your Primary Router 1 you have two routes to the remote site router subnet(10.22.6.0/24)

D 10.22.6.0/24 [90/297246976] via 10.14.22.2, 17:16:05, Tunnel322

[90/297246976] via 10.13.22.2, 17:16:05, Tunnel122

From the Remote Subnet Router(10.22.6.254) you dont have a specific route back to your PC Subnet(10.18.19.0/24) so the default is used... which is only 2MB!!!s

D*EX 0.0.0.0/0 [170/26905600] via 10.14.122.1, 17:40:24, Tunnel4

Now your prod subnets may use different routes, but my point is and as other people have stated, it does look like a bandwidth issue and the routing points to a bandwidth issue, Greater than 10MB out and only 2MB back for this particular traffic flow.

If your not comfortable with sorting out your routing, Ill be frank, the only way you are going to fix this is by having an engineer visit site/remote, understanding your network and services and then implimenting a plan to fix.

Lee.

Sandeep Choudhary · ‎06-27-2012

Hi Lee,

I will try to rsolve this and let you know the result.

Thanks for help.

Regards

Lee Smitherman · ‎06-27-2012

Good luck!