Packet loss on many remote location - Page 2

Sandeep Choudhary · ‎06-19-2012

Hello Everyone,

I am facing this issue from long time and still coudn't find the core issue.

specially we are using CAD servers and some applications(e.g. outlook...email...ftp) on the remote location

Every remote location is connected via gre tunnels and dial back technic from HQ.

can any body give me some suggestion to try something on remote router or HQ router to eliminate these problems.

Regards

Lee Smitherman · ‎06-25-2012

Hi,

Can you pick a remote site, and from your Core router perform an extended ping using a large packet size(1400 will do) to that remote site, let it run for 200-1000 pings and copy paste the results below?

Lee.

Sandeep Choudhary · ‎06-25-2012

Here is the output:

HQVPN2#ping

Protocol [ip]:

Target IP address: 10.22.6.254

Repeat count [5]: 500

Datagram size [100]: 1400

Timeout in seconds [2]:

Extended commands [n]: y

Source address or interface:

Type of service [0]:

Set DF bit in IP header? [no]:

Validate reply data? [no]:

Data pattern [0xABCD]:

Loose, Strict, Record, Timestamp, Verbose[none]:

Sweep range of sizes [n]:

Type escape sequence to abort.

Sending 500, 1400-byte ICMP Echos to 10.22.6.254, timeout is 2 seconds:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!

Success rate is 100 percent (500/500), round-trip min/avg/max = 24/32/280 ms

Earlier it was 80% Success rate but now there is not so much traffice of staa transfer so its 100 but still delay...is 280ms...on 3rd attempt.

280 ms ...this is too much .......................noram is 30 ..

Regards

Lee Smitherman · ‎06-25-2012

Hi,

You mentioned that these are GRE Tunnels, so presuming you are pinging the remote ip address of a device across the tunnel. Are you able to perform the above procedure for the same remote site but for the tunnel endpoint/destination(so the packet is not encapsulated with a GRE Header)???

Lee.

Sandeep Choudhary · ‎06-25-2012

These site are connected with tunnels:

I am pinging the remote vpn router address from the HQ vpn router.

Yes the packet is encapulated with ipsec hearder.

But the problem is while accessing or transfer file from remote to HQ or vice versa,...that time lots of request time out and too much delay.

here is the output from my pc to remote vpn router with size 1400:

Regards

Lee Smitherman · ‎06-25-2012

Hi,

So the output if from a ping thats not being encapsulated with a GRE Headder? Im trying to work out if the delay is circuit based or processing based. Since all sites have the issue, I suspect the issue is with the core router, and since you are getting delays and lost packets with encapsulated and non encapsulated traffic, My first move would be to log a fault with your service provider to check your circuit.

Lee

Lee Smitherman · ‎06-25-2012

Also just out of interest, can you provide a ping from your PC, using 1400 bytes, to the local tunnel interface of the router(Core) that you ran your ping test from?

Sandeep Choudhary · ‎06-25-2012

Hi lee,

Its not with all sites....

its happeing with 2-3 sites.....................every ping is encapsulated......we are only connected via tunnels..........so ..either i am pining from my pc or core router all are encapsulated.

This is the ping from Tunnel start point (Core router) to endpoint(Remote router):

HARCVPN2#ping 10.14.122.2 source 10.14.122.1 repeat 500

Type escape sequence to abort.
Sending 500, 100-byte ICMP Echos to 10.14.122.2, timeout is 2 seconds:
Packet sent with a source address of 10.14.122.1
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!
Success rate is 100 percent (500/500), round-trip min/avg/max = 20/26/212 ms

Here is the ping from my pc to the tunne at core:

Regards

Lee Smitherman · ‎06-25-2012

Ok, so you have multiple sites, which at the core runs multiple tunnels over one physical circuit? What type of circuit is it?

Do you have any QOS on that router? Any chance you could provide a logical network diagram of router, inside lan and circuits + tunnels, also the config of the core router would be good.

Lee.

Sandeep Choudhary · ‎06-25-2012

Hi Lee,

Here i am attaching the LAN_WAN layout:

config of core router:

crypto pki trustpoint TP-self-signed-2121164001
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-2121164001
revocation-check none
rsakeypair TP-self-signed-2121164001
!
!
crypto pki certificate chain TP-self-signed-2121164001
certificate self-signed 01
3082024B 308201B4 A0030201 02020101 300D0609 2A864886 F70D0101 04050030
31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274
69666963 6174652D 32313231 31363430 3031301E 170D3037 30363235 30393331
30325A17 0D323030 31303130 30303030 305A3031 312F302D 06035504 03132649
4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D32 31323131
36343030 3130819F 300D0609 2A864886 F70D0101 01050003 818D0030 81890281
8100C41B 8CD6A883 37388D70 3F4E60A8 1F0B2F35 E90AEC71 AA9EDBF5 2F10AE5D
48D2D343 644D6069 97D814A2 CF79AA3D EAED20E9 C98D57A0 BD682352 2CA30364
F7BF86B3 0E8E6AA6 F8E1AB2A BC6D6CF7 4ECF0D28 3007912C 8E656473 EA561509
22D10837 1BF32C04 3F214438 DFAC856E 8C2F3824 617A77C9 947D4989 B851812F
5F1302
quit
username admin password 7 bbbbbbbbbbbbbbbbb,mmmmm

!
!
controller E1 4/0
pri-group timeslots 1-31
!
!
!
crypto isakmp policy 10
encr 3des
authentication pre-share
!
crypto isakmp policy 20
encr aes 256
authentication pre-share
crypto isakmp key qar20060621hob address xxx.xxx.xxx.xxx no-xauth
crypto isakmp key qar20060621hob address xxx.xxx.xxx.xxy no-xauth
.
!
!
crypto ipsec transform-set ts_xyz esp-3des esp-sha-hmac
mode transport
crypto ipsec transform-set ts_xyz_aes esp-aes esp-sha-hmac
mode transport
!
crypto ipsec profile xyz_vpn_profile
set transform-set ts_xyz!
crypto ipsec profile xyz_vpn_profile_aes
set transform-set ts_xyz_aes
!
!
!
!
!
!
interface Tunnel222
description *** cccvvvvv Tu2 ***
ip address 10.13.122.1 255.255.255.252
ip mtu 1300
ip tcp adjust-mss 1260
tunnel source xxx.xxx.xxx.xxx

tunnel destination xxx.xxx.xxx.xxx

tunnel protection ipsec profile xyz_vpn_profile_aes
.
.
interface Tunnel422
description *** cccvvvvv Tu4 ***
ip address 10.14.122.1 255.255.255.252
ip mtu 1300
ip tcp adjust-mss 1260
tunnel source xxx.xxx.xxx.xxx

tunnel destination xxx.xxx.xxx.xxx

tunnel protection ipsec profile xyz_vpn_profile_aes
.
!
interface GigabitEthernet0/0
description *** Internet Flex2 ***
ip address xxx.xxx.xxx.xxx 255.255.255.240 secondary
ip address xxx.xxx.xxx.xxx 255.255.255.240
ip access-group internet in
no ip redirects
no ip unreachables
no ip proxy-arp
ip inspect myfw out
duplex full
speed 100
media-type rj45
negotiation auto
no cdp enable
!
interface GigabitEthernet0/1
description *** LAN connection to nbjkj1 Giga 12/39 ***
ip address 10.18.2.5 255.255.255.0
no ip redirects
ip route-cache flow
delay 100
duplex auto
speed auto
media-type rj45
negotiation auto
no cdp enable
standby 6 ip 10.18.2.6
standby 6 priority 110
standby 6 preempt
standby 6 authentication VPN
standby 6 track GigabitEthernet0/0
standby 26 ip 10.18.2.26
standby 26 preempt
standby 26 authentication VPN2
standby 26 track GigabitEthernet0/0
!
interface FastEthernet1/0
no ip address
shutdown
duplex auto
speed auto
!
interface FastEthernet1/1
no ip address
shutdown
duplex auto
speed auto
no cdp enable
!
interface Serial4/0:15
description *** PMX 96F 653467376878383-91198-0 ***
no ip address
encapsulation ppp
dialer rotary-group 3
dialer-group 3
isdn switch-type primary-net5
no cdp enable
!
interface Dialer3
description *** ISDN Backup-Interface 09567737 ***
bandwidth 64
ip address 10.13.250.1 255.255.255.0
encapsulation ppp
dialer-group 3
peer default ip address pool backup_einwahl
no cdp enable
ppp authentication chap callin
ppp multilink
!
router eigrp 1
redistribute static metric 100000 100 255 1 1500
passive-interface FastEthernet1/0
network 10.0.0.0
maximum-paths 1
distribute-list prefix no_default out GigabitEthernet0/1
distribute-list prefix filter_eigrp out Dialer3
distribute-list prefix filter_eigrp out Tunnel222
distribute-list prefix filter_eigrp out Tunnel422

no auto-summary
neighbor 10.18.2.4 GigabitEthernet0/1
neighbor 10.18.2.21 GigabitEthernet0/1
neighbor 10.13.122.2 Tunnel222
neighbor 10.14.122.2 Tunnel422

!
router eigrp 26
redistribute static route-map default_only
network 10.13.126.0 0.0.0.3
network 10.18.2.0 0.0.0.255
no auto-summary
neighbor 10.18.2.10 GigabitEthernet0/1
neighbor 10.18.2.11 GigabitEthernet0/1
neighbor 10.18.2.4 GigabitEthernet0/1
!
router eigrp 24
redistribute static route-map default_only
network 10.13.124.0 0.0.0.3
network 10.18.2.0 0.0.0.255
no auto-summary
!
ip local pool backup_einwahl 10.13.250.2 10.13.250.254
ip route 0.0.0.0 0.0.0.0 10.18.2.254

ip route xxx.xxx.xxx.xxx 255.255.255.255 xxx.xxx.xxx.xxx name cccvvvvv_Tu422
ip route xxx.xxx.xxx.xxx 255.255.255.255 xxx.xxx.xxx.xxx name cccvvvvv_Tu222

!
ip flow-export source GigabitEthernet0/1
ip flow-export version 5
ip flow-export destination 10.18.2.99 9995
!
no ip http server
ip http authentication local
ip http secure-server
ip http timeout-policy idle 600 life 86400 requests 10000
!
ip access-list extended internet
..
.
..
.
.
..
.
.
....
!
ip prefix-list default_route seq 5 permit 0.0.0.0/0
!
ip prefix-list filter_eigrp seq 5 permit 0.0.0.0/0
ip prefix-list filter_eigrp seq 10 permit 10.18.2.0/24
logging trap debugging
logging source-interface GigabitEthernet0/1
logging 10.18.4.11
logging 10.18.2.17
access-list 1 permit 10.18.4.11
access-list 101 permit ip 10.18.0.0 0.0.255.255 10.99.0.0 0.0.255.255
access-list 120 permit ip 10.18.0.0 0.0.255.255 10.99.0.0 0.0.255.255
dialer-list 3 protocol ip permit
snmp-server community cwlesen RO
snmp-server community SNMPm!d5vG4% RO 1
snmp-server location DC II
snmp-server enable traps tty
snmp-server enable traps envmon
snmp-server enable traps bgp
snmp-server enable traps config
snmp-server enable traps frame-relay
snmp-server enable traps frame-relay subif
snmp-server enable traps hsrp
snmp-server enable traps rtr
no cdp run
!
route-map default_only permit 10
match ip address prefix-list default_route
set metric 100000 100 255 1 1500
!
!
!
!
control-plane
!
!

Lee Smitherman · ‎06-25-2012

Hi,

thanks for the info, you might want to xxxxxxx out some key IP addressing such as your Internet connection.

Lee

Lee Smitherman · ‎06-25-2012

Hi,

from your PC can you ping 10.13.122.1 with a packet size 1400 and 1200

and 10.13.122.2(should be the remote end of the tunnel at the remote site) with a packet size of 1400 and 1200.

Lee.

Lee Smitherman · ‎06-25-2012

Hi,

Im assuming the above is one of the affected sites. Could from your PC you also run traceroutes to a device on the affected remote site, could you run it a couple of times.

lee

Sandeep Choudhary · ‎06-25-2012

Here is the screenshot:

Tracert remote router from my pc.

Lee Smitherman · ‎06-25-2012

Nice one, and could we also have a "show ip route" from your core.

Sandeep Choudhary · ‎06-25-2012

CoreVPN1#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is 10.18.2.254 to network 0.0.0.0

C       10.14.22.0/30 is directly connected, Tunnel322
D       10.22.6.0/24 [90/297246976] via 10.14.22.2, 17:16:05, Tunnel322
                     [90/297246976] via 10.13.22.2, 17:16:05, Tunnel122

D 10.22.3.0/24 [90/297246976] via 10.14.22.2, 17:16:05, Tunnel322
[90/297246976] via 10.13.22.2, 17:16:05, Tunnel122

S* 0.0.0.0/0 [1/0] via 10.18.2.254