Possible loop issue?

sullyjman12 · ‎09-26-2011

We have an MPLS network that is managed by Verizon network between two different states (VA and TX). We have are hosting Cognos in VA and some of the users over the MPLS network in TX are experiencing issues with accessing the site. In VA I havent heard of any issues so im looking at the MPLS network. Issues include delayed logins, delayed roaming the folders, it just hangs like its waiting for the server to respond. I have seen this first hand at the office.

I was leaning towards a MTU issue but a wireshark caputure didnt show much of restransmission (back of my mind still wants to blame this). Anything tips on what else I should look at with the wireshark I would be greatful.

Also I noticed some strange, when I do a traceroute from a client in TX to the Cognos server here in VA I get these results:

tracert 10.130.12.15

Tracing route to cognosserver [x.x.x.x]

over a maximum of 30 hops:

1    <1 ms    <1 ms    <1 ms x.x.x.x (TX)
2    <1 ms    <1 ms    <1 ms x.x.x.x (TX)
3     7 ms     7 ms     7 ms x.x.x.x (MPLS network)

4 43 ms 43 ms 43 ms x.x.x.x (MPLS Network)

5 45 ms 44 ms 45 ms cognosserver [x.x.x.x] (VA)
6 47 ms 46 ms 44 ms cognosserver [x.x.x.x] (VA)

Yes 5 and 6 is the same information hence why im scratching my head and maybe seeing a loop?

ameya_oke · ‎09-26-2011

From your source 10.130.12.15, check in your gateways routing table ,whether you are having 2 equal cost paths for cognosserver subnet (i.e if load balancing is occuring).

Post results.

AMeya

ameya_oke · ‎09-26-2011

"

3 7 ms 7 ms 7 ms x.x.x.x (MPLS network)

4 43 ms 43 ms 43 ms x.x.x.x (MPLS Network)

"

In above trace response both x.x.x.x are same or different.

I believe they must be different as latency varies, please confirm

Ameya

sullyjman12 · ‎09-26-2011

Line 3 is the exit point for the TX MPLS and 4 is the entrance on the VA side.

ameya_oke · ‎09-26-2011

If you have access to router in line 4(i.e VA router), please post the extended trace output from its wan interface to server.

Check if you are receiving the same response.

Also abt line 5 and 6, does this happen just twice or it goes on and on?

Ameya

ameya_oke · ‎09-26-2011

On VA router put below commands.

On LAn interface: ip tcp adjust-mss 1360 or 1380

On Wan Interface: ip mtu 1436 or 1430.

Ameya

sullyjman12 · ‎09-26-2011

I do have access to the VA router, if I apply these commands will these bring the network interface offline? If so I will have to wait till to do this after hours.

ameya_oke · ‎09-26-2011

Hi,

No it shouldnt,we are just chaning the MTU size.

But if you can wait then in mean time check MTU setting at VA's PE router and let them know about this change and get their opinion.

HTH

Ameya

sullyjman12 · ‎09-26-2011

We arent actually running the MPLS network, Verizon is providing it to us.

sullyjman12 · ‎09-26-2011

VA side:

Serial1/0 is up, line protocol is up

Hardware is DSXPNM Serial

Internet address is x.x.x.x

MTU 4470 bytes, BW 44210 Kbit/sec, DLY 200 usec,

reliability 255/255, txload 2/255, rxload 2/255

Encapsulation PPP, LCP Open

Listen: CDPCP

Open: IPCP, crc 16, loopback not set

Keepalive set (10 sec)

Last input 00:00:02, output 00:00:02, output hang never

Last clearing of "show interface" counters 3d05h

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 5985

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 492000 bits/sec, 227 packets/sec

5 minute output rate 498000 bits/sec, 229 packets/sec

17627294 packets input, 3389136851 bytes, 0 no buffer

Received 0 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

20919274 packets output, 2181931461 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 unknown protocol drops

DSU mode 0, bandwidth 44210, real bandwidth 44210, scramble 0

TX:

Serial1/0 is up, line protocol is up

Hardware is DSXPNM Serial

Internet address is x.x.x.x

MTU 4470 bytes, BW 44210 Kbit/sec, DLY 200 usec,

reliability 255/255, txload 3/255, rxload 1/255

Encapsulation PPP, LCP Open

Listen: CDPCP

Open: IPCP, crc 16, loopback not set

Keepalive set (10 sec)

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 4d04h

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 289000 bits/sec, 93 packets/sec

5 minute output rate 644000 bits/sec, 346 packets/sec

18741919 packets input, 1920149452 bytes, 0 no buffer

Received 0 broadcasts, 0 runts, 0 giants, 0 throttles

943 input errors, 942 CRC, 188 frame, 172 overrun, 0 ignored, 184 abort

31945330 packets output, 1902186475 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 unknown protocol drops

DSU mode 0, bandwidth 44210, real bandwidth 44210, scramble 0

Ivan Krimmel · ‎09-26-2011

before having configured something, it is wise to first check the actual MTU along the path.

sullyjman12 · ‎09-26-2011

as for your traceroute question, it happens twice then stops.

ameya_oke · ‎09-26-2011

Have you provided dual LAN uplinks to the server.

I mean some ethernet redundancy at server end (LPAR/teaming/bridgingetc...)

Some how from Va's router there are 2 requests hitting the server and giving you two replies.

Ameya

ameya_oke · ‎09-26-2011

Okay, as you having serial interfaces, definietly you should take ISP in concurrence while chaning MTU.

Ameya

sullyjman12 · ‎09-26-2011

The traceroute result is not limited to this one server. This "loop" happens between both sides. So if I do a traceroute from the VA office to a domain controller in TX I get the same results (double response)