Re: Possible loop issue? - Page 2

sullyjman12 · ‎09-26-2011

We have an MPLS network that is managed by Verizon network between two different states (VA and TX). We have are hosting Cognos in VA and some of the users over the MPLS network in TX are experiencing issues with accessing the site. In VA I havent heard of any issues so im looking at the MPLS network. Issues include delayed logins, delayed roaming the folders, it just hangs like its waiting for the server to respond. I have seen this first hand at the office.

I was leaning towards a MTU issue but a wireshark caputure didnt show much of restransmission (back of my mind still wants to blame this). Anything tips on what else I should look at with the wireshark I would be greatful.

Also I noticed some strange, when I do a traceroute from a client in TX to the Cognos server here in VA I get these results:

tracert 10.130.12.15

Tracing route to cognosserver [x.x.x.x]

over a maximum of 30 hops:

1    <1 ms    <1 ms    <1 ms x.x.x.x (TX)
2    <1 ms    <1 ms    <1 ms x.x.x.x (TX)
3     7 ms     7 ms     7 ms x.x.x.x (MPLS network)

4 43 ms 43 ms 43 ms x.x.x.x (MPLS Network)

5 45 ms 44 ms 45 ms cognosserver [x.x.x.x] (VA)
6 47 ms 46 ms 44 ms cognosserver [x.x.x.x] (VA)

Yes 5 and 6 is the same information hence why im scratching my head and maybe seeing a loop?

Ivan Krimmel · ‎09-26-2011

to find out whether the MTU size is indeed the culprit here, just issue a usual ping sweep with DF bit enabled.

sullyjman12 · ‎09-26-2011

This is all to the same host (from tx to cognos server)

ping servername -f -l 1472

Pinging servername [x.x.x.x] with 1472 bytes of data:

Reply from x.x.x.x: bytes=1472 time=47ms TTL=123
Reply from x.x.x.x: bytes=1472 time=46ms TTL=123
Reply from x.x.x.x: bytes=1472 time=47ms TTL=123
Reply from x.x.x.x: bytes=1472 time=46ms TTL=123

Ping statistics for 10.130.12.15:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 46ms, Maximum = 47ms, Average = 46ms

ping servername -f -l 1473

Pinging servername [x.x.x.x] with 1473 bytes of data:

Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.

Ping statistics for 10.130.12.15:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

ping servername -f -l 1500

Pinging servername [x.x.x.x] with 1500 bytes of data

Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.

Ping statistics for 10.130.12.15:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

ameya_oke · ‎09-26-2011

Follow below steps.

1) Ask user from location other than TX say NY, to access VA's server, take feedback.

--Also ask VA's user to access a server in NY.

2) From NY ask user to access a server in TX.

-- Also ask TX user to access server in NY.

We would come to know whether there is issue in VA or TX or MPLS as a whole.

Ameya

sullyjman12 · ‎09-26-2011

Im confused by your statement and NY? This is on an MPLS network provided by the ISP to connect our two remote offices together. Nothing is accesible from the outside.

ameya_oke · ‎09-26-2011

Okay,i assumed you hadusers at more than 2 locations .

NY was just an example.

Please take ISP in concurrence and post further updates.

Wish we could have helped.

Ameya

sullyjman12 · ‎09-27-2011

Did a ping from an ubuntu box:

From VA to DC in TX

traceroute to x.x.x.x (x.x.x.x), 30 hops max, 60 byte packets

1 x.x.x.x (x.x.x.x) 1.017 ms 1.514 ms 2.018 ms

2 x.x.x.x (x.x.x.x) 6.521 ms 6.846 ms 6.948 ms

3 x.x.x.x (x.x.x.x) 8.734 ms 8.731 ms 8.725 ms

4 x.x.x.x(x.x.x.x) 50.054 ms 50.052 ms 50.142 ms

5 x.x.x.x (x.x.x.x) 53.599 ms 53.654 ms 53.834 ms

6 x.x.x.x (x.x.x.x) 50.920 ms 49.459 ms 49.434 ms Domain controller in TX

5 and 6 are dupes still

The traceroute hangs between 3 and 4 (like its waiting for the connection to wake up or maybe its just because of the jump) Are these numbers bad?

ameya_oke · ‎09-27-2011

Not really, this a normal latency.

Did you check with your ISP for info on MTU along the path from TX to VA?

Ameya

sullyjman12 · ‎09-27-2011

I am working on getting the circuit information so I can call and inquire!

sullyjman12 · ‎09-27-2011

Interesting tidbit of testing, im messing around with iperf and here are some results I get:

I have an ubuntu box running in VA being the client and I have the domain controller (TX) the server. Here are the results I get:

[1820] local x.x.x.x port 5001 connected with x.x.x.x port 36312

[ ID] Interval Transfer Bandwidth

[1820] 0.0-10.3 sec 7.88 MBytes 6.43 Mbits/sec [1820] local x.x.x.x port 5001 connected with x.x.x.xport 36312
[ ID] Interval Transfer Bandwidth
[1820] 0.0-10.3 sec 7.88 MBytes 6.43 Mbits/sec

If I make the Domain Controller the client and the Ubuntu box the server, here are the results I get:

[1868] local x.x.x.x port 3283 connected with x.x.x.x port 5001

[ ID] Interval Transfer Bandwidth

[1868] 0.0-11.5 sec 15.6 MBytes 11.4 Mbits/sec [1868] local x.x.x.x port 3283 connected with x.x.x.x port 5001
[ ID] Interval Transfer Bandwidth
[1868] 0.0-11.5 sec 15.6 MBytes 11.4 Mbits/sec

I assume we should be getting the same amont of bandwidth both ways corrrect or at least speeds similar. Correct?

Seems like im getting double the speeds in the TX and half in VA.

ameya_oke · ‎09-27-2011

Yes, good piece of info.(How much is the BW promised by the ISP when you commissioned the link)

PLease test upload/download speeds with help of below tool.

First Download Ethernet throuput utility from below link.

http://www.softoxi.com/ethernet-throughput-utility.html

This is an awsome tool which i usually use to check WAN up/down link speeds.

Here is how it works.

topology:

[TX-client-PC]-----RTR1<-------WanLink---->RTR2---[VA-client-PC]

[Also You have to open port 5017 bidirectional on firewall if there is one in between.]

Now install one copy each of this utility on both mentioned PCs.

Open this tool on both PCs and do the below settings.

Step1: To check upload speed from TX to VA

TX-client-PC settings:

Scale: you have to specify the BW scale of your wan link.

In you case select 100Mb.

Connect : type IP of VA-client-PC

Now click "Run Sender" - basically this will start sending raw data on your Wan link.

VA-client-PC settings:

Scale: sameasabove

Connect : type IP of TX-client-PC

Now click "Run Listner" - this will start receivingraw data on your Wan link.(data which PC1 is sending above)

Step2: To check upload speed from VA to TX

Just check the sender and listners.

Ameya

sullyjman12 · ‎09-27-2011

I got a hold of Verizon and we are paying for a 45 Meg MPLS connection.

sullyjman12 · ‎09-29-2011

Its amazing the stuff you find even when you have been on the job for over 6 months, so I noticed the cubes have some kind of VPN

So I found out we have some kind of VPN module installed:

sh crypto engine brief
        crypto engine name: Virtual Private Network (VPN) Module
        crypto engine type: hardware
                     State: Enabled
                  Location: onboard 0
              Product Name: Onboard-VPN
                FW Version: 01100200
              Time running: 46387 seconds
               Compression: Yes
                       DES: Yes
                     3 DES: Yes
                   AES CBC: Yes (128,192,256)
                  AES CNTR: No
     Maximum buffer length: 4096
          Maximum DH index: 0700
          Maximum SA index: 0700
        Maximum Flow index: 1400
      Maximum RSA key size: 2048

        crypto engine name: Cisco VPN Software Implementation
        crypto engine type: software
             serial number: 6C25F8CF
       crypto engine state: installed
     crypto engine in slot: N/A

DALLAS-CUBE#sh crypto engine brief
        crypto engine name: Virtual Private Network (VPN) Module
        crypto engine type: hardware
                     State: Enabled
                  Location: onboard 0
              Product Name: Onboard-VPN
                FW Version: 01100200
              Time running: 46649 seconds
               Compression: Yes
                       DES: Yes
                     3 DES: Yes
                   AES CBC: Yes (128,192,256)
                  AES CNTR: No
     Maximum buffer length: 4096
          Maximum DH index: 0700
          Maximum SA index: 0700
        Maximum Flow index: 1400
      Maximum RSA key size: 2048

        crypto engine name: Cisco VPN Software Implementation
        crypto engine type: software
             serial number: 6C25F8CF
       crypto engine state: installed
--More--

Slot 0:

C3845 Mother board 1GE(TX,SFP),1GE(TX), integrated VPN and 4W Port adapter, 7 ports

Port adapter is analyzed

Port adapter insertion time 12:58:56 ago

Onboard VPN : FW ver01100200

EEPROM contents at hardware discovery:

PCB Serial Number : FOC13304177

Hardware Revision : 1.4

Top Assy. Part Number : 800-23616-07

Board Revision : C0

Deviation Number : 102138

Fab Version : 05

I looked at the config and I dont see any configuration that tell me the VPN module is being used. Is there anything in particular I need to look for?

!

! Last configuration change at 08:31:29 CDT Thu Sep 29 2011 by j

version 12.4

no service pad

service timestamps debug datetime msec localtime

service timestamps log datetime msec localtime

service password-encryption

service sequence-numbers

!

hostname

!

boot-start-marker

boot system flash:c3845-adventerprisek9_ivs-mz.eng-sp-124-20.T2

boot-end-marker

!

card type t3 1

logging message-counter syslog

logging buffered 2000000

no logging rate-limit

!

aaa new-model

!

aaa authentication login default group tacacs+ local

aaa authorization exec default group tacacs+ if-authenticated

aaa accounting exec default start-stop group tacacs+

aaa accounting commands 1 default start-stop group tacacs+

aaa accounting commands 15 default start-stop group tacacs+

aaa accounting network default start-stop group tacacs+

aaa accounting connection default start-stop group tacacs+

aaa accounting system default stop-only group tacacs+

!

aaa session-id common

clock timezone CST -6

clock summer-time CDT recurring

!

dot11 syslog

ip source-route

ip cef

!

ip domain name

ip name-server

no ipv6 cef

!

multilink bundle-name authenticated

!

voice-card 0

dspfarm

dsp services dspfarm

!

voice service voip

allow-connections sip to sip

fax protocol pass-through g711ulaw

h323

sip

bind control source-interface GigabitEthernet0/0

bind media source-interface GigabitEthernet0/0

rel1xx disable

early-offer forced

midcall-signaling passthru

!

voice class codec 1

codec preference 1 g711ulaw

codec preference 2 g729r8

!

voice translation-rule 8

rule 2 /^899\(.*\)/ /\1/

!

voice translation-rule 9

rule 2 /^999\(.*\)/ /\1/

!

voice translation-profile DIGITSTRIP-8

translate called 8

!

voice translation-profile DIGITSTRIP-9

translate called 9

!

crypto pki trustpoint TP-self-signed-1814427855

enrollment selfsigned

subject-name cn=IOS-Self-Signed-Certificate-1814427855

revocation-check none

rsakeypair TP-self-signed-1814427855

!

crypto pki trustpoint TP-self-signed-1589511804

enrollment selfsigned

subject-name cn=IOS-Self-Signed-Certificate-1589511804

revocation-check none

rsakeypair TP-self-signed-1589511804

!

crypto pki certificate chain TP-self-signed-1814427855

certificate self-signed 01

3082024F 308201B8 A0030201 02020101 300D0609 2A864886 F70D0101 04050030

31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274

69666963 6174652D 31383134 34323738 3535301E 170D3039 30383133 31373132

32335A17 0D323030 31303130 30303030 305A3031 312F302D 06035504 03132649

4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D31 38313434

32373835 3530819F 300D0609 2A864886 F70D0101 01050003 818D0030 81890281

8100D304 B6C1AA02 459E284F DE4921F3 8B7EE8C4 456F1C79 284AB9C2 4C20A300

DC2C2A21 3DCC5E26 9AD941E4 EAD17F8B 3234A152 4C573C4E 5BA55EA1 6FEB5694

F9C6ABCA 4BF27035 12A1FE03 540592E3 8FDA5044 08482306 D7151897 38F93CC9

A3B038C2 0F0B57B1 DEDDEE07 6B4DECE2 AE9C1F5C 22CEC30C 15F45C1E DF280AA8

B6730203 010001A3 77307530 0F060355 1D130101 FF040530 030101FF 30220603

551D1104 1B301982 17796F75 726E616D 652E796F 7572646F 6D61696E 2E636F6D

301F0603 551D2304 18301680 1458A4CE 0E86D7A4 C06AC44E 7C39628B 943F5ED4

BA301D06 03551D0E 04160414 58A4CE0E 86D7A4C0 6AC44E7C 39628B94 3F5ED4BA

300D0609 2A864886 F70D0101 04050003 81810011 10CC4251 2455DEA8 56828713

8A85726A 2B3540C3 B407C86F 837E73B2 8ABAE691 2321C5A7 C2CEE8A8 24BA09D9

6AF46448 6C30B81B 7AD162D1 B4A2A666 DCAAE1C6 4106244D C22B46AE 6BDD0075

01AB7E2A 379569F7 2023F5EC 5FC205E3 B34C47D6 9E82CEC8 EEACB88E 8C8CCFD7

27D332FD E574EE23 43324720 6034223D 97FB5C

quit

crypto pki certificate chain TP-self-signed-1589511804

!

archive

log config

hidekeys

!

controller T3 1/0

clock source line

cablelength 200

!

ip ssh time-out 60

ip ssh version 2

!

interface GigabitEthernet0/0

description

ip address

load-interval 30

duplex auto

speed auto

media-type rj45

!

interface GigabitEthernet0/1

no ip address

shutdown

duplex auto

speed auto

media-type rj45

!

interface Serial1/0

description

ip address

ip flow ingress

ip flow egress

encapsulation ppp

no peer neighbor-route

dsu bandwidth 44210

!

router eigrp 100

redistribute bgp 65110 metric 10000 100 255 1 1500

passive-interface default

no passive-interface GigabitEthernet0/0

network 192.168.0.0 0.0.255.255

distance eigrp 90 10

no auto-summary

!

router bgp 65110

no synchronization

no bgp transport path-mtu-discovery

bgp log-neighbor-changes

network 0

network 0 mask 255.255.255.0

network 0 mask 255.255.255.252

aggregate-address summary-only

neighbor remote-as 65000

neighbor soft-reconfiguration inbound

no auto-summary

!

ip forward-protocol nd

no ip http server

no ip http secure-server

ip http timeout-policy idle 60 life 86400 requests 10000

!

ip flow-export source Serial1/0

ip flow-export version 9

ip flow-export destination

!

ip tacacs source-interface GigabitEthernet0/0

!

ip access-list standard SOLARWINDS

logging facility local0

logging source-interface GigabitEthernet0/0

!

tacacs-server host key 7 123E001607133F052325303B

!

control-plane

!

call threshold global cpu-avg low 68 high 75

call threshold global total-mem low 75 high 85

call threshold global total-calls low 9 high 11

!

voice-port 0/0/0

!

voice-port 0/0/1

!

ccm-manager fallback-mgcp

ccm-manager redundant-host

ccm-manager mgcp

ccm-manager music-on-hold

ccm-manager config server

ccm-manager config

!

mgcp

mgcp call-agent 2427 service-type mgcp version 0.1

mgcp dtmf-relay voip codec all mode out-of-band

mgcp rtp unreachable timeout 1000 action notify

mgcp modem passthrough voip mode nse

mgcp package-capability rtp-package

mgcp package-capability sst-package

mgcp package-capability pre-package

no mgcp package-capability res-package

no mgcp timer receive-rtcp

mgcp sdp simple

mgcp fax t38 inhibit

mgcp rtp payload-type g726r16 static

mgcp behavior g729-variants static-pt

!

mgcp profile default

!

sccp local GigabitEthernet0/0

sccp ccm 5 identifier 2 priority 2 version 6.0

sccp ccm identifier 1 version 6.0

sccp ccm identifier 5 priority 1 version 6.0

sccp

!

sccp ccm group 10

bind interface GigabitEthernet0/0

associate ccm 5 priority 1

associate ccm 2 priority 2

associate profile 12 register

associate profile 14 register

associate profile 13 register

!

dspfarm profile 13 transcode

codec g711ulaw

codec g711alaw

codec g729ar8

codec g729abr8

codec g729r8

maximum sessions 4

associate application SCCP

!

dspfarm profile 12 conference

description conference bridge

codec g711ulaw

codec g711alaw

codec g729ar8

codec g729abr8

codec g729r8

codec g729br8

maximum sessions 4

associate application SCCP

!

dspfarm profile 14 mtp

codec g711ulaw

maximum sessions hardware 4

associate application SCCP

!

sullyjman12 · ‎09-29-2011

Also I noticed this:

Interface IP-Address OK? Method Status Protocol

GigabitEthernet0/0 xxxx YES NVRAM up up

GigabitEthernet0/1 unassigned YES NVRAM administratively down down

Serial1/0 xxxxxYES NVRAM up up

SSLVPN-VIF0 unassigned NO unset up up

So is the SSLVPN-VIFO being used?

sullyjman12 · ‎09-29-2011

Looks like its not being used:

SSLVPN-VIF0 is up, line protocol is up

Hardware is SSLVPN_VIF

Interface is unnumbered. Using address of SSLVPN-VIF0 (0.0.0.0)

MTU 1406 bytes, BW 56 Kbit/sec, DLY 5000 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation UNKNOWN, loopback not set

Last input never, output never, output hang never

Last clearing of "show interface" counters never

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

5 minute input rate 0 bits/sec, 0 packets/sec

5 minute output rate 0 bits/sec, 0 packets/sec

0 packets input, 0 bytes, 0 no buffer

Received 0 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

0 packets output, 0 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 unknown protocol drops

0 output buffer failures, 0 output buffers swapped out