CoreSW VL issues / high rxload and slow response moving files internally

[rikdrt1]

i have a pair of large 6807's as the core.

12-15 VLANS.   5 of these have 200 users each. 

the rest are smaller ones.  

issue started when users from a specific VL12 was seeing a 20X slower response to move files from inside the same facility.

VL12 (slowVL)

VL34

VL56

VL78

a host anywhere on the  VL's , originating on VL12 is 55-455kbp to move a 100Mb file as an example. 

anywhere else its seconds and done before you can see how fast it did it. 

after poking around , this is what i found if i check VL12 (problem segment) as comparted to the others. 

why is the RXload so high.   i checked , or am checking every Switch that has these VL's primarily. 

checking for errors on each, even change GBICs, layer 1 stuff and nothing yet. 

any help is appreciated as to where to look.  never seen a VL with anything this high.   i suspect someone is connected a wrong way but i am also checking ports individually.   most of the VL12 users are on 8 4510's we have.    no luck yet on this.    Thanks. 

b8m-s01#sh int vlan 12 | i load
reliability 255/255, txload 10/255, rxload 92/255
b8m-s01#sh int vlan 34 | i load
reliability 255/255, txload 1/255, rxload 1/255

[balaji.bandi]

May be based on the information i can provide some inputs - but good to see your network where you testing port ?

what is the source IP address - what is the destination IP address ?

You need to tell what is VLAN 12 compare to other vlans ? is the VLAN 12 is major transit vlan ? so most of the traffic going via VLAN (this was only guess)

when was this issue started ? what is the change ?

can you provider show interface gx/x  statistics  also show interface vlan 12 and 34

[rikdrt1]

Any source on VL12 to any other internal host .. moving 100Mb.. about 3-5 min moving at Kbps
Any source not on VL12 to any other internal host - moving 100Mb. 1 sec moving at Mbps

They are split somewhat evenly between VL
About 200-250 hosts per VL roughly ..
Total of about 1500 hosts on the top 5 VL's and another 500 on the other smaller 10 VL's

But yes, I think we have a lot of our core firewalls and other components on VL12.
Although this is a PC-VL only so technically maybe 98% are just desktops and the rest is other.

Not sure when it started, but its been gradual I think and its been more noticeable - because of recent issue where one side of this bldg. was trying to move files over to another part and that's how I got wind of it. Anything on VL12 going to VL78 is slow.

I even moved my laptop around the access switches yesterday and noticed that any of the VL12 primary switches (4507/4510's) were slow. As soon as I moved over to vL34 or 56... its perfect and the rxload is <2

In other words. VL12 (primarily) connects in two computer room areas. These two computer rooms have about 5 4510's there. If I plugged my laptop to any of the 5 switches to copy a file to a share on VL78 - slow. When I moved over to the other pair of rooms that connect on VL34 or VL56... those move the same file anywhere in seconds.

I also noticed that internet speed OUTbound of the BLGD onto our routers is not affected. Internet speed out is same everywhere.
Only when moving between VL's internally - which is odd.


b8m-s01#sh int vlan 12
Vlan12 is up, line protocol is up
Hardware is EtherSVI, address is 0896.adf7.5900 (bia 0896.adf7.5900)
Description: PC-TR1-2
Internet address is 10.10.88.1/23
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 47/255, rxload 104/255
Encapsulation ARPA, loopback not set
Keepalive not supported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters 2w2d
Input queue: 0/75/6/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 410497000 bits/sec, 61638 packets/sec
5 minute output rate 185301000 bits/sec, 50165 packets/sec
L2 Switched: ucast: 141839651 pkt, 32808771878 bytes - mcast: 0 pkt, 0 bytes
L3 in Switched: ucast: 98645699220 pkt, 66162373107444 bytes - mcast: 0 pkt, 0 bytes
L3 out Switched: ucast: 54535610394 pkt, 13026876000700 bytes - mcast: 0 pkt, 0 bytes
98710197652 packets input, 65809981225077 bytes, 0 no buffer
Received 0 broadcasts (89867 IP multicasts)
0 runts, 0 giants, 1 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
54597901986 packets output, 12850923421539 bytes, 0 underruns
0 output errors, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
b8m-s01# sh int vlan 34
Vlan34 is up, line protocol is up
Hardware is EtherSVI, address is 0896.adf7.5900 (bia 0896.adf7.5900)
Description: PC-TR3-4
Internet address is 10.10.90.1/23
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not supported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters 2w2d
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 842000 bits/sec, 529 packets/sec
5 minute output rate 5838000 bits/sec, 768 packets/sec
L2 Switched: ucast: 18219017 pkt, 1323174332 bytes - mcast: 0 pkt, 0 bytes
L3 in Switched: ucast: 345779552 pkt, 180531111651 bytes - mcast: 0 pkt, 0 bytes
L3 out Switched: ucast: 431461993 pkt, 420711605961 bytes - mcast: 0 pkt, 0 bytes
348358759 packets input, 179423119373 bytes, 0 no buffer
Received 0 broadcasts (136634 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
433013482 packets output, 419083555078 bytes, 0 underruns
0 output errors, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out

[balaji.bandi]

Can you able to post-show interface g x/x where you mentioned building to building connection both the side.

Just to get my head around ( good to have your small rough network diagram how these are connected)

Some Queue drops :  ( do you have any QoS configured ?)   - Good to have sample configuration with topology as i asked before

Input queue: 0/75/6/0 (size/max/drops/flushes); Total output drops: 0

can you do 1 test. what if Laptop 1 and Laptop connect to the same switch in VLAN 12 and transfer files?  what is the outcome?

[rikdrt1]

I do have qoa on all ports
All bldg switches connect to this pair of 6807 core switches. They are running VSL so every switch has an uplink / port channel to core. 2 fibers per sw

I’ll get a diagram but simple star config.

I did test a file transfer inside VL12 and it’s fine. Second to move 100mb.

If I connect to any dhcp port on all 5 switches .. same slow response. It’s not just a specific port it’s anywhere I pick a vl12 address.

What I didn’t try is connecting laptop then changing VL to other than vl12 to see if it gets improved. My guess is it will. I’ll know Monday.

Thanks

[balaji.bandi]

If this was occured recently and it was working, then there may be something changed in the network.

can you post one of the switch config full to understand

what IOS code running, also post sdm prefer

[rikdrt1]

They are all running the same sort of IOS version 15.3(04r)

As far as the switch configs, they too are somewhat standard throughout.

I believe or it seems that there may be an issue with someone plugging into this VL incorrectly but I cannot prove it.



What is sdm prefer ?

[paul driver]

Hello

Do you have port security /storm control on the switchports for BUM traffic if no, suggest apply some and monitor?
Check for unwarranted span sessions
Turn off any negotiation protocols such a DTP, directed broadcasts ( magic packet/wake on lan), make sure they don’t participate in any stp convergence.
Make each host port an administrative mode of access.

show monitor sessions

int x/x
description Access-posrts
switchport mode access <--------------------------Access administrative mode and disables DTP (dynamic trucking)
switchport access vlan x
switchport nonegotiate <--------------------------Disable DTP (dynamic trunking)
no cdp enable <--- disables cdp discoery
switchport port-security maximum 3<---------------Allows a maximum static/dynamic learned mac address
switchport port-security aging time 10<------------Aging period of 10 mins or port inactivity
switchport port-security aging type inactivity<---Aging period is reached any learned mac address are flushed from the port
switchport port-security<--------------------Port-security enabled
no logging event link-status<-----------------disable logging buffer/syslog etc.. from report link status up/down
udld port aggressive<-----------------------enables Unidirectional links for copper
no snmp trap link-status<-------------------disable snmp trap being sent to snmp managers link status up/down
storm-control broadcast level 10.00<--------- storm control for broadcast/mutlicast traffic 10% of link bandwidth
storm-control multicast level 10.00<---------- storm control for multicast/broadcast 10% of link bandwidth
spanning-tree portfast<---------------------doesn’t participate in stp learning process transition straight into forwarding state
spanning-tree bpduguard enable --- negates BPDUs from being generated on the port

As for the L3 make sure you are at least perfroming fast switching and dont have directed broadcast
int vlan 12
ip route-cache
no ip directed broadcast