6500 wired port config - Page 2

oighodaro1 · ‎01-30-2015

Hello Everyone,

This is my first post and I am hoping I have come to the right place.

Background: I work for a non-profit and we are hoping to get new switches this summer. Our networking infrastructure consist of LAN and WLAN. We are currently using Cisco Catalyst 6509 for the LAN and Meraki APs for WLAN.

Problems/Symptoms:

We having experiencing different issues on different fronts. We are experiencing huge network slow downs when editing over the Network, the connections slow down almost to a halt. Also whenever users attempt to download a file large than 100mb their download halts between 50 to 70 mb. Additionally the log is showing constant flapping.

Log: has been attached.

some questions: What should I be looking for in the network connections ? Is the flapping causing the slow down ?

Any assistance/help would be greatly appreciated.

Thanks

Jon Marshall · ‎02-13-2015

Sincere apologies, I meant "sh spanning-tree detail"

Can you run that and there should be a line in the output detailing the "Number of Topology changes".

That's the line I'm interested in.

You mentioned separate vlans and I was wondering do you have wired clients in your office and also what about the servers ?

I assume they are all on the same vlan and using the same IP subnet.

Ideally the wireless, wired clients and servers would each have their own vlan but then that means their own IP subnet as well.

How feasible is it for you to change the IPs of the wired clients (presumably DHCP) and probably more difficult to change the server IPs.

I am just asking at the moment that's all.

Can you run that command and then we can go from there.

One last point. Just how often are you seeing these mac move notifications in your switch logs ?

Jon

oighodaro1 · ‎02-13-2015

VLAN0001 is executing the rstp compatible Spanning Tree protocol
  Bridge Identifier has priority 8196, address 00d0.0405.0c01
  Configured hello time 2, max age 20, forward delay 15, tranmsit hold-count 6
  We are the root of the spanning tree
  Topology change flag not set, detected flag not set
  Number of topology changes 51346 last change occurred 00:02:41 ago
          from GigabitEthernet1/2
  Times:  hold 1, topology change 35, notification 2
          hello 2, max age 20, forward delay 15 
  Timers: hello 0, topology change 0, notification 0, aging 300

 VLAN0010 is executing the rstp compatible Spanning Tree protocol
  Bridge Identifier has priority 8192, address 00d0.0405.0c0a
  Configured hello time 2, max age 20, forward delay 15, tranmsit hold-count 6
  We are the root of the spanning tree
  Topology change flag not set, detected flag not set
  Number of topology changes 26 last change occurred 3w4d ago
          from GigabitEthernet1/2
  Times:  hold 1, topology change 35, notification 2
          hello 2, max age 20, forward delay 15 
  Timers: hello 0, topology change 0, notification 0, aging 300

VLAN0020 is executing the rstp compatible Spanning Tree protocol
  Bridge Identifier has priority 8192, address 00d0.0405.0c14
  Configured hello time 2, max age 20, forward delay 15, tranmsit hold-count 6
  We are the root of the spanning tree
  Topology change flag not set, detected flag not set
  Number of topology changes 28 last change occurred 3w4d ago
          from GigabitEthernet1/2
  Times:  hold 1, topology change 35, notification 2
          hello 2, max age 20, forward delay 15 
  Timers: hello 0, topology change 0, notification 0, aging 300

VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
1    default                          active    Gi1/1, Gi3/1, Gi3/2, Gi3/5
                                                Gi4/1, Gi4/2, Gi4/3, Gi4/5
                                                Gi4/6, Gi4/7, Gi4/8, Gi4/9
                                                Gi4/10, Gi4/11, Gi4/12, Gi4/13
                                                Gi4/14, Gi4/15, Gi4/16, Gi4/17
                                                Gi4/18, Gi4/19, Gi4/20, Gi4/21
                                                Gi4/22, Gi4/23, Gi4/24, Gi4/25
                                                Gi4/26, Gi4/27, Gi4/28, Gi4/29
                                                Gi4/30, Gi4/31, Gi4/32, Gi4/33
                                                Gi4/34, Gi4/35, Gi4/36, Gi4/37
                                                Gi4/38, Gi4/39, Gi4/40, Gi4/41
                                                Gi4/42, Gi4/43, Gi4/44, Gi4/45
                                                Gi4/46, Gi4/47, Gi4/48, Gi5/1
                                                Gi5/2, Gi5/3, Gi5/4, Gi5/5
                                                Gi5/6, Gi5/7, Gi5/8, Gi5/9
                                                Gi5/10, Gi5/11, Gi5/12, Gi5/13
                                                Gi5/14, Gi5/15, Gi5/16, Gi5/17
                                                Gi5/18, Gi5/19, Gi5/20, Gi5/21
                                                Gi5/22, Gi5/23, Gi5/24, Gi5/25
                                                Gi5/26, Gi5/27, Gi5/28, Gi5/29
                                                Gi5/30, Gi5/31, Gi5/32, Gi5/33
                                                Gi5/34, Gi5/35, Gi5/36, Gi5/37
                                                Gi5/38, Gi5/39, Gi5/40, Gi5/41
                                                Gi5/42, Gi5/43, Gi5/44, Gi5/45
                                                Gi5/46, Gi5/47, Gi5/48, Gi6/1
                                                Gi6/6, Gi6/7, Gi6/8, Gi6/9
                                                Gi6/10, Gi6/11, Gi6/12, Gi6/13
                                                Gi6/14, Gi6/15, Gi6/16, Gi6/17
                                                Gi6/18, Gi6/19, Gi6/20, Gi6/21
                                                Gi6/22, Gi6/23, Gi6/24, Gi6/25
                                                Gi6/26, Gi6/27, Gi6/28, Gi6/29
                                                Gi6/30, Gi6/31, Gi6/32, Gi6/33
                                                Gi6/34, Gi6/35, Gi6/36, Gi6/37
                                                Gi6/38, Gi6/39, Gi6/40, Gi6/41
                                                Gi6/42, Gi6/43, Gi6/44, Gi6/45
                                                Gi6/46, Gi6/47, Gi6/48, Gi7/1
                                                Gi7/2, Gi7/3, Gi7/4, Gi7/5
                                                Gi7/6, Gi7/7, Gi7/8, Gi7/9
                                                Gi7/10, Gi7/11, Gi7/12, Gi7/13
                                                Gi7/14, Gi7/15, Gi7/16, Gi7/17
                                                Gi7/18, Gi7/19, Gi7/20, Gi7/21
                                                Gi7/22, Gi7/23, Gi7/24, Gi7/25
                                                Gi7/26, Gi7/27, Gi7/28, Gi7/29
                                                Gi7/30, Gi7/31, Gi7/32, Gi7/33
                                                Gi7/34, Gi7/35, Gi7/36, Gi7/37
                                                Gi7/38, Gi7/39, Gi7/40, Gi7/41
                                                Gi7/42, Gi7/43, Gi7/44, Gi7/45
                                                Gi7/46, Gi7/47, Gi7/48, Gi8/1
                                                Gi8/2, Gi8/3, Gi8/4, Gi8/5
                                                Gi8/6, Gi8/7, Gi8/8, Gi8/9
                                                Gi8/10, Gi8/11, Gi8/12, Gi8/13
                                                Gi8/14, Gi8/15, Gi8/16, Gi8/17
                                                Gi8/18, Gi8/19, Gi8/20, Gi8/21
                                                Gi8/22, Gi8/23, Gi8/24, Gi8/25
                                                Gi8/26, Gi8/27, Gi8/28, Gi8/29
                                                Gi8/30, Gi8/31, Gi8/32, Gi8/33
                                                Gi8/34, Gi8/35, Gi8/36, Gi8/37
                                                Gi8/38, Gi8/39, Gi8/40, Gi8/41
                                                Gi8/42, Gi8/43, Gi8/44, Gi8/45
                                                Gi8/46, Gi8/47, Gi8/48, Gi9/21
                                                Gi9/32, Gi9/40, Gi9/46, Po3
10   VLAN0010                         active    
20   VLAN0020                         active    
1002 fddi-default                     act/unsup 
1003 trcrf-default                    act/unsup 
1004 fddinet-default                  act/unsup 
1005 trbrf-default                    act/unsup 

VLAN Type  SAID       MTU   Parent RingNo BridgeNo Stp  BrdgMode Trans1 Trans2
---- ----- ---------- ----- ------ ------ -------- ---- -------- ------ ------
1    enet  100001     1500  -      -      -        -    -        0      0   
10   enet  100010     1500  -      -      -        -    -        0      0   
20   enet  100020     1500  -      -      -        -    -        0      0   
1002 fddi  101002     1500  -      -      -        -    -        0      0   
1003 trcrf 101003     4472  1005   3276   -        -    srb      0      0   
1004 fdnet 101004     1500  -      -      -        ieee -        0      0   
1005 trbrf 101005     4472  -      -      15       ibm  -        0      0   


VLAN AREHops STEHops Backup CRF
---- ------- ------- ----------
1003 7       7       off

Remote SPAN VLANs

Yeah the idea of having the different vlans would not work in this situation because they wouldn't be able to communicate.

below is the flapping log, i don't think the flapping is causing the issue because since the meraki is a mesh network and uses mimo, all the near by APs have to negiotate who will take the device in. thats why it happens so often.

104514: Feb 13 17:01:18.746: %MAC_MOVE-SP-4-NOTIF: Host 34a3.95d4.fd04 in vlan 1 is flapping between port Gi5/34 and port Gi4/13
104515: Feb 13 17:02:28.705: %MAC_MOVE-SP-4-NOTIF: Host 5c97.f315.afe4 in vlan 1 is flapping between port Gi5/33 and port Gi5/35
104516: Feb 13 17:03:03.733: %MAC_MOVE-SP-4-NOTIF: Host 7c7a.91e2.df83 in vlan 1 is flapping between port Gi5/34 and port Gi1/2
104517: Feb 13 17:03:38.732: %MAC_MOVE-SP-4-NOTIF: Host 5c97.f315.afe4 in vlan 1 is flapping between port Gi5/35 and port Gi5/33
104518: Feb 13 17:05:23.730: %MAC_MOVE-SP-4-NOTIF: Host 34a3.95d4.fd04 in vlan 1 is flapping between port Gi5/35 and port Gi5/34
104519: Feb 13 17:05:58.730: %MAC_MOVE-SP-4-NOTIF: Host 787e.6147.3468 in vlan 1 is flapping between port Gi4/47 and port Gi5/31
104520: Feb 13 17:05:58.730: %MAC_MOVE-SP-4-NOTIF: Host 0c3e.9f3f.4816 in vlan 1 is flapping between port Gi5/35 and port Gi5/33
104521: Feb 13 17:06:33.729: %MAC_MOVE-SP-4-NOTIF: Host 0c3e.9f3f.4816 in vlan 1 is flapping between port Gi5/33 and port Gi5/35
104522: Feb 13 17:07:08.729: %MAC_MOVE-SP-4-NOTIF: Host 64a3.cb36.e3d1 in vlan 1 is flapping between port Gi4/13 and port Gi5/34
104523: Feb 13 17:07:43.732: %MAC_MOVE-SP-4-NOTIF: Host 64a3.cb36.e3d1 in vlan 1 is flapping between port Gi5/35 and port Gi4/13
104524: Feb 13 17:08:18.728: %MAC_MOVE-SP-4-NOTIF: Host 34a3.95d4.fd04 in vlan 1 is flapping between port Gi1/2 and port Gi5/35
104525: Feb 13 17:08:18.728: %MAC_MOVE-SP-4-NOTIF: Host 0c3e.9f3f.4816 in vlan 1 is flapping between port Gi5/33 and port Gi5/35
104526: Feb 13 17:09:28.727: %MAC_MOVE-SP-4-NOTIF: Host 783a.8444.d58e in vlan 1 is flapping between port Gi5/34 and port Gi5/35
104527: Feb 13 17:09:28.731: %MAC_MOVE-SP-4-NOTIF: Host 7c7a.919d.7894 in vlan 1 is flapping between port Gi5/35 and port Gi5/34
104528: Feb 13 17:09:28.731: %MAC_MOVE-SP-4-NOTIF: Host 9c4e.36c9.ed4c in vlan 1 is flapping between port Gi5/35 and port Gi5/33
104529: Feb 13 17:09:28.731: %MAC_MOVE-SP-4-NOTIF: Host 64a3.cb36.e3d1 in vlan 1 is flapping between port Gi5/34 and port Gi5/35
104530: Feb 13 17:10:03.726: %MAC_MOVE-SP-4-NOTIF: Host 0c3e.9f3f.4816 in vlan 1 is flapping between port Gi5/31 and port Gi5/33
104531: Feb 13 17:10:03.730: %MAC_MOVE-SP-4-NOTIF: Host 0c3e.9f3f.4816 in vlan 1 is flapping between port Gi5/31 and port Gi5/35
104532: Feb 13 17:10:03.730: %MAC_MOVE-SP-4-NOTIF: Host 787e.6147.3468 in vlan 1 is flapping between port Gi5/35 and port Gi4/47
104533: Feb 13 17:11:13.725: %MAC_MOVE-SP-4-NOTIF: Host 9494.26dd.cf61 in vlan 1 is flapping between port Gi4/47 and port Gi7/31
104534: Feb 13 17:11:13.729: %MAC_MOVE-SP-4-NOTIF: Host 787e.6147.3468 in vlan 1 is flapping between port Gi4/13 and port Gi5/35
104535: Feb 13 17:11:48.724: %MAC_MOVE-SP-4-NOTIF: Host 787e.6147.3468 in vlan 1 is flapping between port Gi1/2 and port Gi4/13
104536: Feb 13 17:11:48.724: %MAC_MOVE-SP-4-NOTIF: Host 34a3.95d4.fd04 in vlan 1 is flapping between port Gi5/35 and port Gi1/2
104537: Feb 13 17:12:23.724: %MAC_MOVE-SP-4-NOTIF: Host 7c7a.919d.7894 in vlan 1 is flapping between port Gi5/34 and port Gi5/35
104538: Feb 13 17:14:43.781: %MAC_MOVE-SP-4-NOTIF: Host 28e1.4c69.d949 in vlan 1 is flapping between port Gi1/2 and port Gi4/13
104539: Feb 13 17:15:53.780: %MAC_MOVE-SP-4-NOTIF: Host 8c29.3705.fd9a in vlan 1 is flapping between port Gi4/47 and port Gi5/31
104540: Feb 13 17:18:13.782: %MAC_MOVE-SP-4-NOTIF: Host 0c3e.9f3f.4816 in vlan 1 is flapping between port Gi5/35 and port Gi4/13
104541: Feb 13 17:18:48.782: %MAC_MOVE-SP-4-NOTIF: Host 9494.26dd.cf61 in vlan 1 is flapping between port Gi4/47 and port Gi7/31
104542: Feb 13 17:19:58.780: %MAC_MOVE-SP-4-NOTIF: Host 7c7a.91e2.df83 in vlan 1 is flapping between port Gi1/2 and port Gi5/34
104543: Feb 13 17:21:08.843: %MAC_MOVE-SP-4-NOTIF: Host 64a3.cb36.e3d1 in vlan 1 is flapping between port Gi4/13 and port Gi5/35
104544: Feb 13 17:21:43.803: %MAC_MOVE-SP-4-NOTIF: Host 64a3.cb36.e3d1 in vlan 1 is flapping between port Gi5/35 and port Gi4/13
104545: Feb 13 17:22:53.798: %MAC_MOVE-SP-4-NOTIF: Host 64a3.cb36.e3d1 in vlan 1 is flapping between port Gi5/34 and port Gi5/35
104546: Feb 13 17:22:53.798: %MAC_MOVE-SP-4-NOTIF: Host 64a3.cb36.e3d1 in vlan 1 is flapping between port Gi5/34 and port Gi4/13
104547: Feb 13 17:23:28.797: %MAC_MOVE-SP-4-NOTIF: Host 9c4e.36c9.ed4c in vlan 1 is flapping between port Gi5/33 and port Gi5/35
104548: Feb 13 17:24:03.796: %MAC_MOVE-SP-4-NOTIF: Host 783a.8444.d58e in vlan 1 is flapping between port Gi5/31 and port Gi5/34
104549: Feb 13 17:24:03.796: %MAC_MOVE-SP-4-NOTIF: Host 34a3.95d4.fd04 in vlan 1 is flapping between port Gi4/13 and port Gi5/34
104550: Feb 13 17:24:03.800: %MAC_MOVE-SP-4-NOTIF: Host 0c3e.9f3f.4816 in vlan 1 is flapping between port Gi5/35 and port Gi5/33
104551: Feb 13 17:24:38.796: %MAC_MOVE-SP-4-NOTIF: Host 7c7a.9161.02ec in vlan 1 is flapping between port Gi5/35 and port Gi5/33
104552: Feb 13 17:25:13.795: %MAC_MOVE-SP-4-NOTIF: Host 783a.8444.d58e in vlan 1 is flapping between port Gi5/35 and port Gi5/31
104553: Feb 13 17:25:13.795: %MAC_MOVE-SP-4-NOTIF: Host 34a3.95d4.fd04 in vlan 1 is flapping between port Gi1/2 and port Gi4/13
104554: Feb 13 17:26:23.794: %MAC_MOVE-SP-4-NOTIF: Host 48d7.0520.64f1 in vlan 1 is flapping between port Gi5/35 and port Gi5/31
104555: Feb 13 17:26:58.794: %MAC_MOVE-SP-4-NOTIF: Host 48d7.0520.64f1 in vlan 1 is flapping between port Gi5/31 and port Gi5/35
104556: Feb 13 17:29:18.791: %MAC_MOVE-SP-4-NOTIF: Host 783a.8444.d58e in vlan 1 is flapping between port Gi4/47 and port Gi5/35
104557: Feb 13 17:31:03.854: %MAC_MOVE-SP-4-NOTIF: Host 8c29.3705.fd9a in vlan 1 is flapping between port Gi4/47 and port Gi5/31
104558: Feb 13 17:35:08.858: %MAC_MOVE-SP-4-NOTIF: Host 7c7a.9161.02ec in vlan 1 is flapping between port Gi5/33 and port Gi5/35
104559: Feb 13 17:35:43.873: %MAC_MOVE-SP-4-NOTIF: Host 0c3e.9f3f.4816 in vlan 1 is flapping between port Gi5/35 and port Gi7/31
104560: Feb 13 17:36:18.861: %MAC_MOVE-SP-4-NOTIF: Host f827.9386.f6e5 in vlan 1 is flapping between port Gi1/2 and port Gi5/35
104561: Feb 13 17:36:53.860: %MAC_MOVE-SP-4-NOTIF: Host 64a3.cb36.e3d1 in vlan 1 is flapping between port Gi4/13 and port Gi5/34
104562: Feb 13 17:36:53.864: %MAC_MOVE-SP-4-NOTIF: Host 48d7.0520.64f1 in vlan 1 is flapping between port Gi7/31 and port Gi4/47
104563: Feb 13 17:36:53.864: %MAC_MOVE-SP-4-NOTIF: Host 5c97.f315.afe4 in vlan 1 is flapping between port Gi5/35 and port Gi4/13
104564: Feb 13 17:37:28.860: %MAC_MOVE-SP-4-NOTIF: Host 48d7.0520.64f1 in vlan 1 is flapping between port Gi4/47 and port Gi7/31
104565: Feb 13 17:37:28.860: %MAC_MOVE-SP-4-NOTIF: Host 64a3.cb36.e3d1 in vlan 1 is flapping between port Gi5/35 and port Gi4/13
104566: Feb 13 17:38:03.859: %MAC_MOVE-SP-4-NOTIF: Host 9494.26dd.cf61 in vlan 1 is flapping between port Gi7/31 and port Gi4/47
104567: Feb 13 17:38:03.859: %MAC_MOVE-SP-4-NOTIF: Host 7c7a.9161.02ec in vlan 1 is flapping between port Gi5/35 and port Gi5/33
104568: Feb 13 17:38:38.858: %MAC_MOVE-SP-4-NOTIF: Host 64a3.cb36.e3d1 in vlan 1 is flapping between port Gi5/31 and port Gi5/35
104569: Feb 13 17:39:13.858: %MAC_MOVE-SP-4-NOTIF: Host 48d7.0520.64f1 in vlan 1 is flapping between port Gi7/31 and port Gi4/47
104570: Feb 13 17:39:13.862: %MAC_MOVE-SP-4-NOTIF: Host 64a3.cb36.e3d1 in vlan 1 is flapping between port Gi5/34 and port Gi5/31

Jon Marshall · ‎02-13-2015

The vlans would be able to communicate because your 6500 would route between them.

However you have a very large number of topology changes in the vlan you are using and I think this may well be a large part of your problem.

This many TCNs can severely degrade your network as the network is almost in a permanent state of topology change.

Can you post the configuration of gi1/2 from the 6500 please.

What does that port connect to ?

Jon

oighodaro1 · ‎02-17-2015

Interesting...

what do you mean by the large number of topology changes ?

what are TCN ?

gi1/2 config: (it is connected to the switch on the 6th floor)

GigabitEthernet1/2 is up, line protocol is up (connected)
  Hardware is C6k 1000Mb 802.3, address is 0016.47a9.2c21 (bia 0016.47a9.2c21)
  Description: Uplink to Relay-6FL-3750
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 2/255, rxload 71/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is SX
  input flow-control is off, output flow-control is off
  Clock mode is auto
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:10, output 00:00:00, output hang never
  Last clearing of "show interface" counters 1y27w
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 6253
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 281526000 bits/sec, 25251 packets/sec
  5 minute output rate 9240000 bits/sec, 6385 packets/sec
     69043537372 packets input, 83027782648414 bytes, 0 no buffer
     Received 122842879 broadcasts (31379267 multicasts)
     0 runts, 0 giants, 0 throttles 
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     39947920278 packets output, 29829885200518 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

interface GigabitEthernet1/2
 description Uplink to Relay-6FL-3750
 switchport
 switchport trunk encapsulation dot1q
 switchport mode trunk
 no ip address

Jon Marshall · ‎02-18-2015

Sorry, I missed your last post.

TCNs are part of the STP process.

A certain number of them is fine and expected but your switch is showing a large number of them and this can have an adverse effect on traffic going through the switch.

Can you post the following -

1) the port configuration of the switch on the 6th floor that connects back to the 6500

2) an end host port configuration on the 6500 and on the Dell switch. So that would be for a wired client.

How many wired clients do you have by the way ?

3) the port configuration from one of your APs connecting to the 6500

I am basically trying to work out where all the TCNs are coming from.

Jon

oighodaro1 · ‎02-20-2015

Hey Jon,

Sorry I have been out sick.

6th floor switch port 48 (goes to the catalyst)

show run

interface GigabitEthernet1/2
 description Uplink to Relay-6FL-3750
 switchport
 switchport trunk encapsulation dot1q
 switchport mode trunk
 no ip address

show interface gi1/2

GigabitEthernet1/2 is up, line protocol is up (connected)
  Hardware is C6k 1000Mb 802.3, address is 0016.47a9.2c21 (bia 0016.47a9.2c21)
  Description: Uplink to Relay-6FL-3750
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 114/255, rxload 3/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is SX
  input flow-control is off, output flow-control is off
  Clock mode is auto
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:07, output 00:00:47, output hang never
  Last clearing of "show interface" counters 1y28w
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 6349
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 13493000 bits/sec, 17980 packets/sec
  5 minute output rate 448219000 bits/sec, 37274 packets/sec
     72383422642 packets input, 85965971184275 bytes, 0 no buffer
     Received 124240224 broadcasts (31794823 multicasts)
     0 runts, 0 giants, 0 throttles 
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     43195939841 packets output, 34072957533827 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

The catalyst has 104 ports wired in

Dell powerconnect has 35

AP port config

GigabitEthernet5/35 is up, line protocol is up (connected)
  Hardware is C6k 1000Mb 802.3, address is 0023.0486.bbd2 (bia 0023.0486.bbd2)
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s
  input flow-control is off, output flow-control is off
  Clock mode is auto
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:01:01, output 00:00:25, output hang never
  Last clearing of "show interface" counters 1y28w
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 80000 bits/sec, 40 packets/sec
  5 minute output rate 427000 bits/sec, 87 packets/sec
     21658216 packets input, 9066104545 bytes, 0 no buffer
     Received 562088 broadcasts (234406 multicasts)
     0 runts, 0 giants, 0 throttles 
     0 input errors, 0 CRC, 1546 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     40697797 packets output, 26170828011 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

interface GigabitEthernet5/35
 switchport
 no ip address
!

Jon Marshall · ‎02-21-2015

Just to get an idea of the number of TCNs can you run the "sh spanning-tree detail" command again and just post the lines from the output containing the number of topology changes and the interface the last one was received from ie. -

Number of topology changes 51346 last change occurred 00:02:41 ago
from GigabitEthernet1/2

I just want to see how many more you have had since you ran the command the last time.

Jon

oighodaro1 · ‎02-23-2015

VLAN0001 is executing the rstp compatible Spanning Tree protocol
  Bridge Identifier has priority 8196, address 00d0.0405.0c01
  Configured hello time 2, max age 20, forward delay 15, tranmsit hold-count 6
  We are the root of the spanning tree
  Topology change flag not set, detected flag not set
  Number of topology changes 52872 last change occurred 00:05:18 ago
          from GigabitEthernet8/5
  Times:  hold 1, topology change 35, notification 2
          hello 2, max age 20, forward delay 15 
  Timers: hello 0, topology change 0, notification 0, aging 300

Jon Marshall · ‎02-21-2015

No problem, hope you are feeling better.

Thanks for the above and sorry to keep asking for more info but can you also send a wired client's port configuration from both the 6500 and the Dell switch ?

I want to check what the configuration is because TCNs can be caused by clients but there is a way to stop that happening.

Because everything is in one vlan then any TCNs will affect everyone. What this can mean is if a wired client connects or disconnects, which would mean the port going up or down, it can generate a TCN which causes the switch to flush it's mac address table of all entries except those for the port it received the TCN on.

It should only flush certain entries but it depends on how you have configured the client ports.

The switch then needs to relearn the mac addresses. This is usually a quick thing but while it is learning them any traffic sent to a mac address it doesn't know will be flooded to all ports ie. sent to every device. .

I'm not convinced it is just the wired clients but because it is all one vlan anything we can do to reduce the number of TCNs will help.

This is one reason why people use vlans on their switches. Ideally what I would like to do is implement vlans in your network ie. the servers have their own vlan and depending on the number of users perhaps a couple of vlans for them.

Implementing vlans does mean that each vlan would be using a different IP subnet.

For the servers I think it is doable although you would probably need to readdress them.

For the wired clients also it should be possible, if they are always wired when connected to the network.

But if the wired client can be disconnected and go wireless then they may need to be in the same vlan as the APs unless your wirless supports roaming between subnets and I'm not sure with autonomous APs whether they will.

Again I am not a wireless person at all so I may need to post my own question in those forums to see what is and isn't possible but first I'd like to narrow down what we can do with the wired setup.

So apart from the port configs can you explain about the wired clients ie. do they move between wired and wireless within your office ?

Note your 6500 is quite capable of routing between vlans so if we do decide that is the best way to go then you wont lose connectivity.

Sorry this is taking so long to sort out but as I said in my original response these sorts of issue can take a while to track down.

Jon

oighodaro1 · ‎02-23-2015

6500 wired port config

GigabitEthernet5/8 is up, line protocol is up (connected)
  Hardware is C6k 1000Mb 802.3, address is 0023.0486.bbb7 (bia 0023.0486.bbb7)
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s
  input flow-control is off, output flow-control is on 
  Clock mode is auto
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:50, output hang never
  Last clearing of "show interface" counters 1y28w
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 29000 bits/sec, 34 packets/sec
  5 minute output rate 1439000 bits/sec, 150 packets/sec
     118199327 packets input, 94624128643 bytes, 0 no buffer
     Received 626574 broadcasts (322568 multicasts)
     1 runts, 0 giants, 0 throttles 
     11 input errors, 11 CRC, 861 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     220906499 packets output, 131438856799 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 1326 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

interface GigabitEthernet5/8
 switchport
 no ip address
!

Dell switch wired port

You bring up a good point as far as switch self learning and relearning. The users are constantly moving, they are all using laptops. So when they are docked they are using wired, when they un-dock they are using wifi.

I definitely think it would be worth the try to see how vlans would help.

Jon Marshall · ‎02-23-2015

Okay the fact users are continually connecting and disconnecting from the wired network would explain a lot.

In addition you do not have the right configuration for the wired client ports on the 6500. We need to modify the configuration and then they will not generate TCNs every time they disconnect.

While we are adding the bit you need you may as well add all the other standard bits rather than having to do it later when/if we create new vlans.

So can you just confirm -

1) the wired clients connect directly to the 6500. What I mean is you don't have a VoIP phone connecting to the 6500 and then the laptop connects to the phone ?

2) everything is in the same vlan. So I'm assuming none of the clients need trunks to their laptops. Because of the way you have the ports configured they could be either an access port or a trunk port because you haven't told them what they should be.

I want to make them access ports because I think that's what they should be.

So can you run "sh int trunk" on the 6500 and this will show you all the trunk ports. There shouldn't be that many ie. the Dell connection is one, may be some of the APs although probably not.

I'm hoping you only see the Dell switch and that's it.

3) finally are both Dell switches on the 6th floor the same model and do you have any other switches on the 6th floor.

I'm basically trying to work out what generated the TCN we saw from the first output.

If you can get back to me with the above I can supply a configuration for the client ports on the 6500 that should hopefully address at least some if not most of the problem.

Jon

oighodaro1 · ‎02-24-2015

1. Yes we don't have VOIP phones, the computers connect directly to the 6500 switch.

2. Users do not need trunks to their laptops, they just need access

show int trunk

Port          Mode         Encapsulation  Status        Native vlan
Gi1/2         on           802.1q         trunking      1
Gi4/4         on           802.1q         trunking      1

Port          Vlans allowed on trunk
Gi1/2         1-4094
Gi4/4         1-4094

Port          Vlans allowed and active in management domain
Gi1/2         1,10,20
Gi4/4         1,10,20

Port          Vlans in spanning tree forwarding state and not pruned
Gi1/2         1,10,20
Gi4/4         1,10,20

Port Gi1/2 runs down to the Dell PowerConnect.

3. The 6th floor has two switches and two different models.

Dell PowerConnect 2748 - connects to the 6500

Dell Powerconnect 3548 - connects to the 2748 , i can't log into this switch because the password was never recorded when the first tech installed it.

Jon Marshall · ‎02-24-2015

Okay, all good news except for the Dell.

If your 3548 does run STP then at some stage we may need to get into it because it must be generating TCNs.

However lets concentrate on the 6500.

So you need to configure portfast on your client ports. What happens without it is in effect a double hit -

1) every time a client connects or disconnects it generates a TCN

and

2) the 6500 the flushes the mac addresses of all non edge ports.. The non edge ports is key here because it doesn't flush the entries for edge ports.

And the way the switch knows a port is an edge port is because it is configured with portfast.

Which means by doing this we should not only reduce the number of TCNs but also their effect on your wired clients.

So using gi5/8 as an example you need to add the following -

int gi5/8
switchport mode access
switchport access vlan 1
spanning-tree portfast edge

the 6500 allows you enter a range for the interfaces so you could do "int range gi5/1 - 20" and it would allow you enter the above commands for all interfaces at once.

The "edge" keyword may or may not be needed depending on your IOS version. If you type "spanning-tree portfast ?" and it gives you the option of edge then use it otherwise don't worry.

You can set portfast up globally for the switch but i don't have the full picture so it's easier to do it just for the ports.

The additional switchport commands are optional and are used to tell the port to be an access port so it can't dynamically create a trunk and which vlan it is in. Note by default all ports are in that vlan so it's not strictly needed but if at a later date you started to use vlan 10 for example you would need to modify the above configuration to reflect that.

If you don't want to bother with them that's fine, everything will still work as long as you apply the portfast command part.

The key thing here is only use portfast on the wired client ports. Don't do the APs and definitely don't apply it to the port connecting to the Dell.

What i would do is take a record of the current TCN number as you did before and the port the last TCN was seen on. Together with the other two readings that gives you a baseline as to how many and how quickly they are increasing at the moment.

Then apply the configuration as above and see how the network performs.

If you can perhaps once a day record the TCN number again and as importantly the port the last TCN was seen on and it should give you an idea of whether this is helping.

Like i say, this may not fix the problem fully or even at all but i think you should see some improvement.

Obviously if it doesn't then please come back and we can carry on investigating.

In addition if you want to look into using more vlans then we can discuss that too but i think it's worth seeing if this change makes any difference before doing anything else.

Jon

oighodaro1 · ‎02-24-2015

I like this plan just some questions.

Are we doing it for Gi5/1-20 as a test to see if the TCN doesn't increase as much ?

I have Gi4/0 to Gi9/48 should I do this for all ports except the AP ports ?

Thanks

Jon Marshall · ‎02-24-2015

Sorry, I should have been clearer.

That was just an example of the interface range command.

I have no idea of which ports your clients are connected into so you need to modify it for that.

You should do it for all the wired client ports and nothing else at the moment. I can't really be any more specific than that because I don't know which ports are in use.

However you use the command be careful you only apply it to wired client ports

For example I notice in the "sh int trunk" output you posted gi4/4 is a trunk.

You definitely don't want to apply the commands I provided to a trunk port.

So it may take a bit of work and more than just one interface range command to configure it.

If you are unsure then by all means apply to a subset but what I have given you should not disconnect any clients.

If you do use a subset once you are happy everything is still working in terms of connectivity then yes you need to apply it to all the wired client ports on the 6500 switch.

That way if you are still seeing TCNs and still seeing performance issues (even if things do improve) then we know where to start looking for further causes.

Hope that makes sense.

Jon

Network Problems and Symptoms