Strange issue with new Cisco Catalyst 2960 (IOS)

[longboringusername]

Hello all,

I am upgrading a older 2950(100M) switch replacing it with a gigabit 2960. Installed it in the same rack, the configuration is practically non-existent just set the passwords and IP. We run a single VLAN flat network for this so I started out by patching it to the existing switch, after a few days we had an opportunity to migrate because there was some downtime so I disconnected the cables on the old and moved them to the new.. Everything seemed fine, there is connectivity and things operate, but a few days later we noticed that some network transfer activities are slow. There are no errors or log entries showing on the new switch or the old one, but the low throughput is persistent.

All ports show 1G Full duplex as they should, but what I see when I test is that traffic tests look almost asynchronous when passing switch boundaries with normal read speeds and slow writes. Reversing the direction of the test hosts I get slow reads and fast writes so it seems to 'stick' to one side of the traffic path. Testing the same equipment against differente targets without the switch boundary crossing does not show the problem. All Intra-switch tests look good (gig switches transfer near a gig and 100 switches near 100), but the moment there is a crossing things behave strangely regardless of the target (new switch is center backbone with most hosts, but does no routing). Network layout is essentially a T with everything radiating from the new switch. I can eliminate the old switch soon, but I still need to resolve the problem with the crossing to the other switch.

Everything seems to point at the inter switch links. One is a patch cable under two feet, and the other is a dedicated fiber site link. We had the vendor confirm that the site link showed no issues, but having the same symptoms on both links makes me suspect the switch has something odd happening..

I checked for duplex issues first, but didn't find any. I flushed the arp caches in all of the switches (3 total) and all of the computers as well, but the problem persists.

Could this be an STP issue ?   If so how can I set this switch as the STP root and force a refresh..

Any help would be greatly appreciated.

[longboringusername]

Paul,

Rerunning the tests I found another oddity. I wish it wasn't the case, but it is too unusual not to mention..

Originally I tested between hosts in the new center switch and hosts on both of the others bidirectionally.

I repeated that test and got the same results, but I decided to test between the two end switches directly today and to my surprise no delay.. Traverse one link and you get a delay traverse both and you don't..

The trunk links are all single lines with no etherchannels to be unbalanced and no duplex issues, and now there is a traffic path crossing both links that presents no delay. To me that seems to eliminate most interface/port issues. The tests were restricted to the same three hosts, so the endpoint test was run on the same hosts that presented delays in the other tests..

Getting ready to go and try the config changes, more and mor it looks like a switching/STP issue at the middle.

Dave

[paul driver]

Hello

switch 2960

  Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 12712290

Can you you also try hardcoding the two interfaces between the 2950 and 2960  to 100/full

res

Paul

Please don't forget to rate any posts that have been helpful.

Thanks.

[longboringusername]

Paul,

Don't know how I missed that one.. Good catch !

The odd part is that it is also happening on port 47 which is already fixed config at 100/Full.

Could this new switch be bad ?

[paul driver]

Hello

To summarise:

try hardcoding the two interfaces between the 2950 and 2960  to 100/full and

Don't hardcode the interfaces between the 2960 and the fibre switch

Also apply the config suggested in my previous post and lastly clear the interface counters of all 3 switches then test again

Clear counters

Res

Paul

Sent from Cisco Technical Support iPad App

[longboringusername]

I am at the site now. Tried to post from my iPhone, but the signal died.

Since the link between the two switches is local to the site I decided to try switching that port to 100/full first.  Link dropped and never came back.. Switched to another port and got it running on auto again. Then I configured a hiterhto unused port and grabbed a new (but tested patch) and tried it there. Same thing port does not come up at all, simply stays dark when patched. No flash of yellow followed by green, just darkness..

After making an interuption for the people remaining on this switch I don't want to try the other port today as there is a larger group behind that one. It seems the new cosco switch does not like the older Cisco gear eaither..

The providers fiber line is surfaced ata  Cisco switch too, so this is probably the same issue throughout.

Other than the new switch being an OBF with an odd problem does anythin occur to you ?

The 'remote' switch is less than a year old and seems to exhibit the same issue as well. Is there a compatibility issue between some Cisco models ?

Yikes

Dave

[paul driver]

Hello

"I decided to try switching that port to 100/full first." - Did you do this on either local switch interface?

Anyway did you manage to apply the stp and port config.

Res
Paul

Sent from Cisco Technical Support iPad App

[longboringusername]

Hi Paul,

Working on many things here.. I set the speed and duplex on the port of the New 2960 switch that faces our older 2950 switch (Gi1/0/48). Since the older switch is a 100M switch and is in the same rack I figured the safest place to start was there where I could get a grasp on both sides if anything happened..

Afterwards I set pot Gi1/0/1 as 100 Full and then switched cables. I did not apply any other changes because I don't want to destabilize things. Since the errors are at the switchport level I did not think applying global port changes was a good idea. At the moment there is no indication that STP is the issue, it seems more like link layer.

I will try the STP changes before leaving..

I will be back on Sunday and we have some downtime planned that allows me to do things more freely then.

Dave

[paul driver]

Hello
No worries but the full/duplex setting end to be applied on either end for it to work?

Let me know how you get on

Res
Paul

Sent from Cisco Technical Support iPad App

[longboringusername]

Good day,

After Changing STP priority the new switch became the STP root and the old relinquished the role. The two directly connected switches seem to be handling STP correctly but the change did not resolve the transfer speed problem or the output drops. I looked over the STP detail information and found no inconsistent or blocked ports reported, but the details for the fiber connection shed some light on the presence of two STP roots. It seems that connection is not so transparent.. It seems to be bridging only. I noticed it did not pass CDP, but apparently it does not pass BPDUs either..

On new switch (Site 1)

Port 47 (GigabitEthernet1/0/47) of VLAN0001 is designated forwarding

   Port path cost 19, Port priority 128, Port Identifier 128.47.

   Designated root has priority 1, address f41f.c2dc.9b80

   Designated bridge has priority 1, address f41f.c2dc.9b80

   Designated port id is 128.47, designated path cost 0

   Timers: message age 0, forward delay 0, hold 0

   Number of transitions to forwarding state: 1

   Link type is point-to-point by default

   BPDU: sent 227619, received 0

On remote switch (Site 2)

Port 26 (GigabitEthernet0/2) of VLAN0001 is designated forwarding

   Port path cost 4, Port priority 128, Port Identifier 128.26.

   Designated root has priority 32769, address 1833.9db5.cd80

   Designated bridge has priority 32769, address 1833.9db5.cd80

   Designated port id is 128.26, designated path cost 0

   Timers: message age 0, forward delay 0, hold 0

   Number of transitions to forwarding state: 1

   Link type is point-to-point by default

   BPDU: sent 232210, received 0

Both sending, neither receiving so the result is two separate STP domains.  Doesn't explain the output drops, but probably lets out STP as the source of the problems (I think).

I will be returning today to do other work, Sunday I will re-do the fixed link from both ends to see if it fixes the issue on the patched link. The Fiber link is more awkward because I don't control both ends of those connections and have had trouble coordinating with our provider for these changes. If the change works on the patched line I'll work on the Fiber link on Sunday as well.

Dave

[longboringusername]

Hi Paul,

I looked around for the output drops and found what seems to be a queueng issue. mls qos says Queue3 threshhold3 is where the drops take place. Looking at the ethernet side of the interface I see collissions as well, not an exhorbitant number but none of the other interfaces show any (including the interface to the fiber link).

CISCO-2960-48-GB-ASP#show mls qos int gi1/0/36 statistics

GigabitEthernet1/0/36 (All statistics are in packets)

  dscp: incoming

-------------------------------

  0 -  4 :    40774020            0            0            0            0

  5 -  9 :           0            0            0            0            0

10 - 14 :           0            0            0            0            0

15 - 19 :           0            0            0            0            0

20 - 24 :           0            0            0            0            0

25 - 29 :           0            0            0            0            0

30 - 34 :           0            0            0            0            0

35 - 39 :           0            0            0            0            0

40 - 44 :           0            0            0            0            0

45 - 49 :           0            0            0          391            0

50 - 54 :           0            0            0            0            0

55 - 59 :           0            0            0            0            0

60 - 64 :           0            0            0            0

  dscp: outgoing

-------------------------------

  0 -  4 :    33172079            0            0            0            0

  5 -  9 :           0            0            0            0            0

10 - 14 :           0            0            0            0            0

15 - 19 :           0            0            0            0            0

20 - 24 :           0            0            0            0            0

25 - 29 :           0            0            0            0            0

30 - 34 :           0            0            0            0            0

35 - 39 :           0            0            0            0            0

40 - 44 :           0            0            0            0            0

45 - 49 :           0            0            0            0            0

50 - 54 :           0            0            0            0            0

55 - 59 :           0            0            0            0            0

60 - 64 :           0            0            0            0

  cos: incoming

-------------------------------

  0 -  4 :    40897626            0            0            0            0

  5 -  7 :           0            0            0

  cos: outgoing

-------------------------------

  0 -  4 :    33237304            0            0            1            0

  5 -  7 :           0            0        19025

  output queues enqueued:

queue:    threshold1   threshold2   threshold3

-----------------------------------------------

queue 0:           0           0           0

queue 1:       70655         167       19031

queue 2:           0           0           0

queue 3:           0           0    33213384

  output queues dropped:

queue:    threshold1   threshold2   threshold3

-----------------------------------------------

queue 0:           0           0           0

queue 1:           0           0           0

queue 2:           0           0           0

queue 3:           0           0       29446

Policer: Inprofile:            0 OutofProfile:            0

CISCO-2960-48-GB-ASP#$ers ethernet-controller gigabitEthernet 1/0/36

     Transmit GigabitEthernet1/0/36           Receive

   2921268567 Bytes                       2470246300 Bytes

     33876545 Unicast frames                41805348 Unicast frames

        82757 Multicast frames                 61500 Multicast frames

       118985 Broadcast frames                 13680 Broadcast frames

            0 Too old frames              2464539581 Unicast bytes

          174 Deferred frames                4473742 Multicast bytes

            0 MTU exceeded frames            1232977 Broadcast bytes

          241 1 collision frames                   0 Alignment errors

           64 2 collision frames                   0 FCS errors

           49 3 collision frames                   0 Oversize frames

           49 4 collision frames                   0 Undersize frames

           55 5 collision frames                   0 Collision fragments

           38 6 collision frames

           20 7 collision frames             4934864 Minimum size frames

            3 8 collision frames             8111261 65 to 127 byte frames

            0 9 collision frames             2403315 128 to 255 byte frames

            0 10 collision frames            2916623 256 to 511 byte frames

            0 11 collision frames            1605530 512 to 1023 byte frames

            0 12 collision frames           21908935 1024 to 1518 byte frames

            0 13 collision frames                  0 Overrun frames

            0 14 collision frames                  0 Pause frames

            0 15 collision frames

            0 Excessive collisions                 0 Symbol error frames

          207 Late collisions                      0 Invalid frames, too large

            0 VLAN discard frames                  0 Valid frames, too large

            0 Excess defer frames                  0 Invalid frames, too small

     16051101 64 byte frames                       0 Valid frames, too small

      7154984 127 byte frames

      4355894 255 byte frames                      0 Too old frames

      3074533 511 byte frames                      0 Valid oversize frames

      1411908 1023 byte frames                     1 System FCS error frames

      2030056 1518 byte frames                     0 RxPortFifoFull drop frame

            0 Too large frames

          236 Good (1 coll) frames

          274 Good (>1 coll) frames

Dave

[paul driver]

Hello

Something you not telling me here david... Wasnt aware you have  mls qos enabled?

Now I am not sure how the scheduler works for 2960 however if its the same as 3650  then mls qos is enabled  the ouput queues are used.

3650 has 4 queues, Queue 1 can be the priority queue but is disabled by default, When enabled all traffic that goes to queue 1 has an absolute priority over other traffic, it could consume all of the BW and stave  the other queues.

however each 4 queues can be configured with a shaping weight also and by default queue 1 is confiured with a weight of 25....Unless the priority queue is enabled or this weight is changed, any traffic that goes into queue 1 will have a priority of 1/25 or 4% of the interface BW.

In addition all traffic that goes into queue 1 will be rate limited to 4% of the BW.

If shaped weights are not set for that particular queue then the SHARED weights for that queue becme effective.

Shared weights are relative vaulues, The defaults entries of 25,25,25,25 could be esailly be 1,1,1,1.

By default each queue shares equally the BW remaining after priority and shaped queues have to be serviced.

So my question is now why do have layer 2 qos enabled and if you dont need it turn it off.

no mls qos

res

Paul

Please don't forget to rate any posts that have been helpful.

Thanks.

[glen.grant]

  Your links wont come up when you hardcode them because you are using a straight thru cable and not a crossover . The 2960 can do mdix for cable corrections but this only works when the port is auto/auto , once you hardcode the port will go down and that is normal . the 2950 does not do mdix . Just make sure everything is matched on each end , if it's auto on one end it must be auto on the other end. If it's hardcoded then both ends must be hardcoded.  Seeing that you have a number of collisions on interfaces that indicates sped/duplex mismatches which will slow down your links considrably .

    As far as spanning tree goes are you sure they are all in the same vlan ?  I see you have a 2901 in the drawing  and that is not going to extend the spanning tree doamin from one end to the other .  If I have my undertanding incorrect I apologize.

[longboringusername]

Glen,

Crossover cable.. Makes ver good sense and I will make sure to take several when workign on this. I had used the same cables so you could be right on that one. Feel stupid not thinking to try it, but we get very spoiled to auto MDIX.

Paul,

All configuration suggestions applied with no changes, but if MDIX is inactive without auto that could be what killed me earlier when I forced speed/duplex (and auto with duplex issues could be the main issue).

I am past feeling stupid about it and going shopping for the cables now !

[paul driver]

Hello

Yes for sure -what do they say about assuming! - Apologies David It never crossed my mind you weren't using a crossover cable between these local switches


Res
Paul

Sent from Cisco Technical Support iPad App

[longboringusername]

Armed with cables now..

Tried cat6, cat5e, neglected cross. Will try it on Sunday and see. I sure hope this takes care of the problem..  Still seems odd that some traversals don't show delays and that the other connection shows delays, but I'll have to confirm that the ISP end of the lines aren't on auto somewhere as well creating the same condition. Would be odd since the line worked without problems for several months, but it's not impossible.

Thanks again..

Dave