01-29-2014 08:33 AM - edited 03-07-2019 05:53 PM
Hello all,
I am upgrading a older 2950(100M) switch replacing it with a gigabit 2960. Installed it in the same rack, the configuration is practically non-existent just set the passwords and IP. We run a single VLAN flat network for this so I started out by patching it to the existing switch, after a few days we had an opportunity to migrate because there was some downtime so I disconnected the cables on the old and moved them to the new.. Everything seemed fine, there is connectivity and things operate, but a few days later we noticed that some network transfer activities are slow. There are no errors or log entries showing on the new switch or the old one, but the low throughput is persistent.
All ports show 1G Full duplex as they should, but what I see when I test is that traffic tests look almost asynchronous when passing switch boundaries with normal read speeds and slow writes. Reversing the direction of the test hosts I get slow reads and fast writes so it seems to 'stick' to one side of the traffic path. Testing the same equipment against differente targets without the switch boundary crossing does not show the problem. All Intra-switch tests look good (gig switches transfer near a gig and 100 switches near 100), but the moment there is a crossing things behave strangely regardless of the target (new switch is center backbone with most hosts, but does no routing). Network layout is essentially a T with everything radiating from the new switch. I can eliminate the old switch soon, but I still need to resolve the problem with the crossing to the other switch.
Everything seems to point at the inter switch links. One is a patch cable under two feet, and the other is a dedicated fiber site link. We had the vendor confirm that the site link showed no issues, but having the same symptoms on both links makes me suspect the switch has something odd happening..
I checked for duplex issues first, but didn't find any. I flushed the arp caches in all of the switches (3 total) and all of the computers as well, but the problem persists.
Could this be an STP issue ? If so how can I set this switch as the STP root and force a refresh..
Any help would be greatly appreciated.
01-30-2014 09:47 AM
Paul,
Rerunning the tests I found another oddity. I wish it wasn't the case, but it is too unusual not to mention..
Originally I tested between hosts in the new center switch and hosts on both of the others bidirectionally.
I repeated that test and got the same results, but I decided to test between the two end switches directly today and to my surprise no delay.. Traverse one link and you get a delay traverse both and you don't..
The trunk links are all single lines with no etherchannels to be unbalanced and no duplex issues, and now there is a traffic path crossing both links that presents no delay. To me that seems to eliminate most interface/port issues. The tests were restricted to the same three hosts, so the endpoint test was run on the same hosts that presented delays in the other tests..
Getting ready to go and try the config changes, more and mor it looks like a switching/STP issue at the middle.
Dave
01-30-2014 10:04 AM
Hello
switch 2960
Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 12712290
Can you you also try hardcoding the two interfaces between the 2950 and 2960 to 100/full
res
Paul
Please don't forget to rate any posts that have been helpful.
Thanks.
01-30-2014 10:32 AM
Paul,
Don't know how I missed that one.. Good catch !
The odd part is that it is also happening on port 47 which is already fixed config at 100/Full.
Could this new switch be bad ?
01-30-2014 11:59 AM
Hello
To summarise:
try hardcoding the two interfaces between the 2950 and 2960 to 100/full and
Don't hardcode the interfaces between the 2960 and the fibre switch
Also apply the config suggested in my previous post and lastly clear the interface counters of all 3 switches then test again
Clear counters
Res
Paul
Sent from Cisco Technical Support iPad App
01-30-2014 01:05 PM
I am at the site now. Tried to post from my iPhone, but the signal died.
Since the link between the two switches is local to the site I decided to try switching that port to 100/full first. Link dropped and never came back.. Switched to another port and got it running on auto again. Then I configured a hiterhto unused port and grabbed a new (but tested patch) and tried it there. Same thing port does not come up at all, simply stays dark when patched. No flash of yellow followed by green, just darkness..
After making an interuption for the people remaining on this switch I don't want to try the other port today as there is a larger group behind that one. It seems the new cosco switch does not like the older Cisco gear eaither..
The providers fiber line is surfaced ata Cisco switch too, so this is probably the same issue throughout.
Other than the new switch being an OBF with an odd problem does anythin occur to you ?
The 'remote' switch is less than a year old and seems to exhibit the same issue as well. Is there a compatibility issue between some Cisco models ?
Yikes
Dave
01-30-2014 01:38 PM
Hello
"I decided to try switching that port to 100/full first." - Did you do this on either local switch interface?
Anyway did you manage to apply the stp and port config.
Res
Paul
Sent from Cisco Technical Support iPad App
01-30-2014 02:03 PM
Hi Paul,
Working on many things here.. I set the speed and duplex on the port of the New 2960 switch that faces our older 2950 switch (Gi1/0/48). Since the older switch is a 100M switch and is in the same rack I figured the safest place to start was there where I could get a grasp on both sides if anything happened..
Afterwards I set pot Gi1/0/1 as 100 Full and then switched cables. I did not apply any other changes because I don't want to destabilize things. Since the errors are at the switchport level I did not think applying global port changes was a good idea. At the moment there is no indication that STP is the issue, it seems more like link layer.
I will try the STP changes before leaving..
I will be back on Sunday and we have some downtime planned that allows me to do things more freely then.
Dave
01-30-2014 02:32 PM
Hello
No worries but the full/duplex setting end to be applied on either end for it to work?
Let me know how you get on
Res
Paul
Sent from Cisco Technical Support iPad App
01-31-2014 04:31 AM
Good day,
After Changing STP priority the new switch became the STP root and the old relinquished the role. The two directly connected switches seem to be handling STP correctly but the change did not resolve the transfer speed problem or the output drops. I looked over the STP detail information and found no inconsistent or blocked ports reported, but the details for the fiber connection shed some light on the presence of two STP roots. It seems that connection is not so transparent.. It seems to be bridging only. I noticed it did not pass CDP, but apparently it does not pass BPDUs either..
On new switch (Site 1)
Port 47 (GigabitEthernet1/0/47) of VLAN0001 is designated forwarding
Port path cost 19, Port priority 128, Port Identifier 128.47.
Designated root has priority 1, address f41f.c2dc.9b80
Designated bridge has priority 1, address f41f.c2dc.9b80
Designated port id is 128.47, designated path cost 0
Timers: message age 0, forward delay 0, hold 0
Number of transitions to forwarding state: 1
Link type is point-to-point by default
BPDU: sent 227619, received 0
On remote switch (Site 2)
Port 26 (GigabitEthernet0/2) of VLAN0001 is designated forwarding
Port path cost 4, Port priority 128, Port Identifier 128.26.
Designated root has priority 32769, address 1833.9db5.cd80
Designated bridge has priority 32769, address 1833.9db5.cd80
Designated port id is 128.26, designated path cost 0
Timers: message age 0, forward delay 0, hold 0
Number of transitions to forwarding state: 1
Link type is point-to-point by default
BPDU: sent 232210, received 0
Both sending, neither receiving so the result is two separate STP domains. Doesn't explain the output drops, but probably lets out STP as the source of the problems (I think).
I will be returning today to do other work, Sunday I will re-do the fixed link from both ends to see if it fixes the issue on the patched link. The Fiber link is more awkward because I don't control both ends of those connections and have had trouble coordinating with our provider for these changes. If the change works on the patched line I'll work on the Fiber link on Sunday as well.
Dave
01-31-2014 05:46 AM
Hi Paul,
I looked around for the output drops and found what seems to be a queueng issue. mls qos says Queue3 threshhold3 is where the drops take place. Looking at the ethernet side of the interface I see collissions as well, not an exhorbitant number but none of the other interfaces show any (including the interface to the fiber link).
CISCO-2960-48-GB-ASP#show mls qos int gi1/0/36 statistics
GigabitEthernet1/0/36 (All statistics are in packets)
dscp: incoming
-------------------------------
0 - 4 : 40774020 0 0 0 0
5 - 9 : 0 0 0 0 0
10 - 14 : 0 0 0 0 0
15 - 19 : 0 0 0 0 0
20 - 24 : 0 0 0 0 0
25 - 29 : 0 0 0 0 0
30 - 34 : 0 0 0 0 0
35 - 39 : 0 0 0 0 0
40 - 44 : 0 0 0 0 0
45 - 49 : 0 0 0 391 0
50 - 54 : 0 0 0 0 0
55 - 59 : 0 0 0 0 0
60 - 64 : 0 0 0 0
dscp: outgoing
-------------------------------
0 - 4 : 33172079 0 0 0 0
5 - 9 : 0 0 0 0 0
10 - 14 : 0 0 0 0 0
15 - 19 : 0 0 0 0 0
20 - 24 : 0 0 0 0 0
25 - 29 : 0 0 0 0 0
30 - 34 : 0 0 0 0 0
35 - 39 : 0 0 0 0 0
40 - 44 : 0 0 0 0 0
45 - 49 : 0 0 0 0 0
50 - 54 : 0 0 0 0 0
55 - 59 : 0 0 0 0 0
60 - 64 : 0 0 0 0
cos: incoming
-------------------------------
0 - 4 : 40897626 0 0 0 0
5 - 7 : 0 0 0
cos: outgoing
-------------------------------
0 - 4 : 33237304 0 0 1 0
5 - 7 : 0 0 19025
output queues enqueued:
queue: threshold1 threshold2 threshold3
-----------------------------------------------
queue 0: 0 0 0
queue 1: 70655 167 19031
queue 2: 0 0 0
queue 3: 0 0 33213384
output queues dropped:
queue: threshold1 threshold2 threshold3
-----------------------------------------------
queue 0: 0 0 0
queue 1: 0 0 0
queue 2: 0 0 0
queue 3: 0 0 29446
Policer: Inprofile: 0 OutofProfile: 0
CISCO-2960-48-GB-ASP#$ers ethernet-controller gigabitEthernet 1/0/36
Transmit GigabitEthernet1/0/36 Receive
2921268567 Bytes 2470246300 Bytes
33876545 Unicast frames 41805348 Unicast frames
82757 Multicast frames 61500 Multicast frames
118985 Broadcast frames 13680 Broadcast frames
0 Too old frames 2464539581 Unicast bytes
174 Deferred frames 4473742 Multicast bytes
0 MTU exceeded frames 1232977 Broadcast bytes
241 1 collision frames 0 Alignment errors
64 2 collision frames 0 FCS errors
49 3 collision frames 0 Oversize frames
49 4 collision frames 0 Undersize frames
55 5 collision frames 0 Collision fragments
38 6 collision frames
20 7 collision frames 4934864 Minimum size frames
3 8 collision frames 8111261 65 to 127 byte frames
0 9 collision frames 2403315 128 to 255 byte frames
0 10 collision frames 2916623 256 to 511 byte frames
0 11 collision frames 1605530 512 to 1023 byte frames
0 12 collision frames 21908935 1024 to 1518 byte frames
0 13 collision frames 0 Overrun frames
0 14 collision frames 0 Pause frames
0 15 collision frames
0 Excessive collisions 0 Symbol error frames
207 Late collisions 0 Invalid frames, too large
0 VLAN discard frames 0 Valid frames, too large
0 Excess defer frames 0 Invalid frames, too small
16051101 64 byte frames 0 Valid frames, too small
7154984 127 byte frames
4355894 255 byte frames 0 Too old frames
3074533 511 byte frames 0 Valid oversize frames
1411908 1023 byte frames 1 System FCS error frames
2030056 1518 byte frames 0 RxPortFifoFull drop frame
0 Too large frames
236 Good (1 coll) frames
274 Good (>1 coll) frames
Dave
01-31-2014 06:29 AM
Hello
Something you not telling me here david... Wasnt aware you have mls qos enabled?
Now I am not sure how the scheduler works for 2960 however if its the same as 3650 then mls qos is enabled the ouput queues are used.
3650 has 4 queues, Queue 1 can be the priority queue but is disabled by default, When enabled all traffic that goes to queue 1 has an absolute priority over other traffic, it could consume all of the BW and stave the other queues.
however each 4 queues can be configured with a shaping weight also and by default queue 1 is confiured with a weight of 25....Unless the priority queue is enabled or this weight is changed, any traffic that goes into queue 1 will have a priority of 1/25 or 4% of the interface BW.
In addition all traffic that goes into queue 1 will be rate limited to 4% of the BW.
If shaped weights are not set for that particular queue then the SHARED weights for that queue becme effective.
Shared weights are relative vaulues, The defaults entries of 25,25,25,25 could be esailly be 1,1,1,1.
By default each queue shares equally the BW remaining after priority and shaped queues have to be serviced.
So my question is now why do have layer 2 qos enabled and if you dont need it turn it off.
no mls qos
res
Paul
Please don't forget to rate any posts that have been helpful.
Thanks.
01-31-2014 09:50 AM
Your links wont come up when you hardcode them because you are using a straight thru cable and not a crossover . The 2960 can do mdix for cable corrections but this only works when the port is auto/auto , once you hardcode the port will go down and that is normal . the 2950 does not do mdix . Just make sure everything is matched on each end , if it's auto on one end it must be auto on the other end. If it's hardcoded then both ends must be hardcoded. Seeing that you have a number of collisions on interfaces that indicates sped/duplex mismatches which will slow down your links considrably .
As far as spanning tree goes are you sure they are all in the same vlan ? I see you have a 2901 in the drawing and that is not going to extend the spanning tree doamin from one end to the other . If I have my undertanding incorrect I apologize.
01-31-2014 12:58 PM
Glen,
Crossover cable.. Makes ver good sense and I will make sure to take several when workign on this. I had used the same cables so you could be right on that one. Feel stupid not thinking to try it, but we get very spoiled to auto MDIX.
Paul,
All configuration suggestions applied with no changes, but if MDIX is inactive without auto that could be what killed me earlier when I forced speed/duplex (and auto with duplex issues could be the main issue).
I am past feeling stupid about it and going shopping for the cables now !
01-31-2014 01:15 PM
Hello
Yes for sure -what do they say about assuming! - Apologies David It never crossed my mind you weren't using a crossover cable between these local switches
Res
Paul
Sent from Cisco Technical Support iPad App
01-31-2014 01:57 PM
Armed with cables now..
Tried cat6, cat5e, neglected cross. Will try it on Sunday and see. I sure hope this takes care of the problem.. Still seems odd that some traversals don't show delays and that the other connection shows delays, but I'll have to confirm that the ISP end of the lines aren't on auto somewhere as well creating the same condition. Would be odd since the line worked without problems for several months, but it's not impossible.
Thanks again..
Dave
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide