Hi,
Yesterday we've experienced slow network performance. My core is the 6513 with dual sup2's (Catos 7.6(16) and msfc2 (12.1(11b)E)cards setup in hsrp mode. Switch uptime is 414 days, no issues until yesterday.
In my efforts to troubleshoot this (no changes made to switch/routers in weeks), I've checked the CPU load on my switch and routers, which was normal (7%) on switch and 1% on routers. Sup in slot 1 was active and I have most of my vlans routing through the mfsc2 on slot 1 as well.
My logging buffer on my router was clean, but I've noticed an abnormality in my switch logs (sh logging buff -1023)
Messages here:
2007 Apr 09 15:26:07 edt +06:00 %SYS-3-PORT_OUT_DISCARD:Port 11/11 detected 2424795 output packet(s) discard errors in last 30 minutes
2007 Apr 09 15:26:07 edt +06:00 %SYS-3-PORT_OUT_DISCARD:Port 11/17 detected 2424795 output packet(s) discard errors in last 30 minutes
2007 Apr 09 15:26:07 edt +06:00 %SYS-3-PORT_OUT_DISCARD:Port 11/25 detected 2424795 output packet(s) discard errors in last 30 minutes
2007 Apr 09 15:26:08 edt +06:00 %SYS-3-PORT_OUT_DISCARD:Port 11/35 detected 2424795 output packet(s) discard errors in last 30 minutes
2007 Apr 09 15:26:08 edt +06:00 %SYS-3-PORT_OUT_DISCARD:Port 11/36 detected 2424795 output packet(s) discard errors in last 30 minutes
2007 Apr 09 15:26:08 edt +06:00 %SYS-3-PORT_OUT_DISCARD:Port 11/37 detected 2424795 output packet(s) discard errors in last 30 minutes
2007 Apr 09 15:26:08 edt +06:00 %SYS-3-PORT_OUT_DISCARD:Port 11/39 detected 2424795 output packet(s) discard errors in last 30 minutes
2007 Apr 09 15:26:08 edt +06:00 %SYS-3-PORT_OUT_DISCARD:Port 11/40 detected 2424795 output packet(s) discard errors in last 30 minutes
2007 Apr 09 15:26:08 edt +06:00 %SYS-3-PORT_OUT_DISCARD:Port 11/42 detected 2424795 output packet(s) discard errors in last 30 minutes
2007 Apr 09 15:26:09 edt +06:00 %SYS-3-PORT_OUT_DISCARD:Port 11/45 detected 2424795 output packet(s) discard errors in last 30 minutes
2007 Apr 09 15:26:09 edt +06:00 %SYS-3-PORT_OUT_DISCARD:Port 11/46 detected 2424795 output packet(s) discard errors in last 3
Looking at the output from the "show mac" command, these ports were reading millions of out-discards. All of these ports are in the same VLAN224.
Here's what I've done:
1) Physically reset/reseated module 11.
fo about 3 minutes, network performance was okay
2) Physically removed slot 1 and failed over to module 2. Now module was my active supervisor AND router.
Network performance improved for about 5 minutes.
3) I reinserted module 1, watched and waited for the supervisor and router to come online. Before I attempted to remove module 2, 15 minutes, 30 minutes, etc today passed and since then network performance is fine. I've cleared my switch counters (clear counters) and I haven't seen any incrementation of dropped information.
Therefore, currently module 2 is my active supervisor and session 15 (module 1) is my active router.
Cisco recommends that I power down the switch and reseat all line cards and wait to see what happens. I am reluctant to do this because the nature of our business.
I'm probably going to wait for the next holiday (sucks for administrators) and perform the Cisco recommendation.
But, I'd like to ask this forum if anyone experienced the same issue and know's the likly cause? I don't like being in the dark....
suggestions please....
Regards
Jeff
