Catalyst 4510 networking modules power restoration problem

[chaffner1]

I have a Catalyst 4510 with a bit of an odd issue.  If any of the network modules (that is, any of the modules that are not either of the two supervisors) are powered down via CLI (no hw-module module # power) or physically removed from the shelf, they will not come back on line when restored to the system.  They will either come up as "Faulty" or will repeatedly attempt to "reinsert" themselves.  The only way to restore them to operation is to completely power off the entire Catalyst and then reboot.

It does not appear to be the modules themselves as I have multiple 4510's that I can exchange module between.  When a module will not boot into the problem 4510, I can remove and successfully insert it into another and it will boot.  Contrary to that, I can remove a module from any of my "good" 4510's and, when I attempt to introduce them into the problem unit, they fail to boot.  Yet, if I perform a hard boot (physical power off/on) of my trouble 4510 they will be incorporated with no difficulty.  And, yes, I am swapping like-for-like when it comes to the modules.

Any ideas?  Has anyone seen this behavior before?  I have already replaced the chassis once for another problem related to the supervisors and the backplane.  I am hoping that I don't have to hit up Cisco for another RMA.

[Leo Laohoo]

Contact Cisco TAC and organize to get the chassis RMAed.  
The description provided leads me to suspect the 4510 backplane is faulty.

[marce1000]

                                  >....they will not come back on line when restored to the system

 - Check the logs when that happens, also attach a console and follow-up on messages displayed , also check the module status led-indicator.

 M.

[chaffner1]

Thanks for your reply.

Regarding the module status led-indicator, it starts off red (when powered down), then goes amber during the boot up phase, briefly turns red after the boot up phase fails and then goes amber again when the boot up process restarts....over and over again.

Here is a brief excerpt of the console during the entire process, including a listing of the modules at the beginning.

Mod Ports Card Type Model Serial No.
---+-----+--------------------------------------+------------------+-----------
1 48 10/100/1000BaseT Premium POE E Series WS-X4748-RJ45V+E CAT2227L4AA
2 24 1000BaseX SFP WS-X4724-SFP-E CAT2228L389
3 24 1000BaseX SFP WS-X4724-SFP-E CAT2228L55B
5 8 Sup 8-E 10GE (SFP+), 1000BaseX (SFP) WS-X45-SUP8-E CAT2228L139
6 8 Sup 8-E 10GE (SFP+), 1000BaseX (SFP) WS-X45-SUP8-E CAT2228L157
7 48 10/100/1000BaseT Premium POE E Series WS-X4748-RJ45V+E CAT2227L3YF
8 24 1000BaseX SFP WS-X4724-SFP-E CAT2228L38Q
9 24 1000BaseX SFP WS-X4724-SFP-E CAT2228L386

M MAC addresses Hw Fw Sw Status
--+--------------------------------+---+------------+----------------+---------
1 502f.a878.96cc to 502f.a878.96fb 1.2 Ok
2 502f.a80a.e684 to 502f.a80a.e69b 1.0 Ok
3 502f.a80a.e744 to 502f.a80a.e75b 1.0 Ok
5 acf2.c5df.9d00 to acf2.c5df.9d07 1.3 15.1(1r)SG5 03.08.06.E Ok
6 acf2.c5df.9d08 to acf2.c5df.9d0f 1.3 15.1(1r)SG5 03.08.06.E Ok
7 a093.5106.5850 to a093.5106.587f 1.2 Ok
8 502f.a80a.e654 to 502f.a80a.e66b 1.0 Ok
9 502f.a80a.e6cc to 502f.a80a.e6e3 1.0 Ok

Mod Redundancy role Operating mode Redundancy status
----+-------------------+-------------------+----------------------------------
5 Active Supervisor SSO Active
6 Standby Supervisor SSO Standby hot

NSCore#conf t
NSCore(config)#no hw-module module 7 power
NSCore(config)#
000434: Aug 26 14:58:44.510 PDT: %PARSER-5-CFGLOG_LOGGEDCMD: User:administrator logged command:no hw-module module 7 power
000435: Aug 26 14:58:44.729 PDT: %EC-5-UNBUNDLE: Interface Gi7/18 left the port-channel Po8
000436: Aug 26 14:58:44.732 PDT: %EC-5-UNBUNDLE: Interface Gi7/17 left the port-channel Po7
000437: Aug 26 14:58:44.735 PDT: %EC-5-UNBUNDLE: Interface Gi7/16 left the port-channel Po6
000438: Aug 26 14:58:44.739 PDT: %EC-5-UNBUNDLE: Interface Gi7/15 left the port-channel Po5
000439: Aug
NSCore(config)# 26 14:58:44.742 PDT: %EC-5-UNBUNDLE: Interface Gi7/14 left the port-channel Po4
000440: Aug 26 14:58:44.745 PDT: %EC-5-UNBUNDLE: Interface Gi7/13 left the port-channel Po3
000441: Aug 26 14:58:44.748 PDT: %EC-5-UNBUNDLE: Interface Gi7/12 left the port-channel Po2
000442: Aug 26 14:58:44.752 PDT: %EC-5-UNBUNDLE: Interface Gi7/11 left the port-channel Po1
000443: 000047: Aug 26 14:58:44.753 PDT: %EC-5-UNBUNDLE: STANDBY:Interface Gi7/18 left the port-channel Po8
000444: 000048: Aug 26 14:58:44.756 PDT:
NSCore(config)#%EC-5-UNBUNDLE: STANDBY:Interface Gi7/17 left the port-channel Po7
000445: 000049: Aug 26 14:58:44.763 PDT: %EC-5-UNBUNDLE: STANDBY:Interface Gi7/16 left the port-channel Po6
000446: 000050: Aug 26 14:58:44.766 PDT: %EC-5-UNBUNDLE: STANDBY:Interface Gi7/15 left the port-channel Po5
000447: 000051: Aug 26 14:58:44.771 PDT: %EC-5-UNBUNDLE: STANDBY:Interface Gi7/14 left the port-channel Po4
000448: 000052: Aug 26 14:58:44.774 PDT: %EC-5-UNBUNDLE: STANDBY:Interface Gi7/13 left the port-channel Po3
000449:
NSCore(config)# Aug 26 14:58:45.682 PDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet7/48, changed state to down
000450: Aug 26 14:58:45.712 PDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet7/36, changed state to down
000451: Aug 26 14:58:45.712 PDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet7/34, changed state to down
000452: Aug 26 14:58:45.712 PDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet7/33, changed state to down
000477: 000053: Au
NSCore(config)#g 26 14:58:44.776 PDT: %EC-5-UNBUNDLE: STANDBY:Interface Gi7/12 left the port-channel Po2
000478: 000054: Aug 26 14:58:44.779 PDT: %EC-5-UNBUNDLE: STANDBY:Interface Gi7/11 left the port-channel Po1
000479: Aug 26 14:58:46.699 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/48, changed state to down
000480: Aug 26 14:58:46.700 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/42, changed state to down
000481: Aug 26 14:58:46.712 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/36, changed state to down
00
NSCore(config)#0482: Aug 26 14:58:46.713 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/34, changed state to down
000483: Aug 26 14:58:46.713 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/33, changed state to down
000484: Aug 26 14:58:46.714 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/32, changed state to down
000485: Aug 26 14:58:46.714 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/31, changed state to down
000486: Aug 26 14:58:46.715 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/27, changed state to down
NSCore(config)#
000487: Aug 26 14:58:46.726 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/26, changed state to down
000488: Aug 26 14:58:46.727 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/25, changed state to down
000489: Aug 26 14:58:46.727 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/24, changed state to down
000490: Aug 26 14:58:46.728 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/22, changed state to down
000491: Aug 26 14:58:46.728 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/21, changed state to
NSCore(config)#down
000492: Aug 26 14:58:46.737 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/18, changed state to down
000493: Aug 26 14:58:46.737 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/17, changed state to down
000494: Aug 26 14:58:46.738 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/16, changed state to down
000495: Aug 26 14:58:46.738 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/15, changed state to down
000496: Aug 26 14:58:46.745 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/14, changed state
NSCore(config)# to down
000497: Aug 26 14:58:46.745 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/13, changed state to down
000498: Aug 26 14:58:46.750 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/12, changed state to down
000499: Aug 26 14:58:46.750 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/11, changed state to down
000500: Aug 26 14:58:46.754 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/9, changed state to down
000501: Aug 26 14:58:46.755 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/8, changed sta
NSCore(config)#te to down
000502: Aug 26 14:58:46.759 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/7, changed state to down
000503: Aug 26 14:58:46.759 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/6, changed state to down
000504: Aug 26 14:58:46.760 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/5, changed state to down
000505: Aug 26 14:58:46.765 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/4, changed state to down
000506: Aug 26 14:58:46.766 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/3, changed stat
NSCore(config)#e to down
000507: Aug 26 14:58:46.766 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet7/2, changed state to down
000508: 000055: Aug 26 14:58:46.841 PDT: %C4K_IOSMODPORTMAN-6-MODULEOFFLINE: STANDBY:Module 7 is offline
000509: Aug 26 14:58:48.543 PDT: %C4K_IOSMODPORTMAN-6-MODULEOFFLINE: Module 7 is offline
NSCore(config)#hw-module module 7 power
NSCore(config)#
000510: Aug 26 14:59:13.564 PDT: %PARSER-5-CFGLOG_LOGGEDCMD: User:administrator logged command:hw-module module 7 power
NSCore(config)#
000511: Aug 26 14:59:16.389 PDT: %C4K_IOSMODPORTMAN-6-MODULEREMOVED: Module 7 is removed
000512: 000056: Aug 26 14:59:15.409 PDT: %C4K_IOSMODPORTMAN-6-MODULEREMOVED: STANDBY:Module 7 is removed
NSCore(config)#
000513: Aug 26 14:59:39.003 PDT: %C4K_IOSMODPORTMAN-6-MODULEINSERTED: Module 7 is inserted
NSCore(config)#
000514: Aug 26 14:59:39.706 PDT: %LINK-3-UPDOWN: Interface GigabitEthernet8/8, changed state to up
000515: Aug 26 14:59:40.710 PDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/8, changed state to up
NSCore(config)#
NSCore(config)#
NSCore(config)#
000516: Aug 26 15:05:19.002 PDT: %C4K_CHASSIS-3-MODULEBRINGUPTIMEDOUT: Module 7 software bring up did not complete within allocated time
000517: Aug 26 15:05:19.159 PDT: %C4K_SYSMAN-3-LINECARDDIAGSFAILED: Module in slot 7 failed online diagnostics. Please use 'show diagnostic result module 7 test <tid> detail' command for details.
NSCore(config)#
000518: Aug 26 15:05:20.641 PDT: %C4K_IOSMODPORTMAN-6-MODULEREMOVED: Module 7 is removed
NSCore(config)#
000519: Aug 26 15:05:46.575 PDT: %C4K_IOSMODPORTMAN-6-MODULEINSERTED: Module 7 is inserted
NSCore(config)#

[marce1000]

            - As stated by Leo , it probably is chassis-problem and the whole box should then be replaced.

 M.

[paul driver]

Hello

it could be the line card not the chassis - have you tried reseating the line card!

[chaffner1]

I'm sorry, but I am not familiar with the term "line card".  Would you be more specific?

[Giuseppe Larosa]

Hello @chaffner1 ,

with line card we refer to the modules installed in the chassis that are not supervisors

in your case:

1 48 10/100/1000BaseT Premium POE E Series WS-X4748-RJ45V+E CAT2227L4AA
2 24 1000BaseX SFP WS-X4724-SFP-E CAT2228L389
3 24 1000BaseX SFP WS-X4724-SFP-E CAT2228L55B
5 8 Sup 8-E 10GE (SFP+), 1000BaseX (SFP) WS-X45-SUP8-E CAT2228L139
6 8 Sup 8-E 10GE (SFP+), 1000BaseX (SFP) WS-X45-SUP8-E CAT2228L157
7 48 10/100/1000BaseT Premium POE E Series WS-X4748-RJ45V+E CAT2227L3YF
8 24 1000BaseX SFP WS-X4724-SFP-E CAT2228L38Q
9 24 1000BaseX SFP WS-X4724-SFP-E CAT2228L386

you have line cards in slot 1,2,3 then 7,8,9.

However if the same module/ line card is able to boot on another chassis the issue is related to the chassis.

Hope to help

Giuseppe

[chaffner1]

Thank you, Giuseppe.  That seems to be the direction this is going in (replacing the chassis).

[chaffner1]

First, thanks to everyone for all the help.  I replaced the chassis with one that I know is good.  However, the problem did not go away.  But, in the process, I was able to gather new information.  As I brought up the new chassis, I tested each card as it was added by successfully shutting down and restarting each module.

This is where it gets weird (from the standpoint of an amateur like myself.

After I brought up the entire system (a video security system) I found that the individual modules would no longer boot successfully, again.  If I shutdown all of the server based services related to the application, I was again able to restart all of the modules.  Another observation, with the system fully up and running, If I shutdown all of the network modules (leaving the supervisors up with the 4510), and then rapidly issue HW-MODULE MODULE X POWER commands for each individual card, I am able to bring them all up.  Is it possible that we are overdriving the 4510 some way in that it is simply handling too much to process the reactivation of a card?

[marce1000]

     - Check the software version on the replacement-chassis, make sure it is recent and or an advisory release.

 M.

[chaffner1]

Thank you, but would you please advise, is this different from what is loaded onto the supervisors and, if so, where would I pull this information from?

[marce1000]

 - Check current software version with : show version for the rest go here :

                           https://software.cisco.com/download/home/277241601

  At the right choose correct model+supervisor(-model) , then choose an advisory release.

 M.