Re: Catalyst 6500 Module problems

SDWorx_2 · ‎06-30-2005

Hi,

We are almost a week dealing with a very instable 6500.

What we see is that modules are undergoing a soft reset. When this reset fails the module has a faulty status. Resetting the module mostly solves the problem. But in some cases more then one module has this problem. Even the supervisor module is undergoing the same soft reset some time.

Here are the messages that appear in the logging.

1/07/2005,6:50:41,10.2.0.116,???,LOCAL,NOTICE,2005 Jul 01 06:47:58 %SYS-5-MOD_NOSCPPINGRESPONSE:Module 5 not responding... resetting module

1/07/2005,6:50:41,10.2.0.116,???,LOCAL7,NOTICE,2005 Jul 01 06:47:58 %SYS-5-MOD_RESET:Module 5 reset from Software

1/07/2005,6:50:43,10.2.0.116,???,LOCAL7,NOTICE,2005 Jul 01 06:48:00 %SYS-5-MOD_NOSCPPINGRESPONSE:Module 6 not responding... resetting module

1/07/2005,6:50:43,10.2.0.116,???,LOCAL7,NOTICE,2005 Jul 01 06:48:00 %SYS-5-MOD_RESET:Module 6 reset from Software

1/07/2005,6:53:36,10.2.0.116,???,LOCAL7,NOTICE,2005 Jul 01 06:50:52 %SYS-5-MOD_RMVDNLDSTOP:Download terminated for module 6. Module removed

1/07/2005,6:53:36,10.2.0.116,???,LOCAL7,NOTICE,2005 Jul 01 06:50:52 %SYS-5-MOD_DNLDFAIL:Download failed for module 6

1/07/2005,6:56:44,10.2.0.116,???,LOCAL7,EMERGENCY,2005 Jul 01 06:54:00 %SYS-0-MOD_NOSLCPRESPONSE:Module 1 SLCP not responding... resetting module

1/07/2005,6:56:44,10.2.0.116,???,LOCAL7,NOTICE,2005 Jul 01 06:54:00 %SYS-5-MOD_RESET:Module 1 reset from Software

What has been done:

Since module 5 always reboots first, this card had been replaced. Then we replaced the supervisor. Since still cards where rebooting we replaced the entire chassis. The power modules have also been replaced.

Anyone seen this problem before?

Many thanks!

amit-singh · ‎06-30-2005

Please paste the "show version" and "show module" ,"show power" output from the chassis.

Please see the link below and try the steps listed.

http://www.ciscotaccc.com/lanswitching/result

HTH,

-amit singh

SDWorx_2 · ‎06-30-2005

SBRU_01> (enable) sh ver

WS-C6509 Software, Version NmpSW: 7.3(2)

NMP S/W compiled on Aug 1 2002, 17:18:44

System Bootstrap Version: 5.2(1)

Hardware Version: 2.0 Model: WS-C6509 Serial #: SCA041201PZ

PS1 Module: WS-CAC-1300W Serial #: SON04301066

PS2 Module: WS-CAC-1300W Serial #: SON04290510

Mod Port Model Serial # Versions

--- ---- ------------------- ----------- --------------------------------------

1 2 WS-X6K-SUP1-2GE SAD03384603 Hw : 5.0

Fw : 5.2(1)

Fw1: 5.1(1)CSX

Sw : 7.3(2)

Sw1: 7.3(2)

WS-F6020A SAD03405664 Hw : 2.0

2 48 WS-X6248-TEL SAD04130FP9 Hw : 1.0

Fw : 4.2(0.24)VAI78

Sw : 7.3(2)

3 48 WS-X6248-TEL SAD0412056T Hw : 1.0

Fw : 4.2(0.24)VAI78

Sw : 7.3(2)

4 48 WS-X6248-TEL SAD04130FJH Hw : 1.0

Fw : 4.2(0.24)VAI78

Sw : 7.3(2)

5 48 WS-X6248A-TEL SAL05062YWP Hw : 3.0

Fw : 5.4(2)

Sw : 7.3(2)

6 48 WS-X6248A-TEL SAL05106KV4 Hw : 2.0

Fw : 5.4(2)

Sw : 7.3(2)

DRAM FLASH NVRAM

Module Total Used Free Total Used Free Total Used Free

------ ------- ------- ------- ------- ------- ------- ----- ----- -----

1 65408K 43089K 22319K 16384K 9993K 6391K 512K 377K 135K

Uptime is 0 day, 0 hour, 22 minutes

SBRU_01> (enable) sh mod

Mod Slot Ports Module-Type Model Sub Status

--- ---- ----- ------------------------- ------------------- --- --------

1 1 2 1000BaseX Supervisor WS-X6K-SUP1-2GE yes ok

2 2 48 10/100BaseTX Ethernet WS-X6248-TEL no ok

3 3 48 10/100BaseTX Ethernet WS-X6248-TEL no ok

4 4 48 10/100BaseTX Ethernet WS-X6248-TEL no ok

5 5 48 10/100BaseTX Ethernet WS-X6248A-TEL no other

6 6 48 10/100BaseTX Ethernet WS-X6248A-TEL no other

Mod Module-Name Serial-Num

--- -------------------- -----------

1 SAD03384603

2 SAD04130FP9

3 SAD0412056T

4 SAD04130FJH

5 SAL05062YWP

6 SAL05106KV4

Mod MAC-Address(es) Hw Fw Sw

--- -------------------------------------- ------ ---------- -----------------

1 00-30-80-f7-7a-c2 to 00-30-80-f7-7a-c3 5.0 5.2(1) 7.3(2)

00-30-80-f7-7a-c0 to 00-30-80-f7-7a-c1

00-13-60-48-63-00 to 00-13-60-48-66-ff

2 00-01-97-09-77-f0 to 00-01-97-09-78-1f 1.0 4.2(0.24)V 7.3(2)

3 00-01-97-10-c3-30 to 00-01-97-10-c3-5f 1.0 4.2(0.24)V 7.3(2)

4 00-01-97-09-78-e0 to 00-01-97-09-79-0f 1.0 4.2(0.24)V 7.3(2)

5 00-30-96-37-98-14 to 00-30-96-37-98-43 3.0 5.4(2) 7.3(2)

6 00-02-7e-c1-dc-8c to 00-02-7e-c1-dc-bb 2.0 5.4(2) 7.3(2)

Mod Sub-Type Sub-Model Sub-Serial Sub-Hw

--- ----------------------- ------------------- ----------- ------

1 L2 Switching Engine II WS-F6020A SAD03405664 2.0

SDWorx_2 · ‎06-30-2005

PS1 Capacity: 1153.32 Watts (27.46 Amps @42V)

PS2 Capacity: 1153.32 Watts (27.46 Amps @42V)

PS Configuration : PS1 and PS2 in Redundant Configuration.

Total Power Available: 1153.32 Watts (27.46 Amps @42V)

Total Power Available for Line Card Usage: 1153.32 Watts (27.46 Amps @42V)

Total Power Drawn From the System: 636.30 Watts (15.15 Amps @42V)

Remaining Power in the System: 517.02 Watts (12.31 Amps @42V)

Default Inline Power allocation per port: 7.00 Watts (0.16 Amps @42V)

Slot power Requirement/Usage :

Slot Card Type PowerRequested PowerAllocated CardStatus

Watts A @42V Watts A @42V

---- ------------------- ------- ------ ------- ------ ----------

1 WS-X6K-SUP1-2GE 71.40 1.70 71.40 1.70 ok

2 WS-X6248-TEL 112.98 2.69 112.98 2.69 ok

3 WS-X6248-TEL 112.98 2.69 112.98 2.69 ok

4 WS-X6248-TEL 112.98 2.69 112.98 2.69 ok

5 WS-X6248A-TEL 112.98 2.69 112.98 2.69 other

6 WS-X6248A-TEL 112.98 2.69 112.98 2.69 other

amit-singh · ‎06-30-2005

Hi,

I checked the Hardware/Software and power matix for your chassis and everythign seems to be OK. Its a strange issue and only the module WS-X6248A-TEL are having the issue.

Did you try moving the modules to differnt slots say slot 7, 8 9 or swapping them with a working one. You said that you have swapped the chassis as well, do you have same problem in the new chasis as well.

Did you contact TAC for this.

regards,

-amit singh

SDWorx_2 · ‎07-01-2005

Hello,

Hey many thanks for the checks!

Yes idd only the A modules seem to have this problem.. And the sup as well.

We've placed the blades in different unused slots. But with no result.

We have the same problems with the new chassis and new supervisor. But before more then 2 blades became faulty. The supervisor caused this.

But I have the idea that the blades become less frequent faulty.. But not always. Half an hour ago module 5 and 5 became faulty.

A little later al ports on blade 5 where in status "not connected". A show test 5 did not showed faults.. A reset solved the problem again.

We cannot open TAC cases, our Integrator is aware of this since 5 days. But no solution they provides could help us out...

Thanks

koen

jroyster · ‎07-05-2005

This really calls for a TAC case.

Can you demand the integrator open one? I'd call anyway and provide the serial number. You may have some coverage.

Wish I could help you out her as I've never seen a 6500 have such low-level problems.

Also "show env" to see if you have cooling issues. But one would expect to see messages. What log message do we have when this occurs? 6500 can log locally or syslog depending on how you set it up.

SDWorx_2 · ‎07-06-2005

A TAC case has been opened.

We do not have cooling issues.

Have a look in my next reply, I'll describe what we did till now.

Thanks for your response.

Prashanth Krishnappa · ‎07-05-2005

In the CAT OS you are running, a single faulty line card can cause issues like this. We have made enhancements to better handle this issue through the following bug

http://www.cisco.com/cgi-bin/bugtool/onebug.pl?bugid=CSCea38268

7.6(9) in 7.x has this enhancement but you will need 128MB DRAM for this. Downgrade to 6.4(17) if it as the features you need. Make sure you back up config if you are downgrading since configuration is lost during downgrade.

Also, make sure all the cards in the chassis are seated firmly and screws tightened.

Hope this helps

SDWorx_2 · ‎07-06-2005

Hi,

A TAC case has been opened and I received the same feedback as you write in your reply.

Yesterday we downgraded to version 6-4-11, because for the version you also mention we need 128 and we only have 64MB.

Tonight one module showed up as the reason for all this trouble and was placed in disable by the switch. In the afternoon we replaced the module. 2 hours ago an other module went down. It was the module we replaced yesterday preventive. This module caused a lot of trouble the past days, that's why we decided to replaces this.

What has been replaced till now:

Chassis

Power supplies

Power cables

Supervisor

3 Telco blades.

We have 2 telco blades left that have not been replaced:

Yesterday we also checked the pins and screws, all seem to be fine.

We are almost dealing 2 weeks with this, and I'm running out of idea's..

Thanks for all feedback till now!

Regards

Koen Kips

SD WORX

Belgium

p.ferreira · ‎08-05-2005

Hi, we had experienced the same issue on 4 core switches during 3 days.

"NOSCPPINGRESPONSE","Module X not responding"

These messages occured for all modules

Since the last supervisor software reset, the situation is stable.

We opened a case as well but we received the same response from the TAC :

Upgrade yours switches on 7.6(9) regarding the Bug CSCea38268.

We can't the supervisor have only 64M of Ram

Finally, did you find the solution ?

Regards

Philippe