Line Card PwrDown Status after DFC3BXL Upgrade

[MP13]

Hi,

I know these are old devices and we have a plan to replace them with ASR's over the next 12 months however in the mean time we are trying to upgrade the line cards in a 6503-E chassis to support more routes by installing a SUP720-3BXL, DFC3BXL and DFC3CXL daughter boards on the line cards and upgrading cards to 1GB memory where required too.

The SUP720-3BXL and X6708-10GE with DFC3CXL upgrades all went fine and work correctly however the upgrade of the DFC on the X6748-GE-TX from a DFC3B to a DFC3BXL is not working and the line card ends up in a PwrDown status with the below logged to the console:

*Jan 4 08:52:06.971: %OIR-SP-3-PWRCYCLE: Card in module 2, is being power-cycled 'Off (Module Reset due to exception or user request)'
*Jan 4 08:52:06.971: %C6KPWR-SP-4-DISABLED: power to module in slot 2 set Off (Module Reset due to exception or user request)
*Jan 4 08:53:25.779: %PM_SCP-SP-1-LCP_FW_ERR: System resetting module 2 to recover from error: Linecard received system exception. Errcode =
2509000001
*Jan 4 08:53:25.779: %OIR-SP-3-PWRCYCLE: Card in module 2, is being power-cycled 'Off (Module Reset due to exception or user request)'
*Jan 4 08:53:25.779: %C6KPWR-SP-4-DISABLED: power to module in slot 2 set Off (Module Reset due to exception or user request)
*Jan 4 08:54:46.149: %PM_SCP-SP-1-LCP_FW_ERR: System resetting module 2 to recover from error: Linecard received system exception. Errcode =
2509000001
*Jan 4 08:54:46.149: %OIR-SP-3-PWRCYCLE: Card in module 2, is being power-cycled 'Off (Module Reset due to exception or user request)'
*Jan 4 08:54:46.149: %C6KPWR-SP-4-DISABLED: power to module in slot 2 set Off (Module Reset due to exception or user request)
*Jan 4 08:54:46.157: %OIR-SP-4-RESTART: The card in slot 2 has reached the maximum number of 5 restarts, this card will be disabled

We run these in other devices as well with similar IOS, FW and SW versions so I've matched them up with the other devices just to get them on an identical setup in case there were issues with versions. I'm currently running the below:

Mod Port Model Serial # Versions
---- ---- ------------------ ----------- -------------------------------------
1 2 WS-SUP720-3BXL SAD102007D6 Hw : 5.2
Fw : 8.5(4)
Sw : 15.1(2)SY9
Sw1 : mk1c@1.0.77
WS-SUP720 SAD102200L5 Hw : 2.5
Fw : 12.2(17r)SX7
Sw : 15.1(2)SY9
WS-F6K-PFC3BXL SAD1021086M Hw : 1.8
2 48 WS-X6748-GE-TX SAD080801CH Hw : 1.4
Fw : 12.2(14r)S5
Sw : 15.1(2)SY9
Sw1 : unknown
WS-F6700-DFC3BXL SAD10490B6A Hw : 5.3
3 8 WS-X6708-10GE SAL1109JEKT Hw : 1.3
Fw : 12.2(18r)S1
Sw : 15.1(2)SY9
Sw1 : mk1c@1.0.77
WS-F6700-DFC3CXL SAD111603PR Hw : 1.0

ROM: System Bootstrap, Version 12.2(17r)SX7, RELEASE SOFTWARE (fc1)
BOOTLDR: Cisco IOS Software, s72033_rp Software (s72033_rp-ADVIPSERVICESK9-M), Version 15.1(2)SY9, RELEASE SOFTWARE (fc1)

I have used different X6748-GE-TX line cards, different memory modules, different DFC3BXL's, different slots in the chassis, etc. Nothing hasn't been replaced other than the chassis itself and the issue persists. 

If we put the DFC3B back on the line card then it works fine. It's only when running the DFC3BXL that the issue occurs in any configuration which makes me think it's a compatibility issue however as mentioned we run the same configuration on other devices.

I've been through the data sheets and made sure that there are not any slot restrictions and looked for anything specific in release notes as well but I'm at a bit of a loss and just wondering if anyone had any experience of this to provide any ideas before I roll it back to the DFC3B.

[Leo Laohoo]

---

@MP13 wrote:

*Jan 4 08:54:46.149: %PM_SCP-SP-1-LCP_FW_ERR: System resetting module 2 to recover from error: Linecard received system exception. Errcode =2509000001

---

Move this line card to another slot. 

If the same error message "follow" the line card to another slot, contact Cisco TAC and organize for an RMA.

NOTE:  I can see that someone has taken extra ordinary steps to "edit" the "sh module" output.  If this is a "classified" network, contact Cisco TAC directly.  If this is not a "classified" network, stop editing the output because useful information are being removed unecessarily.

[MP13]

Hi,

We have moved the line card to another slot and the issue persists however we've also used a different line card and different DFC 3BXL and the issue is always present so we've effectively replaced everything already that an RMA would do. The only time it ever works is with a DFC3B.

[Leo Laohoo]

I am really confused with the output provided. 

Please provide the complete output to the command "sh module" -- And DO NOT EDIT ANYTHING.

[MP13]

The output is standard and has not been modified. The previous output was from "sh mod ver".

"sh mod" output is below.

Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 2 Supervisor Engine 720 (Active) WS-SUP720-3BXL SAD102007D6
2 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAD080801CH
3 8 CEF720 8 port 10GE with DFC WS-X6708-10GE SAL1109JEKT

Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
1 0011.9202.668c to 0011.9202.668f 5.2 8.5(4) 15.1(2)SY9 Ok
2 000f.2462.71a8 to 000f.2462.71d7 1.4 12.2(14r)S5 15.1(2)SY9 PwrDown
3 0018.b967.1300 to 0018.b967.1307 1.3 12.2(18r)S1 15.1(2)SY9 Ok

Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
1 Policy Feature Card 3 WS-F6K-PFC3BXL SAD1021086M 1.8 Ok
1 MSFC3 Daughterboard WS-SUP720 SAD102200L5 2.5 Ok
2 Distributed Forwarding Card WS-F6700-DFC3BXL SAD10490B6A 5.3 PwrDown
3 Distributed Forwarding Card WS-F6700-DFC3CXL SAD111603PR 1.0 Ok

Mod Online Diag Status
---- -------------------
1 Pass
2 Not Applicable
3 Pass

[Leo Laohoo]

WTF ... Is this a 6503?

[MP13]

Yes, it's a 6503-E.

[Leo Laohoo]

The WS-X6748-GE-TX is not supported in the Catalyst 6503 switch chassis. The WS-X6748-GE-TX module can occupy any slot in any other Catalyst 6500 or Catalyst 6500-E chassis except the Catalyst 6513 chassis where the module must be installed in slots 9–13. The module does not power up when it is installed in slots 1–8.

---

WS-X6748-GE-TX and WS-X6848-TX-2T/-2TXL Ethernet Modules

Scroll down to the "Chassis/slot restrictions" section.

[MP13]

Thanks. We are on a 6503-E though which is supported. It's only the 6503 which isn't supported.

[Leo Laohoo]

---

@MP13 wrote:

Thanks. We are on a 6503-E though which is supported. It's only the 6503 which isn't supported.

---

Don't believe me?  Raise a TAC Case and get TAC to verify my response.

[MP13]

The configuration has already been validated by our Cisco partner but I can get them to double check it and have logged a case with them.

I have 4 others of these with the exact same configuration already in operation though so I'm fairly certain it's only the 6503 it's not supported in not the 6503-E which would match what the article references as it says 6503 not the 6503-E.

It also works until the DFC3BXL is put on the card and it's been running with the DFC3B for the last few years.

[marce1000]

 - FYI : https://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/41265-186-ErrormsgIOS-41265.html#anc66

 M.