C6509E chassis with WS-SUP32-10GE-3B weird logging messages

fmjiang1966 · ‎07-08-2010

Hi,

I recently installed a new cisco C6509E chassis with WS-SUP32-10GE-3B, there is weird logging messages.

Can someone help me to explain this? It will be greatly appreciated!

Here is the chassis info,

XXX_6509# sh module
Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
1   48 48-port 10/100/1000 RJ45 EtherModule   WS-X6148A-GE-45AF SAL1415F6L4
2   48 48-port 10/100/1000 RJ45 EtherModule   WS-X6148A-GE-45AF SAL1415FE4E
3   48 48-port 10/100/1000 RJ45 EtherModule   WS-X6148A-GE-45AF SAL1415F3XY
4   48 48-port 10/100/1000 RJ45 EtherModule   WS-X6148A-GE-45AF SAL1415FG9R
5    3 Supervisor Engine 32 10GE (Active)     WS-SUP32-10GE-3B   SAL1419HK4T
6    8 8 port 1000mb GBIC Enhanced QoS        WS-X6408A-GBIC     SAL0536B5AY
7   48 48-port 10/100/1000 RJ45 EtherModule   WS-X6148A-GE-45AF SAL1415FG8R
8   48 48-port 10/100/1000 RJ45 EtherModule   WS-X6148A-GE-45AF SAL1415F6GJ
9   48 48-port 10/100/1000 RJ45 EtherModule   WS-X6148A-GE-45AF SAL1415F6K5

Sample logging messages below:

XXX_6509#show logging sys
SEQ: MM/DD/YY HH:MM:SS MOD/SUB: SEV, COMP, MESSAGE
=====================================================

1: 06/30/10 14:33:22    5/-1: MAJ, GOLD, diag_get_port_tx_count_changed: Tx count is unchanged for mod=6 port=7 count=0 prev_coun
2: 06/30/10 14:33:22    5/-1: MAJ, GOLD, diag_get_port_tx_count_changed: Tx count is unchanged for mod=6 port=6 count=0 prev_coun
3: 06/30/10 14:33:22    5/-1: MAJ, GOLD, diag_get_port_tx_count_changed: Tx count is unchanged for mod=6 port=5 count=0 prev_coun
4: 06/30/10 14:33:22    5/-1: MAJ, GOLD, diag_get_port_tx_count_changed: Tx count is unchanged for mod=6 port=4 count=0 prev_coun
5: 06/30/10 14:33:22    5/-1: MAJ, GOLD, diag_get_port_tx_count_changed: Tx count is unchanged for mod=6 port=3 count=0 prev_coun
6: 06/30/10 14:33:22    5/-1: MAJ, GOLD, diag_get_port_tx_count_changed: Tx count is unchanged for mod=6 port=2 count=0 prev_coun
7: 06/30/10 14:32:07    5/-1: MAJ, GOLD, diag_get_port_tx_count_changed: Tx count is unchanged for mod=6 port=7 count=0 prev_coun
8: 06/30/10 14:32:07    5/-1: MAJ, GOLD, diag_get_port_tx_count_changed: Tx count is unchanged for mod=6 port=6 count=0 prev_coun

- (omitted)

147: 06/30/10 06:30:21 5/-1: MAJ, GOLD, check_bpdu_packet_hm[8/5]: newpak is NULL!

-

223: 06/30/10 02:42:24    5/-1: MAJ, GOLD, ds_get_diag_vlan_stats[4/42]: Possible duplex mismatch!
224: 06/30/10 02:42:24    5/-1: MAJ, GOLD, check_bpdu_packet_hm[4/2]: newpak is NULL!
225: 06/30/10 02:42:23    5/-1: MAJ, GOLD, check_bpdu_packet_hm[4/2]: newpak is NULL!

-

-(omitted)

1513: 06/26/10 13:38:20    5/-1: MAJ, GOLD,
test_sp_rp_inband_ping[5]: diag_hit_sp_sys_limit. SP-RP Ping Test skipped. Reason(s): RP CPU is busy (85% util).
1514: 06/26/10 13:36:21    5/-1: MAJ, DIAG_BU, online_diag_flush_pak_queue: flushing a packet from [5/1] when testing [5/-1/1]
1515: 06/26/10 13:36:16    5/-1: NON, DIAG_BU, Compiled Fri 28-May-10 16:27 by prod_rel_team
1516: 06/26/10 13:36:16    5/-1: NON, DIAG_BU, Copyright (c) 1986-2010 by Cisco Systems, Inc.
1517: 06/26/10 13:36:16    5/-1: NON, DIAG_BU, Technical Support: http://www.cisco.com/techsupport
1518: 06/26/10 13:36:16    5/-1: NON, DIAG_BU, Cisco IOS Software, s3223_sp Software (s3223_sp-ADVIPSERVICESK9_WAN-M), Version 12.2)
1519: 06/26/10 11:51:50    5/-1: MAJ, GOLD,
test_sp_rp_inband_ping[5]: diag_hit_sp_sys_limit. SP-RP Ping Test skipped. Reason(s): RP CPU is busy (81% util).
1520: 06/26/10 11:51:45    5/-1: MAJ, GOLD, check_bpdu_packet_hm[7/3]: newpak is NULL!
1521: 06/26/10 11:51:43    5/-1: MAJ, GOLD, check_bpdu_packet_hm[7/3]: newpak is NULL!
1522: 06/26/10 11:51:41    5/-1: MAJ, GOLD, check_bpdu_packet_hm[7/3]: newpak is NULL!
1523: 06/26/10 11:16:19    5/-1: MAJ, GOLD,
test_sp_rp_inband_ping[5]: diag_hit_sp_sys_limit. SP-RP Ping Test skipped. Reason(s): RP CPU is busy (81% util).
1524: 06/26/10 11:04:58    5/-1: MAJ, GOLD,
test_sp_rp_inband_ping[5]: diag_hit_sp_sys_limit. SP-RP Ping Test skipped. Reason(s): RP CPU is busy (84% util).
1525: 06/26/10 11:02:21    5/-1: MAJ, GOLD,
test_sp_rp_inband_ping[5]: diag_hit_sp_sys_limit. SP-RP Ping Test skipped. Reason(s): RP CPU is busy (84% util).

Thanks in advance!

Fuming

Leo Laohoo · ‎07-08-2010

Try reseating the Sup32 card.

rdauman24pb · ‎11-01-2010

Hello,

Just noticed your post. I am getting the same messages, plus the following messages:

08/13 02:01:01.201 E [5]
test_sp_rp_inband_ping[5]: diag_hit_sp_sys_limit
                             . SP-RP Ping Test skipped. Reason(s): RP CPU is b
                             usy (81% util).
08/14 04:18:18.891 E [5]
test_sp_rp_inband_ping[5]: diag_hit_sp_sys_limit
                             . SP-RP Ping Test skipped. Reason(s): RP CPU is b
                             usy (84% util).
08/14 20:28:02.704 E [6]    diag_get_bus_swt_mode(): Error receiving SCP resp
                             onse forgetting the switching mode
08/14 20:28:17.872 E [6]    diag_asic_read_reg16[6/1/42]: set_get_asic_regs()
                              failed (get)
08/14 20:28:17.880 E [6]    queryHyperionSynched[6]: diag_asic_read_reg16 fai
                             led

The main sympton is that the Standby SP on all of my switches are reloading - and the only messages I see in my log after the reload are the following:

Oct 26 13:53:28.375 EDT: %PFREDUN-SP-7-KPA_WARN: RF KPA messages have not been heard for 27 seconds
Oct 26 13:53:55.378 EDT: %PFREDUN-SP-7-KPA_WARN: RF KPA messages have not been heard for 54 seconds
Oct 26 13:54:22.378 EDT: %PFREDUN-SP-7-KPA_WARN: RF KPA messages have not been heard for 81 seconds
Oct 26 13:54:49.378 EDT: %PFREDUN-SP-7-KPA_WARN: RF KPA messages have not been heard for 108 seconds
Oct 26 13:55:16.426 EDT: %PFREDUN-SP-7-KPA_WARN: RF KPA messages have not been heard for 135 seconds
Oct 26 13:55:43.426 EDT: %PFREDUN-SP-7-KPA_WARN: RF KPA messages have not been heard for 162 seconds
Oct 26 13:55:46.158 EDT: %XDR-6-XDRIPCNOTIFY: Message not sent to slot 6/0 (6) because of IPC error timeout. Disabling linecard. (Expected during linecard OIR or system reloads)
Oct 26 13:55:52.770 EDT: %SNMP-5-MODULETRAP: Module 6 [Down] Trap
Oct 26 13:55:52.770 EDT: %DUAL-5-NBRCHANGE: EIGRP-IPv4:(197) 1: Neighbor 10.31.2.57 (TenGigabitEthernet6/1) is down: interface down
Oct 26 13:55:52.579 EDT: %OIR-SP-3-PWRCYCLE: Card in module 6, is being power-cycled 'RF KPA request'
Oct 26 13:55:53.534 EDT: %PFREDUN-SP-6-ACTIVE: Standby processor removed or reloaded, changing to Simplex mode
Oct 26 14:00:49.592 EDT: %PFREDUN-SP-6-ACTIVE: Standby initializing for SSO mode

I am running 12.2(33)SXI1 and wondering if reseating the SUPs did anything for you, as I tried changing the SUP in one switch and it did not make a difference. This seems like it is software and/or traffic related.

Any information would be greatly appreciated.

Thanks,

Rob

fmjiang1966 · ‎11-01-2010

Hi Rob,

You need to re-seat the SUP in question (redundant SUP in your case): loose the screws completely on both sides of the Supervisor engine, then slide out the supervisor blade, slide in/reseat the supervisor engine blade properly.

Using show commands to verify: show module all

One thing to note: "show logging" and "show logging system" tell you different stories. You may find that the message you had only appear when you type "show logging system" but not in "show logging".

Basically it tells you that the Chassis/IOS is testing the sup in question. I believe that if you reseat the sup in question, this isssue will be resolved.

Please let me know if this helps.

Fuming