Hi All,
I have two 6509s with a single CSM in each set up for FT. No changes have been made to the configuration recently but I have noticed the following EMs occuring in the logs. Both logs are from the same period of time so you can see what is happening on one CSM and compare it to the same time stamp on the other.
This problem is reoccuring every couple of days
As far as I can see everything looks good in terms of hardware and config, any help would be greately appreciated
Please note I only have remote access to the switches and the Software version on both CSM is 4.2(2)
Switch uptime is not greater that 828 days and I can see that of the 58,152 heartbeats the Active Master has sent, only 58,056 have been recieved by the Backup/Secondary
I have looked into this for the last 4 hours and the only assumption I can make is that the L2 Etherchannel beteen the Cores is getting swamped with traffic and causing the keep alives to fail, but I don't know how to prove this.
Core 1A
Feb 7 14:49:23.130 UTC: %CSM_SLB-4-REDUNDANCY_WARN: Module 1 FT warning: LRP: no ACK from standby.. standby may be down
Feb 7 14:49:23.730 UTC: %CSM_SLB-4-REDUNDANCY_WARN: Module 1 FT warning: LRP: no ACK from standby.. standby may be down
Feb 7 14:49:26.130 UTC: %CSM_SLB-4-REDUNDANCY_WARN: Module 1 FT warning: LRP: no ACK from standby.. standby may be down
Feb 7 14:49:35.730 UTC: %CSM_SLB-4-REDUNDANCY_WARN: Module 1 FT warning: LRP: no ACK from standby.. standby may be down
Feb 7 14:50:04.691 UTC: %SEC-6-IPACCESSLOGRL: access-list logging rate-limited or missed 210 packets
Feb 7 14:50:14.131 UTC: %CSM_SLB-4-REDUNDANCY_WARN: Module 1 FT warning: LRP: no ACK from standby.. standby may be down
Feb 7 14:51:04.715 UTC: %SEC-6-IPACCESSLOGRL: access-list logging rate-limited or missed 1 packet
Feb 7 14:52:04.739 UTC: %SEC-6-IPACCESSLOGRL: access-list logging rate-limited or missed 3 packets
Feb 7 14:52:08.451 UTC: %CSM_SLB-4-REDUNDANCY_WARN: Module 1 FT warning: LRP: no ACK from standby.. standby may be down
Feb 7 14:52:09.051 UTC: %CSM_SLB-4-REDUNDANCY_WARN: Module 1 FT warning: LRP: no ACK from standby.. standby may be down
Feb 7 14:52:11.447 UTC: %CSM_SLB-4-REDUNDANCY_WARN: Module 1 FT warning: LRP: no ACK from standby.. standby may be down
Feb 7 14:52:21.047 UTC: %CSM_SLB-4-REDUNDANCY_WARN: Module 1 FT warning: LRP: no ACK from standby.. standby may be down
Feb 7 14:52:59.452 UTC: %CSM_SLB-4-REDUNDANCY_WARN: Module 1 FT warning: LRP: no ACK from standby.. standby may be down
Feb 7 14:53:04.764 UTC: %SEC-6-IPACCESSLOGRL: access-list logging rate-limited or missed 224 packets
Feb 7 14:53:37.808 UTC: %C6KPWR-SP-4-DISABLED: power to module in slot 1 set off (Module not responding to Keep Alive polling)
Feb 7 14:55:16.820 UTC: %MLS_RATE-4-DISABLING: The Layer2 Rate Limiters have been disabled.
Feb 7 14:55:15.800 UTC: %DIAG-SP-6-RUN_MINIMUM: Module 1: Running Minimal Diagnostics...
Feb 7 14:55:16.468 UTC: %DIAG-SP-6-DIAG_OK: Module 1: Passed Online Diagnostics
Feb 7 14:55:18.988 UTC: %OIR-SP-6-INSCARD: Card inserted in slot 1, interfaces are now online
Feb 7 14:55:20.356 UTC: %CSM_SLB-6-REDUNDANCY_INFO: Module 1 FT info: State Transition Init -> Standby
Feb 7 14:55:20.700 UTC: %CSM_SLB-6-REDUNDANCY_INFO: Module 1 FT info: State Transition Standby -> Active
Feb 7 14:55:20.864 UTC: %CSM_SLB-6-INFO: Module 1 info: config reload completed successfully.
Feb 7 14:55:23.896 UTC: %CSM_SLB-4-REDUNDANCY_WARN: Module 1 FT warning: LRP: no ACK from standby.. standby may be down
Feb 7 14:55:24.496 UTC: %CSM_SLB-4-REDUNDANCY_WARN: Module 1 FT warning: LRP: no ACK from standby.. standby may be down
Feb 7 14:55:26.896 UTC: %CSM_SLB-4-REDUNDANCY_WARN: Module 1 FT warning: LRP: no ACK from standby.. standby may be down
Feb 7 14:55:36.496 UTC: %CSM_SLB-4-REDUNDANCY_WARN: Module 1 FT warning: LRP: no ACK from standby.. standby may be down
Feb 7 14:56:14.905 UTC: %CSM_SLB-4-REDUNDANCY_WARN: Module 1 FT warning: LRP: no ACK from standby.. standby may be down
COR-1A#show mod contentswitchingModule 1 ft
FT group 1, vlan 4
This box is active
priority 20, heartbeat 1, failover 3, preemption is on
Core 1B
Feb 7 14:49:44.463 UTC: %C6KPWR-SP-4-DISABLED: power to module in slot 1 set off (Module not responding to Keep Alive polling)
Feb 7 14:51:22.467 UTC: %DIAG-SP-6-RUN_MINIMUM: Module 1: Running Minimal Diagnostics...
Feb 7 14:51:24.495 UTC: %MLS_RATE-4-DISABLING: The Layer2 Rate Limiters have been disabled.
Feb 7 14:51:24.156 UTC: %DIAG-SP-6-DIAG_OK: Module 1: Passed Online Diagnostics
Feb 7 14:51:26.657 UTC: %OIR-SP-6-INSCARD: Card inserted in slot 1, interfaces are now online
Feb 7 14:51:28.511 UTC: %CSM_SLB-6-INFO: Module 1 info: config reload completed successfully.
Feb 7 14:51:28.567 UTC: %CSM_SLB-6-REDUNDANCY_INFO: Module 1 FT info: State Transition Init -> Standby
Feb 7 14:52:31.748 UTC: %C6KPWR-SP-4-DISABLED: power to module in slot 1 set off (Module not responding to Keep Alive polling)
Feb 7 14:54:10.782 UTC: %MLS_RATE-4-DISABLING: The Layer2 Rate Limiters have been disabled.
Feb 7 14:54:09.762 UTC: %DIAG-SP-6-RUN_MINIMUM: Module 1: Running Minimal Diagnostics...
Feb 7 14:54:10.554 UTC: %DIAG-SP-6-DIAG_OK: Module 1: Passed Online Diagnostics
Feb 7 14:54:12.870 UTC: %OIR-SP-6-INSCARD: Card inserted in slot 1, interfaces are now online
Feb 7 14:54:14.794 UTC: %CSM_SLB-6-INFO: Module 1 info: config reload completed successfully.
Feb 7 14:55:04.094 UTC: %CSM_SLB-6-REDUNDANCY_INFO: Module 1 FT info: State Transition Elect -> Active
Feb 7 14:55:07.102 UTC: %CSM_SLB-4-REDUNDANCY_WARN: Module 1 FT warning: LRP: no ACK from standby.. standby may be down
Feb 7 14:55:07.698 UTC: %CSM_SLB-4-REDUNDANCY_WARN: Module 1 FT warning: LRP: no ACK from standby.. standby may be down
Feb 7 14:55:10.098 UTC: %CSM_SLB-4-REDUNDANCY_WARN: Module 1 FT warning: LRP: no ACK from standby.. standby may be down
Feb 7 14:55:19.698 UTC: %CSM_SLB-4-REDUNDANCY_WARN: Module 1 FT warning: LRP: no ACK from standby.. standby may be down
Feb 7 14:55:20.502 UTC: %CSM_SLB-4-REDUNDANCY_WARN: Module 1 FT warning: FT configuration might be out of sync.
Feb 7 14:55:20.702 UTC: %CSM_SLB-6-REDUNDANCY_INFO: Module 1 FT info: State Transition Active -> Standby
Feb 7 14:55:45.043 UTC: %C6KPWR-SP-4-DISABLED: power to module in slot 1 set off (Module not responding to Keep Alive polling)
Feb 7 14:57:24.076 UTC: %MLS_RATE-4-DISABLING: The Layer2 Rate Limiters have been disabled.
Feb 7 14:57:23.048 UTC: %DIAG-SP-6-RUN_MINIMUM: Module 1: Running Minimal Diagnostics...
Feb 7 14:57:23.740 UTC: %DIAG-SP-6-DIAG_OK: Module 1: Passed Online Diagnostics
Feb 7 14:57:26.164 UTC: %OIR-SP-6-INSCARD: Card inserted in slot 1, interfaces are now online
Feb 7 14:57:27.736 UTC: %CSM_SLB-6-REDUNDANCY_INFO: Module 1 FT info: State Transition Init -> Standby
Feb 7 14:57:28.104 UTC: %CSM_SLB-6-INFO: Module 1 info: config reload completed successfully.
COR-1B#show module contentSwitchingModule 1 ft
FT group 1, vlan 4
This box is in standby state
priority 10, heartbeat 1, failover 3, preemption is on
Thanks in advance
David
