Buy or Renew
Buy or Renew
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
823
Views
15
Helpful
5
Replies

IPC-SW1_STBY-5-WATERMARK and RPC-SW2-4-CORE_SAT_RPC_FAIL logs

kinshaun0512
Level 1
Level 1

‎11-23-2016 01:35 AM - edited ‎03-08-2019 08:15 AM

Hi,

I got the logging error message about IPC and RPC below. I have checked CPU and memory is working fine.

Nov 23 19:17:45 AEST: %IPC-SW1_STBY-5-WATERMARK: 2882 messages pending in rcv for the port Card22/0:Request(3060000.C) seat 3090000

Nov 23 19:18:24 AEST: %IPC-SW1_STBY-5-WATERMARK: 6191 messages pending in rcv for the port Card22/0:Request(3060000.C) seat 3030000

Nov 23 19:21:27 AEST: %RPC-SW2-4-CORE_SAT_RPC_FAIL: RPC between Core and Remote Switchvs_rp_environmental:env_clear_alarm_rp failed (non-fatal). Expected when VSL goes down.

Nov 23 19:23:54 AEST: %RPC-SW2-4-CORE_SAT_RPC_FAIL: RPC between Core and Remote Switchvs_rp_environmental:env_raise_alarm_rp failed (non-fatal). Expected when VSL goes down.

Possibly is this related to bug id CSCso14087? I couldn't find a bug related to the RPC fail though. Here is a cropped show version output.

SWITCH1#sh ver
Cisco IOS Software, s2t54 Software (s2t54-ADVIPSERVICESK9-M), Version 15.1(2)SY6, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Thu 10-Sep-15 01:36 by prod_rel_team

ROM: System Bootstrap, Version 12.2(50r)SYS3, RELEASE SOFTWARE (fc1)

If you require further outputs to troubleshoot, do let me know.

Thanks in advance, will provide rating for help on answering.

Labels:
0 Helpful
5 Replies 5

Paul Chapman
Level 4
Level 4

‎11-23-2016 07:35 AM

Hi -

Post results of the following commands.

show redundancy states
show platform hardware pfc mode
show switch virtual
show switch virtual role
show switch virtual link

PSC

nmajid
Level 1
Level 1
In response to Paul Chapman

‎11-23-2016 08:44 AM

Paul, here are the requested output from the device where we picked up logs showing these error.

Nov 24 03:38:41 AEST: %RPC-SW2-4-CORE_SAT_RPC_FAIL: RPC between Core and Remote Switchvs_rp_environmental:env_raise_alarm_rp failed (non-fatal). Expected when VSL goes down.
Nov 24 03:29:59 AEST: %IPC-SW1_STBY-5-WATERMARK: 6374 messages pending in rcv for the port Card22/0:Request(3060000.C) seat 3080000
Nov 24 03:30:40 AEST: %IPC-SW1_STBY-5-WATERMARK: 2968 messages pending in rcv for the port Card22/0:Request(3060000.C) seat 3090000
Nov 24 03:31:12 AEST: %IPC-SW1_STBY-5-WATERMARK: 6377 messages pending in rcv for the port Card22/0:Request(3060000.C) seat 3010000
Nov 24 03:32:26 AEST: %IPC-SW1_STBY-5-WATERMARK: 6375 messages pending in rcv for the port Card22/0:Request(3060000.C) seat 3080000
Nov 24 03:32:59 AEST: %IPC-SW1_STBY-5-WATERMARK: 6375 messages pending in rcv for the port Card22/0:Request(3060000.C) seat 3030000
Nov 24 03:33:39 AEST: %IPC-SW1_STBY-5-WATERMARK: 6077 messages pending in rcv for the port Card22/0:Request(3060000.C) seat 3050000
Nov 24 03:34:53 AEST: %IPC-SW1_STBY-5-WATERMARK: 6376 messages pending in rcv for the port Card22/0:Request(3060000.C) seat 3080000
Nov 24 03:35:26 AEST: %IPC-SW1_STBY-5-WATERMARK: 6376 messages pending in rcv for the port Card22/0:Request(3060000.C) seat 3030000
Nov 24 03:36:00 AEST: %IPC-SW1_STBY-5-WATERMARK: 2969 messages pending in rcv for the port Card22/0:Request(3060000.C) seat 3090000
Nov 24 03:37:21 AEST: %IPC-SW1_STBY-5-WATERMARK: 6377 messages pending in rcv for the port Card22/0:Request(3060000.C) seat 3080000
Nov 24 03:37:54 AEST: %IPC-SW1_STBY-5-WATERMARK: 6377 messages pending in rcv for the port Card22/0:Request(3060000.C) seat 3030000
Nov 24 03:38:33 AEST: %IPC-SW1_STBY-5-WATERMARK: 6079 messages pending in rcv for the port Card22/0:Request(3060000.C) seat 3050000
Nov 24 03:42:21 AEST: %RPC-SW2-4-CORE_SAT_RPC_FAIL: RPC between Core and Remote Switchvs_rp_environmental:env_clear_alarm_rp failed (non-fatal). Expected when VSL goes down.

HS-EN-XD-1#show redundancy states
my state = 13 -ACTIVE
peer state = 1 -DISABLED
Mode = Duplex
Unit = Secondary
Unit ID = 38

Redundancy Mode (Operational) = sso
Redundancy Mode (Configured) = sso
Redundancy State = sso
Maintenance Mode = Disabled
Manual Swact = disabled (the peer unit is still initializing)
Communications = Up

client count = 143
client_notification_TMR = 30000 milliseconds
keep_alive TMR = 9000 milliseconds
keep_alive count = 0
keep_alive threshold = 18
RF debug mask = 0x0

HS-EN-XD-1#show platform hardware pfc mode
PFC operating mode : PFC4
Configured PFC operating mode : non-XL


HS-EN-XD-1#show switch virtual
Switch mode : Virtual Switch
Virtual switch domain number : 20
Local switch number : 2
Local switch operational role: Virtual Switch Active
Peer switch number : 1
Peer switch operational role : Virtual Switch Standby
HS-EN-XD-1#show switch virtual role
RRP information for Instance 2

--------------------------------------------------------------------
Valid Flags Peer Preferred Reserved
Count Peer Peer

--------------------------------------------------------------------
TRUE V 1 1 1

Switch Switch Status Priority Role Local Remote
Number Oper(Conf) SID SID
--------------------------------------------------------------------
LOCAL 2 UP 100(100) ACTIVE 0 0
REMOTE 1 UP 200(200) STANDBY 8918 7005

Peer 0 represents the local switch

Flags : V - Valid


In dual-active recovery mode: No

HS-EN-XD-1#show switch virtual link
VSL Status : UP
VSL Uptime : 2 years, 7 weeks, 3 days, 18 hours, 52 minutes
VSL SCP Ping : Pass
VSL ICC Ping : Pass
VSL Control Link : Te2/5/4
VSL Encryption : Configured Mode - On, Operational Mode - On

0 Helpful

Paul Chapman
Level 4
Level 4
In response to nmajid

‎11-23-2016 09:32 AM

Looks like you have failed over to the supervisor on chassis 2 and the sup on chassis 1 is having a problem. You should connect a console cable to the sup on chassis 1 and see what is happening.  Is it constantly rebooting?

PSC

nmajid
Level 1
Level 1
In response to Paul Chapman

‎11-23-2016 02:34 PM

i dont think so..see redundancy status below.

strangely i saw switch environment alert much earlier but then it resolved by itself.

still checking.. any useful suggestions or inputs are very much welcomed.

witch environment alert resolved.

HS-EN-XD-1#sh environment alarm switch 2
environmental alarms:
no alarms

HS-EN-XD-1#sh environment alarm switch 1
environmental alarms:
no alarms

HS-EN-XD-1#sh clo
09:25:54.401 AEST Thu Nov 24 2016

S-EN-XD-1#sh environment alarm switch 2
environmental alarms:
system minor alarm on switch 2 power-supply 1 power-output-fail (raised 00:01:59 ago)

HS-EN-XD-1#sh clo
01:29:13.785 AEST Thu Nov 24 2016

---------------------------------------

HS-EN-XD-1#show redundancy switchover
Switchovers this system has experienced : 5
Last switchover reason : active unit failed
Uptime since this supervisor switched to active : 29 weeks, 1 day, 14 hours, 16 minutes
Total system uptime from reload : 2 years, 7 weeks, 4 days, 50 minutes

HS-EN-XD-1#sh clo
09:30:41.447 AEST Thu Nov 24 2016

0 Helpful

Paul Chapman
Level 4
Level 4
In response to nmajid

‎11-24-2016 09:19 AM

Hi -

Based on your output, I'm not really changing my opinion about your standby chassis.

  1. Switchover occurred 29 weeks ago
  2. Switchover reason: "active unit failed"
  3. Chassis 2 is active, not standby (atypical)

If you do a "show module switch all" what is the status of the supervisor in chassis 1?

Can you connect to the standby supervisor using "remote login standby"?

PSC

Customers Also Viewed These Support Documents
Review Cisco Networking for a $25 gift card