%IP-TCP_NSR-3-OUT_OF_SYNC @ ASR9001 nV Edge

dfranjoso · ‎02-08-2013

Hello,

What does "%IP-TCP_NSR-3-OUT_OF_SYNC One or more NSR control messages could not be delivered to partner TCP" message means in a two ASR9001 nV Edge cluster ?

Does it means that there are problems with both RSP TCP stacks sincronization (control-plane) ? How can i confirm that the Cluster is in sync ?

Thanks !

David

smailmilak · ‎02-08-2013

Hi,

I had the same problem on 4 ASR-s, but I checked a few minutes ago and now it's gone!!

And show redundancy showed this:

v6-routing 0/RSP0/CPU0 0/RSP1/CPU0 Ready

mcast-routing 0/RSP0/CPU0 0/RSP1/CPU0 Ready

netmgmt 0/RSP0/CPU0 0/RSP1/CPU0 Ready

v4-routing 0/RSP0/CPU0 0/RSP1/CPU0 Ready

v4-routing 0/RSP0/CPU0 0/RSP1/CPU0 Not NSR-Ready

central-services 0/RSP0/CPU0 0/RSP1/CPU0 Ready

dlrsc 0/RSP0/CPU0 0/RSP1/CPU0 Ready

dsc 0/RSP0/CPU0 0/RSP1/CPU0 Ready

And now the second "v4-routing" is gone:

dsc 0/RSP0/CPU0 0/RSP1/CPU0 Ready

dlrsc 0/RSP0/CPU0 0/RSP1/CPU0 Ready

central-services 0/RSP0/CPU0 0/RSP1/CPU0 Ready

v4-routing 0/RSP0/CPU0 0/RSP1/CPU0 Ready

netmgmt 0/RSP0/CPU0 0/RSP1/CPU0 Ready

mcast-routing 0/RSP0/CPU0 0/RSP1/CPU0 Ready

v6-routing 0/RSP0/CPU0 0/RSP1/CPU0 Ready

Here is what Cisco says about this issue.

Error Message

%IP-TCP_NSR-3-OUT_OF_SYNC One or more NSR control messages could not be delivered

to partner TCP.

Explanation One or more control messages could not be delivered, after several retries, to the partner

TCP stack. This can leave the TCP stacks in a state that prevents NSR from being restored.

Recommended Action This may be due to a very basic problem with the communications

infrastructure between the TCP stacks running on different nodes. Please collect the following

information and restart the standby TCP process.

smailmilak · ‎02-08-2013

One question.

If I am using 4.3.0, do I have to install and activate the SMUs?

e.g.

asr9k-p-4.1.2.CSCty67156.pie

asr9k-p-4.0.1.CSCto05713.pie

or is this already covered in 4.3.0?

I am asking this because I got this SMUs in the 4.3.0 TAR archive.

dfranjoso · ‎02-11-2013

Hello Smailmilak,

I shouldn't be able to install SMUs from different versions. My scenario is built with two ASR9001 with nV, but i don't think both RSP are synchronizing, since when i cut the power to one node i get lost ~40 seconds of traffic, which is unacceptable.

smailmilak · ‎03-18-2013

Do you know the name of the "standby TCP process" called so I can restart it?

I have the problem again.

xthuijs · ‎03-18-2013

In which version are you seeing this?

I have seen this sporadically in between a few releases.

When NSR is enabled, it is actually the standby RSP that injects the TCP packets for BGP. (this in order to keep sync on seq numbers)

When the standby is not capable or responding to it, the active will do it himself, but that breaks NSR temporarily.

When you have cluster, the standby RSP is in the rack1 (assuming Rack 0 is the active/primary).

What I Am suspecting is that you may have excessive latency on your EOBC links of cluster?

If that is all not the case, then I would recommend collecting a show tech tcp and opening a TAC case,

regards!

xander

smailmilak · ‎03-19-2013

In version 4.3.0.

I don't know the needed commands to check eobc, I am checking some guides right now.

And I have to add the this ASR failed this morning and it had to be reloaded in order to get it to work.

ICMP to directly connected interfaces was not working.

These are the error messages from yesterday.

RP/0/RSP0/CPU0:Mar 18 07:52:38.026 : tcp[426]: %IP-TCP_NSR-3-OUT_OF_SYNC : One or more NSR control messages could not be delivered to partner TCP.

RP/0/RSP0/CPU0:Mar 18 07:52:57.008 : tcp[426]: %IP-TCP_NSR-3-OUT_OF_SYNC : One or more NSR control messages could not be delivered to partner TCP.

RP/0/RSP0/CPU0:Mar 18 07:53:07.007 : tcp[426]: %IP-TCP_NSR-3-OUT_OF_SYNC : One or more NSR control messages could not be delivered to partner TCP.

RP/0/RSP0/CPU0:ASR9010_PE_1#sh redundancy

Mon Mar 18 09:58:02.032 GMT

Redundancy information for node 0/RSP0/CPU0:

==========================================

Node 0/RSP0/CPU0 is in ACTIVE role

Node Redundancy Partner (0/RSP1/CPU0) is in STANDBY role

Standby node in 0/RSP1/CPU0 is ready

Standby node in 0/RSP1/CPU0 is NSR-ready

Node 0/RSP0/CPU0 is in process group PRIMARY role

Process Redundancy Partner (0/RSP1/CPU0) is in BACKUP role

Backup node in 0/RSP1/CPU0 is ready

Backup node in 0/RSP1/CPU0 is not NSR-ready

Group Primary Backup Status

--------- --------- --------- ---------

dsc 0/RSP0/CPU0 0/RSP1/CPU0 Ready

dlrsc 0/RSP0/CPU0 0/RSP1/CPU0 Ready

central-services 0/RSP0/CPU0 0/RSP1/CPU0 Ready

v4-routing 0/RSP0/CPU0 0/RSP1/CPU0 Ready

v4-routing 0/RSP0/CPU0 0/RSP1/CPU0 Not NSR-Ready

netmgmt 0/RSP0/CPU0 0/RSP1/CPU0 Ready

mcast-routing 0/RSP0/CPU0 0/RSP1/CPU0 Ready

v6-routing 0/RSP0/CPU0 0/RSP1/CPU0 Ready

Process Group Details

---------------------

Current primary rmf state: Ready

All backup not-ready bits clear - backup should be ready

Current primary rmf state for NSR: Not Ready

Reason for backup not NSR-ready

1011 0/RSP0/CPU0 ospf v4-routing No connection to standby

Not ready set Sat Mar 9 04:59:58 2013: 1 week, 2 days, 4 hours, 58 minutes ago

1044 0/RSP0/CPU0 mpls_ldp v4-routing LDP NSR sessions not synchronized

Not ready set Sat Mar 9 05:00:21 2013: 1 week, 2 days, 4 hours, 57 minutes ago

1012 0/RSP0/CPU0 ospf v4-routing No connection to standby

Not ready set Sat Mar 9 05:02:46 2013: 1 week, 2 days, 4 hours, 55 minutes ago

1049 0/RSP0/CPU0 bgp v4-routing BGP NSR sessions not synchronized : inst_name=default, inst_id=0

Not ready set Sat Mar 9 05:03:33 2013: 1 week, 2 days, 4 hours, 54 minutes ago

1128 0/RSP0/CPU0 l2vpn_mgr v4-routing L2VPN ATOM sessions not synchronized

Not ready set Fri Mar 15 15:33:42 2013: 2 days, 18 hours, 24 minutes ago

Reload and boot info

----------------------

A9K-RSP440-TR reloaded Fri Feb 8 11:44:50 2013: 5 weeks, 2 days, 22 hours, 13 minutes ago

Active node booted Fri Feb 8 11:44:50 2013: 5 weeks, 2 days, 22 hours, 13 minutes ago

Standby node boot Fri Feb 8 11:44:53 2013: 5 weeks, 2 days, 22 hours, 13 minutes ago

Standby node last went not ready Fri Feb 8 11:48:34 2013: 5 weeks, 2 days, 22 hours, 9 minutes ago

Standby node last went ready Fri Feb 8 11:48:34 2013: 5 weeks, 2 days, 22 hours, 9 minutes ago

There have been 0 switch-overs since reload

I will open a TAC case for this!