Error when adding Subscriber to Cluster

jp.barraja · ‎10-06-2010

Hello,

I am adding a subscriber MCS7835 5.1.3 to a Cluster over WAN (MPLS - no firewall between). The error screen appears (see attached JPEG) during subscriber setup and Publisher reports the error below - Help would be really helpfull:

Date: Oct 6 16:58:38
Machine Name: xxxcm1001
Severity: Error
Process:
Message: 599 Oct 06 145838.26 UTC %CCM_CLUSTERMANAGER-CLUSTERMANAGER-3-CLM_MsgIntChkError ClusterMgr message integrity check error. Sender's IP address10.109.50.12 App IDCisco Cluster Manager Cluster ID Node IDxxxcm1001

Pings are ok:

xxxRT1000#ping vrf CIO 10.104.93.11 size 1500 repeat 100

Type escape sequence to abort.
Sending 100, 1500-byte ICMP Echos to 10.104.93.11, timeout is 2 seconds:
!!!!!.!!!!!.!!!!!.!!!!!.!!!!!.!!!!!.!!!!!.!!!!!.!!!!!.!!!!!.
Success rate is 83 percent (50/60), round-trip min/avg/max = 12/12/12 ms

Just traffic is not correctly marked and go in Default traffic:

amsqs1001# traffic flow -tupixCA 10.104.93.11
Num TCP Flows total = 1 (all classes)

InAddr Port OutAddr Port Idle ClasI ClasO Svc
-------------------------------------------------------------------------------
10.109.50.12 1099 10.104.93.11 55948 5s /Inbound/Default /Outbound/Default -

Num UDP Flows total   =      2 (all classes)
InAddr           Port OutAddr          Port Idle ClasI ClasO Svc
-------------------------------------------------------------------------------
10.109.50.14    25757 10.104.93.11     8500 23m - /Outbound/Default -
10.109.50.12     8500 10.104.93.11     8500 4s /Inbound/Default /Outbound/Default

Many thanks

CHRIS CHARLEBOIS · ‎10-06-2010

Correct me if I'm wrong, but you are losing every sixth ping. If you have a drop rate of 16%, there are going to be problems with UDP. I mean, TCP would also have problems, but those would be masked somewhat.

Considering it is every sixth packets, I would guess there is some packet shaping and/or policing going on.

jp.barraja · ‎10-07-2010

Thx ccharlebois, I will raise the point to network team.

Regards,

JPB

Sajjit Rajagopal · ‎10-06-2010

Most of the time, this is due to MTU mismatch between the nodes. 
What is the mtu on pub and whats the limit over WAN. 
BTW, do you have the sub added under system>server of pub?
How many nodes in the cluster? What is the status of replication?
Do the security passwords match ?
You can also try restarting cluster manager on pub , reboot sub and then try the install again.  
Sometimes the Pub can get into a bad state and prevent nodes from
 joining -"admin:utils service restart Cluster Manager"

-Sajjit

jp.barraja · ‎10-07-2010

Hi Sajjit,

Many thanks for your help.

What is the mtu on pub and whats the limit over WAN. I will check the config and involve our Global Network Team!
BTW, do you have the sub added under system>server of pub? Yes
How many nodes in the cluster? 1 pub + 2 sub + 1 I need to add. What is the status of replication? The status is correct, just the new subscriber is not in the replication process.
Do the security passwords match ? Yes

You can also try restarting cluster manager on pub , reboot sub and then try the install again. Really good point!
Sometimes the Pub can get into a bad state and prevent nodes from
joining -"admin:utils service restart Cluster Manager"

I will come back to you with answer to differnet investigation, I just have to wiatthe next available window next Monday.

Regards,

JP