ICM Service down on CVP Server

[harresh123]

PG is unable to establish relation with one of the CVP server. It is as if it connects to VRU and it drops the connection with a protocol error. has anyone else ran into a similar situation before. I have checked the config multiple times and nothing jumps out. TAC is clueless either. below is what I see in PIM logs.

23:23:25:075 PG10A-pim1 Trace: ICM_QoS_Policies::UpdatePolicy(CISCO-ICM-QOS-RPORT-5000) succeeded
23:23:25:079 PG10A-pim1 Connection to VRU established.
23:23:25:080 PG10A-pim1 Trace: PG->VRU:OPEN_REQ(12 bytes):InvID=328353250 Version=8 IdleTimeout=20000
23:23:25:080 PG10A-pim1 Protocol error. Length of message received from VRU: 1213486168, exceeds maximum: 65535.
23:23:25:081 PG10A-pim1 TCP connection to VRU has been broken.
23:23:34:989 PG10A-pim1 Timeout occurred waiting for response to message: Open Req (= Message Type 3).

[bill.king1]

Was it working before and just stopped? Does either side work fine? What version?

[harresh123]

NO, it never worked. and I don't understand why it is throwing an error. rebuilt the entire CVP server and still the same. I just do not see an OPEN_CONF back from CVP to VRU

23:42:25:586 PG10A-pim1 Attempting to connect to VRU at IP address: 10.x.x.x, port: 5000.
23:42:25:586 PG10A-pim1 Connection to VRU established.
23:42:25:588 PG10A-pim1 Trace: PG->VRU:OPEN_REQ(12 bytes):InvID=3505756375 Version=8 IdleTimeout=20000
23:42:25:588 PG10A-pim1 Protocol error. Length of message received from VRU: 1213486168, exceeds maximum: 65535.
23:42:25:588 PG10A-pim1 TCP connection to VRU has been broken.
23:42:35:512 PG10A-pim1 Timeout occurred waiting for response to message: Open Req (= Message Type 3).
23:42:56:578 PG10A-pim1 Process PIM set idle request: peripheral: 5019, going idle.
23:42:56:578 PG10A-pim1 Trace: Forcing mutex release during deactivation

[david.macias]

65535 looks like a port number, but the 123486168 makes no sense. Can you post shots of the pim config?

david

[therealptindall]

The fact you're getting an invalid length field in the response suggests you're not connecting to the right place.  I would check you're actually connecting to the intended address and if so, what is running on that server that is listening on port 5000.  Also, I'd suggest getting a wireshark trace at the CVP end.

[piyush aghera]

Can you increase log level and see if you find any other details regarding protocol error ?

[harresh123]

This is what I see in wireshark. http bad request from PG. Not sure what could be causing it.

[piyush aghera]

Can you share screenshots of PIM config from PG as David suggested ?  Also is it possible for you to increase log level in PG and collect PIM logs ?