SIPtrunkOOS on one Subscriber, but Full Service in CUCM.

[catkins101]

Hello,

I have a SIP trunk between CUCM and Xmedius Fax server that has shown stable for over two days (Full Service in CUCM). However, the application logs in Syslog viewer show the SIP trunk alarms repeatedly coming in on one subscriber and completely OOS on the other subscriber. The Publisher has it bouncing but not as much as the first subscriber. I am unable to locate any information on the protocols from each node to report back the status of Full Service. If each node is using Options Ping to check the status, which is responsible for the status Full Service. I have Run On All Nodes checked, but I am not sure this has anything to do with it. I ran a ping from the CLI on the second subscriber that is reporting it OOS, but I ping fine. Solarwinds shows the trunk state from that subscriber is Unknown. It seems each server is getting different results.

[Anthony Holloway]

You need to use RTMT to see per node SIP status of a trunk, and any node, and at least one node needs to have OPTIONS working for the entire trunk to show In Service.

ICMP is ok for testing, but the problem is not likely at layer 3, rather within the SIP stack. Either Xmedius is not replying to all of your OPTIONS (overloading it? self protection?), and maybe even doesn't have all of your CUCM nodes defined within it.

I don't know xmedius at all, but if it's like CUCM or CUBE, then all of the CUCM peer addresses (run on all nodes) need to be in xmedius in order for it to accept SIP traffic from CUCM.

You should pull CM traces too, for like a 15 minute window, from all of your CUCM nodes, and see what the OPTIONS request/response towards xmedius look like (if present at all).

[catkins101]

I have all of the nodes defined in XMedius the same and their support has looked them over. Can you explain how to look for the OPTION request/response in RTMT?

[Anthony Holloway]

Sure, just open RTMT and go to: Voice/Video > Device > Device Search > {Your Cluster} > SIP Trunk

In the wizard to select the SIP trunk, select whatever you need to to find your SIP trunk. For me, I'll just select Any Status + Any CallManager and click Finish, since I don't have many SIP trunks.

At this point, you should notice that the SIP trunk name is in the list two or more times, once for each node in the CUCM cluster, each with their own status.

[catkins101]

Thanks for all your help on this issue. I was able to collect some traces and pulled them up in Translator X. I see good outgoing and incoming TCP messages for one of my subscribers and the publisher, but the other subscriber shows no attempts on SIP OPTIONS. That would explain why RTMT shows that node down for several days. I am going to try and capture a time when one of these other two drop service. Do you know any reason why there would be no SIP TCP chatter from the one subscriber? I have even looked into the firewall rules on each node and they appear the same, but I am not experienced with them at all. The default SIP signaling source port is 5060 for XMedius.

139  ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:5060 flags:0x02/0x02 limit: up to 35/sec burst 7500 mode srcip-dstport
140  LOG        tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:5060 flags:0x02/0x02 limit: avg 1/min burst 1 LOG flags 0 level 4 prefix ` Exceeded hashlimit '
141  DROP       tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:5060 flags:0x02/0x02
142  ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0           udp dpt:5060 limit: up to 1000/sec burst 10000 mode srcip-dstport
143  LOG        udp  --  0.0.0.0/0            0.0.0.0/0           udp dpt:5060 limit: avg 1/min burst 1 LOG flags 0 level 4 prefix ` Exceeded hashlimit '
144  DROP       udp  --  0.0.0.0/0            0.0.0.0/0           udp dpt:5060

[Anthony Holloway]

Yes, if the TCP handshake cannot be established, then CUCM will not send any TCP SIP messages. You should do a packet capture on that one node, from the CLI with: utils network capture, and then see if there is a TCP connection attempting to be established. You may want to reset the SIP trunk at the beginning of your capture to force a connection attempt, else you'll just need to wait for the SIP OPTIONS delay.

[catkins101 · Accepted solution]

Thanks for your assistance with this. I ended up restarting the Call Manager service on that node and now the SIP trunk is showing up and stable for nearly 24 hours.