Calls to FXS port - Fax line locks Callmanager

[jasonhilt69]

Have an issue with our FXS ports that are used for fax machines.  If a call is active on the fax line and another call comes in to the same number, it locks up the phone system - phones can't do anything, buttons dont work for a minute or so and no calls can be made.  There is no busy signal when the second call comes in to the fax line.  Just dead air.  I even tried it with the phone line disconnected from the FXS port and same issue.  I even tried it on a port not connected to a fax machine with same result.

A fax machine on an ATA device works fine. I get a busy signal.

Another school with it's own ISR4331 gateway and NIM-4FXS card doesn't have this issue.  It runs off the same CallManager as our main building.

We are on CallManager 10.5

Gateway is ISR4351/K9 runninng Version 15.4(3)S2 with two NIM-4FXS cards.

Where should I look to fix this as it's a major problem with school starting?

Below is some config I think is important.  Anything else just let me know what you need.

voice service voip
 ip address trusted list
  ipv4 10.1.2.150
  ipv4 10.1.2.151
 allow-connections h323 to h323
 allow-connections h323 to sip
 allow-connections sip to h323
 allow-connections sip to sip
 fax protocol t38 version 0 ls-redundancy 0 hs-redundancy 0 fallback none
 modem passthrough nse codec g711ulaw
!

mgcp
mgcp call-agent 10.1.2.151 2427 service-type mgcp version 0.1
mgcp dtmf-relay voip codec all mode out-of-band
mgcp rtp unreachable timeout 1000 action notify
mgcp modem passthrough voip mode nse
mgcp package-capability rtp-package
mgcp package-capability sst-package
mgcp package-capability pre-package
no mgcp package-capability res-package
no mgcp package-capability fxr-package
no mgcp timer receive-rtcp
mgcp sdp simple
mgcp validate domain-name
mgcp validate call-agent source-ipaddr
mgcp fax t38 inhibit
mgcp rtp payload-type g726r16 static
mgcp behavior rsip-range tgcp-only
mgcp behavior comedia-role none
mgcp behavior comedia-check-media-src disable
mgcp behavior comedia-sdp-force disable
!
mgcp profile default

ccm-manager music-on-hold
!
ccm-manager fallback-mgcp
ccm-manager redundant-host 10.1.2.150
ccm-manager mgcp
no ccm-manager fax protocol cisco
ccm-manager config server 10.1.2.150 10.1.2.151
ccm-manager config

voice-port 0/1/2
 station-id name Athl-Fax
 station-id number 7243
 caller-id enable

dial-peer voice 7243 pots
 description Ath-Fax
 destination-pattern 7243
 port 0/1/2

[Mohammed al Baqari]

Can you share traces to see what is happening when the second call comes in?

Also, from RTMT do see anything such as CPU spike, etc.

[jasonhilt69]

Cpu and network both skyrocket when the system hangs.  VM monitor shows 100% usage up to 6000 MHz.  Normal usage is below 500 MHz.  Network packets jump to over 80,000.  Normal is below 5000.

Attached is a trace..hope it's the right one.  I did notice a whole bunch of these:  Phone number being called is 7243 internal.

Time in trace log is 15:50.14.357  or line 10011 in Notepad++ for the call I made. I left the rest of the log in case you wanted to compare a normal call

2015/07/28 15:50:22.571|CC|SETUP|46836399|46836400|3178627243|7243|7243
2015/07/28 15:50:22.578|CC|OFFERED|46836399|46836400|3178627243|7243|7243|SEP001C581C2C54|10.1.100.1
2015/07/28 15:50:22.587|CC|SETUP|46836401|46836402|3178627243|7243|7243
2015/07/28 15:50:22.591|CC|OFFERED|46836401|46836402|3178627243|7243|7243|10.1.100.1|10.1.100.1
2015/07/28 15:50:22.597|CC|SETUP|46836403|46836404|3178627243|7243|7243
2015/07/28 15:50:22.600|CC|OFFERED|46836403|46836404|3178627243|7243|7243|10.1.100.1|10.1.100.1
2015/07/28 15:50:22.606|CC|SETUP|46836405|46836406|3178627243|7243|7243
2015/07/28 15:50:22.610|CC|OFFERED|46836405|46836406|3178627243|7243|7243|10.1.100.1|10.1.100.1
2015/07/28 15:50:22.618|CC|SETUP|46836407|46836408|3178627243|7243|7243
2015/07/28 15:50:22.623|CC|OFFERED|46836407|46836408|3178627243|7243|7243|10.1.100.1|10.1.100.1
2015/07/28 15:50:22.632|CC|SETUP|46836409|46836410|3178627243|7243|7243
2015/07/28 15:50:22.636|CC|OFFERED|46836409|46836410|3178627243|7243|7243|10.1.100.1|10.1.100.1
2015/07/28 15:50:22.642|CC|SETUP|46836411|46836412|3178627243|7243|7243
2015/07/28 15:50:22.646|CC|OFFERED|46836411|46836412|3178627243|7243|7243|10.1.100.1|10.1.100.1
2015/07/28 15:50:22.653|CC|SETUP|46836413|46836414|3178627243|7243|7243

[jasonhilt69]

It noticed I didn't upload all the logs, but when I try to upload the one big file, website just sits there and spins with no network activity.

Let me know if you need anything else.

[Mohammed al Baqari]

Hi,

I think we are having a call routing loop here which is causing the problem. Can you please share the router configuration? Also, please share a DNA report from CUCM for me to understand the call routing

[jasonhilt69]

Router config and DNA attached.

[Mohammed al Baqari]

Hi,

Thanks for sharing the files. I understand the topology. You are using MGCP for calls coming from PRI to CUCM. Then CUCM connects using H323 back to the same gateways and from their to the fax machine. Can you please share the following outputs from the gateway during the problem:

1. debug mgcp packets

2. debug cch323 h225

3. debug voice dailpeer all

PS: Due to the nature of your problem the output of debugs might be massive and your router might go to 100% CPU.

[jasonhilt69]

file attached.

I called the fax line with my cell and once it connected, called it from my desk phone dialing out then back in.

I also tried calling it direct, 7243.  Both times I never received a busy signal even though I know the line was in use.

Odd thing is it didn't lock up the phones like it use to.

[Mohammed al Baqari]

Please do the following:

1. On your fax dialpeers please put the command  'fax rate disable'

2. In you MGCP configuration please remove the port 2427. Please use the default port.

I can see that CUCM keeps signaling H323 to the gateway but can't understand exactly why. Do you have an overlapping route patterns in which the call manager is hunting between? Also, can you share the service parameters configuration:

Stop Routing on Out of Bandwidth Flag
Stop Routing on Unallocated Number Flag
Stop Routing on User Busy Flag

[jasonhilt69]

On fax dial-peers, fax rate disable command is not available for pots.  Only see it on a dial-peer voice voip config.

removed port on mgcp config

Only overlapping route patterns I see for 7243 are on the gateway itself.  Dial-peer voice 111 voip and 1110 voip.

Stop Routing on Out of Bandwidth Flag - False

Stop Routing on Unallocated Number Flag - True

Stop Routing on User Busy Flag - True

All three at suggested values.

[Mohammed al Baqari]

Sorry for asking too many questions but can u plz share debug isdn q931 during the problem in addition to the other debugs. Also please indicate calling and called numbers 

[jasonhilt69]

Debug isdn log is attached.

My call from 4592222 to 7243 starts at line 129 in Notepad++.  It connects and I hear the fax tones. 

I then call from my desk phone, 8626646 to 7243. Starts at line 633.  This call I get dead air.

Same is true if I call from my desk phone first then cell.  Cell gets dead air.  No busy signal.

Surprisingly it didn't lock the system up.

[Mohammed al Baqari · Accepted solution]

Hi,

Looking at the full debug, the problem is the FXS port isn't sending proper busy tone to the gateway. Hence the CUCM isn't aware that the line is busy keeps sending H323 Setup messages to the gateway which is causing a call routing loop and hogging the CPU of your cluster.

Now, the bad news that H323 doesn't support Setup retry limit configuration, i.e. you can't limit the number of Setup messages sent before terminating the call.

The good news which I suggest is to register your FXS ports directly with the CUCM using MGCP or SCCP protocol. This will make CUCM taking decisions whether to route the call or not rather than waiting for H323 gateway to send a busy signal again.

You can do this config by navigating to Device > Gateway > Add New > Select the Gateway type > Select the Gateway Protocol (MGCP/SCCP) > In slot 0 select NM-4VWIC MBD. After you save the gateway, you can select the FXS module from the sub slot.

On the gateway you need to configure SCCP or MGCP (already present) for signaling.

[jasonhilt69]

I was able to reconfigure the fax lines on the one gateway that uses MGCP.  I now get a busy signal on the second call and no system lockup.

Now the problem is, the other gateways are on h.323 and a second call to those pots lines locks the system.  There are no PRIs going to those.  They only handle 911, fax, and if the main network link goes down.

Any easy fix for those like the MGCP change?

[Mohammed al Baqari]

Great news !! We are thinking in the right direction.

You can still use MGCP for the other gateways to register FXS with CUCM even if you don't have PRI lines. The flow can be as follow:

911 Line --- Gateway --- H323 ---- CUCM ---- MGCP --- FXS (Fax)

*** Please rate useful posts