Problems with blocking cic in pgw2200

[rollenybbe]

Hi,

We have a delicat problem every day, around peakhours in our pgw2200 the cic:s is being blocked, we have configured 2000 cic:s to each AXE:s (we have 2) and have license for 2000 simultaneous calls. Last weeks our pgw2200 is behaving very strange and block cic BLK=GATEWAY, and we can't do anything it is just keep blocking more and more cic:s and will stay like this until it shots /opt/CiscoMGC/bin/startAudit.sh around 04:10 in the morning and all the cic:s are back to normal.

My question is why PGW2200 is behaving like this suddenly? It is very annoying, it is blocking cic due that we still have a lot off free ones.

Does anyone experienced this before, please help us, we don't know what to do.

/Roland Nyberg

[Daniele Giordano]

BLK=GATEWAY gives you an indication that the Cisco PGW 2200 received RSIP (RM:forced) or RSIP (RM:graceful) from the gateway. This state is cleared when the Cisco PGW 2200 receives RSIP (RM:restart) from the gateway.

GATEWAY indicates that the CIC is blocked locally due to a gateway event.

You can use debug mgcp packet to check mgcp messages.

What are your gateways? What IOS is installed?

Regards.

[rollenybbe]

Ok, thanx for the answer. We have in our PGW 2200:

   MGC-01 - Media Gateway Controller 2011-12-05 09:50:34.828 MET

M  RTRV

   "Type:MGC"

   "Hardware platform:sun4u sparc SUNW,Netra-240"

   "Vendor:"Cisco Systems, Inc.""

   "Location:MGC-01 - Media Gateway Controller"

   "Version:"9.6(1)""

and in our AS5850:

SESTO0803-GV1#show version
Cisco IOS Software, 5850tb Software (C5850TB-P9-M), Version 12.4(15)T11, RELEASE          SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Thu 29-Oct-09 12:11 by prod_rel_team

ROM: System Bootstrap, Version 12.3(1r)T1, RELEASE SOFTWARE (fc1)
ROM: Cisco IOS Software, 5850tb Software (C5850TB-BOOT-M), Version 12.4(15)T1, R         ELEASE SOFTWARE (fc2)

SESTO0803-GV1 uptime is 1 week, 3 days, 6 hours, 33 minutes
System returned to ROM by reload at 03:23:06 CET Fri Nov 25 2011
System restarted at 03:24:47 CET Fri Nov 25 2011
System image file is "disk0:c5850tb-p9-mz.124-15.T11.bin"
Last reload reason: Reason unspecified

Cisco c5850 (BCM1250) processor (revision 0.255) with 917504K/131072K bytes of m         emory.
Processor board ID JAB104005CZ
SB-1 CPU at 650MHz, Implementation 1025, Rev 0.2, 512KB L2 Cache
Last reset from Mbus reset
2 FastEthernet interfaces
2 Gigabit Ethernet interfaces
1944 terminal lines
62 Channelized E1/PRI ports
1 Channelized STM1 port
507K bytes of NVRAM.

62592K bytes of Compact Flash card at slot 0 (Sector size 512).
16384K bytes of Flash internal SIMM (Sector size 512k).
Configuration register is 0x2102

[Daniele Giordano]

I've same config: PGW 9.6(1) + 3 x AS5400 with IOS 124-15.T15.

Occasionally, the CICs block happens in the same way but are 3 cics blocked over 500.

Can you try to upgrade your IOS software?

Regards.

[rollenybbe]

We have the discussion with our supplier to upgrade.

Unfortunately I don't really understand what you mean with:

"Occasionally, the CICs block happens in the same way but are 3 cics blocked over 500"

and the thing is that the RSC:s cpu is up to 99% when this problem occur

[abrajan]

Hey Roland,

Can you paste this command when thsi issue takes place rtrv-tc and rtrv-cic. Like Daniele says its due to the block message which is not getting cleared from the GW. RSIP (restart) would help to clear the cics.

This is an example of this command with the rtrv-tc output:

   "ss7p-veri:CIC=377,PST=IS,CALL=IDLE,GW_STAT=INTERFACE_DISABLED,BLK=GATEWAY"

   "ss7p-veri:CIC=378,PST=IS,CALL=IDLE,GW_STAT=INTERFACE_DISABLED,BLK=GATEWAY"

   "ss7p-veri:CIC=379,PST=IS,CALL=IDLE,GW_STAT=INTERFACE_DISABLED,BLK=GATEWAY"

   "ss7p-veri:CIC=380,PST=IS,CALL=IDLE,GW_STAT=INTERFACE_DISABLED,BLK=GATEWAY"

   "ss7p-veri:CIC=381,PST=IS,CALL=IDLE,GW_STAT=INTERFACE_DISABLED,BLK=GATEWAY"

Try to take sniffer traces between the gw and PGW during high peak time. Run a syslog server so that you dont face any issues.

Also run debug mgcp packets at the GW so we can check the RSIP messages.

Cheers,

Aby

[rollenybbe]

Thanx, I will do this when our load and the problem occurs again, we have now routed some of our customers through other equipment and the ciscos are stable.

Thanx again

/Roland

[j116464tdc]

Hi Roland

You can run the audit script manually from mml

sta-aud

wait 2 minutes

stp-aud

Also You can alter the cronjob so the audit script runs more often

10 4 * * *   /opt/CiscoMGC/bin/startAudit.sh    #$CSCOga001$

for the rootcause check for MGCP 400 responsecodes

/Henning

[julio.vargas23]

Hi Roland,

I have the same problem,Did you solved this problem in your equipments?  do you have any additional procedure to solve this issue?

Thanks in advanced,

/Julio