SPA122 ATA's ignoring incomming SIP requests.

vianet.ca · ‎01-15-2015

Hi all. We currently have approximately 2000 122's in service. That is not an exaggerated number by any means.

Over the past few years, we have noticed a rather problematic bug. These, ATA's, At random tend to lockup in a very specific manner. While browsing to the ATA's web interface, is fine, it Can make calls, it remains registered, and the user can still surf through the ATA ( although not in all cases was it being used as a router), the ata IGNORES all incoming sip packets. The SIP packet rx count doesn't budge, unless there is a re-registration, or the user calls out.

We believe this is ONLY happening with ATA's on PPPoE, however due to the randomness of when it occurs, we can not say for sure.

Even making a call out, does not bring it back. A power cycle is needed.

When in this erred state, It is as though the ATA is not receiving the Invite when a call is placed to it. our traces tell us the calls time out while waiting for the endpoint to respond to INVITE request.

With syslog enabled, the debug show nothing on an incoming call.

We have tried multiple firmwares, and still the issue persists.

Any Suggestion on how we can troubleshoot / fix this?

mdobiac · ‎01-15-2015

Hello vianet.ca,

You stated that you enabled a syslog on the ATA but you did not received the invite, have you attempted a packet capture in front of the ATA to see if the request is leaving the modem? If it is not then the modem is not forwarding the traffic and there is nothing for the ATA to respond to.

You may need to look into the modems and see if there if it is specific modem that this is reoccurring on.

Hope this hopes,

Michael D.

vianet.ca · ‎01-16-2015

Thanks for the Reply Michael. The modems are in bridge mode. Also, we have had it on different makes / models of dsl modem, as well as customer on fixed wireless service.

Further to that, The ATA's are still registered, and can still make outbound calls. Only incomming calls are affected.

Chris

mdobiac · ‎01-16-2015

Hello vianet.ca,

Thank you for the follow up. Well it is a concern if the ATA does not see the invite we still need to find out if the request is leaving the modem to verify for sure that the ATA is ignoring the invite.

Other things that can possibly be done to eliminate a few things: first upgrade to 1.3.5 (if applicable), perform a factory reset and reconfiguration of the ATA to clear out any potential errors. Also reconfigure the syslog in case the issue does resurface.

Regards,

Michael D.

Dan Miley · ‎03-11-2015

Frequently if outbound works, and inbound does not after a period of time, there is a session aware firewall in front of the ATA or phone. After a certain time the SIP/UDP session in the firewall times out, then inbound calls are blocked.

you said inbound calls work after an outbound call or reregister, right?

a couple options if this is the issue,

- port forward sip ports to the ATA through the firewall

-enable nat mapping and nat keepalives on the ata line ( you may need to do both extensions).

voice--> line x --> nat parameters

if you have an entitled serial number for a device either Mike or I can open a case for you. Spa122 have a 1 year phone support and hardware warranty. Spa122-rc only have a 90 day hardware warranty. There are not any small biz devices that have 30 day warranty that I know of.

Call in 866-606-1866 with your Cisco ID and serial number or PM it to Mike or me and we'll open a case for you. Sorry you had difficulty opening a case.

Dan

carllitt · ‎03-12-2015

Hi Daniel,

As vianet.ca and I both pointed out, the SPA122 itself is the PPPoE connection - there is no NAT or firewall, it is directly exposed to the Internet. Furthermore, in my packet captures I am demonstrating that the SIP INVITE packet is observed on the Ethernet link directly into the SPA122's Internet port. There is no chance that the packets are being dropped upstream.

Dan Miley · ‎03-16-2015

reference: case 634055483

Carl, if you have a device that is still under warranty, you can call in. ask for the small business support center. The direct # is 866-606-1866 in the US, http://www.cisco.com/c/en/us/support/web/tsd-cisco-small-business-support-center-contacts.html

There is a chat option, which may be easier.

The enterprise TAC does require a contract for most devices.

With this ATA there is one year phone support, to open a case, you do need the serial of a device that is under warranty, but we can definitely open a case for you. If the front line will not allow you to open a case, ask to speak to a duty manager, or be transferred to the small business TAC. You can also reference this post when you open the case. that will help the engineer come up to speed more quickly.

dlm...

vianet.ca · ‎03-30-2015

Any updates on this?

vianet.ca · ‎03-12-2015

Dan I have PMed you a MAC and Serial number of a 122

carllitt · ‎03-05-2015

Hi All,

We have several hundred SPA122's deployed, and we are seeing this issue intermittently as well - SIP INVITEs are sent to the SPA122 but the SPA122 acts as though it doesn't receive them. For example, with the 'SIP_Debug_Option_1_' set to 'full' (syslog all SIP packets), there is nothing logged by the SPA122 when the SIP INVITE packet is sent to it.

I have observed this issue on number of SPA122's which are connected to a DSL modem in bridge mode, with the SPA122 doing PPPoE. Most (possibly all) of these SPA122's also have some sort of customer equipment plugged into the LAN port for which they are doing NAT.

I have reproduced this issue a couple times in our lab. I have a Comtrend 5071T modem in bridge mode, with the LAN port connected to an Ethernet hub. Also connected to the hub are the SPA122 and a Linux system for packet capturing. (Note this is a dumb hub, not a switch, for the purposes of mirroring packets between the modem and the SPA to a capture device)

Here's a rough diagram:

DSL Modem	------	Hub	-----	SPA122
		\|
		Capture Box

The Linux capture box is using 'tcpdump' to write packets to disk, which I then analyze with Wireshark. The packets captured are the PPPoE session towards the SPA122.

First, the SPA was factory reset using the IVR, configured for PPPoE, then bootstrapped to our base configuration using our Profile_Rule. The SPA122 is powered up and the PPPoE session is observed to be connected. I also observe that the SIP line is successfully registered and I dial an outbound call to verify that it works. Inbound calls within the first few minutes always seem to complete normally. But I have observed inbound calls after 10-15 minutes may fail in this manner.

With these captures I have proven that the SIP INVITE is forwarded through the DSL modem to the SPA within the PPPoE session, but the SPA does not react at all to the SIP INVITE, eg. the SPA doesn't respond to the SIP INVITE and no SIP INVITE is logged with the debug option. This proves that the issue is with the SPA122 and not with the modem or any packet loss upstream of the SPA122.

When this happens, I can dial an outbound call from the ATA immediately afterward and the call completes normally. This coincides with our customer reports as well.

The issue seems to happen somewhat intermittently. There will be times when the SPA seems to function normally, but then other times it ignores calls this way. We have not identified any particular trigger that causes this.

This issue has been observed on 11 different SPA122's at different customer locations in the past week, so this is not a hardware issue (ie. there is no point in replacing the SPA with another SPA). I have also tried several firmware versions, including the latest, and the issue is observed on all of them.

The issue has been observed on all of the following SPA firmware versions:

1.3.2(014)

1.3.2(014n)

1.3.3(015)

1.3.5(004)

1.3.5(004p) (the latest version)

Stepping back, so far the issue is only observed when the SPA122 is configured for PPPoE. It may also be relevant that there is customer equipment passing through the SPA122 with NAT.

I have all packet captures and logs saved for analysis, and I can most likely reproduce this again for further troubleshooting. We are highly motivated to work with Cisco to find a solution.

vianet.ca · ‎03-05-2015

Thanks for your efforts carllit. We have yet to be able to capture this, so this should go a long way. When re-creating this, was there any thing special you had to do in the lab to get it to fail? Did it only fail in this configuration? By that, I mean, behind the Comtrend modem? We have attempted to capture the data in between the ATA's WAN and a switch port configured on a pppoe access vlan, but have been unsuccessful in re-creating it.

I don't think even looking at the PCAP files would de us anygood anyway though, being as you have managed to get a capture and can clearly see the INVITE making it to the SPA122. I think that being said, It would be up to Cisco now.

In response to the previous comment from Cisco, no we have not menaged to get a capture, however we have seen it on the most recent firmware versions, as well as on virgin ATA's that were freshly provisioned.

Chris

mdobiac · ‎03-05-2015

vianet.ca and carllitt,

Thank you for the feedback, the best I can offer now is to have you both open a case with Cisco Support and we can work towards a solution in regards to this issue. With the troubleshooting done this will help in finding a resolution.

If you can have all you packet captures available and the topology mostly involved ready go that will help move things along.

Thank you,

carllitt · ‎03-06-2015

Hi Michael,

What's the easiest way for me to open a case with Cisco Suppport? I can't open a CSC from this thread because I'm not the original author. Any anytime I call into Cisco support they refuse to open a ticket unless I provide the serial number of an SPA122 that they can identify is covered under the 1-year hardware warranty (which is of course senseless when this is a firmware issue and not a hardware replacement issue).

mdobiac · ‎03-06-2015

carllitt and vianet.ca,

Thank you for the feedback and I understand the frustration. Yes as you both have described you will need a SPA122 that is under warranty or a contract and a Cisco User ID. That way we will be open a case. After that we will need to be able to trouble shoot the issue or if able replicated in your labs for us to at least take a look at and try to resolve the issue.

Also you can email me at mdobiac@cisco.com and I will be able to provide more information and I can also work on getting a case open and get the process started to try to find a resolution for this issue.

Regards,

Michael D.

vianet.ca · ‎03-06-2015

Hi Michael. As mentioned by Carllit, we as well have tried to open a TAC with Cisco support, and being as there is no support contract in place, we are refused. We are also asked for the Serial. We could potentially use ANY spa122 MAC, however we get refused due to the issue not actually being present at the time on that particular unit.

I concur with Carllit. This is 100% an issue, be it firmware or hardware, on the ATA. Any suggestions as to how we can open a TAC when we keep getting rejected?

As i am sure you can appreciate, from a service provider perspective, to have multiple thousands of customers relying on this device that just 'stops working' is not really acceptable.

Chris