Call Quality Issues on New UC560 & SPA588

[ciscojoesauer]

Hello Everyone,

This is my first UC560 install, and unfortunately it has been a trying experience.  The UC and companion ESW500 series switches and SPA508 phones (16 of them) are all new and all updated to latest firmware.  The version on the UC is 15.1(2)T2.  In addition to the built in FXO ports I have an additional 4-port FXO VIC installed, and all 8 FXO ports are populated and are the only connectivity to the PSTN (no SIP trunks in use).

The biggest issue is call quality, which ranges from not great to horrible.  I believe that there may be more than one issue.

First, there can be echo on the VoIP extension side (the PSTN side party does not seem to hear).  I suspect that the line impedance is mismatched, and I have been working with Cisco support to rectify the problem,  Unfortunately, when we run the test voice port x/y/z -sweep verbose command, we get a ERL very low. set_impedance to 600r failed !!!error.  I was working this last Saturday when the client office is closed, and unfortunately, CSBS was undergoing a system upgrade as well, so the support specialist did not have access to resources that allowed us to troubleshoot this test.

Second, the call quality deteriorates as the call volume increases.  When there is one or two active calls, the quality is on the better side.  However, as the volume of active calls increases, conversations are clipped and distorted (lots of drop-out) and results in nearly impossible communications.  Also, we notice that phone commands / soft buttons begin to lag or not function during high call volume.  I am wondering if CoS or QoS might be an issue here as network traffic increases, and I need to learn how to investigate if those parameters are properly set-up and functioning.

Today was the "final day" for a promo the client was advertising.  The call volume was high, and the frustration higher.  My back is up against a wall here, and I will be dispatching tomorrow (Saturday) morning to troubleshoot during closed hours.

Any advice would be greatly appreciated!

[David Trad]

Hi Joseph,

I feel your frustration right now and the only encouragement I can offer you right now is to hang in there, know that you will be looked after by a great team

So to your problem, lets work through this in conjunction with what the support team is doing

1. Please take note of all the serial numbers of each phone and provide them to support and check and see if any of them fall within a batch of phones that were released with bad hand pieces, I am going through this right now with one client and the response from the Cisco support team and the RMA team is outstanding and the client is so happy with this level of support
2. Secondly Are you running the latest SW pack?
3. Have you upgraded the phones to the latest loads? (This may help)

Further questions if you can answer them would be great as well

1. Is the echo persistent or does it slowly fade away as the conversation keeps going?
2. Is it an actual echo effect or is it just hearing your own voice when you talk and the (B) end does not hear it? Echo is often heard by the (B) end as well and hearing your own voice is often seen as being echo when it is more closer to feedback
3. Can you post your voice-port configuration? It would be good to see what commands CCA is put in there, I am not certain but I am concerned with some of the configurations I have noticed of late, they seem to be missing some configurations to that of which I am used too

The other thing I have learnt to do is use the Live Record feature and capture the problem and provide the .WAV file to support, being able to hear it can help with the diagnosis, unless they are calling in on the system and are able to hear it first hand

The other thing I wouldn't mind if you can try is install CIPC (NOT version 8, get 7.06 version) and see if the problem happens with the CIPC client, this may help with factoring out the SPA phones.

Also if you have handy a 7900 series phone, can you please plug one of them in and give that a try... Again I have some concerns about the SPA phones as all the installs I have done so far the client feedback is that the voice is too soft and the clients complain from time to time of not being able to hear them, so they have to talk louder down the phone, this issue I am working on daily at the moment to try and get to the bottom of it (NOTE: DO not have this problem with the 7900 series phones)

Just keep soldiering on, you will resolve it I promise

Cheers,

David.

[bill.roland]

I agree with David.  If you have any 7900 series phone available that you could try, I would be interested in seeing if that eases up the symptoms you customer is having.  I recently got rid of all my SPA508s for similar reasons, in my case the 7900s (and even the SPA525G's) produced far better audio quality.

[ciscojoesauer]

Thanks again for all the help.

I spent a couple hours on the phone with CSBS on Saturday, and there are a number of issues I need to work on starting today.

For some reason, there is an problem with the support contract, so my Cisco ID will not allow me to download the latest software pack.  They could not help me with that on a weekend.  Also, I did download the latest phone load 7.4.8a - but I get a "this is not a valid file" error when I try and upload to the phone loads in CCA.  The Cisco tech could not fix this either.  I could not get any feedback on the  test voice port x/y/z -sweep verbosecommand, so I went ahead a did a great deal of testing on my own, by manually selecting all of the different impedance settings and making test calls (with echo cancellation turned off).  I was disappointed to learn that there was little to no perceptible difference between all the different choices - so I defaulted back to 600r.

I did make a big discovery though -

In testing the FXO ports,I learned that port 0/2/0 on the VIC is bad.  I assumed the POTS line connected to that port had a serious impedance mismatch, but I swapped with several other POTS lines, and the problem did not move.  Also, the assumed offensive line works fine on one of the built-on FXO.  In summary, the built-on FXO ports work better than the VIC, and one port on the VIC is unusable.  I will hit CSBS with that today as well.

The tech also acknowledged that there was a bad run on the SPA508, but he could not elaborate on the serial number range (again, a weekend thing).

So I will camp out at the client today and start hammering away at the list of issues.

Joe

[paolo bevilacqua]

Friendly advice - return all phones, only use 79xx for safe results.

Asl, it would be great if you could get ISDN BRI or PRI instead of FXO.

[ciscojoesauer]

I am getting the feeling that the SPA series is not up to the task -

Anyone else have any input on this?

I am also looking into digital PSTN.  At the client location in Connecticut, there are really only two (non Internet-based) providers:  Comcast and AT&T.  I need 8 trunks, but AT&T does not provide a fractional PRI.  I am looking into their IP Flex product line to see if it might be competitive with the clients POTS recurring.  Comcast has started to offer fractional PRI (and maybe even SIP trunking) but their reliability is questionable.

[David Trad]

Hi Joseph,

I am getting the feeling that the SPA series is not up to the task -
Anyone else have any input on this?

As stated previously, I am currently going through an RMA process on 12 handsets where the hand pieces are being replaced, the client has advised that on the two we have already changed the voice quality improved slightly, but still not to a level they would be happy with. Installing a 7962 phone maybe caused me more problems than that of what I wanted as they are now looking at potentially replacing to them, however the cost differences are too great for us to wear that cost and replace them (Direct swap), so I have to hope that Cisco will resolve this issue.

I am also looking into digital PSTN.  At the client location in 
Connecticut, there are really only two (non Internet-based) providers:  
Comcast and AT&T.  I need 8 trunks, but AT&T does not provide a 
fractional PRI.  I am looking into their IP Flex product line to see if 
it might be competitive with the clients POTS recurring.  Comcast has 
started to offer fractional PRI (and maybe even SIP trunking) but their 
reliability is questionable.

Hmmm not a very friendly position to be in sadly

If you can go SIP trunk you might be happier with that, I am surprised that none of the carriers do fractional PRI, it actually makes no sense to not provide it... You cannot look at ISDN services as your system is an FXO system and not an out of factory BRI system, so that excludes that possibility.

It would appear your only possibly chance to progress beyond analogue lines is to go SIP trunk, and in doing so I would recommend that you retain at at least a couple of analogue lines as a backup if the services do not appear to be stable, this way you at least have some form of redundant connectivity.

Cheers,

David.

[Dan Smith]

for what its worth I have installed several hundred of the SPA series phones and have yet to RMA any related to voice quality issues. I have never had an issue with echo on a PRI circuit, but have found quite frequently that with pots line you have to adjust the input gain/output attenuation for inbound/outbound volume control on the voice ports. This can be adjusted on the fly for real time testing until you find a balance with the volume on the ports. this took me a while to figure out and I went through the same process of replacing the SPA phones with 7900 series phones to no avail. I would recommend changing the input gain and output attenuation on the voice ports first to make sure that its not a volume issue with the pots phone lines.

just my 2 cents.

[ciscojoesauer]

dsmith@fincomm.net
for what its worth I have installed several hundred of the SPA series phones and have yet to RMA any related to voice quality issues. I have never had an issue with echo on a PRI circuit, but have found quite frequently that with pots line you have to adjust the input gain/output attenuation for inbound/outbound volume control on the voice ports. This can be adjusted on the fly for real time testing until you find a balance with the volume on the ports. this took me a while to figure out and I went through the same process of replacing the SPA phones with 7900 series phones to no avail. I would recommend changing the input gain and output attenuation on the voice ports first to make sure that its not a volume issue with the pots phone lines.
just my 2 cents.

At the moment, the situation has stabilized somewhat.  Towards the end of my work this past Saturday, I did exactly what Dan described above.  By adjusting the outbound and inbound attenuation and gain, I was able to get some real improvement with only a slight loss of volume.  Overall, the result was well received by the staff.  I do have one user who is sensitive to the reduced side tone, but we are working on that.

Joe

[ciscojoesauer]

wicomoz@hotmail.com
Hi Joseph,
I am getting the feeling that the SPA series is not up to the task -
Anyone else have any input on this?
As stated previously, I am currently going through an RMA process on 12 handsets where the hand pieces are being replaced, the client has advised that on the two we have already changed the voice quality improved slightly, but still not to a level they would be happy with. Installing a 7962 phone maybe caused me more problems than that of what I wanted as they are now looking at potentially replacing to them, however the cost differences are too great for us to wear that cost and replace them (Direct swap), so I have to hope that Cisco will resolve this issue.
I am also looking into digital PSTN.  At the client location in Connecticut, there are really only two (non Internet-based) providers:  Comcast and AT&T.  I need 8 trunks, but AT&T does not provide a fractional PRI.  I am looking into their IP Flex product line to see if it might be competitive with the clients POTS recurring.  Comcast has started to offer fractional PRI (and maybe even SIP trunking) but their reliability is questionable.
Hmmm not a very friendly position to be in sadly 
If you can go SIP trunk you might be happier with that, I am surprised that none of the carriers do fractional PRI, it actually makes no sense to not provide it... You cannot look at ISDN services as your system is an FXO system and not an out of factory BRI system, so that excludes that possibility.
It would appear your only possibly chance to progress beyond analogue lines is to go SIP trunk, and in doing so I would recommend that you retain at at least a couple of analogue lines as a backup if the services do not appear to be stable, this way you at least have some form of redundant connectivity.
Cheers,

David.

Yes, you would be surprised just how few carrier choices we have in this part of CT (unless you want to go over the public Internet, which is not a viable solution in my opinion).  AT&T still thinks like a monopoly, so why offer a fractional PRI when you can bang everyone for a full?  And BRI never took off in this area - probably because every voice call has a per-minute fee (again, AT&T still stuck in monopoly mode).  In fact, I am trying to turn up a BRI for a NASCAR / MRN radio broadcast at the same client (they are a race track) and the C.O. doesn't have capability!  So AT&T is working on a "total reach" circuit from a C.O. about 12 miles away.

I am still gathering pricing on anything I can find, but I am hopeful that AT&T IP Flex might be affordable - and in this area, they would deliver over a T1 with an SLA.

I have other issues to work out - like I have not figured out how to manage the phone templates from CCA (perhaps this is not possible)?  The users are looking to rearrange soft buttons, and this will need to be addressed soon.  Also, when I browse directly to a phone, there does not appear to be an administrative login?  Perhaps another issues with the default template configuration?

Most of this will be on hold until mid next week - the track is hosting the Rolex Grand-Am race this weekend, so the focus is on that.  I also provide voice and data services for the event, so I will not be able to do much until next week either.

Thanks again for all the input!

[David Trad]

Hi Joseph,

Thanks for the update

I have other issues to work out - like I have not figured out how to 
manage the phone templates from CCA (perhaps this is not possible)?

Not yet possible (Well not that I am aware of, still haven't found it yet), and I do not know how CCA would react to a CLI modification to this, maybe the Cisco team can advise correctly on this

Also, when I browse directly to a phone, there does not appear to be an 
administrative login?  Perhaps another issues with the default template 
configuration?

I believe that you are unable to make any configuration directly on the phone, browsing to the phone itself will give you configuration information and some stats, but that is as far as it goes. Although if I am not mistaken with a SIP load installed you can configure the phones directly... I would hope someone from Cisco would clarify this.

Most of this will be on hold until mid next week - the track is hosting 
the Rolex Grand-Am race this weekend, so the focus is on that.  I also 
provide voice and data services for the event, so I will not be able to 
do much until next week either.

Good Luck Mate

Cheers,

David.

[ciscojoesauer]

I believe that you are unable to make any configuration directly on the phone, browsing to the phone itself will give you configuration information and some stats, but that is as far as it goes. Although if I am not mistaken with a SIP load installed you can configure the phones directly... I would hope someone from Cisco would clarify this.

There was documentation on browsing and logging into the phones administrative functionality - but it was probably referring to the SIP load.  This is where I was baffled - it kept mentioning the "login" button on the home page, but I couldn't find it!  I will get some feedback from Cisco on this (probably next week).

Thanks -
Joe

[paolo bevilacqua]

Thanks for sharing the results of your port setting tuning, and the insight about service availability in your area (that matches a lot the situation in many part of the US). Actually I'm surprised that BRI was offered. What is a "total reach" circuit? Are they willing to make a special extension to deliver it to you? Will you be using it for two B channels wideband audio to broadcast ?

You can also ask for a partial T1 CAS service, with ANI*DNIS number delivery. That gives incoming CLID (and DID of course), and there is a method to get it working on the Cisco despite official  lack of support for it.

However at the end of they day, there is nothing wrong with a well working analog circuit. Some telco also offer "trunk grade" ground start analog for business users.