UC520 Router SPA525G2 and SSL VPN's, Very Dissapointed!!

[synmedenfield]

We have been having an issue with our VOIP system.  We purchased a SPA525G2 phone in the hope that it would connect back remotely to our UC520 and could be used everywhere in the world.

I have over 15 years configuring Cisco gear and have a CCNA and CCNP - so i configured this myself using a combination of the CCA and confirming in the CLI to make sure i understood what was going on.

The set-up seems to have worked - the phone connects and registers okay, however if i try and make a call all hell breaks loose.

First of all incoming traffic to the phone fails after 15 seconds but outgoing traffic carries on (i.e. I can't ping the phone anymore, but the other person at the other end is able to hear me)

After about 1minute 20 seconds the phone says there is a network error and then it says connecting

At the same time the router throws off any other SSL VPN client which is attached to it

About 10 seconds after that all the phones in the office start loosing buttons and have to re-register.

I have logged this call with TAC.  I have been working on it with them for over a week, but no one is able to come up with a solution.

Finally made some headway when we were able to see that the buffers where failing due to lack of memory during all of this.  The Engineer said he would put my call through to someone who knew about hardware/memory issues but has forwarded me to the SPA team who look after the phone????

It's all a bit rediculous.

My personal belief is this router can't take it.

It can't take the workload.....

has anyone got this set-up running?

Before anyone asks i'm running the latest software on the router and on the SPA525G2...

I'm really frustrated about this - Having read around this board other people seem to have expereienced simular but there are no solutions out there.

[David Trad]

Hi Geoffrey,

Your problem is not uncommon, but also not running rampant as well. I am aware of one issue locally and have reviewed this particular end-users system and the configuration.

It is to my understanding that after much testing with SBS and various actions taken to mitigate the problem, it come down to one thing in particular, and that was downgrading the phone load of the 525G to a previous version, as soon as this was done the UC560 stopped doing memory dumps and resetting all the phones and creating instability across the entire system.

Their setup is Local system in Australia with 525G in various other countries connecting back into the system, and their issue sounded very similar to yours.

I point out that the UC520 can handle it and should be able to handle it, the concern here has been in the quality of the IOS and phone loads being released, as more and more features are engaged, it would seem faults that we are not used to seeing are surfacing, most of the problems I am seeing today are due to all the new features coming out and one of them is the inbuilt SSL into the 525G and other higher level feature sets. Don't take me the wrong way, not having a go at this as I LOVE the progress, just a tab bit concerned about the quality of the work, it does not quite seem the same as 4 years ago.... Or maybe that's just me becoming cynical

You could try 7-4-8a which is a revision phone load, or go back to 7-4-7 and test it with that load.

From what I have seen so far (and I have not diagnosed this on many systems) It is almost like the UC is being hit with a broadcast storm and goes into melt down, but I have no way of capturing the debug before the system fails, but turning of the VPN made the system purr like a kitten.

Can you give either of the two suggest phone loads a try and let us know, would like to know if this brings back some stability to your system or if it continues to do the same thing.

Cheers,

David.

[synmedenfield]

thanks for your response!

I've tried the older firmware (7-4-7) and repeated exactly the same issue...

I can't find 7-4-8a - it doesn't appear in the downloads list for this particular phone.

I'm probably going to install a third party open source SSL VPN on my edge network and see if that works.

the SPA525 works fine within the office - its just is the VPN that can't take it.  I bet it has something to do with voice calls being a stream of small packets which the router can't process.  It can deal with moving files around on a PC through the SSL VPN, but then that's different (fewer larger packets as opposed to lots of small packets).

If you think about it - the router has to decrypt every packet that it recieves - if there is a large volume of them it can't deal and falls over..   The Buffer failures relate to lack of memory....

What's the difference in hardware spec between the 520 and 540/560?  I would imagine RAM.  I remember thinking when I originally installed the device that it seemed 'light' for everything it was supposed to do.  I installed an 857 as the edge router and have all the cisco easy VPN stuff going on there to help keep the pressure off.

What upset's me most about this is TAC's response.  I haven't ever needed to use them, most of the time I can find the solution to a problem out here in the communities or a bit of time on google usually does the trick.  But this is a proper problem and I just thought that the TAC would be on top of it.  They aren't - I seem to be getting passed from team to team and don't seem to be getting any closer to a solution.

Very Frustrating!

[David Trad]

Hi Geoffrey,

I can't find 7-4-8a - it doesn't appear in the downloads list for this particular phone

This is my fault, this revision is only meant for the 50X series, not the 525G series... Sorry

the SPA525 works fine within the office - its just is the VPN that can't
 take it.  I bet it has something to do with voice calls being a stream 
of small packets which the router can't process.  It can deal with 
moving files around on a PC through the SSL VPN, but then that's 
different (fewer larger packets as opposed to lots of small packets).

This does not seem right, I am aware of one system that has 5X SPA-525G2 operating remotely and it works quite well, it is only a UC-540W and it handles it quite well.

This is my perspective only and others may not share it, but the UC-500 series to me are first and foremost Switches/Routers come Telephony systems, to me the Telephony side is a feature set add-on to an already proven and competent system, for them not to handle the small amounts of VPN traffic (And it is only small) would be an absolute surprise to me.

If you think about it - the router has to decrypt every packet that it 
recieves - if there is a large volume of them it can't deal and falls 
over..   The Buffer failures relate to lack of memory....

Again I point out not for one device 20 maybe but 1 no way, there is another driving factor to your issues and we need to drill down on this.

What's the
 difference in hardware spec between the 520 and 540/560?  I would 
imagine RAM.  I remember thinking when I originally installed the device
 that it seemed 'light' for everything it was supposed to do.  I 
installed an 857 as the edge router and have all the cisco easy VPN 
stuff going on there to help keep the pressure off.

Main differences would be DSP resources, and DRAM size hence why the 560 can handle more VPN tunnels than say a 520 or 540, the 540 though is a little light and it is the primary reason why it can only handle 32 users, essentially it is a redressed 520 with reduced hardware capacity, but the 560 has a heavy emphasis on the CUE side as well.

a Cisco 857 has less resources than a UC-540/560, the memory mapping on the UC is much higher, and I would even go as far as to say that even with everything turned up to full capacity it will still hold its own against a 857 for VPN routing.

What 
upset's me most about this is TAC's response.  I haven't ever needed to 
use them, most of the time I can find the solution to a problem out here
 in the communities or a bit of time on google usually does the trick.  
But this is a proper problem and I just thought that the TAC would be on
 top of it.  They aren't - I seem to be getting passed from team to team
 and don't seem to be getting any closer to a solution.

SBS is different to TAC, SBS support operate within a set charter, TAC operate on your large systems, anything above UC's such as your 2800 series and above, and they are CLI support only, SBS is GUI with CLI elevation to an engineering level.

The community to me is the single most best resources, the people on here are full of wealth in technical knowldge, and it is absolutely brilliant, the Cisco reps are amazing on here as well

I would really...really...really like to do a remote session with you one time, SSH or Team Viewer with SSH access, and with the 525G operational, the goal would be to capture the debug data and analyses it right at the time of the system becoming unstable and during... Also your configuration capture would be good.

You have plenty of help on this, and you will find other people on here more than willing like me to go to any ends to see your problem resolved, and this can all be done in tandem with the SBS support team, please do not abandon them as if there is a problem found they will handle this with speed as they have with me on many occasions.

I hope to be able to help you

Cheers,

David.

[synmedenfield]

Thanks Again for your responses...

I do agree with you that these boxes are really good for what they do.  I am pushing the boundries possibly - we use as many of the bell's and wistles as we can.

I do have various packet captures and logs/configs etc.  I can upload them somewhere...

Here's something to chew on though:

When the connection first fire's up these values remain stable, however as the connection begins to break down the buffer fails start to increase.

these are two sho buffer's as we see the effect....

Public buffer pools:
Small  buffers, 104 bytes (total 104, permanent 50, peak 129 @ 1d00h):
25 in  free list (20 min, 150 max allowed)
8136282 hits, 11357 misses, 1067  trims, 1121 created
0 failures (0 no memory)
Middle buffers, 600  bytes (total 47, permanent 25, peak 82 @ 5d16h):
39 in free list (10  min, 150 max allowed)
53786782 hits, 288 misses, 714 trims, 736  created
8 failures (0 no memory)
Big buffers, 1536 bytes (total 1036,  permanent 500, peak 3362 @ 5d00h):
1000 in free list (500 min, 1000 max  allowed)
3456286 hits, 76870 misses, 74599 trims, 75135 created
40275 failures (72561 no memory)
VeryBig buffers, 4520 bytes (total 10,  permanent 10):
10 in free list (0 min, 100 max allowed)
5129  hits, 40206 misses, 0 trims, 0 created
40206 failures (66289 no  memory)
Large buffers, 5024 bytes (total 1, permanent 0, peak 1 @  4d21h):
1 in free list (0 min, 10 max allowed)
6 hits, 40200  misses, 42 trims, 43 created
40200 failures (66287 no memory)
Huge  buffers, 18024 bytes (total 1, permanent 0, peak 1 @ 4d21h):
1 in free  list (0 min, 4 max allowed)
13 hits, 59635 misses, 42 trims, 43  created
59635 failures (67661 no memory)

Public buffer pools:
Small buffers,  104 bytes (total 113, permanent 50, peak 129 @ 1d00h):
29 in free list  (20 min, 150 max allowed)
8158905 hits, 11417 misses, 1067 trims, 1130  created
0 failures (0 no memory)
Middle buffers, 600 bytes (total 47,  permanent 25, peak 82 @ 5d16h):
39 in free list (10 min, 150 max  allowed)
53853278 hits, 288 misses, 714 trims, 736 created
8  failures (0 no memory)
Big buffers, 1536 bytes (total 3342, permanent 500,  peak 3362 @ 5d00h):
0 in free list (500 min, 1000 max allowed)
3463874 hits, 82703 misses, 74599 trims, 77441 created
44588 failures  (75750 no memory)
VeryBig buffers, 4520 bytes (total 10, permanent  10):
0 in free list (0 min, 100 max allowed)
5249 hits, 44515  misses, 0 trims, 0 created
44515 failures (69189 no memory)
Large  buffers, 5024 bytes (total 1, permanent 0, peak 1 @ 4d21h):
0 in free  list (0 min, 10 max allowed)
7 hits, 44508 misses, 42 trims, 43  created
44508 failures (69187 no memory)
Huge buffers, 18024 bytes  (total 1, permanent 0, peak 1 @ 4d21h):
0 in free list (0 min, 4 max  allowed)
15 hits, 66065 misses, 42 trims, 43 created
66065  failures (70570 no memory)

you can see the values increase and you can see the no memory statement next to the failure acount..

Moments after this - as the router slows down you see a load of these appear in the logs:

116375: May 23 11:52:25.970:  %TCP-6-NOBUFF: TTY0, no buffer available -Process= "",  ipl= 4
116376: May 23 11:52:25.994: %TCP-6-NOBUFF: TTY0, no buffer available  -Process= "", ipl= 4
116377: May 23 11:52:26.014:  %TCP-6-NOBUFF: TTY0, no buffer available -Process= "",  ipl= 4
116378: May 23 11:52:26.038: %TCP-6-NOBUFF: TTY0, no buffer available  -Process= "", ipl= 4
116379: May 23 11:52:26.054:  %TCP-6-NOBUFF: TTY0, no buffer available -Process= "",  ipl= 4
116380: May 23 11:52:26.070: %TCP-6-NOBUFF: TTY0, no buffer available  -Process= "", ipl= 4
116381: May 23 11:52:26.098:  %TCP-6-NOBUFF: TTY0, no buffer available -Process= "",  ipl= 4
116382: May 23 11:52:26.114: %TCP-6-NOBUFF: TTY0, no buffer available  -Process= "", ipl= 4
116383: May 23 11:52:26.130:  %TCP-6-NOBUFF: TTY0, no buffer available -Process= "",  ipl= 4
116384: May 23 11:52:26.150: %TCP-6-NOBUFF: TTY0, no buffer available  -Process= "", ipl= 4
116385: May 23 11:52:26.174:  %TCP-6-NOBUFF: TTY0, no buffer available -Process= "",  ipl= 4
116386: May 23 11:52:26.198: %TCP-6-NOBUFF: TTY0, no buffer available  -Process= "", ipl= 4
116387: May 23 11:52:26.218:  %TCP-6-NOBUFF: TTY0, no buffer available -Process= "",  ipl= 4
116388: May 23 11:52:26.230: %TCP-6-NOBUFF: TTY0, no buffer available  -Process= "", ipl= 4
116389: May 23 11:52:26.250:  %TCP-6-NOBUFF: TTY0, no buffer available -Process= "",  ipl= 4
116390: May 23 11:52:26.270: %TCP-6-NOBUFF: TTY0, no buffer available  -Process= "", ipl= 4
116391: May 23 11:52:26.290:  %TCP-6-NOBUFF: TTY0, no buffer available -Process= "",  ipl= 4
116392: May 23 11:52:26.310: %TCP-6-NOBUFF: TTY0, no buffer available  -Process= "", ipl= 4
116393: May 23 11:52:26.338:  %TCP-6-NOBUFF: TTY0, no buffer available -Process= "",  ipl= 4

Also - I have managed to repeat this with a cisco IP communicator setup through the webvpn service on the router (If i use my easyVPN connection it works fine).  It's not as repeatable as with the SPA, but it still happens.

Interesting that other people are experiencing simular problems.   I wonder if we are 'bleeding edge' as opposed to 'leading edge'....   Only time will tell!...

I also can attach some packet captures.  As the VPN goes down on the SPA other clients that are connected to the VPN go down as well.  I have a ping capture going on from my laptop to a server and back the other way.  If you're interested I can attach.

Can you read Visio files?

Thanks for your help on this one!

Regards

[David Trad]

Hi Geoffrey,

I can read Visio files

What UC Software pack are you running by the way?

The information you provide is good thanks for that

Cheers,

David.

[synmedenfield]

Hi,

I'm still waiting for a response from the support team.  I spoke to them yesterday and they assured me that this has been escalated.  My guess is its with the developers who are probably trying to recreate the issue and then hopefully fix it...

In the mean time I've attached a visio which might help people get a feel for the network and i've attached a running config.

We are using UC520_8.1.0 as the pack - the IOS you get out of that is: uc500-advipservicesk9-mz.151-2.T2

I have tried the both 7.4.8 and 7.4.7 on the SPA525G2 phone.....

If I hear anything from the support team I will let people know.

Regards,

[tabernus2010]

Hi Geoffrey!

We are having issues nearly identical to yours.  Our phones are running 7.4.5.  How exactly did you update your firmware? 

Please post any responses you receive from the Cisco peaps.  Would love to use the VPN phone but as it is now it is a great looking paper weight.

Thanks!

[pierrescotland]

I am seeing much the same symptoms when connecting to a 2811 using a 525G2, after 15 seconds of perfect audio, the SPA users audio stream gets choppy or fails, yet they can always hear the other side.  (since it's a 2811 I doubt it is resources on the router, also we have it working fine for months to an 877 as well!)

The strange thing is we have identical set-ups using exact same IOS and phone firmware which work without problem.  Occasionally we see long (3 -5 seconds) delays on call-pickup and dialing on the handset, but the audio is fine once the call goes through.

I would be most interested to hear a solution for this!

[seguinmario]

Does anybody try with the new phone load 7.4.9c?

Thanks,

Mario

[pierrescotland]

We upgraded to this firmware and it solved the 15 sec audio problem.   Strange as some sites appear to work with very old firmware.   I would certainly upgrade if you are still having problems.

[seguinmario]

Tks Peter.

Mario

De : pierrescotland

Envoyé : 2 novembre 2011 11:48

À : Mario Séguin

Objet : - Re: UC520 Router SPA525G2 and SSL VPN's, Very Dissapointed!! [eh8m66-857t-22ljk]

Home<>

Re: UC520 Router SPA525G2 and SSL VPN's, Very Dissapointed!!

created by PETER ROWE<> in SBCS - UC500 - View the full discussion<>

[tabernus2010]

We recently investigating to move on with 3CX cloud phones/services and ditching our old Cisco UC520 since we never get it to work (since 2011) correctly with our teleworkers phones SPA525G2 and had a lot of downtime due to reboots -whenever we had this SPA525G2 online.

Since we no longer has the service contract active, we just download what is available to us.

Anyway, as a latch ditch effort, we upgraded the firmware of SPA525G2 to the latest version: SPA525g_7.5.6  (none BT version)

And updated the IOS firmware on UC520 to 15.1(4)M7-ADV-IP-SERV-CRYPTO.  - which was released not too long ago to address other catastrophic issue on UC500 series.

Summary Software version.

Software Pack: 8.1.0

IOS Image      : uc500-advipservicesk9-mz.151-4.M7

CME Version  : 8.6

Voicemail version: 8.0.3

We are using CCA version 3.2(3) to configure the phones, SSL VPN etc.

And the Full tunnel mode SSL VPN client : CISCO STC win2k+ (Version: 2,5,2019) 

Voila!.. it's working like it supposed to be now...ok great !..now I jinx it...we'll see.

Perhaps this post help whomever land on this page while researching the solution to this issue.

This would give us few more years of service onthe UC520...hopefully.

[seguinmario]

Hi Geoffry,

We have the same problem. Did you try with the new phone load 7.4.9c?

Thanks,

Mario

[Jeremy Lizzotte]

I also want to point out that the SSL VPN should be version 2.5 i believe. if its 3.something thats only for the ASAs (i had this installed, and it kept giving me memory full issues for ram, for VPN connections)