Packet loss and jitter on phones connected to 4506e switch

[Steven Gould]

Hello all,

I have an issue with a 4506e chassis that was put in to replace a failing 3750 stack. The chassis is running a SUP 7 (WS-X45-SUP7-E) with five 48 port line cards (WS-X4648-RJ45V+E). The software version on the SUP is 03.06.05.E (cat4500e-universal.SPA.03.06.05.E.152-2.E5.bin).

The VoIP is based on CallManager with a mixture of 7940 and 7960 Cisco IP phones - these are the 100mb models. We are seeing a lot of jitter and packet loss on internal and external calls made with the phones, even between phones on the same line card. This is present whether AutoQoS is applied or removed. Not that it should be necessary as we are not going through any low bandwidth trunks. It is also present after I have set the interface speed and duplex settings on the switchport and phone.

Does anyone have any insight into why this may be happening and any possible remedial steps I could take?

Many thanks,

Steve

[m.batts]

I have had some strange issues when swopping out new switches while still using the old Generation phones.

First of all , what version of firmware are you running on the phones? is it relatively current?

Secondly and this may sound odd but have you power booted or reset any of the phones since you plugged them in to the switch.I have managed to cure a few issues by just disconnecting and then reconnecting them.

I assume the Phones are in their own  vlan and you have checked there isn't any traffic swamping the switch??

[Steven Gould]

Thanks for the reply.

The version is 7.2(3.0) on both models (CP-7960G and CP-7940G).

Yes, done several resets and re-connects during the troubleshooting i've undertaken so far - made no difference.

Yes, we have one vlan for phones and another for PCs etc, The load on the switch is minimal. We are a thin client environment and all switchports show hardly any load.

[m.batts]

I just noticed that your experiencing issues pinging devices in general so I wouldn't bother looking at anything to do with phones or QOS

What you need to do is get a laptop , plus it straight into the switch , i.e go into the comms room with a nice new patch cable and ping the switch. Then ping  another device  on the same vlan so it doesn't go through the SVI . Do you still get packet loss???

Was the structured cabling changed at the same time as the switch change?

[Steven Gould]

Structured cabling hasn't changed, it is CAT6 and was installed during 2006 when the building was purchased.

I connected a laptop directly to the switch with a new 2m patch cable and then connected it to a phone that was also connected with a new 2m patch cable, ping results as follows:

Blade 2

Directly connected, 1gb

Ping statistics for 192.168.2.2:
    Packets: Sent = 100, Received = 100, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 13ms, Average = 0ms

Directly connected, 100mb

Ping statistics for 192.168.2.2:
    Packets: Sent = 100, Received = 100, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 307ms, Average = 11ms

Connected through phone, 100mb

Ping statistics for 192.168.2.2:
    Packets: Sent = 100, Received = 98, Lost = 2 (2% loss),
Approximate round trip times in milli-seconds:
    Minimum = 1ms, Maximum = 81ms, Average = 2ms

Blade 3

Directly connected, 1gb

Ping statistics for 192.168.2.2:
    Packets: Sent = 100, Received = 100, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 16ms, Average = 1ms

Directly connected, 100mb

Ping statistics for 192.168.2.2:
    Packets: Sent = 100, Received = 100, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 1ms, Maximum = 355ms, Average = 24ms

Connected through phone, 100mb

Ping statistics for 192.168.2.2:
    Packets: Sent = 100, Received = 96, Lost = 4 (4% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 107ms, Average = 4ms

Blade 4

Directly connected, 1gb

Ping statistics for 192.168.2.2:
    Packets: Sent = 100, Received = 100, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 14ms, Average = 0ms

Directly connected, 100mb

Ping statistics for 192.168.2.2:
    Packets: Sent = 100, Received = 100, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 1ms, Maximum = 351ms, Average = 17ms

Connected through phone, 100mb

Ping statistics for 192.168.2.2:
    Packets: Sent = 100, Received = 94, Lost = 6 (6% loss),
Approximate round trip times in milli-seconds:
    Minimum = 1ms, Maximum = 310ms, Average = 10ms

Blade 5

Directly connected, 1gb

Ping statistics for 192.168.2.2:
    Packets: Sent = 100, Received = 100, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 16ms, Average = 0ms

Directly connected, 100mb

Ping statistics for 192.168.2.2:
    Packets: Sent = 100, Received = 100, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 1ms, Maximum = 237ms, Average = 6ms

Connected through phone, 100mb

Ping statistics for 192.168.2.2:
    Packets: Sent = 100, Received = 89, Lost = 11 (11% loss),
Approximate round trip times in milli-seconds:
    Minimum = 1ms, Maximum = 316ms, Average = 8ms

Blade 6

Directly connected, 1gb

Ping statistics for 192.168.2.2:
    Packets: Sent = 100, Received = 100, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 6ms, Average = 0ms

Directly connected, 100mb

Ping statistics for 192.168.2.2:
    Packets: Sent = 100, Received = 100, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 1ms, Maximum = 348ms, Average = 22ms

Connected through phone, 100mb

Ping statistics for 192.168.2.2:
    Packets: Sent = 100, Received = 92, Lost = 8 (8% loss),
Approximate round trip times in milli-seconds:
    Minimum = 1ms, Maximum = 277ms, Average = 9ms

[m.batts]

Mmmm , if I'm reading this right it looks like it only occurs via the phone which leads me back to the original issue of old phones and new switches.You could try upgrading the phone firmware but to be honest i'm not convinced that would do much.

Just checking you're not getting duplex or speed mismatch messages right?

Do you have any other phone types other than 7940 and 7960's??

[Steven Gould]

I have set two interfaces (on the switch and phone) as speed 100 duplex full and tried test calls between them with the same results.

No i'm afraid, just those two models. Our CallManager is fairly old and I suspect it would struggle to support more modern phones. Specifically ones with an integrated 1GB switch.

[m.batts]

Sorry I'm out of ideas.... I would try newer phones , you should be able to get some later 7961GE phones for very little money as a test.What version of CUCM do you have?

[Steven Gould]

I'm running (hold your breath) CCM 4.1(3)sr2. This is on the basis of - if it ain't broke don't fix it.

I'll see if our supplier can lend me a CP-7961G-GE for testing surposes. I believe these are supported by our version of CCM.

Thanks for your responses - much appreciated.

[m.batts]

Wow an old windows one , haven't seen one of those for a few years..

I doubt you're supplier will be able to source any , you're best of trying ebay.

[Steven Gould]

I have an update on my issue.

I have been in touch with my supplier and he is sending me some grade c 7961 phones to test with. Until they arrive I thought I would give everything one last shot.

I removed the auto qos voip cisco-phone from a couple of the interfaces and re-added it. As it was re-added it also added the service policy lines for the AutoQoS back in. I made some test calls between these phones and the jitter only reached 5 and there was no packet loss. This improvement made me curious about what i've changed that could make this so. Aside from correcting the option 150 on the voice DHCP, I couldn't really think of much.

I did some ping tests from the PCs connected through the phones on the newly configured interfaces and the pings are still getting dropped anywhere between 10 and 20% but it makes me wonder if I just need to change the AutoQoS configuration or potentially change traffic classifications to ease this problem.

Any either of you have any further thoughts on this?

[m.batts]

You really shouldn't be getting any ping loss across the switch in your environment.You might be able to mask issues possibly with QOS  but there is definitely something wrong, personally I think its switch in the phones talking to the 4506E but the only way to prove that would be to move to Gen2 phones.

[Steven Gould]

My supplier sent two graded CP-7941G-GE phones for me to test with. They load and are supported by my CCM (phew). Ping tests from the connected workstations don't drop any packets and show a minimal delay so it looks like it was the switch inside the phones that was causing the drops.

This unfortunately means I am now faced with the task (and expense) of replacing all my handsets.

[chrihussey]

Before you take the route of replacing your phones, it may just be how the switch handles the slower 100mb vs gig connection from a buffering and queuing perspective. If you're still getting the occasional high delay packets when pinging from just a 100mb PC (and not with gig) then you may simply be putting a bandaid on the issue.

Since you have a spare SUP, I'd suggest swapping it first. You have nothing to lose. Just my thought.

[m.batts]

Mmm , I suspected as much . I did some digging around using Gen1 phones and the later switches but there's very little info as most people have upgraded by now.

You might want to look into how the switch handles 100MB connections in general to see if anything needs tuning.Clutching at straws really though :-)