cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
5023
Views
0
Helpful
27
Replies

Packet loss and jitter on phones connected to 4506e switch

Steven Gould
Level 1
Level 1

Hello all,

I have an issue with a 4506e chassis that was put in to replace a failing 3750 stack. The chassis is running a SUP 7 (WS-X45-SUP7-E) with five 48 port line cards (WS-X4648-RJ45V+E). The software version on the SUP is 03.06.05.E (cat4500e-universal.SPA.03.06.05.E.152-2.E5.bin).

The VoIP is based on CallManager with a mixture of 7940 and 7960 Cisco IP phones - these are the 100mb models. We are seeing a lot of jitter and packet loss on internal and external calls made with the phones, even between phones on the same line card. This is present whether AutoQoS is applied or removed. Not that it should be necessary as we are not going through any low bandwidth trunks. It is also present after I have set the interface speed and duplex settings on the switchport and phone.

Does anyone have any insight into why this may be happening and any possible remedial steps I could take?

Many thanks,

Steve

27 Replies 27

Interesting indeed, but that just raises more questions.

Did the phones get new loads since you changed the option 150?

If you ping from just a PC at 100mb, no phone, do you still get 100% but with occasional long delay packets?

Clear the counters on the switch ports and run the pings through the phones and verify you see the packet drops on the switch port.

Run test calls with the phones periodically to make sure there is consistency in those results.

The phones should have been operating with the cached loads from their last communication with the TFTP server prior to me changing the switch. Nothing about the load has changed so it shouldn't have changed anything.

Yes, pings are still giving the same results as before. Looking at the stats all the traffic is either going in queue 1 or queue 8.

Test calls between handsets on the same and different blades are fine - minimal jitter and no loss.

I was going to ask about the possibility of duplex mismatches but read the other thread and see that you have that covered.

What I do notice is that when doing the ping test at gig speed you get minimal delay and no packet loss. However at 100mb without the phone, you have no packet loss, but there is a very high maximum delay factor. With the phone you have the same high maximum delay along with the packet loss. So at 100mb, these sporadic delays are amplified with the phones and are why you are seeing the jitter and packet loss .

The only things I can suggest are:

1- Try the "speed auto 10 100" on the phone interfaces with the phones set to auto and see if this makes a difference. (A very long shot, but easy to try).

2- Not sure if it is an option but if you have a spare SUP7, try replacing it. The scheduling and queuing is controlled by the supervisor.

3- Try an older rev blade like an WS-X4248-RJ45V with the older phones. It's a 10/100 blade and should be compatible with the SUP 7E (https://www.cisco.com/c/en/us/products/collateral/switches/catalyst-4500-series-switches/data_sheet_c78-612364.html)

4- Open a CCO case to see if they can determine if there is a bug or even a back plane issue. You've done enough here to describe the issue in greater detail.

Not sure what else to suggest but hope this helps.

1. I had tried the speed auto 100 with the phone set to auto and also with the phone set to 100 FULL - made no difference.

2. As it happens I do have a spare SUP7. I called our supplier for an RMA and the new unit arrived the day before yesterday. It is running an older version of IOS which I plan to test with.

3. I'll investigate this if the other SUP7 is unsuccessful.

4. I'll proceed with that is the 1-3 doesn't work out.

Thanks for the responses - much appreciated.

chrihussey
VIP Alumni
VIP Alumni

Aside from a bug or hardware issue here are some suggestions:

1- Clear counters and verify the physical layer is clean on all phone and uplink interfaces. (Input, output errors, runts, etc.)

2- Verify that this is not only a VoIP issue and that the similar packet loss isn't being experienced during data transfers or normal usage.

3- Get sniffer traces of calls and verify that the QoS settings are being preserved and not being altered or changed to best effort.

4- Verify (with a sniffer trace) that the phones on the same blade are communicating directly to each other and not having to go through an intermediate resource.

5- Try the "sh int gx/x counter detail" on one of the interfaces that has a phone to determine if traffic is in the proper queues and which queues are dropping the packets.

6- If you want, provide the switch's config for review.

Thanks for the reply.

1. I am getting output drops but I expect to see that given that the bandwidth is being reduced to 100mb. Aside from that the interfaces are clean.

2. The packet loss is being experienced on the voice and data vlans. The PCs (connected to the switchport on the phones) are dropping packets while pinging the switch.

3. I'll look at doing this but don't have a great deal of experience tracing and such.

4. As above.

5.

6. Here is the config. I've taken out the majority of similarly configured interfaces for ease of reading:

4506e#sh run
Building configuration...

Current configuration : 48165 bytes
!
! Last configuration change at 14:43:12 UTC Tue Oct 11 2016
!
version 15.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
service compress-config
!
hostname 4506e
!
boot-start-marker
boot-end-marker
!
!
vrf definition mgmtVrf
 !
 address-family ipv4
 exit-address-family
 !
 address-family ipv6
 exit-address-family
!
enable secret 5 $1$4xW2$r8sninHTDPvENjxnBBXNA1
!
username admin privilege 15 secret 5 $1$.cN8$KNVH8upXn3h6u1JJaJ9li0
no aaa new-model
!
!
!
!
!
!
ip domain-name .co.uk
ip dhcp excluded-address 10.102.1.1 10.102.1.9
ip dhcp excluded-address 10.102.1.193 10.102.1.254
ip dhcp excluded-address 192.168.2.1 192.168.2.64
ip dhcp excluded-address 192.168.2.219 192.168.2.254
!
ip dhcp pool DATA
 import all
 network 192.168.2.0 255.255.255.0
 default-router 192.168.2.2
 domain-name .co.uk
 dns-server 192.168.2.33 192.168.2.34
 netbios-name-server 192.168.1.15
!
ip dhcp pool VOICE
 import all
 network 10.102.1.0 255.255.255.0
 default-router 10.102.1.2
 domain-name .co.uk
 option 150 ip 10.102.1.2
 dns-server 192.168.2.33
 lease 8
!
!
!
power redundancy-mode redundant
!
spanning-tree mode rapid-pvst
no spanning-tree optimize bpdu transmission
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!         
!
class-map match-all AutoQos-4.0-Scavenger-Classify
 match access-group name AutoQos-4.0-ACL-Scavenger
class-map match-all AutoQos-4.0-Signaling-Classify
 match access-group name AutoQos-4.0-ACL-Signaling
class-map match-any AutoQos-4.0-Priority-Queue
 match cos  5
 match dscp ef
 match dscp cs5
 match dscp cs4
class-map match-all AutoQos-4.0-VoIP-Data-Cos
 match cos  5
class-map match-any AutoQos-4.0-Multimedia-Stream-Queue
 match dscp af31
 match dscp af32
 match dscp af33
class-map match-all AutoQos-4.0-Network-Mgmt
 match dscp cs2
class-map match-all AutoQos-4.0-VoIP-Signal-Cos
 match cos  3
class-map match-any AutoQos-4.0-Multimedia-Conf-Queue
 match cos  4
 match dscp af41
 match dscp af42
 match dscp af43
 match access-group name AutoQos-4.0-ACL-Multimedia-Conf
class-map match-any AutoQos-4.0-Transaction-Data
 match dscp af21
 match dscp af22
 match dscp af23
class-map match-all AutoQos-4.0-Network-Ctrl
 match dscp cs7
class-map match-all AutoQos-4.0-Scavenger
 match dscp cs1
class-map match-all AutoQos-4.0-Default-Classify
 match access-group name AutoQos-4.0-ACL-Default
class-map match-any AutoQos-4.0-Signaling
 match dscp cs3
 match cos  3
class-map match-any AutoQos-4.0-Bulk-Data-Queue
 match cos  1
 match dscp af11
 match dscp af12
 match dscp af13
 match access-group name AutoQos-4.0-ACL-Bulk-Data
class-map match-all AutoQos-4.0-Transaction-Classify
 match access-group name AutoQos-4.0-ACL-Transactional-Data
class-map match-all AutoQos-4.0-Broadcast-Vid
 match dscp cs5
class-map match-any AutoQos-4.0-Bulk-Data
 match dscp af11
 match dscp af12
 match dscp af13
class-map match-any AutoQos-4.0-Scavenger-Queue
 match dscp cs1
 match cos  1
 match access-group name AutoQos-4.0-ACL-Scavenger
class-map match-any AutoQos-4.0-VoIP
 match dscp ef
 match cos  5
class-map match-any AutoQos-4.0-Multimedia-Conf
 match dscp af41
 match dscp af42
 match dscp af43
class-map match-any AutoQos-4.0-Control-Mgmt-Queue
 match cos  3
 match dscp cs7
 match dscp cs6
 match dscp cs3
 match dscp cs2
 match access-group name AutoQos-4.0-ACL-Signaling
class-map match-all AutoQos-4.0-Bulk-Data-Classify
 match access-group name AutoQos-4.0-ACL-Bulk-Data
class-map match-any AutoQos-4.0-Trans-Data-Queue
 match cos  2
 match dscp af21
 match dscp af22
 match dscp af23
 match access-group name AutoQos-4.0-ACL-Transactional-Data
class-map match-any AutoQos-4.0-Multimedia-Stream
 match dscp af31
 match dscp af32
 match dscp af33
class-map match-any AutoQos-4.0-VoIP-Data
 match dscp ef
 match cos  5
class-map match-all AutoQos-4.0-Internetwork-Ctrl
 match dscp cs6
class-map match-all AutoQos-4.0-Realtime-Interact
 match dscp cs4
class-map match-all AutoQos-4.0-Multimedia-Conf-Classify
 match access-group name AutoQos-4.0-ACL-Multimedia-Conf
class-map match-any AutoQos-4.0-VoIP-Signal
 match dscp cs3
 match cos  3
!
policy-map AutoQos-4.0-Output-Policy
 class AutoQos-4.0-Scavenger-Queue
  bandwidth remaining percent 1
 class AutoQos-4.0-Priority-Queue
  priority
  police cir percent 30 bc 33 ms
 class AutoQos-4.0-Control-Mgmt-Queue
  bandwidth remaining percent 10
 class AutoQos-4.0-Multimedia-Conf-Queue
  bandwidth remaining percent 10
 class AutoQos-4.0-Multimedia-Stream-Queue
  bandwidth remaining percent 10
 class AutoQos-4.0-Trans-Data-Queue
  bandwidth remaining percent 10
    dbl
 class AutoQos-4.0-Bulk-Data-Queue
  bandwidth remaining percent 4
    dbl
 class class-default
  bandwidth remaining percent 25
    dbl
policy-map AutoQos-4.0-Input-Policy
 class AutoQos-4.0-VoIP
 class AutoQos-4.0-Broadcast-Vid
 class AutoQos-4.0-Realtime-Interact
 class AutoQos-4.0-Network-Ctrl
 class AutoQos-4.0-Internetwork-Ctrl
 class AutoQos-4.0-Signaling
 class AutoQos-4.0-Network-Mgmt
 class AutoQos-4.0-Multimedia-Conf
 class AutoQos-4.0-Multimedia-Stream
 class AutoQos-4.0-Transaction-Data
 class AutoQos-4.0-Bulk-Data
 class AutoQos-4.0-Scavenger
policy-map AutoQos-4.0-Cisco-Phone-Input-Policy
 class AutoQos-4.0-VoIP-Data-Cos
  set dscp ef
  police cir 128000 bc 8000
   exceed-action set-dscp-transmit cs1
   exceed-action set-cos-transmit 1
 class AutoQos-4.0-VoIP-Signal-Cos
  set dscp cs3
  police cir 32000 bc 8000
   exceed-action set-dscp-transmit cs1
   exceed-action set-cos-transmit 1
 class class-default
  set dscp default
  set cos 0
!
!
!
!
!
!
!
interface Port-channel1
 switchport
 switchport trunk allowed vlan 20,102
 switchport mode trunk
!
interface FastEthernet1
 vrf forwarding mgmtVrf
 no ip address
 speed auto
 duplex auto
!
interface TenGigabitEthernet1/1
!
interface TenGigabitEthernet1/2
!
interface TenGigabitEthernet1/3
!
interface TenGigabitEthernet1/4
!
interface GigabitEthernet2/1
 description Trunk link to server room
 switchport trunk allowed vlan 20,102
 switchport mode trunk
 qos trust device cisco-phone
 channel-group 1 mode active
!
interface GigabitEthernet2/2
 switchport access vlan 20
 switchport voice vlan 102
 auto qos voip cisco-phone
 qos trust device cisco-phone
 spanning-tree portfast
!
interface GigabitEthernet2/3
 switchport access vlan 20
 switchport voice vlan 102
 auto qos voip cisco-phone
 qos trust device cisco-phone
 spanning-tree portfast
!
interface GigabitEthernet2/4
 switchport access vlan 20
 switchport voice vlan 102
 auto qos voip cisco-phone
 qos trust device cisco-phone
 spanning-tree portfast
!
interface GigabitEthernet2/5
 description CallManager Subscriber
 switchport access vlan 102
 switchport voice vlan 102
 speed 100
 duplex full
 spanning-tree portfast
!
interface GigabitEthernet2/6
 description CallManager Publisher
 switchport access vlan 102
 switchport voice vlan 102
 speed 100
 duplex full
 spanning-tree portfast
!
interface GigabitEthernet2/7
 description Unity
 switchport access vlan 102
 speed 100
 duplex full
 spanning-tree portfast
!
interface GigabitEthernet2/8
 description CUBE Router
 switchport access vlan 102
 switchport voice vlan 102
 auto qos trust
 spanning-tree portfast
 service-policy input AutoQos-4.0-Input-Policy
!
interface GigabitEthernet2/9
 description IPT Resource
 switchport access vlan 20
 speed 100
 duplex full
 spanning-tree portfast
!
interface GigabitEthernet2/10
 description ISDN Router
 switchport mode trunk
!
interface GigabitEthernet2/11
 switchport access vlan 20
 switchport voice vlan 102
 auto qos voip cisco-phone
 qos trust device cisco-phone
 spanning-tree portfast
!
interface GigabitEthernet2/12
 switchport access vlan 20
 switchport voice vlan 102
 auto qos voip cisco-phone
 qos trust device cisco-phone
 spanning-tree portfast
!
interface GigabitEthernet2/13
 switchport access vlan 20
 switchport voice vlan 102
 auto qos voip cisco-phone
 qos trust device cisco-phone
 spanning-tree portfast
!
interface GigabitEthernet2/14
 switchport access vlan 20
 switchport voice vlan 102
 auto qos voip cisco-phone
 qos trust device cisco-phone
 spanning-tree portfast
!
interface GigabitEthernet2/15
 switchport access vlan 20
 switchport voice vlan 102
 auto qos voip cisco-phone
 qos trust device cisco-phone
 spanning-tree portfast
!
interface GigabitEthernet6/47
 switchport access vlan 20
 switchport voice vlan 102
 auto qos voip cisco-phone
 qos trust device cisco-phone
 spanning-tree portfast
!
interface GigabitEthernet6/48
 switchport access vlan 20
 switchport voice vlan 102
 auto qos voip cisco-phone
 qos trust device cisco-phone
 spanning-tree portfast
!
interface Vlan1
 no ip address
 shutdown
!
interface Vlan20
 description ACCESS VLAN
 ip address 192.168.2.2 255.255.255.0
!
interface Vlan102
 description VOICE VLAN
 ip address 10.102.1.2 255.255.255.0
!
ip default-gateway 192.168.2.1
ip forward-protocol nd
no ip http server
ip route 0.0.0.0 0.0.0.0 192.168.2.7
ip route 93.95.124.81 255.255.255.255 192.168.2.7
!
ip access-list extended AutoQos-4.0-ACL-Bulk-Data
 permit tcp any any eq ftp
 permit tcp any any eq ftp-data
 permit tcp any any eq 22
 permit tcp any any eq smtp
 permit tcp any any eq 465
 permit tcp any any eq 143
 permit tcp any any eq 993
 permit tcp any any eq pop3
 permit tcp any any eq 995
 permit tcp any any eq 1914
ip access-list extended AutoQos-4.0-ACL-Default
 permit ip any any
ip access-list extended AutoQos-4.0-ACL-Multimedia-Conf
 permit udp any any range 16384 32767
ip access-list extended AutoQos-4.0-ACL-Scavenger
 permit tcp any any eq 1214
 permit udp any any eq 1214
 permit tcp any any range 2300 2400
 permit udp any any range 2300 2400
 permit tcp any any eq 3689
 permit udp any any eq 3689
 permit tcp any any range 6881 6999
 permit tcp any any eq 11999
 permit tcp any any range 28800 29100
ip access-list extended AutoQos-4.0-ACL-Signaling
 permit tcp any any range 2000 2002
 permit tcp any any range 5060 5061
 permit udp any any range 5060 5061
ip access-list extended AutoQos-4.0-ACL-Transactional-Data
 permit tcp any any eq 443
 permit tcp any any eq 1521
 permit udp any any eq 1521
 permit tcp any any eq 1526
 permit udp any any eq 1526
 permit tcp any any eq 1575
 permit udp any any eq 1575
 permit tcp any any eq 1630
 permit udp any any eq 1630
!
!
!
!
line con 0
 stopbits 1
line vty 0 4
 password
 login
line vty 5 15
 password
 login
!
end

If you are dropping packets when simply pinging the switch from PCs that would indicate a larger problem and quite probably just not a phone issue.

Check the CPU utilization (sh proc cpu & sh proc cpu history). Also check for high traffic utilization on any port, especially trunk or server ports. That would be a good start.

I don't understand why the Voice DHCP scope has the option 150 pointing to the voice VLAN interface on the switch. It usually points to a TFTP server. Not sure what you are doing there.

From the counter output, it looks like the majority of packets being dropped are the best effort (data) packets. I agree if port utilization is minimal, this shouldn't be happening, but like I said there appears to be a bigger issue here. 

Also, some minor points:

On G2/1, the trunk link to the server room. The qos trust device cisco phone will result in all incoming packets to be marked best effort. I'd remove it if you don't want that to happen.

On G2/5,6 & 8, you just need the access VLAN and not the voice VLAN.

The phone ports have "auto qos voip cisco phone" but I don't see any of the auto generated service policies applied. Not sure why that is. Try removing the command and re-applying it to one interface to see if that changes anything on the interface.

Good catch on the option 150. That is supposed to point at our CallManager - i'll change that.

The changes you suggest to the ports make sense so i'll go ahead and make those.

When I issued the auto qos voip cisco-phone command the policies were applied. I had removed them to establish whether the policies were having an effect on the jitter/packet loss. As it turns out it didn't make much difference whether the policies were applied or not.

CPU output as follows:

4506e#sh proc cpu         
Core 0: CPU utilization for five seconds: 9%; one minute: 9%; five minutes: 9%
Core 1: CPU utilization for five seconds: 9%; one minute: 10%; five minutes: 10%
PID     Runtime(ms) Invoked   uSecs  5Sec 1Min 5Min TTY   Process
1       853         728       1172   0.00 0.00 0.00 0     init               
2       9           246       38     0.00 0.00 0.00 0     kthrd           
3       263         38545     6      0.00 0.00 0.00 0     migration/0        
4       0           3         3      0.00 0.00 0.00 0     sirq-high/0        
5       1218968     235727009 5      0.00 0.00 0.00 0     sirq-timer/0       
6       0           9         3      0.00 0.00 0.00 0     sirq-net-tx/0      
7       69          5127      13     0.00 0.00 0.00 0     sirq-net-rx/0      
8       0           7         0      0.00 0.00 0.00 0     sirq-block/0       
9       0           8         0      0.00 0.00 0.00 0     sirq-block-iopo    
10      8           2780      3      0.00 0.00 0.00 0     sirq-tasklet/0     
11      215288      64449630  3      0.00 0.00 0.00 0     sirq-sched/0       
12      0           11        0      0.00 0.00 0.00 0     sirq-hrtimer/0     
13      774673      235727016 3      0.00 0.00 0.00 0     sirq-rcu/0         
14      521         59616     8      0.00 0.00 0.00 0     desched/0          
15      56          8467      6      0.00 0.00 0.00 0     migration/1        
16      0           15        0      0.00 0.00 0.00 0     sirq-high/1        
17      593524      235726839 2      0.00 0.00 0.00 0     sirq-timer/1       
18      0           21        1      0.00 0.00 0.00 0     sirq-net-tx/1      
19      163         12520     13     0.00 0.00 0.00 0     sirq-net-rx/1      
20      0           19        0      0.00 0.00 0.00 0     sirq-block/1       
21      0           20        0      0.00 0.00 0.00 0     sirq-block-iopo    
22      3           905       3      0.00 0.00 0.00 0     sirq-tasklet/1     
23      192560      71151539  2      0.00 0.00 0.00 0     sirq-sched/1       
24      0           28        2      0.00 0.00 0.00 0     sirq-hrtimer/1     
25      617997      235726845 2      0.00 0.00 0.00 0     sirq-rcu/1         
26      64          8763      7      0.00 0.00 0.00 0     desched/1          
27      436930      2895494   150    0.00 0.00 0.00 0     events/0           
28      36000       1021794   35     0.00 0.00 0.00 0     events/1           
29      0           28        0      0.00 0.00 0.00 0     cpuset             
30      18          1467      12     0.00 0.00 0.00 0     khelper            
33      0           30        0      0.00 0.00 0.00 0     netns              
34      0           31        0      0.00 0.00 0.00 0     async/mgr          
154     1346        157184    8      0.00 0.00 0.00 0     sync_supers        
156     66          3541      18     0.00 0.00 0.00 0     bdi-default        
158     2197        3730      589    0.00 0.00 0.00 0     kblockd/0          
159     328         309       1064   0.00 0.00 0.00 0     kblockd/1          
169     0           36        0      0.00 0.00 0.00 0     ksuspend_usbd      
174     0           38        0      0.00 0.00 0.00 0     khubd              
177     0           38        0      0.00 0.00 0.00 0     kseriod            
179     3           43        71     0.00 0.00 0.00 0     kmmcd              
205     0           59        3      0.00 0.00 0.00 0     rpciod/0           
206     0           57        2      0.00 0.00 0.00 0     rpciod/1           
241     0           44        0      0.00 0.00 0.00 0     kswapd0            
242     0           45        0      0.00 0.00 0.00 0     aio/0              
243     0           46        0      0.00 0.00 0.00 0     aio/1              
244     0           47        0      0.00 0.00 0.00 0     nfsiod             
246     0           48        0      0.00 0.00 0.00 0     xfs_mru_cache      
247     0           49        0      0.00 0.00 0.00 0     xfslogd/0          
248     0           50        0      0.00 0.00 0.00 0     xfslogd/1          
249     0           51        0      0.00 0.00 0.00 0     xfsdatad/0         
250     0           52        0      0.00 0.00 0.00 0     xfsdatad/1         
251     0           53        0      0.00 0.00 0.00 0     xfsconvertd/0      
252     0           54        0      0.00 0.00 0.00 0     xfsconvertd/1      
253     0           55        0      0.00 0.00 0.00 0     crypto/0           
254     0           56        0      0.00 0.00 0.00 0     crypto/1           
436     0           61        0      0.00 0.00 0.00 0     mtdblockd          
488     0           62        0      0.00 0.00 0.00 0     usbhid_resumer     
558     93          824       113    0.00 0.00 0.00 0     udevd              
706     163         5255      31     0.00 0.00 0.00 0     loop1              
722     0           72        1      0.00 0.00 0.00 0     loop2              
738     332         6898      48     0.00 0.00 0.00 0     loop3              
1587    5           758       6      0.00 0.00 0.00 0     udevd              
1597    2           755       3      0.00 0.00 0.00 0     udevd              
1663    705         700       1007   0.00 0.00 0.00 0     dbus-daemon        
2061    66          603       109    0.00 0.00 0.00 0     klogd              
2073    2           601       4      0.00 0.00 0.00 0     portmap            
2338    1           771       2      0.00 0.00 0.00 0     platform.mount     
2343    1           771       1      0.00 0.00 0.00 0     inotifywait        
2372    0           580       1      0.00 0.00 0.00 0     sshd               
2384    2           575       3      0.00 0.00 0.00 0     xinetd             
2388    1           576       3      0.00 0.00 0.00 0     xinetd             
2417    0           70        0      0.00 0.00 0.00 0     lockd              
2418    1           333       4      0.00 0.00 0.00 0     nfsd               
2419    1           334       3      0.00 0.00 0.00 0     nfsd               
2420    1           335       3      0.00 0.00 0.00 0     nfsd               
2421    1           336       3      0.00 0.00 0.00 0     nfsd               
2422    1           337       3      0.00 0.00 0.00 0     nfsd               
2423    1           338       3      0.00 0.00 0.00 0     nfsd               
2424    1           339       4      0.00 0.00 0.00 0     nfsd               
2425    2           339       6      0.00 0.00 0.00 0     nfsd               
2431    0           597       0      0.00 0.00 0.00 0     rpc.mountd         
2795    1           698       2      0.00 0.00 0.00 0     sh                 
2798    1           698       2      0.00 0.00 0.00 0     inotifywait        
3505    0           79        0      0.00 0.00 0.00 0     edac-poller        
3527    0           80        0      0.00 0.00 0.00 0     vsi work/0         
3528    0           81        0      0.00 0.00 0.00 0     vsi work/1         
3899    9644        944363    10     0.00 0.00 0.00 0     watchdog           
4274    144523      1886961   76     0.00 0.01 0.01 0     system_mgr         
4485    14956       1046883   14     0.00 0.00 0.00 1088  slproc             
4520    32783       1550712   21     0.00 0.00 0.01 1088  hwcontrol          
4526    14746       1040706   14     0.00 0.00 0.00 1088  ns_oir_proxy       
4528    14934       1043185   14     0.00 0.00 0.00 0     profiled           
4529    59908       90875     659    0.00 0.01 0.05 0     oom_poll.sh        
4530    16041       1046693   15     0.00 0.00 0.00 0     obfld              
4531    14607       1043184   14     0.00 0.02 0.00 0     consoled           
4532    14986       1043182   14     0.00 0.00 0.00 0     system_mgr_cli     
4533    244509      6080184   40     0.00 0.00 0.01 0     plogd              
4534    14393       1043228   13     0.00 0.00 0.00 0     psdprov            
4535    239448      22247769  10     0.04 0.01 0.04 0     pdsd               
4536    15270       1043253   14     0.00 0.01 0.02 0     osinfo-provider    
4537    14718       1043211   14     0.00 0.00 0.00 0     oscore_p           
4538    26667       1155404   23     0.00 0.03 0.04 0     netd               
4539    322616      1890199   170    0.04 0.04 0.04 0     mem_mgmt           
4540    42223       1983546   21     0.00 0.00 0.02 0     mgmte_tap          
4541    22816       1072338   21     0.00 0.01 0.00 0     licensed           
4542    14918       1060315   14     0.00 0.00 0.00 0     installer          
4543    629296      24454520  376    0.12 0.12 0.12 0     ffm                
4544    18494       1075988   17     0.00 0.08 0.28 0     eicored            
4545    15785       1076406   14     0.00 0.00 0.00 0     dtmgr              
4546    572065      1984535   288    0.04 0.03 0.04 0     cpumemd            
4547    43955       2001064   21     0.00 0.00 0.02 0     liin_tap           
4549    16118       1049655   15     0.00 0.06 0.06 0     cli_agent          
4551    15412       5351150   2      0.00 0.00 0.00 0     snmp_subagent      
4552    30940       1985917   15     0.00 0.01 0.02 0     erm_p              
4553    25983       1985908   13     0.00 0.00 0.00 0     ngdumper_provid    
4554    25757       1985917   12     0.00 0.03 0.04 0     sysinfo_p          
4576    14918       1039280   14     0.00 0.00 0.00 0     fs_cmd             
4587    16524       1145074   14     0.00 0.00 0.00 0     licenseagentd      
4588    15220       1045335   14     0.00 0.00 0.00 0     iifd               
4589    2600001     888453590 210    9.23 9.78 9.78 0     iosd               
4590    15305       1063394   14     0.00 0.00 0.01 0     ha_mgr             
6048    15          877       17     0.00 0.00 0.00 0     epc_sm_liaison     
6072    0           878       0      0.00 0.00 0.00 34816 script             
6078    4           877       4      0.00 0.00 0.00 34816 sh                 
6084    1           883       2      0.00 0.00 0.00 34816 tail               
6269    98          810       121    0.00 0.00 0.00 34816 nova_sntp.sh       
6399    212         901       236    0.00 0.00 0.00 34816 rollback_timer.    
6494    0           808       0      0.00 0.00 0.00 0     nova_sntp.sh       
6498    50          808       63     0.00 0.00 0.00 0     xlogger            
6499    3           814       4      0.00 0.00 0.00 0     sntp               
6554    1           889       1      0.00 0.00 0.00 0     rollback_timer.    
6557    51          889       57     0.00 0.00 0.00 0     xlogger            
6561    2           893       2      0.00 0.00 0.00 0     inotifywait        
15085   0           258       0      0.00 0.00 0.00 0     flush-3:0          
15132   1           90875     0      0.00 0.00 0.00 0     sleep              

4506e#sh proc cpu history

History information for system:

                                                                
         11111                                                  
    999990000088888888887777777777777779999977777777776666677777
100                                                             
 90                                                             
 80                                                             
 70                                                             
 60                                                             
 50                                                             
 40                                                             
 30                                                             
 20                                                             
 10 ************************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5    
               CPU% per second (last 60 seconds)

                                                                
    22112  1112 1 1 1  1 11 1 1  1 1  1 1 11111 1 1111 1  1 1111
    713029922929081819759539495995809979188703396921257089180008
100                                                             
 90                                                             
 80                                                             
 70                                                             
 60                                                             
 50                                                             
 40                                                             
 30 *                                                           
 20 **  *    **        * *    *  *    *   **    *    *         *
 10 ############################################################
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5    
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

                                                                            
    22231111122111 21 12223222411  11 1111111111413432311 11111 11111 11111
    159490375294319559313231810799933861440140253802068679044449070088100488
100                                                                         
 90                                                                         
 80                                                                         
 70                                                                         
 60                                                                         
 50                                                                         
 40                           *                 *  *  *                     
 30  ***      *    *      * * *                 * *****                     
 20 *****  ****    **  **********     *        **********        *  *     *
 10 ########################################################################
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0
                       CPU% per hour (last 72 hours)
                    * = maximum CPU%   # = average CPU%


History information for core 0:

                                                                
    1     11111                                                 
    399999000009999999999777778888877777999997777777777666667777
100                                                             
 90                                                             
 80                                                             
 70                                                             
 60                                                             
 50                                                             
 40                                                             
 30                                                             
 20                                                             
 10 ************************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5    
               CPU% per second (last 60 seconds)

                                                                
    111111 111111 1  1 1 1111 1  1 11 1   111111111111    1 1111
    973071922880190890928310494985800939997501314001248999081012
100                                                             
 90                                                             
 80                                                             
 70                                                             
 60                                                             
 50                                                             
 40                                                             
 30                                                             
 20 **  *    **                  *        **                    
 10 ############################################################
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5    
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

                                                                            
    12223111112111121111313221311 111 1 11 11111212322311  11111 1  11111111
    616821033901101240296700693699123949329030138654522549922340949960101350
100                                                                         
 90                                                                         
 80                                                                         
 70                                                                         
 60                                                                         
 50                                                                         
 40                     *                                                   
 30   ***               * * * *                 * *** *                     
 20 *****    **    *   **********               ********            *     *
 10 ########################################################################
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0
                       CPU% per hour (last 72 hours)
                    * = maximum CPU%   # = average CPU%


History information for core 1:

                                                                
    1     11111                         11111                   
    099999000008888877777777777777777777000008888877777777778888
100                                                             
 90                                                             
 80                                                             
 70                                                             
 60                                                             
 50                                                             
 40                                                             
 30                                                             
 20                                                             
 10 ************************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5    
               CPU% per second (last 60 seconds)

                                                                
    32112 1112211 111  1111111111111112 1 22111 1 1111 1 11 1112
    698188022261282048880760407106012029381005389973378390282004
100                                                             
 90                                                             
 80                                                             
 70                                                             
 60                                                             
 50                                                             
 40 *                                                           
 30 **  *     *                                                 
 20 *** *    **        * **   *  *    *   ** *  * *  *         *
 10 ############################################################
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5    
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

                                                                            
    233421121231111211123242325221111 11111111115235334121111111121121111121
    704142619688530971445704654100054993671251398461524920267562202230312631
100                                                                         
 90                                                                         
 80                                                                         
 70                                                                         
 60                                             *                           
 50                           *                 *  *                        
 40    *      *         * * * *                 * *** *                     
 30 ****     **    *    *** ***                 * *****                     
 20 ***** *******  **  **********  *  * **  *  **********  ****  *  *    **
 10 ########################################################################
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0
                       CPU% per hour (last 72 hours)
                    * = maximum CPU%   # = average CPU%

Just curious, how much packet loss is there when you ping the switch from a PC or simply go PC to PC on the same switch? Does it happen whether or not you are going through a phone? Is there a chance it can be isolated to a particular blade?

 

The packet loss varies. When pinging from a workstation to the switch it will drop anywhere from 5 - 10%. The packet loss/jitter on the phones looks like this:

I have posted the results of some directed connected and phone connected ping tests I ran in the other thread below (i'd post the results again but I don't want to make this even longer). I tested all blades to try and isolate, they are all performing similarly.

Leo Laohoo
Hall of Fame
Hall of Fame

The chassis is running a SUP 7 (WS-X45-SUP7-E) with five 48 port line cards (WS-X4648-RJ45V+E).

I've got a 4510R+E with Sup7E and six of these X4648 but I'm running 3.7.2E. 

My chassis has phones and wireless access points in them.  I have no issue about call quality.

Thanks for the reply. 

That is interesting. Can you share a little more about your setup. Are you using AutoQoS or do you have a custom policy (or no policy?). What model of phone do you use? Are you SIP or ISDN for your external calls?

What model of phone do you use? Are you SIP or ISDN for your external calls?

All Cisco shop.  Cisco phones 7945/65/75 talking to a full CUCM cluster.  Aside from the corded phones, some sites have 7925G wireless phones.  Work OK.  

Are you using AutoQoS or do you have a custom policy (or no policy?).

We've laid down our own fibre optic links (plural form) all over the city.  Some people like to put QoS on the switches but when I see them I rip them out. 

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: