11-21-2011 04:23 AM
Hi
We are running one linux PC with 3 VLC streams for our company Add, last couple days I notices that our multicast source router, where this VLC PC connected is running on high cpu utilization, and I did the following troubleshooting
After I disconneted this PC form network, the CPU process is normal.
The PC is running on Linux, what could be the probelm.
A-DISTT1#sh processes cpu sorted | exclude 0.00%
CPU utilization for five seconds: 92%/91%; one minute: 25%; five minutes: 22%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
395 97350452 389572702 249 0.15% 0.74% 0.69% 0 PIM Process
155 2974377924264655218 0 0.15% 0.96% 1.57% 0 IP Input
396 57168908 68962609 828 0.15% 0.39% 0.26% 0 MLSM Process
3 12376 2411 5133 0.07% 0.41% 0.47% 1 SSH Process
350 121028320 392568888 308 0.07% 0.53% 0.53% 0 Port manager pe
A-DISTT2#sh processes cpu sorted | exclude 0.00%
CPU utilization for five seconds: 45%/33%; one minute: 38%; five minutes: 23%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
155 347591256 330574202 1051 10.87% 2.20% 1.46% 0 IP Input
395 154942084 526796334 294 0.47% 0.70% 0.53% 0 PIM Process
397 34611708 35864894 965 0.15% 0.09% 0.09% 0 MLSM Stat Proces
396 39788256 153516402 259 0.15% 0.21% 0.16% 0 MLSM Process
A-DISTT1#sh processes cpu history
2222222111112222211111222222222211111111112222211111111111
5577777999991111166666333332222299999555556666633333555552
100
90
80
70
60
50
40
30 ******* *****
20 *********************************************** *****
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
2159989999899998899999999999999889997999989999999999999999
6247377575427883157615748084586688842776862785125887682872
100 * **** *** *** ** * * *** ** **** *** ****** **
90 ******* **** ******************* *********************
80 ********************************* *********************
70 *******************************************************
60 *******************************************************
50 ************************************#************#***#**
40 *##*##****###****#*#####*####**#****###****#*##*##*####*
30 * *#######*#########################*####*################
20 # *#######################################################
10 ##########################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
9224236422232233272499999999999999999999999999999999999999999999999999
8770918257566815646499999999999999999999999999999999999899999999999999
100 * **************************************************
90 * **************************************************
80 * **************************************************
70 * * * **************************************************
60 * * * **************************************************
50 * * * **************************************************
40 * * ** * * * **#################################################
30 *********************#################################################
20 #*******************##################################################
10 ######################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
-------------------------------------------------------------------------------------
A-DISTT2#sh processes cpu history
111
1114444444444333339999911111888884444466666444444444422222
100
90
80
70
60
50
40
30
20
10 *** ***** ***** *****
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
9989999999999999999999979999999998889998899998999899999
9798707677387474683822447660644006864779868968758886289656
100 ** **** ** * ** * ** * *** *** **** *** *****
90 ** ******************** ********* *************** *****
80 *******************************************************
70 *******************************************************
60 *******************************************************
50 ******************************************#************
40 ##*#####*###***##***###*####**###***###*####**###***##*
30 #######################################################
20 #######################################################
10 ***#######################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
9922417242123212422199999999999999999999999999999999999999999999999999
9733570000945192502989999999999999999999999999999999999999999999899999
100 ** **************************************************
90 ** **************************************************
80 ** **************************************************
70 ** * **************************************************
60 ** * **************************************************
50 ** * * * **************************************************
40 ** * * * * * *#################################################
30 #* * * * * * *#################################################
20 #********************#################################################
10 ###########*##########################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
You can analysis by seeing the packet drops
A-DISTT1#sh int | in up | drop
Vlan120 is up, line protocol is up
Input queue: 0/75/1452/1452 (size/max/drops/flushes); Total output drops: 0
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
source and destination and where the packet drop is ocurring
A-DISTT1#sh buffers input-interface vlan 120 header
Buffer information for Big buffer at 0x45A30684
data_area 0x826A144, refcount 1, next 0x5026A528, flags 0x200
linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1
if_input 0x47B08364 (Vlan120), if_output 0x0 (None)
inputtime 00:00:00.000 (elapsed never)
outputtime 21w5d (elapsed 21:33:01.704), oqnumber 65535
datagramstart 0x826A1BA, datagramsize 1358, maximum size 1740
mac_start 0x826A1BA, addr_start 0x826A1BA, info_start 0x0
network_start 0x826A1C8, transport_start 0x826A1DC, caller_pc 0x41FFDF6C
source: 192.168.120.200, destination: 239.232.3.2, id: 0xFA3C, ttl: 1,
TOS: 0 prot: 17, source port 39598, destination port 5000
Buffer information for Big buffer at 0x45A5FA00
data_area 0x8324744, refcount 1, next 0x5027A814, flags 0x200
linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1
if_input 0x47B08364 (Vlan120), if_output 0x0 (None)
inputtime 00:00:00.000 (elapsed never)
outputtime 17w5d (elapsed 4w0d), oqnumber 65535
datagramstart 0x83247BA, datagramsize 1358, maximum size 1740
mac_start 0x83247BA, addr_start 0x83247BA, info_start 0x0
network_start 0x83247C8, transport_start 0x83247DC, caller_pc 0x41FFDF6C
source: 192.168.120.200, destination: 239.232.3.3, id: 0x156B, ttl: 1,
TOS: 0 prot: 17, source port 50428, destination port 5000
A-DISTT2#sh buffers input-interface vlan 120 header
Buffer information for Big buffer at 0x45A76AF0
data_area 0x839B744, refcount 1, next 0x502883FC, flags 0x200
linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1
if_input 0x52C86C6C (Vlan120), if_output 0x0 (None)
inputtime 00:00:00.000 (elapsed never)
outputtime 14w5d (elapsed 21:21:01.044), oqnumber 65535
datagramstart 0x839B7BA, datagramsize 1358, maximum size 1740
mac_start 0x839B7BA, addr_start 0x839B7BA, info_start 0x0
network_start 0x839B7C8, transport_start 0x839B7DC, caller_pc 0x41FFDF6C
source: 192.168.120.200, destination: 239.232.3.2, id: 0xDDD3, ttl: 1,
TOS: 0 prot: 17, source port 46214, destination port 5000
Buffer information for Big buffer at 0x45A77208
data_area 0x839D344, refcount 1, next 0x502827C4, flags 0x200
linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1
if_input 0x52C86C6C (Vlan120), if_output 0x0 (None)
inputtime 00:00:00.000 (elapsed never)
outputtime 14w5d (elapsed 16:51:56.084), oqnumber 65535
datagramstart 0x839D3BA, datagramsize 1358, maximum size 1740
mac_start 0x839D3BA, addr_start 0x839D3BA, info_start 0x0
network_start 0x839D3C8, transport_start 0x839D3DC, caller_pc 0x41FFDF6C
source: 192.168.120.200, destination: 239.232.3.3, id: 0xF9AD, ttl: 1,
TOS: 0 prot: 17, source port 39010, destination port 5000
11-22-2011 07:26 AM
Hi there,
the issue is that the mcast streams sourced by the PC have TTL=1
source: 192.168.120.200, destination: 239.232.3.2, id: 0xFA3C, ttl: 1,
...
source: 192.168.120.200, destination: 239.232.3.3, id: 0x156B, ttl: 1,
...
source: 192.168.120.200, destination: 239.232.3.2, id: 0xDDD3, ttl: 1,
...
source: 192.168.120.200, destination: 239.232.3.3, id: 0xF9AD, ttl: 1,
Those packets cannot be handled in hardware, and therefore require CPU assistance > High CPU, as packets with TTL=1 cannot be routed and must be sent to the CPU for it to generate the TTL expired ICMP message.
Easiest solution is to configure your VLC streams to use a TTL>1 in order to avoid TTL expiration in vlan120.
Problem is pretty common on all Cisco switches, not only on cat6k.
regards,
Riccardo
11-25-2011 06:36 AM
Hi there,
please share with us your comments or flag this question as answered.
Riccardo
12-01-2011 09:03 AM
along with the above comments, for future reference you might start adding the following line to the interface
storm-control multicast level 10.00 (10 being a %, change to your preference)
good practice for multicast storm control.
Jude
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide