02-07-2012 06:34 AM - edited 03-07-2019 04:46 AM
We are experiencing with high CPU input due to ARP input between 20:30 and 22:30 every day
At this time we have a lot of backup operations. When I look the netflow report, I can't see anything anormal.
We are changing our backup server's NIC card from 1gig to 10Gig. The backup operation's traffic is high (approx 2Gbps level) but 6509 has to be handle this size of traffic.
We are using two 6509E in VSS mode
and our image version is s72033-adventerprisek9_wan-mz.122-33.SXJ.bin
What the problem cause may be?
20:00
show ip arp summary
--------------
2588 IP ARP entries, with 166 of them incomplete
show process cpu sorted
--------------
CPU utilization for five seconds: 11%/3%; one minute: 9%; five minutes: 10%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
3 124 130 953 1.67% 0.13% 0.02% 6 SSH Process
9 7713920 452047 17064 1.19% 0.47% 0.37% 0 Check heaps
306 45535792 118795990 383 0.79% 0.73% 0.73% 0 Earl NDE Task
327 18046456 124766366 144 0.71% 0.54% 0.50% 0 IP Input
558 9185532 35575478 258 0.71% 0.36% 0.35% 0 Port manager per
12 263195052 123093541 2138 0.47% 0.52% 0.55% 0 ARP Input
20:30
show ip arp summary
--------------
2587 IP ARP entries, with 165 of them incomplete
show process cpu sorted
--------------
CPU utilization for five seconds: 88%/29%; one minute: 82%; five minutes: 83%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
12 263956908 123363086 2139 42.39% 39.49% 40.28% 0 ARP Input
397 26557084 28759093 923 4.71% 4.33% 4.32% 0 XDR mcast
9 7721604 452477 17065 3.11% 0.52% 0.38% 0 Check heaps
306 45582404 118903810 383 2.79% 2.38% 2.00% 0 Earl NDE Task
3 168 130 1292 1.67% 0.13% 0.02% 6 SSH Process
21:00
show ip arp summary
--------------
2589 IP ARP entries, with 167 of them incomplete
show process cpu sorted
--------------
CPU utilization for five seconds: 81%/27%; one minute: 83%; five minutes: 79%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
12 265046164 123738558 2141 37.91% 39.18% 37.08% 0 ARP Input
397 26664908 28858941 923 5.43% 4.55% 4.26% 0 XDR mcast
306 45645772 119037173 383 3.11% 2.09% 1.95% 0 Earl NDE Task
3 176 130 1353 1.67% 0.13% 0.02% 6 SSH Process
327 18089292 124960978 144 1.11% 0.71% 0.70% 0 IP Input
21:30
show ip arp summary
--------------
2589 IP ARP entries, with 168 of them incomplete
--------------
show process cpu sorted
--------------
CPU utilization for five seconds: 98%/40%; one minute: 83%; five minutes: 79%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
12 266025804 124084184 2143 41.67% 37.30% 36.26% 0 ARP Input
397 26760700 28950445 924 4.07% 3.98% 3.92% 0 XDR mcast
24 18692320 127459398 146 1.75% 1.07% 0.83% 0 IPC Seat Manager
3 160 130 1230 1.67% 0.13% 0.02% 6 SSH Process
327 18110704 125057205 144 1.27% 0.78% 0.70% 0 IP Input
22:00
show ip arp summary
--------------
2586 IP ARP entries, with 164 of them incomplete
--------------
show process cpu sorted
--------------
CPU utilization for five seconds: 23%/9%; one minute: 65%; five minutes: 72%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
12 266948752 124452798 2144 7.27% 28.77% 32.71% 0 ARP Input
3 144 130 1107 1.67% 0.13% 0.02% 6 SSH Process
306 45782092 119315110 383 1.59% 2.04% 2.08% 0 Earl NDE Task
397 26863336 29062080 924 1.11% 3.87% 4.44% 0 XDR mcast
327 18132460 125159162 144 0.79% 0.66% 0.69% 0 IP Input
02-07-2012 10:28 AM
Hi ,
You can check :
sh ip traffic | s ARP
Also you can use control-plane policy for ARP :
mls qos protocol ARP police
Dan
02-07-2012 11:04 PM
Hi, thanks for your answer,
When I look for ip traffic, I see that Receive Replies is increasing rapidly..
***
C6509-251-DC#clear ip traffic
Clear "show ip traffic" counters [confirm]
C6509-251-DC#
C6509-251-DC#
C6509-251-DC#
C6509-251-DC#
C6509-251-DC#sh ip traffic | s ARP
ARP statistics:
Rcvd: 36 requests, 8199 replies, 3 reverse, 0 other
Sent: 14 requests, 8 replies (0 proxy), 0 reverse
Drop due to input queue full: 0
C6509-251-DC#
C6509-251-DC#
C6509-251-DC#sh ip traffic | s ARP
ARP statistics:
Rcvd: 77 requests, 21586 replies, 11 reverse, 0 other
Sent: 61 requests, 22 replies (0 proxy), 0 reverse
Drop due to input queue full: 0
C6509-251-DC#
C6509-251-DC#
C6509-251-DC#
C6509-251-DC#sh ip traffic | s ARP
ARP statistics:
Rcvd: 228 requests, 57298 replies, 24 reverse, 0 other
Sent: 213 requests, 77 replies (2 proxy), 0 reverse
Drop due to input queue full: 0
C6509-251-DC#
C6509-251-DC#
C6509-251-DC#
C6509-251-DC#
C6509-251-DC#sh ip traffic | s ARP
ARP statistics:
Rcvd: 335 requests, 82834 replies, 31 reverse, 0 other
Sent: 344 requests, 114 replies (5 proxy), 0 reverse
Drop due to input queue full: 0
C6509-251-DC#
C6509-251-DC#
C6509-251-DC#
C6509-251-DC#sh ip traffic | s ARP
ARP statistics:
Rcvd: 482 requests, 121919 replies, 46 reverse, 0 other
Sent: 491 requests, 166 replies (6 proxy), 0 reverse
Drop due to input queue full: 0
Erdal
02-07-2012 11:07 PM
At normal time (when CPU traffic is normal)
this ARP reply count is not increasing;
C6509-251-DC#show ip traffic | section ARP
ARP statistics:
Rcvd: 161 requests, 0 replies, 17 reverse, 0 other
Sent: 33 requests, 57 replies (2 proxy), 0 reverse
Drop due to input queue full: 0
C6509-251-DC#show ip traffic | section ARP
ARP statistics:
Rcvd: 199 requests, 0 replies, 22 reverse, 0 other
Sent: 38 requests, 75 replies (2 proxy), 0 reverse
Drop due to input queue full: 0
C6509-251-DC#show ip traffic | section ARP
ARP statistics:
Rcvd: 206 requests, 0 replies, 22 reverse, 0 other
Sent: 44 requests, 78 replies (2 proxy), 0 reverse
Drop due to input queue full: 0
C6509-251-DC#show ip traffic | section ARP
ARP statistics:
Rcvd: 211 requests, 0 replies, 23 reverse, 0 other
Sent: 46 requests, 78 replies (2 proxy), 0 reverse
Drop due to input queue full: 0
C6509-251-DC#show ip traffic | section ARP
ARP statistics:
Rcvd: 277 requests, 0 replies, 25 reverse, 0 other
Sent: 49 requests, 137 replies (35 proxy), 0 reverse
Drop due to input queue full: 0
C6509-251-DC#show ip traffic | section ARP
ARP statistics:
Rcvd: 295 requests, 0 replies, 25 reverse, 0 other
Sent: 61 requests, 143 replies (35 proxy), 0 reverse
Drop due to input queue full: 0
02-07-2012 11:45 PM
Hi ,
You could apply the ARP rate limit, but first of all you must know where are they from :
show interfaces accounting | i Ethe|ARP|Vlan
The second column will show you the number of packets IN. I would try using accounting identify the source VLAN and also source interface.
Dan
02-08-2012 05:19 AM
Hi,
I get this output;
Vlan1 VL001(MGMT)
Protocol PktsIn CharsIn PktsOut CharsOut
Other 160 9600 0 0
IP 388089272 282981891991 253012817 134920433824
DECMOP 418 32186 142 18318
ARP 47795 2960482 30521 3418352
Vlan2 VL002
Protocol PktsIn CharsIn PktsOut CharsOut
Other 10 600 0 0
IP 1933102896 727656700757 4009256393 5467437676205
DECMOP 125 9625 142 18318
ARP 82061561 4923744674 496931 55656272
and export to excel and sorted, I see the ARP input is coming from VLAN2, I will try to get info when the problem occour (22:00 PM at GMT+2)
Protocol | PktsIn | CharsIn | PktsOut | CharsOut | |
Vlan2 | ARP | 82061561 | 4923744674 | 496931 | 55656272 |
Vlan111 | ARP | 351433 | 21113488 | 56252 | 6300224 |
Vlan211 | ARP | 265659 | 15939540 | 100196 | 11221952 |
Vlan102 | ARP | 140205 | 8412432 | 18156 | 2033472 |
Vlan110 | ARP | 113548 | 6812880 | 78163 | 8754256 |
unfurtunately I cannot see any arp traffic on Layer2 ports, how can i find the source of this arp packet?
Whould I have to enable anything to apply ARP rate limit or "mls qos protocol ARP police" is enoug for
02-08-2012 05:20 AM
Hi,
I want to ask this, how can I clear this counters?
Regards..
02-08-2012 09:11 AM
Hi ,
Clear : clear counters
To enable arp rate-limit you need to enable "mls qos".
Dan
02-08-2012 11:53 AM
You might try , having in mind that you now know that the problem is on Vlan 2 , using :
show buffers input-interface vlan2
Dan
02-08-2012 12:00 PM
Hi,
I configured
mls qos protocol ARP police 64000
but it was distrupt the treaffic, so i had to be remove command.
i thing i has to be configure the correct value, which value whould i use?
this is the show buffers command;
C6509-251-DC#show buffers input-interface vlan 2
Header DataArea Pool Rcnt Size Link Enc Flags Input Output
47950894 802BCA4 Small 1 60 1 1 200 Vl2 None
47950C28 802BF64 Small 1 60 1 1 200 Vl2 None
479521A0 802CFE4 Small 1 60 1 1 200 Vl2 None
47952534 802D2A4 Small 1 60 1 1 200 Vl2 None
47953E40 802E5E4 Small 1 60 1 1 200 Vl2 None
4795659C 8030424 Small 1 60 1 1 200 Vl2 None
47956930 80306E4 Small 1 60 1 1 200 Vl2 None
4795823C 8031A24 Small 1 60 1 1 200 Vl2 None
4795A998 8033864 Small 1 60 1 1 200 Vl2 None
4795C9CC 8035124 Small 1 60 1 1 200 Vl2 None
4795E2D8 8036464 Small 1 60 1 1 200 Vl2 None
4795F128 8036F64 Small 1 60 1 1 200 Vl2 None
47964E30 803B6E4 Small 1 60 1 1 200 Vl2 None
47965C80 803C1E4 Small 1 60 1 1 200 Vl2 None
47966E64 803CFA4 Small 1 60 1 1 200 Vl2 None
4796A7A4 803FBA4 Small 1 60 1 1 200 Vl2 None
4796AECC 8040124 Small 1 60 1 1 200 Vl2 None
479704AC 8044324 Small 1 60 1 1 200 Vl2 None
47974180 80471E4 Small 1 60 1 1 200 Vl2 None
4797C978 804DA64 Small 1 60 1 1 200 Vl2 None
4797E618 804F064 Small 1 60 1 1 200 Vl2 None
479822EC 8051F24 Small 1 60 1 1 200 Vl2 None
479846B4 8053AA4 Small 1 60 1 1 200 Vl2 None
47989900 80579E4 Small 1 60 1 1 200 Vl2 None
4798C05C 8059824 Small 1 60 1 1 200 Vl2 None
4798D968 805AB64 Small 1 60 1 1 200 Vl2 None
4799248C 805E524 Small 1 60 1 1 200 Vl2 None
479956A4 8060BA4 Small 1 60 1 1 200 Vl2 None
47996FB0 8061EE4 Small 1 60 1 1 200 Vl2 None
479AC39C 8072424 Small 1 60 1 1 200 Vl2 None
479AC730 80726E4 Small 1 60 1 1 200 Vl2 None
479B197C 8076624 Small 1 60 1 1 200 Vl2 None
479B2EF4 80776A4 Small 1 60 1 1 200 Vl2 None
479B40D8 8078464 Small 1 60 1 1 200 Vl2 None
479B52BC 8079224 Small 1 60 1 1 200 Vl2 None
479B9DE0 807CBE4 Small 1 60 1 1 200 Vl2 None
479BAFC4 807D9A4 Small 1 60 1 1 200 Vl2 None
479BCC64 807EFA4 Small 1 60 1 1 200 Vl2 None
479BCFF8 807F264 Small 1 60 1 1 200 Vl2 None
Header DataArea Pool Rcnt Size Original Flags caller_pc
02-08-2012 12:05 PM
There is no "correct" value, the value is linked with how many hosts are behind this system.
Let's try:
int vlan 2
ip accounting mac-address input
exit
show int vl 2 mac-accounting
Dan
02-08-2012 12:29 PM
When i use this command i cant see any anomal trafic at input way, i see anarmous traffic on output side.
When I track this ip address, i saw that this is out backup servers mac address...
direction | mac | packets | bytes | last seen |
output | 0000.c9f2.f38a(66 ) | 3831844 | 5813M | last: 8340ms ago |
output | 0000.c9f2.f38e(70 ) | 3745234 | 5681M | last: 6252ms ago |
output | e41f.1331.4890(1 ) | 67631 | 33061779 | last: 6248ms ago |
output | 001a.64c1.d544(46 ) | 33941 | 10921794 | last: 6252ms ago |
output | 5cf3.fc0b.e724(155) | 31055 | 8412955 | last: 972ms ago |
output | e41f.13b3.b6f8(21 ) | 28986 | 14008037 | last: 6248ms ago |
Yes, backup server can be able to get great trafficbut how the 6500 can't be able to handle this size of traffic?
02-08-2012 01:13 PM
As I understood until now , you problem is INPUT traffic , explicitly ARP Input on Vlan 2.
So I do not belive that the accounted traffic is the one that generates the issue.
Also until now , I've understand that you issue is interminttent , am I right ?
Dan
02-08-2012 01:23 PM
Hi,
I see that you have a great deal of traffic punted to the CPU at the time of the issue, you can try to use
"debug netdr capture" to see what packets are getting to the CPU, it will give you more info on the flow
https://supportforums.cisco.com/docs/DOC-15608
cheers
Dejan
02-08-2012 11:24 PM
Hi,
Yes our problem is occours specially at backup time, between 20:30 PM to 22:30 PM and, 00:30 AM to 02:30 AM.
Other than this time range, we have no problem. Also this CPU problem is not interrupting normal traffic at this specific time range..
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide