7609-S - High CPU - netdr - ARP Flood

[FrankyF]

Hi Cisco community,

Recently we had a customer flooding one of our routers with ARP requests, which led to High CPU on the device, plus instabilities on BGP for the rest of the customers.

To resolve it we shutted-down the sub-interface of the customer, few days later a field engineer has just reseted the NTU before we bring the interface up again. We never found what the root cause was due to EOL EOF.

Any idea how we could prevent this from happening again in the future? Maybe "storm-control broadcast"?

Few logs for your reference:

#################################################################

CPU utilization for five seconds: 99%/53%; one minute: 99%; five minutes: 99%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
12 44889508 162607466 276 37.77% 40.06% 40.01% 0 ARP Input
563 1111170872 38756609 28670 3.91% 1.77% 1.38% 0 BGP Scanner

#################################################################

------- dump of incoming inband packet -------
interface Te1/2.2113, routine process_rx_packet_inline, timestamp 08:15:08.472
dbus info: src_vlan 0x647(1607), src_indx 0x1(1), len 0x40(64)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x4647(17991)
E8020400 06470000 00010000 40000000 E0000540 01000040 00000008 46470000 
destmac FF.FF.FF.FF.FF.FF, srcmac C8.9C.1D.19.31.40, protocol 0806
layer 3 data: 00010800 06040001 C89C1D19 3140D989 90530000 00000000
D9899052 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000647

------- dump of incoming inband packet -------
interface Te1/2.2113, routine process_rx_packet_inline, timestamp 08:15:08.472
dbus info: src_vlan 0x647(1607), src_indx 0x1(1), len 0x40(64)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x4647(17991)
F0020400 06470000 00010000 40000000 E0000540 01000040 00000008 46470000 
destmac FF.FF.FF.FF.FF.FF, srcmac C8.9C.1D.19.31.40, protocol 0806
layer 3 data: 00010800 06040001 C89C1D19 3140D989 90530000 00000000
D9899052 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000647

#################################################################

Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 2 7600 ES+ 76-ES+XT-2TG3CXL SAL1XXXXXXX
2 4 CEF720 4 port 10-Gigabit Ethernet WS-X6704-10GE SAL1XXXXXXX
3 4 CEF720 4 port 10-Gigabit Ethernet WS-X6704-10GE SAL1XXXXXXX
4 2 7600 ES+ 76-ES+XT-2TG3CXL SAL1XXXXXXX
5 2 Route Switch Processor 720 (Active) RSP720-3CXL-GE SAL1XXXXXXX
6 2 Route Switch Processor 720 (Hot) RSP720-3CXL-GE SAL1XXXXXXX
7 4 CEF720 4 port 10-Gigabit Ethernet WS-X6704-10GE SAL1XXXXXXX
8 4 CEF720 4 port 10-Gigabit Ethernet WS-X6704-10GE SAL1XXXXXXX
9 4 CEF720 4 port 10-Gigabit Ethernet WS-X6704-10GE SAL1XXXXXXX

Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
1 a0ec.f9bd.38c0 to a0ec.f9bd.38cf 3.0 12.2(33r)SRE 12.2(33)SRE6 Ok
2 d0d0.fd62.f3f4 to d0d0.fd62.f3f7 3.2 12.2(14r)S5 12.2(33)SRE6 Ok
3 001a.6cf5.dcf4 to 001a.6cf5.dcf7 2.5 12.2(14r)S5 12.2(33)SRE6 Ok
4 1005.ca77.8690 to 1005.ca77.869f 1.7 12.2(33r)SRE 12.2(33)SRE6 Ok
5 0022.bdec.abd4 to 0022.bdec.abd7 5.13 12.2(33r)SRE 12.2(33)SRE6 Ok
6 f40f.1b7a.025c to f40f.1b7a.025f 6.1 12.2(33r)SRE 12.2(33)SRE6 Ok
7 6c20.5699.ef0c to 6c20.5699.ef0f 3.2 12.2(14r)S5 12.2(33)SRE6 Ok
8 6c20.5699.b6f8 to 6c20.5699.b6fb 3.2 12.2(14r)S5 12.2(33)SRE6 Ok
9 001a.a10e.d34c to 001a.a10e.d34f 2.5 12.2(14r)S5 12.2(33)SRE6 Ok

Mod Sub-Module Model Serial Hw Status 
---- --------------------------- ------------------ ----------- ------- -------
1 7600 ES+ DFC XL 7600-ES+3CXL SAL1XXXXXXX 3.0 Ok
1 7600 ES+ 2x10GE XFP ITU 7600-ES+ITU-2TG-LK SAL1XXXXXXX 3.0 Ok
2 Distributed Forwarding Card WS-F6700-DFC3CXL SAL1XXXXXXX 1.0 Ok
3 Distributed Forwarding Card WS-F6700-DFC3CXL SAL1XXXXXXX 1.7 Ok
4 7600 ES+ DFC XL 7600-ES+3CXL SAL1XXXXXXX 1.4 Ok
4 7600 ES+ 2x10GE XFP ITU 7600-ES+ITU-2TG-LK SAL1XXXXXXX 1.2 Ok
5 Policy Feature Card 3 7600-PFC3CXL SAL1XXXXXXX 1.1 Ok
5 C7600 MSFC4 Daughterboard 7600-MSFC4 SAL1XXXXXXX 5.0 Ok
6 Policy Feature Card 3 7600-PFC3CXL SAL1XXXXXXX 2.0 Ok
6 C7600 MSFC4 Daughterboard 7600-MSFC4 SAL1XXXXXXX 8.0 Ok
7 Distributed Forwarding Card WS-F6700-DFC3CXL SAL1XXXXXXX 1.7 Ok
8 Distributed Forwarding Card WS-F6700-DFC3CXL SAL1XXXXXXX 1.7 Ok
9 Distributed Forwarding Card WS-F6700-DFC3CXL SAL1XXXXXXX 1.7 Ok

#################################################################

interface TenGigabitEthernet1/2
description "xxxxxxxx"
mtu 4470
no ip address
load-interval 30
no cdp enable
end
!
!
interface TenGigabitEthernet1/2.2113
description CFL CAL0xxxxx
bandwidth 30000
encapsulation dot1Q 2113
ip address 2xx.xxx.xxx.xx 255.255.255.254
ip access-group 179 in
no ip redirects
no ip unreachables
no ip proxy-arp
no cdp enable
service-policy input MIA-30Mb-Parent
service-policy output MIA-30Mb-Parent
end

#################################################################

Thank you in advance

[MHM Cisco World]

I see Bgp scanner in cpu, please check cisco where there are some case and there are restrictions for bgp scanner,

Bgp scanner can lead to high cpu utilize.

[FrankyF]

Hi,

Thanks for your time to reply.

The cause of the High CPU has been confirmed by Cisco that is caused from the ARP flooding.

But as said, because that device has dozens of BGP peerings the high CPU caused instabilities on them, hence you see BGP consuming CPU too.

Because of EOF EOS, Cisco cannot recommend a workaround.

My question is on how we can prevent ARP Flooding on the customer facing interfaces ?

Thank you

[Georg Pauwen]

Hello,

dynamic ARP inspection is typically configured to prevent this, I don't know if that feature is available on your platform, as it is usually configured at the Vlan level.

Did you get an idea of where the ARP floods came from ?

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/release/12-2_52_se/configuration/guide/3750scg/swdynarp.html

[MHM Cisco World]

https://www.cisco.com/c/en/us/support/docs/dial-access/floating-static-route/118263-technote-nexthop-00.html

if the costumer config Static route toward you.

[paul driver · Accepted solution]

Hello
To negate any BUM traffic it can be done via with L2 storm control or port security, thats one of the reasons we have L2 WAN Handoff switches between us and the ISP’s
Also if the client is pointing a default towards it wan interface with a next hop of the physical port instead of specifying an next hop ip address then the rtr will "arp" for everything it wants to connect to via that interface – not good!