Hey guys, I'm running a pair of Cisco Nexus 9K's in HSRP mode as the core routers. I created a a 10.29.0.0/19 subnet to accomodate our ever-increasing list of BOYD devices. I noticed that the latency is extremely high when pinging clients in this subnet:
64 bytes from 10.29.28.184: icmp_seq=6627 ttl=63 time=424.914 ms 64 bytes from 10.29.28.184: icmp_seq=6628 ttl=63 time=289.047 ms 64 bytes from 10.29.28.184: icmp_seq=6629 ttl=63 time=219.505 ms 64 bytes from 10.29.28.184: icmp_seq=6630 ttl=63 time=396.309 ms 64 bytes from 10.29.28.184: icmp_seq=6631 ttl=63 time=251.964 ms 64 bytes from 10.29.28.184: icmp_seq=6632 ttl=63 time=379.557 ms 64 bytes from 10.29.28.184: icmp_seq=6633 ttl=63 time=549.037 ms 64 bytes from 10.29.28.184: icmp_seq=6634 ttl=63 time=436.171 ms 64 bytes from 10.29.28.184: icmp_seq=6635 ttl=63 time=635.576 ms 64 bytes from 10.29.28.184: icmp_seq=6636 ttl=63 time=470.314 ms 64 bytes from 10.29.28.184: icmp_seq=6637 ttl=63 time=390.080 ms 64 bytes from 10.29.28.184: icmp_seq=6638 ttl=63 time=304.494 ms 64 bytes from 10.29.28.184: icmp_seq=6639 ttl=63 time=478.791 ms 64 bytes from 10.29.28.184: icmp_seq=6640 ttl=63 time=556.774 ms 64 bytes from 10.29.28.184: icmp_seq=6641 ttl=63 time=469.451 ms 64 bytes from 10.29.28.184: icmp_seq=6642 ttl=63 time=1564.194 ms 64 bytes from 10.29.28.184: icmp_seq=6643 ttl=63 time=559.249 ms 64 bytes from 10.29.28.184: icmp_seq=6644 ttl=63 time=423.030 ms
I took a PCAP and discovered there are tons of arp requests from devices asking who the default gateway is (ex: Who has 10.29.0.1? Tell 10.29.28.53). It looks like the mac address of the default gateway is not being resolved. Could this be the culprit to our latency issue? I am able to ping 10.29.0.1 but this address only shows as active in the arp table on the standby router for some reason.
Address Age MAC Address Interface Flags 10.29.0.1 00:07:48 0000.0c9f.f320 Vlan800
Here is our VLAN configuration. The standby router looks identical for the most part with the exception of the interface IP and HSRP priority (10.29.31.252, priority 125). Any help would be much appreciated. Thanks in advance!
IP Interface Status for VRF "default"(1) Vlan800, Interface status: protocol-up/link-up/admin-up, iod: 139, IP address: 10.29.31.251, IP subnet: 10.29.0.0/19 route-preference: 0, tag: 0 IP broadcast address: 255.255.255.255 IP multicast groups locally joined: 220.127.116.11 IP MTU: 1500 bytes (using link MTU) IP primary address route-preference: 0, tag: 0 IP proxy ARP : disabled IP Local Proxy ARP : disabled IP multicast routing: disabled IP icmp redirects: disabled IP directed-broadcast: disabled IP Forwarding: disabled IP icmp unreachables (except port): disabled IP icmp port-unreachable: enabled IP unicast reverse path forwarding: none IP load sharing: none IP interface statistics last reset: never IP interface software stats: (sent/received/forwarded/originated/consumed) Unicast packets : 62804/155833/646/62238/359075 Unicast bytes : 42174958/68353903/187096/41994582/159289835 Multicast packets : 1/2149160/0/1/753830 Multicast bytes : 82/369070078/0/82/30153200 Broadcast packets : 0/0/0/0/0 Broadcast bytes : 0/0/0/0/0 Labeled packets : 0/0/0/0/0 Labeled bytes : 0/0/0/0/0 WCCP Redirect outbound: disabled WCCP Redirect inbound: disabled WCCP Redirect exclude: disabled
Vlan800 - Group 800 (HSRP-V2) (IPv4) Local state is Active, priority 150 (Cfged 150), may preempt Forwarding threshold(for vPC), lower: 0 upper: 150 Hellotime 3 sec, holdtime 10 sec Next hello sent in 1.556000 sec(s) Virtual IP address is 10.29.0.1 (Cfged) Active router is local Standby router is 10.29.31.252 , priority 125 expires in 7.817000 sec(s) Authentication text "cisco" Virtual mac address is 0000.0c9f.f320 (Default MAC) 8 state changes, last state change 1w0d IP redundancy name is hsrp-Vlan800-800 (default)
interface Vlan800 description no shutdown no ip redirects ip address 10.29.31.251/19 ip ospf passive-interface ip router ospf 100 area 0.0.0.0 hsrp version 2 hsrp 800 preempt priority 150 ip 10.29.0.1 ip dhcp relay address 172.18.0.19 ip dhcp relay address 172.18.0.127
RTT wouldn't be much affected by an ARP request-reply since that would happen once per ARP flush time (at most). Besides, that should take a handful of milliseconds if the device is on the same LAN.
I'd consider the following options:
1) Pinging the IOT device from a seperate device on the same LAN as the HSRP switches. That will confirm if the latency has anything to do with the Nexus switches.
2.1) If 1 returns that the high latency only exists with the Nexus switches involved, then you may have a performance issue with those switches. Could be you need to use a CoPP that will remediate the issue.
2.2) If 1 returns that there is always high latency, start checking where along the path that latency no longer exists. Could be that ICMP in general is being policied somewhere, being queued for some time, and that's affecting your RTT.
Agile Networking with Cloud-managed IT [Meraki The Future is here] Cisco Meraki The Future Is Here: Agile Networking With Cloud-managed IT Your customer’s employees are in hybrid workstyle from home, the road, and the office and IT teams need t...
1. Smart Licensing
1.1. What is Smart Licensing?
A. Cisco Smart Licensing is a flexible licensing model that provides you with an easier, faster, and more consistent way to purchase and manage software across the Cisco portfolio and across your orga...
Cisco DNA - Cisco SD-WAN: Connect to any cloud, anywhere, securely Cisco offers on-prem and cloud-managed WAN edge solutions to meet these new demands. Connect any user to any application with integrated capabilities for multicloud, security, ...
Join us for this virtual event as cloud providers, integrators, ecosystem technology partners and customers discuss what tomorrow's cloud will be and what you need to know to prepare. Get ready to hear about innovations for faster operations, mult...
Let´s talk about spanning-treeLeave your comment in english and in your own language.------------------------------------------------Vamos falar sobre spanning-tree Deixe seu comentário em inglês e no seu idioma.