Hey guys, I'm running a pair of Cisco Nexus 9K's in HSRP mode as the core routers. I created a a 10.29.0.0/19 subnet to accomodate our ever-increasing list of BOYD devices. I noticed that the latency is extremely high when pinging clients in this subnet:
64 bytes from 10.29.28.184: icmp_seq=6627 ttl=63 time=424.914 ms
64 bytes from 10.29.28.184: icmp_seq=6628 ttl=63 time=289.047 ms
64 bytes from 10.29.28.184: icmp_seq=6629 ttl=63 time=219.505 ms
64 bytes from 10.29.28.184: icmp_seq=6630 ttl=63 time=396.309 ms
64 bytes from 10.29.28.184: icmp_seq=6631 ttl=63 time=251.964 ms
64 bytes from 10.29.28.184: icmp_seq=6632 ttl=63 time=379.557 ms
64 bytes from 10.29.28.184: icmp_seq=6633 ttl=63 time=549.037 ms
64 bytes from 10.29.28.184: icmp_seq=6634 ttl=63 time=436.171 ms
64 bytes from 10.29.28.184: icmp_seq=6635 ttl=63 time=635.576 ms
64 bytes from 10.29.28.184: icmp_seq=6636 ttl=63 time=470.314 ms
64 bytes from 10.29.28.184: icmp_seq=6637 ttl=63 time=390.080 ms
64 bytes from 10.29.28.184: icmp_seq=6638 ttl=63 time=304.494 ms
64 bytes from 10.29.28.184: icmp_seq=6639 ttl=63 time=478.791 ms
64 bytes from 10.29.28.184: icmp_seq=6640 ttl=63 time=556.774 ms
64 bytes from 10.29.28.184: icmp_seq=6641 ttl=63 time=469.451 ms
64 bytes from 10.29.28.184: icmp_seq=6642 ttl=63 time=1564.194 ms
64 bytes from 10.29.28.184: icmp_seq=6643 ttl=63 time=559.249 ms
64 bytes from 10.29.28.184: icmp_seq=6644 ttl=63 time=423.030 ms
I took a PCAP and discovered there are tons of arp requests from devices asking who the default gateway is (ex: Who has 10.29.0.1? Tell 10.29.28.53). It looks like the mac address of the default gateway is not being resolved. Could this be the culprit to our latency issue? I am able to ping 10.29.0.1 but this address only shows as active in the arp table on the standby router for some reason.
Address Age MAC Address Interface Flags
10.29.0.1 00:07:48 0000.0c9f.f320 Vlan800
Here is our VLAN configuration. The standby router looks identical for the most part with the exception of the interface IP and HSRP priority (10.29.31.252, priority 125). Any help would be much appreciated. Thanks in advance!
IP Interface Status for VRF "default"(1)
Vlan800, Interface status: protocol-up/link-up/admin-up, iod: 139,
IP address: 10.29.31.251, IP subnet: 10.29.0.0/19 route-preference: 0, tag: 0
IP broadcast address: 255.255.255.255
IP multicast groups locally joined:
224.0.0.102
IP MTU: 1500 bytes (using link MTU)
IP primary address route-preference: 0, tag: 0
IP proxy ARP : disabled
IP Local Proxy ARP : disabled
IP multicast routing: disabled
IP icmp redirects: disabled
IP directed-broadcast: disabled
IP Forwarding: disabled
IP icmp unreachables (except port): disabled
IP icmp port-unreachable: enabled
IP unicast reverse path forwarding: none
IP load sharing: none
IP interface statistics last reset: never
IP interface software stats: (sent/received/forwarded/originated/consumed)
Unicast packets : 62804/155833/646/62238/359075
Unicast bytes : 42174958/68353903/187096/41994582/159289835
Multicast packets : 1/2149160/0/1/753830
Multicast bytes : 82/369070078/0/82/30153200
Broadcast packets : 0/0/0/0/0
Broadcast bytes : 0/0/0/0/0
Labeled packets : 0/0/0/0/0
Labeled bytes : 0/0/0/0/0
WCCP Redirect outbound: disabled
WCCP Redirect inbound: disabled
WCCP Redirect exclude: disabled
Vlan800 - Group 800 (HSRP-V2) (IPv4)
Local state is Active, priority 150 (Cfged 150), may preempt
Forwarding threshold(for vPC), lower: 0 upper: 150
Hellotime 3 sec, holdtime 10 sec
Next hello sent in 1.556000 sec(s)
Virtual IP address is 10.29.0.1 (Cfged)
Active router is local
Standby router is 10.29.31.252 , priority 125 expires in 7.817000 sec(s)
Authentication text "cisco"
Virtual mac address is 0000.0c9f.f320 (Default MAC)
8 state changes, last state change 1w0d
IP redundancy name is hsrp-Vlan800-800 (default)
interface Vlan800
description
no shutdown
no ip redirects
ip address 10.29.31.251/19
ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
hsrp version 2
hsrp 800
preempt
priority 150
ip 10.29.0.1
ip dhcp relay address 172.18.0.19
ip dhcp relay address 172.18.0.127
