cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
4780
Views
0
Helpful
17
Replies

HSRP Failover on Nexus 7k for no reason

We had major Network outage on April 9 and 10th due to HSRP failover on our distribution layer Nexus 7k switches. Few Nutanix hosts were unable to reach their gateway and as a result multiple applications that hosted these Nutanix environment were affected. Any idea what could be the reason of this behavior?

 

2019 Apr 9 22:22:05 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan1010 IPV4 Grp 1010 Standby router is 0.0.0.0
2019 Apr 9 22:22:05 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBYROUTERPRIORITYCHANGE: Interface Vlan1010 IPV4 Grp 1010 Standby router priority changed to 0
2019 Apr 9 22:22:05 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan1010 IPV4 Grp 1010 state changed from Standby to Active reason Act Timer E
xpired.
2019 Apr 9 22:22:05 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVE_CHANGE: Interface Vlan1010 IPV4 Grp 1010 Active router is 10.100.20.2
2019 Apr 9 22:22:05 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVEROUTERPRIORITY: Interface Vlan1010 IPV4 Grp 1010 Active router priority changed to 120
2019 Apr 9 22:22:05 AM-US-SAT_6-DD-35-BDRLFSW01 %ARP-3-DUP_VADDR_SRC_IP: arp [10967] Source address of packet received from 0000.0c9f.f1c9 on Vlan457(Gateway Port-Ch
annel1:1100) is duplicate of local virtual ip, 172.28.157.1
2019 Apr 9 22:22:05 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan452 IPV4 Grp 452 Standby router is 172.28.152.3
2019 Apr 9 22:22:06 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan3765 IPV4 Grp 3765 Standby router is 10.12.20.3
2019 Apr 9 22:22:06 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan458 IPV4 Grp 458 Standby router is 172.28.158.3
2019 Apr 9 22:22:06 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan951 IPV4 Grp 951 Standby router is 10.12.250.131
2019 Apr 9 22:22:06 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan3763 IPV4 Grp 3763 Standby router is 10.12.18.3
2019 Apr 9 22:22:06 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan560 IPV4 Grp 560 Standby router is 172.28.160.3
2019 Apr 9 22:22:06 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan3014 IPV4 Grp 3014 Standby router is 10.12.14.3
2019 Apr 9 22:22:06 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan3006 IPV4 Grp 3006 Standby router is 0.0.0.0
2019 Apr 9 22:22:06 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBYROUTERPRIORITYCHANGE: Interface Vlan3006 IPV4 Grp 3006 Standby router priority changed to 0
2019 Apr 9 22:22:06 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan3006 IPV4 Grp 3006 state changed from Standby to Active reason Act Timer E
xpired.
2019 Apr 9 22:22:06 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVE_CHANGE: Interface Vlan3006 IPV4 Grp 3006 Active router is 10.12.5.3
2019 Apr 9 22:22:06 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVEROUTERPRIORITY: Interface Vlan3006 IPV4 Grp 3006 Active router priority changed to 100
2019 Apr 9 22:22:07 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan439 IPV4 Grp 439 Standby router is 172.28.139.3
2019 Apr 9 22:22:07 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan1410 IPV4 Grp 1410 Standby router is 0.0.0.0
2019 Apr 9 22:22:07 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBYROUTERPRIORITYCHANGE: Interface Vlan1410 IPV4 Grp 1410 Standby router priority changed to 0
2019 Apr 9 22:22:07 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan1410 IPV4 Grp 1410 state changed from Standby to Active reason Act Timer E
xpired.
2019 Apr 9 22:22:07 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVE_CHANGE: Interface Vlan1410 IPV4 Grp 1410 Active router is 10.100.220.2
2019 Apr 9 22:22:07 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVEROUTERPRIORITY: Interface Vlan1410 IPV4 Grp 1410 Active router priority changed to 120
2019 Apr 9 22:22:07 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan410 IPV4 Grp 410 Standby router is 172.28.110.3
2019 Apr 9 22:22:07 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally c
onfigured address 172.28.219.1
2019 Apr 9 22:22:07 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan3764 IPV4 Grp 3764 Standby router is 10.12.19.3
2019 Apr 9 22:22:08 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan432 IPV4 Grp 432 Standby router is 172.28.132.3
2019 Apr 9 22:22:09 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVE_CHANGE: Interface Vlan1010 IPV4 Grp 1010 Active router is 10.100.20.3
2019 Apr 9 22:22:09 AM-US-SAT_6-DD-35-BDRLFSW01 last message repeated 1 time
2019 Apr 9 22:22:09 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVEROUTERPRIORITY: Interface Vlan1010 IPV4 Grp 1010 Active router priority changed to 120
2019 Apr 9 22:22:09 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan1010 IPV4 Grp 1010 state changed from Active to Speak reason Rx Hello-Hi P
ri Act.
2019 Apr 9 22:22:10 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally c
onfigured address 172.28.219.1
2019 Apr 9 22:22:11 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVE_CHANGE: Interface Vlan1410 IPV4 Grp 1410 Active router is 10.100.220.3
2019 Apr 9 22:22:11 AM-US-SAT_6-DD-35-BDRLFSW01 last message repeated 1 time
2019 Apr 9 22:22:11 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVEROUTERPRIORITY: Interface Vlan1410 IPV4 Grp 1410 Active router priority changed to 120
2019 Apr 9 22:22:11 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan1410 IPV4 Grp 1410 state changed from Active to Speak reason Rx Hello-Hi P
ri Act.
2019 Apr 9 22:22:13 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally c
onfigured address 172.28.219.1
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 last message repeated 2 times
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan1010 IPV4 Grp 1010 state changed from Speak to Standby reason Sby Timer Ex
pired.
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan1010 IPV4 Grp 1010 Standby router is 10.100.20.2
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBYROUTERPRIORITYCHANGE: Interface Vlan1010 IPV4 Grp 1010 Standby router priority changed to 120
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel411: first operational port changed from Ethernet1/12 to none
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel411: Ethernet1/12 is down
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 10, vPC 411 configuration is consistent
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel411 is down (No operational members)
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 10, vPC 411 configuration is consistent
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 last message repeated 1 time
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %STP-6-PORT_RANGE_DELETED: Interface port-channel411 removed from vlan=110,751-752
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %STP-6-PORT_RANGE_ROLE: new_role=designated interface=port-channel411 vlan=110,751-752
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 10, vPC 411 configuration is consistent
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %STP-6-PORT_RANGE_STATE: new_state=disabled interface=port-channel411 vlan=110
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %STP-6-PORT_RANGE_STATE: new_state=disabled interface=port-channel411 vlan=751-752
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 10, vPC 411 configuration is consistent
2019 Apr 9 22:22:21 AM-US-SAT_6-DD-35-BDRLFSW01 last message repeated 6 times
2019 Apr 9 22:22:21 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan1410 IPV4 Grp 1410 state changed from Speak to Standby reason Sby Timer Ex
pired.
2019 Apr 9 22:22:21 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan1410 IPV4 Grp 1410 Standby router is 10.100.220.2
2019 Apr 9 22:22:21 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBYROUTERPRIORITYCHANGE: Interface Vlan1410 IPV4 Grp 1410 Standby router priority changed to 120
2019 Apr 9 22:22:21 AM-US-SAT_6-DD-35-BDRLFSW01 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel411,bandwidth changed to 100000 Kbit
2019 Apr 9 22:22:21 AM-US-SAT_6-DD-35-BDRLFSW01 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/12 is down (Link failure)
2019 Apr 9 22:22:21 AM-US-SAT_6-DD-35-BDRLFSW01 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel411 is down (No operational members)
2019 Apr 9 22:22:21 AM-US-SAT_6-DD-35-BDRLFSW01 %CDP-5-NEIGHBOR_REMOVED: CDP Neighbor amussat-cube03.kci.com on port Ethernet1/12 has been removed
2019 Apr 9 22:22:21 AM-US-SAT_6-DD-35-BDRLFSW01 %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 10, vPC 411 configuration is consistent
2019 Apr 9 22:22:22 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally c
onfigured address 172.28.219.1
2019 Apr 9 22:22:22 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan3004 IPV4 Grp 3004 Standby router is 0.0.0.0
2019 Apr 9 22:22:22 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBYROUTERPRIORITYCHANGE: Interface Vlan3004 IPV4 Grp 3004 Standby router priority changed to 0
2019 Apr 9 22:22:22 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan3004 IPV4 Grp 3004 state changed from Standby to Active reason Act Timer E
xpired.
2019 Apr 9 22:22:22 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVE_CHANGE: Interface Vlan3004 IPV4 Grp 3004 Active router is 10.12.4.3
2019 Apr 9 22:22:22 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVEROUTERPRIORITY: Interface Vlan3004 IPV4 Grp 3004 Active router priority changed to 100
2019 Apr 9 22:22:25 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally c
onfigured address 172.28.219.1
2019 Apr 9 22:22:30 AM-US-SAT_6-DD-35-BDRLFSW01 last message repeated 1 time
2019 Apr 9 22:22:30 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan1010 IPV4 Grp 1010 Standby router is 0.0.0.0
2019 Apr 9 22:22:30 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBYROUTERPRIORITYCHANGE: Interface Vlan1010 IPV4 Grp 1010 Standby router priority changed to 0
2019 Apr 9 22:22:30 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan1010 IPV4 Grp 1010 state changed from Standby to Active reason Act Timer E
xpired.
2019 Apr 9 22:22:30 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVE_CHANGE: Interface Vlan1010 IPV4 Grp 1010 Active router is 10.100.20.2
2019 Apr 9 22:22:30 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVEROUTERPRIORITY: Interface Vlan1010 IPV4 Grp 1010 Active router priority changed to 120
2019 Apr 9 22:22:31 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally c
onfigured address 172.28.219.1
2019 Apr 9 22:22:32 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan1410 IPV4 Grp 1410 Standby router is 0.0.0.0
2019 Apr 9 22:22:32 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBYROUTERPRIORITYCHANGE: Interface Vlan1410 IPV4 Grp 1410 Standby router priority changed to 0
2019 Apr 9 22:22:32 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan1410 IPV4 Grp 1410 state changed from Standby to Active reason Act Timer E
xpired.
2019 Apr 9 22:22:32 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVE_CHANGE: Interface Vlan1410 IPV4 Grp 1410 Active router is 10.100.220.2
2019 Apr 9 22:22:32 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVEROUTERPRIORITY: Interface Vlan1410 IPV4 Grp 1410 Active router priority changed to 120
2019 Apr 9 22:22:34 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally c
onfigured address 172.28.219.1
2019 Apr 9 22:22:44 AM-US-SAT_6-DD-35-BDRLFSW01 last message repeated 3 times
2019 Apr 9 22:22:44 AM-US-SAT_6-DD-35-BDRLFSW01 %ETHPORT-5-SPEED: Interface Ethernet1/15, operational speed changed to 10 Gbps
2019 Apr 9 22:22:44 AM-US-SAT_6-DD-35-BDRLFSW01 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/15, operational duplex mode changed to Full
2019 Apr 9 22:22:44 AM-US-SAT_6-DD-35-BDRLFSW01 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/15, operational Receive Flow Control state changed to off
2019 Apr 9 22:22:44 AM-US-SAT_6-DD-35-BDRLFSW01 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/15, operational Transmit Flow Control state changed to off
2019 Apr 9 22:22:46 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally c
onfigured address 172.28.219.1
2019 Apr 9 22:22:47 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVE_CHANGE: Interface Vlan1010 IPV4 Grp 1010 Active router is 10.100.20.3
2019 Apr 9 22:22:47 AM-US-SAT_6-DD-35-BDRLFSW01 last message repeated 1 time
2019 Apr 9 22:22:47 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVEROUTERPRIORITY: Interface Vlan1010 IPV4 Grp 1010 Active router priority changed to 120
2019 Apr 9 22:22:47 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan1010 IPV4 Grp 1010 state changed from Active to Speak reason Rx Hello-Hi P
ri Act.
2019 Apr 9 22:22:49 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally c
onfigured address 172.28.219.1
2019 Apr 9 22:22:54 AM-US-SAT_6-DD-35-BDRLFSW01 last message repeated 4 times
2019 Apr 9 22:22:54 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVE_CHANGE: Interface Vlan599 IPV4 Grp 599 Active router is 172.28.219.4
2019 Apr 9 22:22:54 AM-US-SAT_6-DD-35-BDRLFSW01 last message repeated 1 time
2019 Apr 9 22:22:54 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVEROUTERPRIORITY: Interface Vlan599 IPV4 Grp 599 Active router priority changed to 120
2019 Apr 9 22:22:54 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan599 IPV4 Grp 599 state changed from Active to Speak reason Rx Hello-Hi Pri
Act.
2019 Apr 9 22:22:56 AM-US-SAT_6-DD-35-BDRLFSW01 %CDP-5-NEIGHBOR_ADDED: Device amussat-p6kvss.kci.com discovered of type cisco WS-C6506-E with port TenGigabitEthernet1/
5/15 on incoming port Ethernet1/15 with ip addr 192.168.67.1 and mgmt ip 192.168.67.1
2019 Apr 9 22:22:56 AM-US-SAT_6-DD-35-BDRLFSW01 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel499: Ethernet1/15 is up
2019 Apr 9 22:22:57 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally c
onfigured address 172.28.219.1
2019 Apr 9 22:22:57 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan1010 IPV4 Grp 1010 state changed from Speak to Standby reason Sby Timer Ex
pired.
2019 Apr 9 22:22:57 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan1010 IPV4 Grp 1010 Standby router is 10.100.20.2
2019 Apr 9 22:22:57 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBYROUTERPRIORITYCHANGE: Interface Vlan1010 IPV4 Grp 1010 Standby router priority changed to 120
2019 Apr 9 22:22:57 AM-US-SAT_6-DD-35-BDRLFSW01 %ETHPORT-5-IF_TRUNK_UP: Interface Ethernet1/15, vlan 1,27,75-76,84,110,112,127,246-249,251-255,261,299,400-427,429-459,
491-505,534,553,555,557,560,599,627,649,651,654,667,700,711 up
2019 Apr 9 22:22:57 AM-US-SAT_6-DD-35-BDRLFSW01 %ETHPORT-5-IF_TRUNK_UP: Interface Ethernet1/15, vlan 751-753,777,800,866,888-890,912,950-951,1005,1010,1103-1104,1305,1
403,1410,1710,1722,1728,3004-3006,3012,3014,3751-3771 up
2019 Apr 9 22:22:57 AM-US-SAT_6-DD-35-BDRLFSW01 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel499,bandwidth changed to 20000000 Kbit
2019 Apr 9 22:22:57 AM-US-SAT_6-DD-35-BDRLFSW01 %ETHPORT-5-IF_UP: Interface Ethernet1/15 is up in mode trunk
2019 Apr 9 22:22:59 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally c
onfigured address 172.28.219.1
2019 Apr 9 22:23:05 AM-US-SAT_6-DD-35-BDRLFSW01 last message repeated 1 time
2019 Apr 9 22:23:05 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan599 IPV4 Grp 599 state changed from Speak to Standby reason Sby Timer Expi
red.

17 Replies 17

Jaderson Pessoa
VIP Alumni
VIP Alumni

sharath.babu9022@gmail.com  hello,

 

I have read line by line and there are many insconsistency on your configuration, like it;

 

2019 Apr 9 22:22:13 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally configured address 172.28.219.1

 

2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan1010 IPV4 Grp 1010 state changed from Speak to Standby reason Sby Timer Expired



I think that you have had wrong configuration between your peers hsrp. Could you provide the configuration on both devices?

 

 

Thanks in advance.

Jaderson Pessoa
*** Rate All Helpful Responses ***

balaji.bandi
Hall of Fame
Hall of Fame

As suggested you have inconsistance config on HSRP, please post the both the switchs HSRP config to have look.

 

is this 7K configured as vPC ?

 

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Hi Balaji and Jaderson,

 

Thanks a ton for looking into this issue. As per your request, I have uploaded the HSRP and vPc Set up, please have a look.

i suggest to send full running config of both the switch along with show spanning-tree output.

 

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Attached running and spanning tree output.

sharath.babu9022@gmail.com , could you share your spanning-tre configuration and ip of the virtual interfaces (vlans)

 

maybe you have a spanning-tre issue or a routing issue, but we need more information to suggest you the better solution.

 

Thanks in advance.

Jaderson Pessoa
*** Rate All Helpful Responses ***

I have attached the requested information.

i think those attached files has just piece of information about hsrp and vpc.
Jaderson Pessoa
*** Rate All Helpful Responses ***

Could you please let me know if you need more outputs to investigate further? I am suspecting that there was a connectivity loss between the active and standy HSRP routers and both became active at the same time and might have resulted in the outage. Possibility?

Hello,

 

post the output of:

 

sh run | include vlan 599

 

from both switches...

AM-US-SAT_6-DD-35-BDRLFSW01# sh run vlan 599

!Command: show running-config vlan 599
!Time: Mon Apr 15 02:17:25 2019

version 6.2(16)
vlan 599
vlan 599
mode fabricpath
name VMWARE-5-1-PROD

 

AM-US-SAT_6-GD-35-BDRLFSW02# sh running-config vlan 599

!Command: show running-config vlan 599
!Time: Mon Apr 15 02:18:25 2019

version 6.2(16)
vlan 599
vlan 599
mode fabricpath
name VMWARE-5-1-PROD

Thanks Georg! I have uploaded the requested information.

On the Quick look i did not see any issue in terms of config.

But we are not sure how your Esxi  / Physical connected, 

 

If you lost connection between switches there may be chance of Split brian situation, that can only view with log files.

 

Nexus have good log storage space. pull the logs for the incident time and post here to look, or you can also verify same.

 

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Problem Description :

====================

[+] Few servers were not able to reach its gateway (172.28.219.1) in VLAN 599.

 

Action Taken :

============

 

[+] Noticed HSRP for vlan 599 is standby and listen in our Datacenter.

 

AM-US-SAT_6-GD-35-BDRLFSW02# sh hsrp interface vlan 599 brief    (This is our Datacenter switch)

*:IPv6 group   #:group belongs to a bundle

                     P indicates configured to preempt.

                     |

Interface   Grp  Prio P State    Active addr      Standby addr     Group addr

  Vlan599     599  100  P Listen   172.28.219.4     172.28.219.2     172.28.219.1        (conf)

 

 

AM-US-SAT_6-DD-35-BDRLFSW01# sh hsrp interface vlan 599 brief            (This is our Datacenter switch)

*:IPv6 group   #:group belongs to a bundle

                     P indicates configured to preempt.

                     |

Interface   Grp  Prio P State    Active addr      Standby addr     Group addr

  Vlan599     599  120  P Standby  172.28.219.4     local            172.28.219.1    (conf)              << was standby

AM-US-SAT_6-DD-35-BDRLFSW01#

 

 

[+] HSRP was active in another DC as seen below also group address is different which case all servers/application to stops working.

 

AM-US-SCF_01-0726-35-BDRLFSW01# sh hsrp interface vlan 599 brief       (This is our disaster recovery Datacenter switch)

*:IPv6 group   #:group belongs to a bundle

                     P indicates configured to preempt.

                     |

Interface   Grp  Prio P State    Active addr      Standby addr     Group addr

  Vlan599     599  120  P Active   local            172.28.219.2     172.28.219.254  (conf)              << now become active..

 

 

[+] And VIP of newly active is different than Active one in problematic DC.

 

show logging log :

-----------------

2019 Mar 28 21:58:32 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally configured address 172.28.219.1

2019 Mar 28 21:58:33 AM-US-SAT_6-DD-35-BDRLFSW01  %AUTHPRIV-3-SYSTEM_MSG: pam_aaa:Authentication failed from 172.28.219.209 - sshd

2019 Mar 28 21:58:34 AM-US-SAT_6-DD-35-BDRLFSW01  %USER-2-SYSTEM_MSG: fatal: Write failed: Broken pipe .Client is 172.28.219.209,length of packet causing error 496 496  - sshd

2019 Mar 28 21:58:35 AM-US-SAT_6-DD-35-BDRLFSW01  %AUTHPRIV-3-SYSTEM_MSG: pam_aaa:Authentication failed from 172.28.219.209 - sshd

2019 Mar 28 21:58:35 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally configured address 172.28.219.1

2019 Mar 28 21:58:50 AM-US-SAT_6-DD-35-BDRLFSW01 last message repeated 4 times

2019 Mar 28 21:58:50 AM-US-SAT_6-DD-35-BDRLFSW01  %AUTHPRIV-3-SYSTEM_MSG: pam_aaa:Authentication failed from 172.28.219.209 - sshd

2019 Mar 28 21:58:50 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally configured address 172.28.219.1

2019 Mar 28 21:58:52 AM-US-SAT_6-DD-35-BDRLFSW01  %USER-2-SYSTEM_MSG: fatal: Write failed: Broken pipe .Client is 172.28.219.209,length of packet causing error 496 496  - sshd

2019 Mar 28 21:58:53 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally configured address 172.28.219.1

2019 Mar 28 21:58:53 AM-US-SAT_6-DD-35-BDRLFSW01  %AUTHPRIV-3-SYSTEM_MSG: pam_aaa:Authentication failed from 172.28.219.209 - sshd

2019 Mar 28 21:58:56 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally configured address 172.28.219.1

 

 

[+] Increase HSRP priority on BDRLFSW01, results in to become HSRP active on this device.

[+] After changing the priority everything starts working.

 

 

[+] Server which were not pingable starts pinging.

 

AM-US-SAT_6-DD-35-BDRLFSW01# ping 172.28.219.207

PING 172.28.219.207 (172.28.219.207): 56 data bytes

64 bytes from 172.28.219.207: icmp_seq=0 ttl=127 time=0.995 ms

64 bytes from 172.28.219.207: icmp_seq=1 ttl=127 time=0.849 ms

64 bytes from 172.28.219.207: icmp_seq=2 ttl=127 time=0.792 ms

64 bytes from 172.28.219.207: icmp_seq=3 ttl=127 time=1.411 ms

64 bytes from 172.28.219.207: icmp_seq=4 ttl=127 time=1.794 ms

 

--- 172.28.219.207 ping statistics ---

5 packets transmitted, 5 packets received, 0.00% packet loss

round-trip min/avg/max = 0.792/1.168/1.794 ms

 

VLAN 599 is spanned between our datacenter at San Antonio and DR at Scottsdale with OTV. The gateway for VLAN 599 is .1 in San Antonio but it is .254 in DR. This is by design.

 

We are trying to understand why did the HSRP events occur on our Nexus Environment on April 7th? And what made VLAN 599 to become active at our DR and listen & standy state in our DR.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: