04-13-2019 04:14 AM
We had major Network outage on April 9 and 10th due to HSRP failover on our distribution layer Nexus 7k switches. Few Nutanix hosts were unable to reach their gateway and as a result multiple applications that hosted these Nutanix environment were affected. Any idea what could be the reason of this behavior?
2019 Apr 9 22:22:05 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan1010 IPV4 Grp 1010 Standby router is 0.0.0.0
2019 Apr 9 22:22:05 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBYROUTERPRIORITYCHANGE: Interface Vlan1010 IPV4 Grp 1010 Standby router priority changed to 0
2019 Apr 9 22:22:05 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan1010 IPV4 Grp 1010 state changed from Standby to Active reason Act Timer E
xpired.
2019 Apr 9 22:22:05 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVE_CHANGE: Interface Vlan1010 IPV4 Grp 1010 Active router is 10.100.20.2
2019 Apr 9 22:22:05 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVEROUTERPRIORITY: Interface Vlan1010 IPV4 Grp 1010 Active router priority changed to 120
2019 Apr 9 22:22:05 AM-US-SAT_6-DD-35-BDRLFSW01 %ARP-3-DUP_VADDR_SRC_IP: arp [10967] Source address of packet received from 0000.0c9f.f1c9 on Vlan457(Gateway Port-Ch
annel1:1100) is duplicate of local virtual ip, 172.28.157.1
2019 Apr 9 22:22:05 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan452 IPV4 Grp 452 Standby router is 172.28.152.3
2019 Apr 9 22:22:06 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan3765 IPV4 Grp 3765 Standby router is 10.12.20.3
2019 Apr 9 22:22:06 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan458 IPV4 Grp 458 Standby router is 172.28.158.3
2019 Apr 9 22:22:06 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan951 IPV4 Grp 951 Standby router is 10.12.250.131
2019 Apr 9 22:22:06 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan3763 IPV4 Grp 3763 Standby router is 10.12.18.3
2019 Apr 9 22:22:06 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan560 IPV4 Grp 560 Standby router is 172.28.160.3
2019 Apr 9 22:22:06 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan3014 IPV4 Grp 3014 Standby router is 10.12.14.3
2019 Apr 9 22:22:06 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan3006 IPV4 Grp 3006 Standby router is 0.0.0.0
2019 Apr 9 22:22:06 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBYROUTERPRIORITYCHANGE: Interface Vlan3006 IPV4 Grp 3006 Standby router priority changed to 0
2019 Apr 9 22:22:06 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan3006 IPV4 Grp 3006 state changed from Standby to Active reason Act Timer E
xpired.
2019 Apr 9 22:22:06 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVE_CHANGE: Interface Vlan3006 IPV4 Grp 3006 Active router is 10.12.5.3
2019 Apr 9 22:22:06 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVEROUTERPRIORITY: Interface Vlan3006 IPV4 Grp 3006 Active router priority changed to 100
2019 Apr 9 22:22:07 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan439 IPV4 Grp 439 Standby router is 172.28.139.3
2019 Apr 9 22:22:07 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan1410 IPV4 Grp 1410 Standby router is 0.0.0.0
2019 Apr 9 22:22:07 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBYROUTERPRIORITYCHANGE: Interface Vlan1410 IPV4 Grp 1410 Standby router priority changed to 0
2019 Apr 9 22:22:07 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan1410 IPV4 Grp 1410 state changed from Standby to Active reason Act Timer E
xpired.
2019 Apr 9 22:22:07 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVE_CHANGE: Interface Vlan1410 IPV4 Grp 1410 Active router is 10.100.220.2
2019 Apr 9 22:22:07 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVEROUTERPRIORITY: Interface Vlan1410 IPV4 Grp 1410 Active router priority changed to 120
2019 Apr 9 22:22:07 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan410 IPV4 Grp 410 Standby router is 172.28.110.3
2019 Apr 9 22:22:07 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally c
onfigured address 172.28.219.1
2019 Apr 9 22:22:07 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan3764 IPV4 Grp 3764 Standby router is 10.12.19.3
2019 Apr 9 22:22:08 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan432 IPV4 Grp 432 Standby router is 172.28.132.3
2019 Apr 9 22:22:09 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVE_CHANGE: Interface Vlan1010 IPV4 Grp 1010 Active router is 10.100.20.3
2019 Apr 9 22:22:09 AM-US-SAT_6-DD-35-BDRLFSW01 last message repeated 1 time
2019 Apr 9 22:22:09 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVEROUTERPRIORITY: Interface Vlan1010 IPV4 Grp 1010 Active router priority changed to 120
2019 Apr 9 22:22:09 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan1010 IPV4 Grp 1010 state changed from Active to Speak reason Rx Hello-Hi P
ri Act.
2019 Apr 9 22:22:10 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally c
onfigured address 172.28.219.1
2019 Apr 9 22:22:11 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVE_CHANGE: Interface Vlan1410 IPV4 Grp 1410 Active router is 10.100.220.3
2019 Apr 9 22:22:11 AM-US-SAT_6-DD-35-BDRLFSW01 last message repeated 1 time
2019 Apr 9 22:22:11 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVEROUTERPRIORITY: Interface Vlan1410 IPV4 Grp 1410 Active router priority changed to 120
2019 Apr 9 22:22:11 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan1410 IPV4 Grp 1410 state changed from Active to Speak reason Rx Hello-Hi P
ri Act.
2019 Apr 9 22:22:13 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally c
onfigured address 172.28.219.1
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 last message repeated 2 times
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan1010 IPV4 Grp 1010 state changed from Speak to Standby reason Sby Timer Ex
pired.
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan1010 IPV4 Grp 1010 Standby router is 10.100.20.2
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBYROUTERPRIORITYCHANGE: Interface Vlan1010 IPV4 Grp 1010 Standby router priority changed to 120
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel411: first operational port changed from Ethernet1/12 to none
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel411: Ethernet1/12 is down
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 10, vPC 411 configuration is consistent
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel411 is down (No operational members)
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 10, vPC 411 configuration is consistent
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 last message repeated 1 time
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %STP-6-PORT_RANGE_DELETED: Interface port-channel411 removed from vlan=110,751-752
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %STP-6-PORT_RANGE_ROLE: new_role=designated interface=port-channel411 vlan=110,751-752
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 10, vPC 411 configuration is consistent
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %STP-6-PORT_RANGE_STATE: new_state=disabled interface=port-channel411 vlan=110
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %STP-6-PORT_RANGE_STATE: new_state=disabled interface=port-channel411 vlan=751-752
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 10, vPC 411 configuration is consistent
2019 Apr 9 22:22:21 AM-US-SAT_6-DD-35-BDRLFSW01 last message repeated 6 times
2019 Apr 9 22:22:21 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan1410 IPV4 Grp 1410 state changed from Speak to Standby reason Sby Timer Ex
pired.
2019 Apr 9 22:22:21 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan1410 IPV4 Grp 1410 Standby router is 10.100.220.2
2019 Apr 9 22:22:21 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBYROUTERPRIORITYCHANGE: Interface Vlan1410 IPV4 Grp 1410 Standby router priority changed to 120
2019 Apr 9 22:22:21 AM-US-SAT_6-DD-35-BDRLFSW01 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel411,bandwidth changed to 100000 Kbit
2019 Apr 9 22:22:21 AM-US-SAT_6-DD-35-BDRLFSW01 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/12 is down (Link failure)
2019 Apr 9 22:22:21 AM-US-SAT_6-DD-35-BDRLFSW01 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel411 is down (No operational members)
2019 Apr 9 22:22:21 AM-US-SAT_6-DD-35-BDRLFSW01 %CDP-5-NEIGHBOR_REMOVED: CDP Neighbor amussat-cube03.kci.com on port Ethernet1/12 has been removed
2019 Apr 9 22:22:21 AM-US-SAT_6-DD-35-BDRLFSW01 %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 10, vPC 411 configuration is consistent
2019 Apr 9 22:22:22 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally c
onfigured address 172.28.219.1
2019 Apr 9 22:22:22 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan3004 IPV4 Grp 3004 Standby router is 0.0.0.0
2019 Apr 9 22:22:22 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBYROUTERPRIORITYCHANGE: Interface Vlan3004 IPV4 Grp 3004 Standby router priority changed to 0
2019 Apr 9 22:22:22 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan3004 IPV4 Grp 3004 state changed from Standby to Active reason Act Timer E
xpired.
2019 Apr 9 22:22:22 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVE_CHANGE: Interface Vlan3004 IPV4 Grp 3004 Active router is 10.12.4.3
2019 Apr 9 22:22:22 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVEROUTERPRIORITY: Interface Vlan3004 IPV4 Grp 3004 Active router priority changed to 100
2019 Apr 9 22:22:25 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally c
onfigured address 172.28.219.1
2019 Apr 9 22:22:30 AM-US-SAT_6-DD-35-BDRLFSW01 last message repeated 1 time
2019 Apr 9 22:22:30 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan1010 IPV4 Grp 1010 Standby router is 0.0.0.0
2019 Apr 9 22:22:30 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBYROUTERPRIORITYCHANGE: Interface Vlan1010 IPV4 Grp 1010 Standby router priority changed to 0
2019 Apr 9 22:22:30 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan1010 IPV4 Grp 1010 state changed from Standby to Active reason Act Timer E
xpired.
2019 Apr 9 22:22:30 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVE_CHANGE: Interface Vlan1010 IPV4 Grp 1010 Active router is 10.100.20.2
2019 Apr 9 22:22:30 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVEROUTERPRIORITY: Interface Vlan1010 IPV4 Grp 1010 Active router priority changed to 120
2019 Apr 9 22:22:31 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally c
onfigured address 172.28.219.1
2019 Apr 9 22:22:32 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan1410 IPV4 Grp 1410 Standby router is 0.0.0.0
2019 Apr 9 22:22:32 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBYROUTERPRIORITYCHANGE: Interface Vlan1410 IPV4 Grp 1410 Standby router priority changed to 0
2019 Apr 9 22:22:32 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan1410 IPV4 Grp 1410 state changed from Standby to Active reason Act Timer E
xpired.
2019 Apr 9 22:22:32 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVE_CHANGE: Interface Vlan1410 IPV4 Grp 1410 Active router is 10.100.220.2
2019 Apr 9 22:22:32 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVEROUTERPRIORITY: Interface Vlan1410 IPV4 Grp 1410 Active router priority changed to 120
2019 Apr 9 22:22:34 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally c
onfigured address 172.28.219.1
2019 Apr 9 22:22:44 AM-US-SAT_6-DD-35-BDRLFSW01 last message repeated 3 times
2019 Apr 9 22:22:44 AM-US-SAT_6-DD-35-BDRLFSW01 %ETHPORT-5-SPEED: Interface Ethernet1/15, operational speed changed to 10 Gbps
2019 Apr 9 22:22:44 AM-US-SAT_6-DD-35-BDRLFSW01 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/15, operational duplex mode changed to Full
2019 Apr 9 22:22:44 AM-US-SAT_6-DD-35-BDRLFSW01 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/15, operational Receive Flow Control state changed to off
2019 Apr 9 22:22:44 AM-US-SAT_6-DD-35-BDRLFSW01 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/15, operational Transmit Flow Control state changed to off
2019 Apr 9 22:22:46 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally c
onfigured address 172.28.219.1
2019 Apr 9 22:22:47 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVE_CHANGE: Interface Vlan1010 IPV4 Grp 1010 Active router is 10.100.20.3
2019 Apr 9 22:22:47 AM-US-SAT_6-DD-35-BDRLFSW01 last message repeated 1 time
2019 Apr 9 22:22:47 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVEROUTERPRIORITY: Interface Vlan1010 IPV4 Grp 1010 Active router priority changed to 120
2019 Apr 9 22:22:47 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan1010 IPV4 Grp 1010 state changed from Active to Speak reason Rx Hello-Hi P
ri Act.
2019 Apr 9 22:22:49 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally c
onfigured address 172.28.219.1
2019 Apr 9 22:22:54 AM-US-SAT_6-DD-35-BDRLFSW01 last message repeated 4 times
2019 Apr 9 22:22:54 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVE_CHANGE: Interface Vlan599 IPV4 Grp 599 Active router is 172.28.219.4
2019 Apr 9 22:22:54 AM-US-SAT_6-DD-35-BDRLFSW01 last message repeated 1 time
2019 Apr 9 22:22:54 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-ACTIVEROUTERPRIORITY: Interface Vlan599 IPV4 Grp 599 Active router priority changed to 120
2019 Apr 9 22:22:54 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan599 IPV4 Grp 599 state changed from Active to Speak reason Rx Hello-Hi Pri
Act.
2019 Apr 9 22:22:56 AM-US-SAT_6-DD-35-BDRLFSW01 %CDP-5-NEIGHBOR_ADDED: Device amussat-p6kvss.kci.com discovered of type cisco WS-C6506-E with port TenGigabitEthernet1/
5/15 on incoming port Ethernet1/15 with ip addr 192.168.67.1 and mgmt ip 192.168.67.1
2019 Apr 9 22:22:56 AM-US-SAT_6-DD-35-BDRLFSW01 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel499: Ethernet1/15 is up
2019 Apr 9 22:22:57 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally c
onfigured address 172.28.219.1
2019 Apr 9 22:22:57 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan1010 IPV4 Grp 1010 state changed from Speak to Standby reason Sby Timer Ex
pired.
2019 Apr 9 22:22:57 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBY_CHANGE: Interface Vlan1010 IPV4 Grp 1010 Standby router is 10.100.20.2
2019 Apr 9 22:22:57 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-STANDBYROUTERPRIORITYCHANGE: Interface Vlan1010 IPV4 Grp 1010 Standby router priority changed to 120
2019 Apr 9 22:22:57 AM-US-SAT_6-DD-35-BDRLFSW01 %ETHPORT-5-IF_TRUNK_UP: Interface Ethernet1/15, vlan 1,27,75-76,84,110,112,127,246-249,251-255,261,299,400-427,429-459,
491-505,534,553,555,557,560,599,627,649,651,654,667,700,711 up
2019 Apr 9 22:22:57 AM-US-SAT_6-DD-35-BDRLFSW01 %ETHPORT-5-IF_TRUNK_UP: Interface Ethernet1/15, vlan 751-753,777,800,866,888-890,912,950-951,1005,1010,1103-1104,1305,1
403,1410,1710,1722,1728,3004-3006,3012,3014,3751-3771 up
2019 Apr 9 22:22:57 AM-US-SAT_6-DD-35-BDRLFSW01 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel499,bandwidth changed to 20000000 Kbit
2019 Apr 9 22:22:57 AM-US-SAT_6-DD-35-BDRLFSW01 %ETHPORT-5-IF_UP: Interface Ethernet1/15 is up in mode trunk
2019 Apr 9 22:22:59 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally c
onfigured address 172.28.219.1
2019 Apr 9 22:23:05 AM-US-SAT_6-DD-35-BDRLFSW01 last message repeated 1 time
2019 Apr 9 22:23:05 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan599 IPV4 Grp 599 state changed from Speak to Standby reason Sby Timer Expi
red.
04-13-2019 06:35 AM
sharath.babu9022@gmail.com hello,
I have read line by line and there are many insconsistency on your configuration, like it;
2019 Apr 9 22:22:13 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally configured address 172.28.219.1
2019 Apr 9 22:22:20 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-5-GRPSTATECHANGE: Interface Vlan1010 IPV4 Grp 1010 state changed from Speak to Standby reason Sby Timer Expired
I think that you have had wrong configuration between your peers hsrp. Could you provide the configuration on both devices?
Thanks in advance.
04-13-2019 12:39 PM
As suggested you have inconsistance config on HSRP, please post the both the switchs HSRP config to have look.
is this 7K configured as vPC ?
04-13-2019 01:53 PM
04-13-2019 02:51 PM
i suggest to send full running config of both the switch along with show spanning-tree output.
04-13-2019 03:25 PM - edited 04-14-2019 11:25 PM
Attached running and spanning tree output.
04-13-2019 03:13 PM - edited 04-13-2019 03:14 PM
sharath.babu9022@gmail.com , could you share your spanning-tre configuration and ip of the virtual interfaces (vlans)
maybe you have a spanning-tre issue or a routing issue, but we need more information to suggest you the better solution.
Thanks in advance.
04-13-2019 03:45 PM
04-13-2019 04:18 PM
04-14-2019 11:25 PM
Could you please let me know if you need more outputs to investigate further? I am suspecting that there was a connectivity loss between the active and standy HSRP routers and both became active at the same time and might have resulted in the outage. Possibility?
04-15-2019 12:15 AM
Hello,
post the output of:
sh run | include vlan 599
from both switches...
04-15-2019 12:18 AM
AM-US-SAT_6-DD-35-BDRLFSW01# sh run vlan 599
!Command: show running-config vlan 599
!Time: Mon Apr 15 02:17:25 2019
version 6.2(16)
vlan 599
vlan 599
mode fabricpath
name VMWARE-5-1-PROD
AM-US-SAT_6-GD-35-BDRLFSW02# sh running-config vlan 599
!Command: show running-config vlan 599
!Time: Mon Apr 15 02:18:25 2019
version 6.2(16)
vlan 599
vlan 599
mode fabricpath
name VMWARE-5-1-PROD
04-15-2019 12:19 AM
04-15-2019 12:29 AM
On the Quick look i did not see any issue in terms of config.
But we are not sure how your Esxi / Physical connected,
If you lost connection between switches there may be chance of Split brian situation, that can only view with log files.
Nexus have good log storage space. pull the logs for the incident time and post here to look, or you can also verify same.
04-15-2019 01:20 AM
Problem Description :
====================
[+] Few servers were not able to reach its gateway (172.28.219.1) in VLAN 599.
Action Taken :
============
[+] Noticed HSRP for vlan 599 is standby and listen in our Datacenter.
AM-US-SAT_6-GD-35-BDRLFSW02# sh hsrp interface vlan 599 brief (This is our Datacenter switch)
*:IPv6 group #:group belongs to a bundle
P indicates configured to preempt.
|
Interface Grp Prio P State Active addr Standby addr Group addr
Vlan599 599 100 P Listen 172.28.219.4 172.28.219.2 172.28.219.1 (conf)
AM-US-SAT_6-DD-35-BDRLFSW01# sh hsrp interface vlan 599 brief (This is our Datacenter switch)
*:IPv6 group #:group belongs to a bundle
P indicates configured to preempt.
|
Interface Grp Prio P State Active addr Standby addr Group addr
Vlan599 599 120 P Standby 172.28.219.4 local 172.28.219.1 (conf) << was standby
AM-US-SAT_6-DD-35-BDRLFSW01#
[+] HSRP was active in another DC as seen below also group address is different which case all servers/application to stops working.
AM-US-SCF_01-0726-35-BDRLFSW01# sh hsrp interface vlan 599 brief (This is our disaster recovery Datacenter switch)
*:IPv6 group #:group belongs to a bundle
P indicates configured to preempt.
|
Interface Grp Prio P State Active addr Standby addr Group addr
Vlan599 599 120 P Active local 172.28.219.2 172.28.219.254 (conf) << now become active..
[+] And VIP of newly active is different than Active one in problematic DC.
show logging log :
-----------------
2019 Mar 28 21:58:32 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally configured address 172.28.219.1
2019 Mar 28 21:58:33 AM-US-SAT_6-DD-35-BDRLFSW01 %AUTHPRIV-3-SYSTEM_MSG: pam_aaa:Authentication failed from 172.28.219.209 - sshd
2019 Mar 28 21:58:34 AM-US-SAT_6-DD-35-BDRLFSW01 %USER-2-SYSTEM_MSG: fatal: Write failed: Broken pipe .Client is 172.28.219.209,length of packet causing error 496 496 - sshd
2019 Mar 28 21:58:35 AM-US-SAT_6-DD-35-BDRLFSW01 %AUTHPRIV-3-SYSTEM_MSG: pam_aaa:Authentication failed from 172.28.219.209 - sshd
2019 Mar 28 21:58:35 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally configured address 172.28.219.1
2019 Mar 28 21:58:50 AM-US-SAT_6-DD-35-BDRLFSW01 last message repeated 4 times
2019 Mar 28 21:58:50 AM-US-SAT_6-DD-35-BDRLFSW01 %AUTHPRIV-3-SYSTEM_MSG: pam_aaa:Authentication failed from 172.28.219.209 - sshd
2019 Mar 28 21:58:50 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally configured address 172.28.219.1
2019 Mar 28 21:58:52 AM-US-SAT_6-DD-35-BDRLFSW01 %USER-2-SYSTEM_MSG: fatal: Write failed: Broken pipe .Client is 172.28.219.209,length of packet causing error 496 496 - sshd
2019 Mar 28 21:58:53 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally configured address 172.28.219.1
2019 Mar 28 21:58:53 AM-US-SAT_6-DD-35-BDRLFSW01 %AUTHPRIV-3-SYSTEM_MSG: pam_aaa:Authentication failed from 172.28.219.209 - sshd
2019 Mar 28 21:58:56 AM-US-SAT_6-DD-35-BDRLFSW01 %HSRP_ENGINE-4-DIFFVIP1: Vlan599 Grp 599 active routers virtual IP address 172.28.219.254 is different to the locally configured address 172.28.219.1
[+] Increase HSRP priority on BDRLFSW01, results in to become HSRP active on this device.
[+] After changing the priority everything starts working.
[+] Server which were not pingable starts pinging.
AM-US-SAT_6-DD-35-BDRLFSW01# ping 172.28.219.207
PING 172.28.219.207 (172.28.219.207): 56 data bytes
64 bytes from 172.28.219.207: icmp_seq=0 ttl=127 time=0.995 ms
64 bytes from 172.28.219.207: icmp_seq=1 ttl=127 time=0.849 ms
64 bytes from 172.28.219.207: icmp_seq=2 ttl=127 time=0.792 ms
64 bytes from 172.28.219.207: icmp_seq=3 ttl=127 time=1.411 ms
64 bytes from 172.28.219.207: icmp_seq=4 ttl=127 time=1.794 ms
--- 172.28.219.207 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 0.792/1.168/1.794 ms
VLAN 599 is spanned between our datacenter at San Antonio and DR at Scottsdale with OTV. The gateway for VLAN 599 is .1 in San Antonio but it is .254 in DR. This is by design.
We are trying to understand why did the HSRP events occur on our Nexus Environment on April 7th? And what made VLAN 599 to become active at our DR and listen & standy state in our DR.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide