Re: Loopguard kicked in temporarily

aok · ‎09-24-2018

Hello

We had a 10 minute outage on our network and looking in the logs saw these messages:

2018 Sep 24 14:11:36.797 CA1CORE1 %STP-2-LOOPGUARD_BLOCK: Loop guard blocking port Ethernet1/41 on VLAN0400.
2018 Sep 24 14:11:42.327 CA1CORE1 %OSPF-5-ADJCHANGE: ospf-100 [3677] Nbr 172.20.0.17 on Ethernet1/16 went DOWN
2018 Sep 24 14:11:42.341 CA1CORE1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/16 is down (Link failure)
2018 Sep 24 14:11:42.586 CA1CORE1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/15 is down (Link failure)
2018 Sep 24 14:11:42.615 CA1CORE1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/41 is down (Link failure)
2018 Sep 24 14:12:01.743 CA1CORE1 %ARP-2-DUP_SRC_IP: arp [3505] Source address of packet received from f07f.0644.4b81 on Vlan2312(port-channel4090) is duplicate of local, 172.23.12.2
2018 Sep 24 14:17:46.989 CA1CORE1 %ETHPORT-5-SPEED: Interface Ethernet1/41, operational speed changed to 1 Gbps
2018 Sep 24 14:17:46.989 CA1CORE1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/41, operational duplex mode changed to Full
2018 Sep 24 14:17:46.989 CA1CORE1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/41, operational Receive Flow Control state changed to off
2018 Sep 24 14:17:46.989 CA1CORE1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/41, operational Transmit Flow Control state changed to off
2018 Sep 24 14:17:47.020 CA1CORE1 %ETHPORT-5-IF_UP: Interface Ethernet1/41 is up in mode access
2018 Sep 24 14:17:47.117 CA1CORE1 %ETHPORT-5-SPEED: Interface Ethernet1/15, operational speed changed to 1 Gbps
2018 Sep 24 14:17:47.117 CA1CORE1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/15, operational duplex mode changed to Full
2018 Sep 24 14:17:47.117 CA1CORE1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/15, operational Receive Flow Control state changed to off
2018 Sep 24 14:17:47.117 CA1CORE1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/15, operational Transmit Flow Control state changed to off
2018 Sep 24 14:17:47.157 CA1CORE1 %ETHPORT-5-IF_UP: Interface Ethernet1/15 is up in mode trunk
2018 Sep 24 14:17:48.063 CA1CORE1 %ETHPORT-5-SPEED: Interface Ethernet1/16, operational speed changed to 1 Gbps
2018 Sep 24 14:17:48.063 CA1CORE1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/16, operational duplex mode changed to Full
2018 Sep 24 14:17:48.063 CA1CORE1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/16, operational Receive Flow Control state changed to off
2018 Sep 24 14:17:48.064 CA1CORE1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/16, operational Transmit Flow Control state changed to off
2018 Sep 24 14:17:48.094 CA1CORE1 %ETHPORT-5-IF_UP: Interface Ethernet1/16 is up in Layer3
2018 Sep 24 14:17:51.696 CA1CORE1 %OSPF-5-ADJCHANGE: ospf-100 [3677] Nbr 172.20.0.17 on Ethernet1/16 went EXSTART
2018 Sep 24 14:17:52.259 CA1CORE1 %OSPF-5-ADJCHANGE: ospf-100 [3677] Nbr 172.20.0.17 on Ethernet1/16 went FULL
2018 Sep 24 14:18:54.100 CA1CORE1 %ARP-2-DUP_SRC_IP: arp [3505] Source address of packet received from f07f.0644.4b81 on Vlan2312(port-channel4090) is duplicate of local, 172.23.12.2
2018 Sep 24 14:44:17 CA1CORE1 %AUTHPRIV-3-SYSTEM_MSG: pam_aaa:Authentication failed for user admin from 172.23.18.140 - dcos_sshd[4622]
2018 Sep 24 14:44:41.493 CA1CORE1 %ARP-2-DUP_SRC_IP: arp [3505] Source address of packet received from f07f.0644.4b81 on Vlan2312(port-channel4090) is duplicate of local, 172.23.12.2
CA1CORE1#

___

By the time we figured out what device the issue was on the problem had resolved itself. Any suggestions on what may have caused it or things we can check to prevent it from happening again?

Thanks

A

Leo Laohoo · ‎09-24-2018

@aok wrote:

2018 Sep 24 14:11:36.797 CA1CORE1 %STP-2-LOOPGUARD_BLOCK: Loop guard blocking port Ethernet1/41 on VLAN0400.
2018 Sep 24 14:11:42.327 CA1CORE1 %OSPF-5-ADJCHANGE: ospf-100 [3677] Nbr 172.20.0.17 on Ethernet1/16 went DOWN
2018 Sep 24 14:11:42.341 CA1CORE1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/16 is down (Link failure)
2018 Sep 24 14:11:42.586 CA1CORE1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/15 is down (Link failure)
2018 Sep 24 14:11:42.615 CA1CORE1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/41 is down (Link failure)

Are these three ports meant to be connected in an EtherChannel?

aok · ‎09-24-2018

Hi Leo

No they're separate connections to different devices. Here are the interface configs:

interface Ethernet1/16
speed 1000
description Layer3 to HQ1Core2 e1/16
no switchport
no ip redirects
ip address 172.20.0.22/29
ip ospf cost 10
ip ospf hello-interval 1
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
no shutdown

CA1CORE1# show run int eth1/15

interface Ethernet1/15
speed 1000
description Layer 2 connect to legacy 3750G g1/0/43
switchport mode trunk
switchport trunk allowed vlan 700
spanning-tree port type normal
no shutdown

CA1CORE1# show run int eth1/41

interface Ethernet1/41

speed 1000
description vlan 400 test pc
switchport access vlan 400
spanning-tree port type edge
spanning-tree guard loop
no shutdown

Thanks
A

Leo Laohoo · ‎09-24-2018

If they're all separate, post the complete output to the command "sh interface E1/41".
Also, all three ports went down simultaneously, did all three downstream clients lose power or something?

aok · ‎09-25-2018

Hi Leo

No, none of them are in a port-channel. Here is the output you requested:

Ethernet1/41 is up
Dedicated Interface
Hardware: 100/1000/10000/40000 Ethernet, address: f07f.0644.4a50 (bia f07f.0644.4a50)
Description: vlan 400 test pc
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec
reliability 255/255, txload 15/255, rxload 2/255
Encapsulation ARPA
Port mode is access
full-duplex, 1000 Mb/s, media type is 1G
Beacon is turned off
Input flow-control is off, output flow-control is off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
Last link flapped 22:51:59
Last clearing of "show interface" counters 39w6d
4 interface resets
Load-Interval #1: 30 seconds
30 seconds input rate 3446112 bits/sec, 989 packets/sec
30 seconds output rate 13804456 bits/sec, 5779 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 9.66 Mbps, 1.54 Kpps; output rate 61.45 Mbps, 11.45 Kpps
RX
50355512144 unicast packets 21752980 multicast packets 100360984 broadcast packets
50477625554 input packets 42896568105888 bytes
0 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
203776130155 unicast packets 290428467 multicast packets 468945353 broadcast packets
204535503975 output packets 74637874320358 bytes
0 jumbo packets
0 output errors 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 1107168 output discard
0 Tx pause

----------------

Here's a diagram of what I've mapped out so far:

Leo Laohoo · ‎09-25-2018

Why can't E1/15 & E1/41 be in an EtherChannel?
I can see why the links went into error-disable.

aok · ‎09-26-2018

I'm not sure why it was set up this way and the people who configured it aren't around anymore. Please would you explain why you think E1/41 loopguard kicked in and the other ports also took a dive?

Leo Laohoo · ‎09-26-2018

There are three links going from the switch to the 3750. Each link carries their own "tiny" VLANs. One flips and loopguard kicks in because of the way the links are configured incorrectly.
Ideally, those three links (two would even work and this already includes redundancy), should be in an EtherChannel.

@aok wrote:

I'm not sure why it was set up this way and the people who configured it aren't around anymore.

If you want to be a good network engineer then it's time for you to "step up" and ask the hard question(s). If no one cares to answer, then think about improving the network.

aok · ‎10-04-2018

Hi Leo

When you say the links are configured incorrectly what exactly do you mean? I get that they're sub-optimal and needs improving but it's supporting legacy equipment and will be going away once someone gets around to moving all the old servers off. In the meantime I'd like to fully understand what happened with loopguard and what caused one of the ports to flip in the first place. They have been working this way for a long time and this was the first issue we've seen.

Thanks

A

Leo Laohoo · ‎10-04-2018

@aok wrote:

When you say the links are configured incorrectly what exactly do you mean?

There are three links coming from one physical switch and goes to the another physical switch. Each link are configured with individual VLANs.

Have a read about EtherChannel.