We are facing intermittent issue of network service loss, approx. once or twice a week. We experience what appears to be a port flapping issue, which then triggers a series of failovers between switches on the network, which results in users losing internet/WAN access for 5-8 mins in duration and is now having a significant impact. We have changed the network patch cable and port on the primary link, but issue keeps occurring intermittently.
Network Detail:
ASA
| |
L3-SW1 -- L3-SW2
| |
Access Layer
Network design follows 2 tier design with collapsed core and separate access layer. The ASA 5510 is working as a WAN demarcation device which has an outside & inside zone.
Each L3 switch has a single port which is configured as a trunk link and connected the ASA, from the ASA’s perspective the two links make a port-channel.
(ASA port 1 -> L3Sw1 port 23, ASA port 2 -> L3Sw2 port24
Core switches (Cisco c3560G – hostnames: SH-L3-SW1, SH-L3-SW2) are responsible for fast switching and routing between VLANs.
For redundancy HSRP is configured between ASA and Core switches on a per/VLAN basis with priority of 150 and decrement of 70.
An Ether channel is configured between L3 switches.
Logs for L3Sw1
Oct 22 16:01:12.158 BST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 10.10.1.1 (Vlan601) is down: holding time expired
Oct 22 16:01:53.959 BST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 10.10.1.1 (Vlan601) is up: new adjacency
Oct 22 16:01:54.026 BST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 10.10.1.1 (Vlan601) is down: Interface Goodbye received
Oct 22 16:03:35.076 BST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 10.10.1.1 (Vlan601) is up: new adjacency
Oct 22 16:04:35.357 BST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 10.10.1.1 (Vlan601) is down: holding time expired
Oct 22 16:05:20.270 BST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 10.10.1.1 (Vlan601) is up: new adjacency
Oct 22 16:05:20.279 BST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 10.10.1.1 (Vlan601) is down: Peer goodbye received
Oct 22 16:05:20.279 BST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 10.10.1.1 (Vlan601) is down: Interface Goodbye received
Oct 22 16:07:00.632 BST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 10.10.1.1 (Vlan601) is up: new adjacency
Oct 22 16:07:48.532 BST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 10.10.1.1 (Vlan601) is down: holding time expired
Oct 22 16:08:30.894 BST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 10.10.1.1 (Vlan601) is up: new adjacency
Oct 22 16:08:30.903 BST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 10.10.1.1 (Vlan601) is down: Interface Goodbye received
Oct 22 16:10:10.292 BST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 10.10.1.1 (Vlan601) is up: new adjacency
Logs for L3Sw2
Oct 22 16:06:52.065 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/24, changed state to down
Oct 22 16:06:53.071 BST: %LINK-3-UPDOWN: Interface GigabitEthernet0/24, changed state to down
Oct 22 16:06:55.320 BST: %LINK-3-UPDOWN: Interface GigabitEthernet0/24, changed state to up
Oct 22 16:06:56.326 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/24, changed state to up
Oct 22 16:07:00.638 BST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 10.10.1.1 (Vlan601) is up: new adjacency
Oct 22 16:07:48.537 BST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 10.10.1.1 (Vlan601) is down: holding time expired
Oct 22 16:08:30.900 BST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 10.10.1.1 (Vlan601) is up: new adjacency
Oct 22 16:08:30.900 BST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 10.10.1.1 (Vlan601) is down: Peer goodbye received
Oct 22 16:08:30.900 BST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 10.10.1.1 (Vlan601) is down: Interface Goodbye received
Oct 22 16:08:54.682 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/24, changed state to down
Oct 22 16:08:55.688 BST: %LINK-3-UPDOWN: Interface GigabitEthernet0/24, changed state to down
Oct 22 16:09:05.109 BST: %LINK-3-UPDOWN: Interface GigabitEthernet0/24, changed state to up
Oct 22 16:09:06.115 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/24, changed state to up
Oct 22 16:09:55.088 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/24, changed state to down
Oct 22 16:09:56.086 BST: %LINK-3-UPDOWN: Interface GigabitEthernet0/24, changed state to down
Oct 22 16:09:58.544 BST: %LINK-3-UPDOWN: Interface GigabitEthernet0/24, changed state to up
Oct 22 16:09:59.551 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/24, changed state to up
Oct 22 16:10:02.629 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/24, changed state to down
Oct 22 16:10:03.636 BST: %LINK-3-UPDOWN: Interface GigabitEthernet0/24, changed state to down
Oct 22 16:10:05.926 BST: %LINK-3-UPDOWN: Interface GigabitEthernet0/24, changed state to up
Any help or advice on how to isolate the cause and resolve this would be very much appreciated- thank you! J
