Hi,
I have two office locations which are approximately a kilometer apart and is connected by two Point-to-Point WAN links provided by two different ISP's.
Each links is of 45 Mbps. I am trying to bundle both of these links into a Ether-channel, however, I am facing some challenges.
I have tested both links individually and its working fine. Even i was able to bundle it without using negotiation protocols-PAgP or LACP (using ON). But the problem is if any one of the link or interface goes down in either end, I am losing the entire connectivity between sites. Also what i have observed is that if i shutdown either ISP_1 or ISP_2 interfaces on both ends (when it is bundled) then there is no issue.
So I am not able to achieve the actual goal of having an Ether-channel.
Below is the current Configuration which is working fine at the moment.
Site-A
3850-SW1#sh run int gi 1/0/9
Building configuration...
Current configuration : 152 bytes
!
interface GigabitEthernet1/0/9
description ***ISP_1***
switchport mode trunk
speed 100
duplex full
channel-group 5 mode on
end
3850-SW1#sh run int gi 1/0/10
Building configuration...
Current configuration : 151 bytes
!
interface GigabitEthernet1/0/10
description ***ISP_2***
switchport mode trunk
speed 100
duplex full
channel-group 5 mode on
end
3850-SW1#sh run int po5
Building configuration...
Current configuration : 112 bytes
!
interface Port-channel5
description ***To_3850-SW2***
switchport mode trunk
speed 100
duplex full
end
3850-SW1#sh etherchannel 5 su
Flags: D - down P - bundled in port-channel
I - stand-alone s - suspended
H - Hot-standby (LACP only)
R - Layer3 S - Layer2
U - in use f - failed to allocate aggregator
M - not in use, minimum links not met
u - unsuitable for bundling
w - waiting to be aggregated
d - default port
Number of channel-groups in use: 5
Number of aggregators: 5
Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
5 Po5(SU) - Gi1/0/9(P) Gi1/0/10(P)
3850-SW1#sh int gi 1/0/9
GigabitEthernet1/0/9 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 189c.5d17.8989 (bia 189c.5d17.8989)
Description: ***ISP_1***
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 7/255, rxload 95/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 22:20:54, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 37395000 bits/sec, 4653 packets/sec
5 minute output rate 2787000 bits/sec, 1861 packets/sec
1193866499 packets input, 1240289447 bytes, 0 no buffer
Received 529960 broadcasts (364319 multicasts)
0 runts, 0 giants, 0 throttles
3 input errors, 2 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 364319 multicast, 9302883 pause input
0 input packets with dribble condition detected
1240533949 packets output, 4274348333 bytes, 0 underruns
0 output errors, 0 collisions, 13 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
3850-SW1#sh int gi 1/0/10
GigabitEthernet1/0/10 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 189c.5d17.898a (bia 189c.5d17.898a)
Description: ***ISP_2***
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 100/255, rxload 12/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:14, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 4865000 bits/sec, 1041 packets/sec
5 minute output rate 39453000 bits/sec, 3924 packets/sec
625518180 packets input, 3078590463 bytes, 0 no buffer
Received 4726585 broadcasts (3860846 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 3860846 multicast, 0 pause input
0 input packets with dribble condition detected
839274849 packets output, 72907943 bytes, 0 underruns
0 output errors, 0 collisions, 7 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
3850-SW1#sh int po5
Port-channel5 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 189c.5d17.8989 (bia 189c.5d17.8989)
Description: ***To_3850-SW2***
MTU 1500 bytes, BW 200000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 53/255, rxload 52/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, link type is auto, media type is
input flow-control is off, output flow-control is unsupported
Members in this channel: Gi1/0/9 Gi1/0/10
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 41503000 bits/sec, 5634 packets/sec
5 minute output rate 42280000 bits/sec, 5761 packets/sec
1819416500 packets input, 37549348 bytes, 0 no buffer
Received 5256594 broadcasts (4225195 multicasts)
0 runts, 0 giants, 0 throttles
3 input errors, 2 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 4225195 multicast, 9302883 pause input
0 input packets with dribble condition detected
2079838549 packets output, 82510093 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
Site-B
3850-SW2#sh run int gi 1/0/24
Building configuration...
Current configuration : 157 bytes
!
interface GigabitEthernet1/0/24
description ***ISP_1***
switchport mode trunk
speed 100
duplex full
channel-group 5 mode on
end
3850-SW2#sh run int gi 2/0/24
Building configuration...
Current configuration : 155 bytes
!
interface GigabitEthernet2/0/24
description ***ISP_2***
switchport mode trunk
speed 100
duplex full
channel-group 5 mode on
end
3850-SW2#sh run int po5
Building configuration...
Current configuration : 116 bytes
!
interface Port-channel5
description ***TO_3850-SW1***
switchport mode trunk
speed 100
duplex full
end
3850-SW2#sh etherchannel 5 su
Flags: D - down P - bundled in port-channel
I - stand-alone s - suspended
H - Hot-standby (LACP only)
R - Layer3 S - Layer2
U - in use f - failed to allocate aggregator
M - not in use, minimum links not met
u - unsuitable for bundling
w - waiting to be aggregated
d - default port
Number of channel-groups in use: 1
Number of aggregators: 1
Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
5 Po5(SU) - Gi1/0/24(P) Gi2/0/24(P)
3850-SW2#sh int gi 1/0/24
GigabitEthernet1/0/24 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 8480.2d94.5b98 (bia 8480.2d94.5b98)
Description: ***ISP_1***
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 122/255, rxload 7/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:05, output never, output hang never
Last clearing of "show interface" counters 2w5d
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 2758000 bits/sec, 2033 packets/sec
5 minute output rate 47996000 bits/sec, 5577 packets/sec
1189754195 packets input, 4252242003 bytes, 0 no buffer
Received 4228735 broadcasts (3889387 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 3889387 multicast, 0 pause input
0 input packets with dribble condition detected
1175349750 packets output, 1127653607 bytes, 0 underruns
0 output errors, 0 collisions, 3 interface resets
3 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
3850-SW2#sh int gi 2/0/24
GigabitEthernet2/0/24 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 6899.cdf7.6918 (bia 6899.cdf7.6918)
Description: ***ISP_2***
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 13/255, rxload 104/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:02, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 40999000 bits/sec, 4105 packets/sec
5 minute output rate 5212000 bits/sec, 1095 packets/sec
836857372 packets input, 1648069837 bytes, 0 no buffer
Received 42717801 broadcasts (35073673 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 35073673 multicast, 0 pause input
0 input packets with dribble condition detected
625207137 packets output, 2864794486 bytes, 0 underruns
0 output errors, 0 collisions, 6 interface resets
3 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
3850-SW2#sh int po5
Port-channel5 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 8480.2d94.5b98 (bia 8480.2d94.5b98)
Description: ***TO_3850-SW1***
MTU 1500 bytes, BW 200000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 67/255, rxload 55/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, link type is auto, media type is
input flow-control is off, output flow-control is unsupported
Members in this channel: Gi1/0/24 Gi2/0/24
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 43725000 bits/sec, 6139 packets/sec
5 minute output rate 53294000 bits/sec, 6672 packets/sec
2072525594 packets input, 3576579744 bytes, 0 no buffer
Received 46967137 broadcasts (38981141 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 38981141 multicast, 0 pause input
0 input packets with dribble condition detected
1839423586 packets output, 659594591 bytes, 0 underruns
150224 output errors, 3467 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 1046 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
When i used LACP / PAgP , i was not able to bundle both interfaces and when i tried to add the second link using "channel-group" command I got below error which i received on both ends and lost connectivity.
*May 26 06:54:31.171: %SW_MATM-4-MACFLAP_NOTIF: Host 0060.2d01.342f in vlan 1 is flapping between port Po5 and port Gi1/0/24 (3850-SW2-2)
*May 26 06:54:33.893: %SW_MATM-4-MACFLAP_NOTIF: Host d067.e5ea.9ef1 in vlan 1 is flapping between port Gi1/0/24 and port Po5
*May 26 06:54:34.434: %SW_MATM-4-MACFLAP_NOTIF: Host 5ca4.8a10.54b0 in vlan 1 is flapping between port Gi1/0/24 and port Po5
*May 26 06:54:33.937: %SW_MATM-4-MACFLAP_NOTIF: Host d067.e5ea.9ef1 in vlan 1 is flapping between port Gi1/0/24 and port Po5 (3850-SW2-2)
*May 26 06:54:34.444: %SW_MATM-4-MACFLAP_NOTIF: Host 5ca4.8a10.54b0 in vlan 1 is flapping between port Gi1/0/24 and port Po5 (3850-SW2-2)
*May 26 06:54:35.192: %SW_MATM-4-MACFLAP_NOTIF: Host c434.6b65.bd80 in vlan 1 is flapping between port Gi1/0/24 and port Po5
Any suggestion / guidelines are most welcome :)
Thanks in Advance..
