L3 portchannel across provider clouds = blackholed traffic

[Larry Sullivan]

Hi,

Need advice on an issue.  We have a L3 port-channel (fiber metro ethernet) across two separate providers clouds.  ASR to ASR.  1 link is provider A and other link in the port-channel is provider B.  When something in the provider's cloud breaks on one of the links, the interface will still be up, thus black holing traffic on that link.  Are there any suggestions on how to detect when a link is black holed?  Can't do it by line protocol state and can't do it by IP.  Any special configs to assist in detection or shifting the traffic to the working link, pulling the black holed link out of the port-channel?  Alternate configs or methods altogether?  Thanks.     

[paul driver]

Hello

Are you using lacp, or negating any speed/duplex interface negotiation?

Can you post the etherchannel config?

[Larry Sullivan]

Please note, we had to take the bad link out of the port-channel, so you'll notice that config missing along with ip addresses added (to monitor progress of fix).  Hope that isn't too confusing.  But it config seems like static LAG. 

interface Port-channel1
 description
 ip address 192.168.202.1 255.255.255.252
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nbar protocol-discovery
 ip pim sparse-dense-mode
 ip igmp version 3
 no negotiation auto
end

interface GigabitEthernet0/0/6
 description
 ip address 10.3.1.1 255.255.255.252
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 negotiation auto
 cdp enable

interface GigabitEthernet0/0/7
 description
 no ip address
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 negotiation auto
 cdp enable
 channel-group 1

--------------------------

interface Port-channel1
 description
 ip address 192.168.202.2 255.255.255.252
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat outside
 ip nbar protocol-discovery
 ip pim sparse-dense-mode
 ip igmp version 3
 no negotiation auto

interface GigabitEthernet0/0/6
 description
 ip address 10.3.1.2 255.255.255.252
 negotiation auto
 cdp enable

interface GigabitEthernet0/0/7
 description
 no ip address
 negotiation auto
 cdp enable
 channel-group 1

[dbeattie · Accepted solution]

Hi,

There are basically three ways around this:

1. The easiest is to ensure that your etherchannel uses a negotiation protocol such as LACP. This is done by changing your "channel-group 1" command to "channel-group 1 mode active". In order to do this you will have to take all links out of the channel and then add them back in. Please note that if it is available you should use LACP fast rate.

2. Use UDLD to detect when a link goes unidirectional. This is dependent upon your providers supporting end-to-end forwarding of BPDUs. It also helps if your provider supports Link Loss Forwarding (LLF), which will propagate a middle-of-path failure to the hand-off interface

3. separate the links out of the channel and run them independently as L3 links. Then use a routing protocol (like OSPF) that allows ECMP to use both links simultaneously. This then means that you can configure BFD on both links to ensure rapid convergence in a failure scenario and ensures that middle-of-path failures are detected rapidly.

Ultimately, I would tend to go for the last option because it puts you in charge of your network rather than relying on your provider. (but I am a bit paranoid in that way).

Hope this helps

Dave

[Larry Sullivan]

Awesome.  Some interesting options to explore there that I haven't considered.  Thanks dbeattie.