xthuijs
Cisco Employee
Cisco Employee

 

Introduction

Multichassis LAG is a tricky concept. In general the members of a bundle (also called LAG, Link Aggregation Group, Etherchannel, Portchannel) are between 2 distinct devices. The advantage of using a bundle is that there is a single routing peering, no worries about spanning tree and things like that. However the redundancy is compromised when either one of the peers fail. Using ECMP (Equal Cost Multipath) in L3 scenarios allows me to dual home to 2 different devices so I have a back up also when one of the peers fail for me, but that negates the benefit of using bundle having a single routing peering.

MC-LAG attempts to provide a means to allow me to dual home a device (DHD, the dual homed device) to two different peer devices (the POA, or Point of Attachment), so basically allowing me to have the benefits of node redundancy, while maintaining single peerings which makes my L2 (Spanning Tree/ STP) or L3 (no dual peerings) life a lot easier.

Does it come with restrictions? Of course! It's technology, nothing comes for free...! So in this document we will highlight how to set it up, what the restrictions are that you need to be aware of and how to troubleshoot and verify MC-LAG scenarios.

 

Overview

MC-LAG & ICCP enable a switch/router to use standard Ethernet Link Aggregation for device dual-homing, with active/standby redundancy

Dual-homed Device (DHD) operates as if it is connected to single virtual device and runs IEEE std. 802.1AX-2008 (LACP)

Point of Attachment (PoA) nodes run Inter-chassis Communication Protocol (ICCP) to synchronize state & form a Redundancy Group (RG)

 

Screen Shot 2013-07-15 at 8.36.30 AM.png

Idea is to let the peer “device” feel that it’s connected to a single “device” •à need information sync between two PoA.

MC-LAG uses ICCP to synchronize LACP configuration & operational state between PoAs, to provide DHD the perception of being connected to a single switch. All PoAs use the same System MAC Address & System Priority when communicating with DHD

Configurable or automatically synchronized via ICCP

Every PoA in the RG is configured with a unique Node ID (value 0 to 7). Node ID + 8 forms the most significant nibble of the Port Number.
For a given bundle, all links on the same PoA must have the same Port Priority.
 

1. Port Status

Currently all vendor’s MC-LAG implementation is based on active/standby mode. All the member ports on standby POA must be in standby mode
which doesn’t forward any packet.
The POA need to be configured with higher system priority (low value) than DHD. So POA determine which link is in active/standby state
By default, all member ports on the active POA is in active state unless:
– If active POA has “maximum active-links” configured, then it only put that amount of ports into active
– If DHD has “maximum active-links” configured, then only that amount of ports can be put into active
Active POA determine which ports goes to active or standby based on the port priority and port number on its local port as normal LACP
Failure events can cause bundle member port active/standby state failover
 
 
Coupled vs. Decoupled

MC-LAG bundle (sub-)interface can be configured for both L2 and L3 service

 

Service redundancy status may or may not be tied to PoA/bundle active/standby status

 

P2P PW (coupled mode): bundle state determine the PW state. If bundle is in active state, then it advertise “active” PW status message.

 

Otherwise it will advertise “standby” PW status message to its peer Routers

 

H-VPLS access P2P PW (coupled & one-way mode): PW and its backup PW are in regular “one-way” PW redundancy mode  on active POA.

 

On the standby POA, both of itse PWs are in standby state

 

VPLS service (de-coupled mode): regardless if bundle is active or standby, VPLS PWs are always in active forwarding state

 

H-VPLS access PW (PW under bridge-domain): same de-coupled mode as VPLS

 

L3 service (coupled mode): bundle state determine the L3 sub-interface state.

 

If bundle is in active state, then bundle L3 interface/sub-interface keep up. Otherwise, it keeps in protocol “down” state

Screen Shot 2013-07-15 at 8.43.33 AM.png
This example above is for p2p two-way PW redundancy. Both sides must run MC-LAG. It’s in coupled mode. Bundle state decide PW
redundancy state. Active POA send active PW status to remote Router. Standby POA send standby PW status. PW become active ONLY if
local and remote Routers are both active. The rest of 3 PWs are in standby mode. When active link fail, the PoA and bundle will switchover.
Then it will advertise new active or standby PW status message to the peer device, which cause PW switchover as well
Screen Shot 2013-07-15 at 8.44.03 AM.png
This example is the VPLS service.
The remote Routers don’t have to run MC-LAG. The remote side could be singe or dual home Router, could be vpls or spoke PW
VPLS PW status is independent of the link bundle or POA status. They are all in active state unless remote Router signal a standby status. This is called decouple mode. ASR9K only support decouple mode for VPLS PW and bridge-domain access PW
Note, 7600 support both couple and decouple mode. And coupled mode is on by default. In coupled mode, the Bundle AC state will decide the
PW state. With coupled mode, if POA is in standby mode, then its VPLS PW will be in standby mode as well
POA (bundle) failover will trigger LDP MAC withdrawal message to remote Routers
 
 
 
 

Configuration

 

The following configuration is recommended for mLACP operation. 

 

2. ICCP setup (POAs)

 

First,  an ICCP group must be set up. This configuration is not owned by the  bundle infra, so only a simple example setup is provided here. For more  details please refer to the information provided by the ICCP team.

On each POA, the following is required:

  • An IPv4 address at each end of the link between the POAs to provide IP connectivity.
  • An IP address set up on a loopback interface as the endpoint for the LDP link over which ICCP operates;
  • A  routing protocol so that the routes can be determined (static route in  the example, for which proxy-arp on the connection between POAs is  required);
  • An LDP session to connect to the peer.

 

RP/0/0/CPU0:ios(config)#interface <link connected to peer POA>
RP/0/0/CPU0:ios(config-if)#ipv4 address <IP address; e.g. 4.0.0.[1-2]/8>
RP/0/0/CPU0:ios(config-if)#proxy-arp
RP/0/0/CPU0:ios(config-if)#root
RP/0/0/CPU0:ios(config)#interface Loopback <ID> ipv4 address <IP address; e.g. 5.4.3.[1-2]/32>
RP/0/0/CPU0:ios(config)#router static address-family ipv4 unicast <IP of peer POA, e.g. 5.4.3.[2-1]/32> <IP of far end of link between POAs, e.g. 4.0.0.[2-1]>
RP/0/0/CPU0:ios(config)#mpls ldp
RP/0/0/CPU0:ios(config-ldp)#router-id <unique ID, e.g. 5.4.3.<1-2>>
RP/0/0/CPU0:ios(config-ldp)#discovery targeted-hello accept
RP/0/0/CPU0:ios(config-ldp)#log neighbor
RP/0/0/CPU0:ios(config-ldp)#interface <link connected to other POA>
RP/0/0/CPU0:ios(config-ldp)#commit

The next step is to add an ICCP redundancy group on each POA; using the same group ID for both:

RP/0/0/CPU0:poa1(config)#redundancy iccp group <group-id> member neighbor <LDP router-id of other POA>
RP/0/0/CPU0:poa1(config)#commit

NB: The same ICCP group ID must be used on both POAs.

At  this point the two POAs should connect and start running ICCP. However,  it may take a little while for the ICCP connection to be established.  Once connected, the peer POA's information should appear in the show iccp group <group-id> command, and its state should be up (connected). NB:  The configuration and commands up to this point are not owned by the  bundle infra; please refer to the information from the ICCP team for  more details. (However, if you investigate and find these instructions  incorrect or out of date, please let us know!)

 

3. mLACP setup (POAs)

 

The  underlying ICCP session has now been established. To enable the mLACP  session, some further configuration settings are required under the ICCP  group:

  • The system MAC, which should be the same on both POAs;
  • The system priority, which should be the same on both POAs, and a priority of 1 is recommended;
  • The node ID, which must be different on each POA.

 

RP/0/0/CPU0:ios(config)#redundancy iccp group <ICCP group ID>
RP/0/0/CPU0:ios(config-redundancy-iccp-group)#mlacp system mac <ID unique to group, same on both POAs>
RP/0/0/CPU0:ios(config-redundancy-iccp-group)#mlacp system priority 1
RP/0/0/CPU0:ios(config-redundancy-iccp-group)#mlacp node <ID unique to each device in the group>
RP/0/0/CPU0:ios(config-redundancy-iccp-group)#commit

Next,  a bundle needs to be added to the ICCP group, with a configured MAC  address. In order to protect against flaps during switchover operations,  please add the recommended configuration items below. Matching  configuration is required on both POAs. This includes using the same  number for the bundle interface, and the same MAC address.

RP/0/0/CPU0:ios(config)#interface Bundle-Ether <x>
RP/0/0/CPU0:ios(config-if)#mac-address 0001.0002.0003
RP/0/0/CPU0:ios(config-if)#bundle wait-while 100
RP/0/0/CPU0:ios(config-if)#lacp switchover suppress-flaps 300
RP/0/0/CPU0:ios(config-if)#mlacp iccp-group <ICCP group ID>
RP/0/0/CPU0:ios(config-if)#commit

On the POAs, the last step is to add members to the bundle, as you would with normal LACP. The period configuration is optional; it enables faster LACP timeouts.

RP/0/0/CPU0:ios(config)#interface <Ethernet interface>
RP/0/0/CPU0:ios(config-if)#bundle id <x> mode active
RP/0/0/CPU0:ios(config-if)#lacp period short

 

4. DHD setup

 

All  that's left is to add the bundle and member configuration on the DHD.  If the DHD is an XR box running the new software, the configuration is  as follows:

RP/0/0/CPU0:ios(config)#interface Bundle-Ether <y>
RP/0/0/CPU0:ios(config-if)#bundle wait-while 100
RP/0/0/CPU0:ios(config-if)#lacp switchover suppress-flaps 300
RP/0/0/CPU0:ios(config-if)#root
RP/0/0/CPU0:ios(config)#interface <each interface connected to POAs>
RP/0/0/CPU0:ios(config-if)#bundle id <y> mode active
RP/0/0/CPU0:ios(config-if)#lacp period short
RP/0/0/CPU0:ios(config-if)#commit

Any  non-XR device (or XR up to R3.9) should have a bundle and members  configured similarly. NB: Against some implementations the expected  behavior is that the bundle will flap on a switchback event for up to 2  seconds. To avoid this, lacp fast-switchover (IOS) or equivalent configuration is required.

If lacp fast-switchover or similar configuration is not available on the DHD, lacp switchover suppress-flaps 2500 configuration can be added on the bundle on the POAs to avoid the state  flap. However, this will result in ~2 seconds of traffic loss on the  switchback event (though the bundle stays up).

If the lacp switchover suppress-flaps configuration or some kind of state dampening is not used or not  available on the DHD, a bundle flap on the DHD on a switchover or  switchback event is expected.

 

5. Checking status

 

The  members added to the bundle on one POA will go Active, and the members  on the other POA will be in Standby state. This can be verified using show bundle on either POA to display the membership information for correctly configured members on both the POAs:

RP/0/0/CPU0:ios#show bundle 
Mon Jun  7 06:02:46.507 PDT

Bundle-Ether1
  Status:                                    Up
  Local links <active/standby/configured>:   1 / 0 / 1
  Local bandwidth <effective/available>:     1000000 (1000000) kbps
  MAC address (source):                      0000.deaf.0000 (Configured)
  Minimum active links / bandwidth:          1 / 1 kbps
  Maximum active links:                      64
  Wait while timer:                          100 ms
  Load balancing:                            Default
  LACP:                                      Operational
    Flap suppression timer:                  300 ms
    Cisco extensions:                        Disabled
  mLACP:                                     Operational
    ICCP Group:                              1
    Role:                                    Active
    Foreign links <active/configured>:       0 / 1
    Switchover type:                         Non-revertive
    Recovery delay:                          300 s
    Maximize threshold:                      Not configured
  IPv4 BFD:                                  Not configured

  Port                  Device           State        Port ID         B/W, kbps
  --------------------  ---------------  -----------  --------------  ----------
  Gi0/0/0/0             Local            Active       0x8001, 0x9001     1000000
      Link is Active
  Gi0/0/0/0             5.4.3.2          Standby      0x8002, 0xa001     1000000
      Link is marked as Standby by mLACP peer
RP/0/0/CPU0:ios#


Switchover

 


 

To switch which POA is active you can use the following CLI command on the currently active router:

mlacp switchover Bundle-Ether 1

The following example illustrates a switchover using this command.

RP/0/0/CPU0:ios#show bundle 
Mon Jun  7 06:02:46.507 PDT

Bundle-Ether1
  Status:                                    Up
  Local links <active/standby/configured>:   1 / 0 / 1
  Local bandwidth <effective/available>:     1000000 (1000000) kbps
  MAC address (source):                      0000.deaf.0000 (Configured)
  Minimum active links / bandwidth:          1 / 1 kbps
  Maximum active links:                      64
  Wait while timer:                          100 ms
  Load balancing:                            Default
  LACP:                                      Operational
    Flap suppression timer:                  300 ms
    Cisco extensions:                        Disabled
  mLACP:                                     Operational
    ICCP Group:                              1
    Role:                                    Active
    Foreign links <active/configured>:       0 / 1
    Switchover type:                         Non-revertive
    Recovery delay:                          300 s
    Maximize threshold:                      Not configured
  IPv4 BFD:                                  Not configured

  Port                  Device           State        Port ID         B/W, kbps
  --------------------  ---------------  -----------  --------------  ----------
  Gi0/0/0/0             Local            Active       0x8001, 0x9001     1000000
      Link is Active
  Gi0/0/0/0             5.4.3.2          Standby      0x8002, 0xa001     1000000
      Link is marked as Standby by mLACP peer
RP/0/0/CPU0:ios#mlacp switchover Bundle-Ether 1
Sun Jan 31 23:46:43.706 PST
This will trigger the peer device (0/0/CPU0 (0x0)) to become active for Bundle-Ether1. This may result in packet loss on the specified bundle.

Proceed with switch over? [confirm]

RP/0/0/CPU0:Jan 31 23:46:44.666 : BM-DISTRIB[282]: %L2-BM-5-MLACP_BUNDLE_ACTIVE : This device is no longer the active device for Bundle-Ether1
RP/0/0/CPU0:Jan 31 23:46:44.668 : BM-DISTRIB[282]: %L2-BM-6-ACTIVE : Gi0/0/0/0 (0x20000020) is no longer Active as part of Bundle-Ether1 (Not enough links available to meet minimum-active threshold)

RP/0/0/CPU0:ios#show bundle 
Mon Jun  7 06:04:17.778 PDT

Bundle-Ether1
  Status:                                    mLACP hot standby
  Local links <active/standby/configured>:   0 / 1 / 1
  Local bandwidth <effective/available>:     0 (0) kbps
  MAC address (source):                      0000.deaf.0000 (Configured)
  Minimum active links / bandwidth:          1 / 1 kbps
  Maximum active links:                      64
  Wait while timer:                          100 ms
  Load balancing:                            Default
  LACP:                                      Operational
    Flap suppression timer:                  300 ms
    Cisco extensions:                        Disabled
  mLACP:                                     Operational
    ICCP Group:                              1
    Role:                                    Standby
    Foreign links <active/configured>:       1 / 1
    Switchover type:                         Non-revertive
    Recovery delay:                          300 s
    Maximize threshold:                      Not configured
  IPv4 BFD:                                  Not configured

  Port                  Device           State        Port ID         B/W, kbps
  --------------------  ---------------  -----------  --------------  ----------
  Gi0/0/0/0             Local            Standby      0x8003, 0x9001     1000000
      mLACP peer is active
  Gi0/0/0/0             5.4.3.2          Active       0x8002, 0xa001     1000000
      Link is Active
RP/0/0/CPU0:ios#

This  command should cover most of the required information. However, for a  more detailed look, there is also an mLACP-specific command available.  This gives you information about each router in each redundancy group,  and the state and configuration each has advertized for each bundle.

 

RP/0/0/CPU0:ios#show mlacp 
Mon Jun  7 06:05:36.901 PDT

ICCP Group 1
  Connect timer:  Off

  Node  LDP ID           State         System ID                 Sync   Vers
  ----  ---------------  ------------  ------------------------  -----  ----
     1  Local            Up            0x0001,00-0d-00-0e-00-0f  Done   -   
     2  5.4.3.2          Up            0x0001,00-0d-00-0e-00-0f  Done   1   


  Bundle-Ether1 (ROID: 0000.0001.0000.0000)
    Node  Aggregator Name       State       Agg ID  MAC Address
    ----  --------------------  ----------  ------  ---------------
       1  Bundle-Ether1         Up          0x0001  0000.deaf.0000
       2  BE1                   Up          0x0001  0000.deaf.0000

RP/0/0/CPU0:ios#

 

Troubleshooting

In this section some basic verification tips that can be checked out that hopefully pinpoint to the culprit when MC-LAG is not working as we expected.

 

6. Information to Collect

Please collect the following information to be provided in any triage request for mLACP issues. Before submiting a TAC case please follow the steps set out in the following sections to diagnose some more common faults and misconfigurations: 

 

  1. Details of the symptoms, and which POA is experiencing them (with timestamps if possible). 
  2. The sequence of events that triggered the issue. [mLACP states can be history-dependent so it can be important to know the previous bundle configuration.] 
  3. The following logs from both of the POAs: 

    1. show tech bundle 

    2. show tech bundle in admin mode. 

    3. show iccp group 

    4. show iccp counters 

    5. show iccp trace 

  4. The following logs from the DHD: 
    1. show tech bundle 

    2. show tech bundle in admin mode.  

 

7. Is this a bundle issue?

Checking ICCP

If you are seeing unexpected mLACP behavior, the first thing to check the health of the ICCP layer (which mLACP relies on) to ensure it is in the expected state. Normally ICCP should be up with a group member present (unless you are testing split brain or device-level failure):

 

RP/0/0/CPU0:ios#show iccp group
Wed Feb 10 22:58:58.845 PST
Redundancy Group 1
  member ip:5.4.3.2 (ios), up (connected)
    monitor: route-watch (up)
  No backbone interfaces.
  enabled applications: mLACP 
RP/0/0/CPU0:ios#

 

If you have a backbone interface configured, it should normally be showing as up unless core isolation is in effect:

 

RP/0/0/CPU0:ios#show iccp group
Wed Feb 10 23:04:03.406 PST
Redundancy Group 1
  member ip:5.4.3.2 (ios), up (connected)
    monitor: route-watch (up)
  backbone interface Gi0/0/0/3: up
  enabled applications: mLACP 
RP/0/0/CPU0:ios#

 

If the output of this command is not what you are expect, the issue needs to be dealt with by the ICCP team. Please refer to information from the ICCP team, or contact them if you believe there is a problem.

 

Features & Protocols

 

If there is a problem with features that run on the bundle but the bundle IM state/LACP state is correct, then the team for the particular feature should be contacted. As with normal bundle interfaces, the Bundle Infra controls the bundle interface itself, and other components are responsible for running their services over the bundle.

This info will help you direct the TAC where to send the case to. Is it really an MC-LAG issue, or a feature issue (that might exist on a regular bundle also) or is there just an issue with teh bundle itself. These items are handled by different development groups, so it helps to drill down the failure issue to a most common set of problems that will inherently mean a faster case resolution as the sw problem can be dealt with immediatley with the right group.

 

8. Bundle Infra issues

The bundle is down

 

To identify the issue, please go through the following steps:

 

  1. Check whether ICCP is healthy, as described above. If not, this is likely to be the cause of the issue. Please correct this and see if the problem persists.
  2. Verify that the configuration is correct for the ICCP group and the bundles. Correct any misconfigurations and see if the issue persists.

  3. Check that the minimum-active links/bandwidth configuration is set to an appropriate value (i.e. smaller than the number of links in the bundle/their total bandwidth).

  4. The output of the show bundle command should tell you the reason any member links are not Active. The output of this command (combined with the table of reasons with additional information should indicate the cause of the problem and how to fix it.

  5. If you are still seeing that the bundle is down, please collect the information listed above.

 

 

9. The bundle flapped on the POA

 

Bundle flaps are sometimes expected on mLACP events, but they are usually the result of misconfigurations. So the first thing to do is check that all your configuration is correct. Please refer to the configuration guidance for details, and specifically take note of the following:

 

  1. If a protocol flap is seen (e.g. OSPF) but not a flap in the state of the bundle interface itself, please refer to the feature-specific wiki and follow up with the team responsible for the feature.
  2. The mlacp system priority and mlacp system mac configured under redundancy iccp group <x> must be configured to be the same value on both devices in the RG.

  3. The same mac-address should  be configured on the bundle interface on both POAs.

  4. If the DHD supports bundle wait-while 100 (XR 4.0) or lacp fast-switchover (XR up to 3.9 & IOS) or equivalent, this should be configured on the DHD.

  5. If the DHD has bundle wait-while 100 configuration, lacp switchover suppress-flaps 300 should be configured on the POAs. Otherwise, lacp switchover suppress-flaps 2500 is required.

  6. The bundle should be configured with bundle wait-while 100 on the POAs.

  7. Check for any syslogs or bistate alarms that could indicate the cause of the flap.

 

If you're still seeing a flap, please collect the information requested above.

 

 

10. The bundle flapped on the DHD

 

Bundle flaps on the DHD are usually expected during switchover events. If an XR 4.0 device is in use, you can configure bundle wait-while 100 and lacp switchover suppress-flaps 300 on the bundle to avoid a flap (assuming the POAs are set up correctly as above). If other DHD software provides similar functionality it can be used, otherwise a flap cannot be avoided.

 

 

11. Switchover did not occur

 

This is likely to mean one of the following:

 

  1. The bundle is Down on the Standby POA (and was Down before the switchover attempt). The output of show bundle on the Standby POA should indicate the reason for this.

  2. The two POAs are not communicating over ICCP. See the "Checking ICCP" section above for details.

 

12. Both POAs are Active

 

This should only happen when the POAs are failing to communicate over ICCP. See the "Checking ICCP" section above for details.

 

13. The bundle is Down on the Active POA

 

This can happen if:

 

  1. There is no Standby POA to switch over to, or the bundle is Down on the Standby POA.
  2. The required number of links have been put in LACP Selected state, but have not been Activated (e.g. due to failing to progress further in LACP negotiations). A switchover cannot be performed in this scenario. This could indicate a misconfiguration on the DHD.

 

14. The bundle is Up on the Standby POA

 

This is expected. Please see the section on Interface State in the events and scenarios section.

 

Events and Scenarios

15. Initial Bringup

 

By  default, when the first device is added to the ICCP group, it does not  enable any bundles in that ICCP group until negotiations with a peer  have been completed. This is to reduce churn when adding a device to the  group (or reloading the device).

This means that without an mLACP peer, mLACP bundles cannot be enabled. It could also lead to the following:

  1. Two routers (A and B ) are configured as mLACP peers and start operating their mLACP bundles.
  2. Router B crashes. Router A takes over operation of all bundles and continues running on its own.
  3. Router  A is reloaded. After it comes up it waits indefinitely for the peer,  and therefore does not bring up any of the mLACP bundles.

If this is undesirable, the following configuration can be used:

RP/0/0/CPU0:ios(config)#redundancy iccp group 1 mlacp connect timeout ?
  <0-65534>  Number of seconds to wait before assuming mLACP peer is down.
RP/0/0/CPU0:ios(config)#

If  a connect timeout is configured and no peer device is present, mLACP  bundles will be enabled once the timeout has elapsed after joining the  ICCP group.

 

16. mLACP Active and Standby

 

Each  of the mLACP peer devices is either the Active or the Standby device  for each bundle. On each device, its status (Active or Standby) is  displayed in the mLACP section of the show bundle output.

 

Interface State

 

The bundle interface state (as displayed in show interfaces)  can be Up or Down on either the Active or the Standby device for the  bundle. On both devices, under normal conditions, the bundle should be  Up.

The bundle interface state will be Down on the Active device if:

  1. The  minimum link threshold could not be met (or no links could be brought  up) and there is no mLACP peer advertising that it is ready to bring the  bundle Up (so a switchover cannot be performed).
  2. The  required number of links have been put in LACP Selected state, but have  not been Activated (e.g. due to failing to progress further in LACP  negotiations). Again, a switchover cannot be performed in this case.  This could indicate a misconfiguration on the DHD.

The bundle interface will be Down on the Standby device if:

  1. The device is not ready to forward bundle traffic; if the bundle goes down on the Active device, traffic will be lost.
  2. The device is cold standby for the bundle (see next subsection).

 

Hot vs Cold Standby

 

In the show bundle output, the state of the bundle on the standby POA could be either of the following:

  • mLACP Hot Standby:  This bundle has enough links available, it is Up in IM, and it has the  required POA configs to take over without a flap if the active router  goes down. NB: This does not guarantee that the bundle will take over without a flap. Incorrect/incompatible configuration on the DHD could still lead to a flap.

  • mLACP Cold Standby:  The bundle has enough links available to take over if required.  However, it is marked as Down in IM because it will have to be Down  initially after the switchover while LACP negotiations are in progress.  This is due to some missing configuration that is required for a  seamless switchover; e.g. lacp switchover suppress-flaps.

Please see the

 

17. Switchover

 

The  following subsections provide more details on mLACP switchover and  switchback events and the behavior under various conditions.

 

Switchover Types

 

The mLACP switchover method can be modified using the following CLI:

RP/0/0/CPU0:ios(config)#interface bundle-ether 1 mlacp switchover type ?
  brute-force    Force switchover by disabling all local member links
  revertive      Revert based on configured priority values
RP/0/0/CPU0:ios(config)#interface bundle-ether 1 mlacp switchover type 

These  options (and the default, when this configuration is not present) are  mutually exclusive, and the value must match on the bundle on both POAs.   They determine whether the dynamic priority management or brute force mechanism is used, and whether the behavior is revertive or non-revertive.

The behavior can also be modified to maximize the links or bandwidth available using one of the following CLIs:

interface <bundle> mlacp switchover maximize bandwidth [threshold <kbps>]
interface <bundle> mlacp switchover maximize links [threshold <count>]

This  causes the active device to be the one with more bandwidth or more  links available in the bundle. If a threshold is specified, this  behavior only takes effect once the available links/bandwidth on one  device falls below the threshold. This must be configured symmetrically  on both devices.

 

Dynamic Priority Management

 

Dynamic priority management means that a switchover is achieved gracefully by modifying the LACP port priorities.

  • Before any switchovers have occurred, the LACP port priorities for the members on each POA match the value configured using the mlacp port-priority configuration for the bundle interface, or 32768 as a default if there is no value configured.
  • To  perform a switchover, the priorities at which the links are operating  on at least one POA are changed automatically by the POA software so  that the links to the POA which is to go active are higher-priority  (have a smaller LACP priority number) than the links attached to the  other POA. (This means the configured value can differ from the  operational value.)
  • LACP  demands the highest-priority links must be selected ahead of  lower-priority ones (by the DHD as well as the POAs), which is why this  action triggers a switchover.
  • In  almost all cases, the port priority of the currently-active POA is  reduced (the operational priority number is increased) to trigger a  switchover to the peer POA. This avoids churn in the LACP state machines  on the links to the POA becoming active.
  • If  this is not possible (i.e. if the operating priority of the standby POA  is 65535 so there are no larger numbers available) the priorities of  both are modified; the newly active POA's links take priority 1 and the  other POA's take priority 2.

It  is possible to reset the priorities used on both POAs to their initial  values. Doing so will cause a switchover if the device with the higher  configured port priority is in the standby role at the time the command  is issued.

 

RP/0/0/CPU0:ios#mlacp reset priority Bundle-ether 1 
Sun Aug 28 16:12:58.110 BST

Warning: this will reset all negotiated mLACP sessions on Bundle-Ether1.

If traffic is flowing over this bundle, traffic loss WILL occur.

Proceed with priority reset? [confirm]

RP/0/0/CPU0:Aug 28 16:13:02.983 : BM-DISTRIB[1132]: %L2-BM-6-MLACP_BUNDLE_ACTIVE : This device is now the active device for Bundle-Ether1 

 

Brute Force

 

The brute force mechanism does not involve any priority changes. When using brute  force, the operational priority always matches the configured priority. A  switchover is performed as follows:

  • When  the bundle goes down on the active POA (e.g. the minimum-active links  threshold is breached), the LACP state machine is updated for any ports  that are not encountering problems on the active POA. (Any ports which  have lost connectivity to trigger the condition should already have been  deselected.)
  • These updates to the LACP states are transmitted over mLACP to the other POA.
  • A  final LACP packet is sent to the DHD on each of the links to the Active  POA to cause the DHD to deselect them as well (and select links to the  other POA in preference).
  • After  this, there is no further LACP transmission on those links. This means  that the state of these links is not being monitored by LACP.
  • To  recover from this situation, once the link that failed has recovered  (enough to be LACP Selected) LACP operations are restarted on the  remaining links.

 

Revertive Behavior

 

Revertive behavior means that the bundle effectively has a "primary" and a  "secondary" POA. The bundle is active on the primary POA whenever  possible. It is only active on the secondary POA when it is down on the  primary, or the secondary has more available links and the maximize threshold has been breached.

The calculation to determine which POA is the "primary" for the bundle is as follows.

  • If the POAs have different mlacp port-priority configuration for the bundle, the one with the higher priority configured (smaller number) is the primary.
  • If the configured bundle mLACP priorities match, the POA with the smaller mlacp node configuration in the redundancy group is the primary.
  • NB: Between two POAs in an RG sharing multiple bundles, it's possible for each POA to be a primary for some of the bundles.

Bu  default, when the primary POA recovers, it delays going active for 300  seconds to allow higher-layer ICCP-aware protocols (e.g. routing  protocols) to sync their state between devices. This helps avoid churn  and downtime at higher layers. The delay time can be modified using the  following configuration setting:

interface <bundle> mlacp switchover recovery-delay <time in seconds>

 

Non-revertive Behavior

 

Non-revertive behavior does not consider POAs to be "primary" and "secondary"; only  "active" or "standby". The configured priorities are only used to  determine which POA is initially active. After this point, the active POA remains active until it encounters a failure, or mlacp maximize settings are in effect and the peer router has more links or bandwidth available (see later for details).

So without mlacp maximize there is no "switchback" in this mode; when a POA recovers from a failure, it remains standby until the other POA fails.

This mode of operation causes least churn, and is the recommended option.

 

Notes

 

The configuration options map to the following:

  • Default: Dynamic priority management with non-revertive behavior.
  • brute-force: Brute force mechanism with revertive behavior.
  • revertive: Dynamic priority management with revertive behavior.

Two things to note about the available settings:

  • Brute  force operation is not compatible with non-revertive operation; it is  inherently revertive because the POA with the higher operational  priority is fixed by configuration. So a combination of these options is  not available.
  • If  the DHD is the higher priority LACP system the LACP port priorities set  by the POAs for the members are ignored. So the only mechanism that can  be used is brute force, regardless of the configuration. An alarm is  raised by the Bundle Infra on the bundle to indicate when this happens.

 

18. Switchover Triggers

 

The following events can trigger a switchover to the mLACP peer:

  1. Link failure: A port or link between the DHD and one of the POAs fails.
  2. Device failure: Meltdown or reload of one of the POAs, with total loss of connectivity (to the DHD, the core and the other POA).
  3. Core isolation:  A POA loses its connectivity to the core network and therefore is of no  value, being unable to forward traffic to or from the DHD.

NB: Other documents (e.g. from IOS) may discuss 5 failure cases, listed as A-E. These map onto the above as follows:

  1. Failure of upstream port on the DHD: Covered by link failure case.
  2. Failure of link between DHD and POA: Covered by link failure case.
  3. Failure of downstream port on the POA: Again considered a type of link failure in the above set.
  4. Node failure: Described here as device failure, to clarify that it describes the router failing.
  5. PE isolation: Named core isolation here.

The  following sections give more detail on each type of failure, how they  can be produced, and the expected results. Each of these events only  result in a switchover if the bundle is in (hot or cold) standby state  on the standby POA; otherwise the bundle will be Down on both routers.

 

 

This can be triggered for testing purposes by disabling an active bundle member when:

  • The bundle on standby POA is listed in an mLACP Standby state in show bundle.
  • There are no Standby links connected to the currently active POA.
  • Either of the following is the case:        
    • Once the link is removed, there is not enough bandwidth/links in the bundle on that POA to meet a configured minimum-active threshold.
    • Once the link is removed, the amount of links/bandwidth in the bundle is below the mlacp maximize threshold and the standby POA has more links/bandwidth available in the bundle than the active POA.

The member can be removed from the bundle using any of the following events:

  • Fiber pull at either end of the POA-DHD link. (NB: This is the most representative way of testing this scenario.)
  • Shut down the POA-DHD link on the DHD.
  • Shut down the POA-DHD link on the POA.
  • Remove the bundle membership configuration on either end of the POA-DHD link.

 

Device Failure

 

This  represents a meltdown of the router carrying the traffic; total loss of  the node. The only valid test trigger for this event is a hard reset  (power down) of the device, even a router reload is not truly  representative as the device will tend to go down in a series of  indeterminate stages.

 

Core Isolation

 

This  is an ICCP event meaning that connectivity to the network core has been  lost. The configuration for core connectivity monitoring is owned by  ICCP and not the Bundle Infra, so the following is only a brief  description of the basic steps required; please see ICCP documentation  for more details.

One or more "backbone interfaces" can be configured under the redundancy group for monitoring by ICCP:

RP/0/0/CPU0:ios(config)#redundancy iccp-group <id> 
RP/0/0/CPU0:ios(config-redundancy-iccp-group)# backbone interface <core-facing interface name>

A  core isolation event will be declared by ICCP when all the specified  interfaces are operationally down, or not present (e.g. node reload).  (If the interfaces do not exist or are down when they are configured,  core isolation is declared immediately. In all cases it can be cleared  by removing the backbone interface configuration.)

If  a POA experiences this event it sends dying gasp LACP packet to the DHD  on its member links and stops LACP negotiations on them, using the same  mechanism as if a brute force switchover was being performed. However,  it also changes port priorities on all links if dynamic priority  management is in effect (in accordance with the revertive or  non-revertive behavior described above) as appropriate.

 

19. User-Controlled Switchover

 

There  are a couple of configuration commands that can be used to bring a  bundle down on one POA, triggering a switchover to its mLACP peer.  Additionally, in some cases exec CLI commands can be used to control the  switchovers.

 

Bundle Interface Shutdown

 

The first is simply:

RP/0/0/CPU0:ios(config)# interface Bundle-Ether <x>
RP/0/0/CPU0:ios(config-if)# shutdown

This  brings down all the links in the bundle interface and forces a  switchover to the peer. This can be used in all cases to force a  switchover. However, this means that neither the LACP states nor even  the link states of the bundle members can be monitored while the bundle  is down.

This works for all switchover modes.

 

Bundle Interface Bundle-level Shutdown

 

If a dynamic priority management form of switchover is in use, i.e. the bundle is not configured to use  brute force switchover and the POAs are the higher priority LACP system,  there is an alternative command:

RP/0/0/CPU0:ios(config)# interface Bundle-Ether <x>
RP/0/0/CPU0:ios(config-if)# bundle shutdown

With  this slight variation, instead of bringing the member links down, they  are kept in LACP Standby state. The bundle is declared down so the mLACP  peer takes over. However, LACP continues to operate and the health of  the links can be verified before allowing them to come up again.

This  works for revertive and non-revertive switchover modes (not brute  force). NB: The DHD also has to be the lower priority LACP system,  otherwise brute force will implicitly be used.

 

Non-revertive Switchover CLI

 

CLI  commands are also available for performing a switchover on user demand.  To use these commands, the bundle must be using the default mLACP  switchover behavior (non-revertive). (Otherwise the bundle would have to  simply revert to the originally active router immediately.)

A switchover to the standby POA can be performed by issuing the following command on the active POA:

RP/0/0/CPU0:ios# mlacp switchover Bundle-Ether <x>

If required, the same operation can be performed on the standby POA to become active:

RP/0/0/CPU0:ios# mlacp switchback Bundle-Ether <x>

However, the switchback command could cause a bundle flap on the POA that is becoming active, so it is preferable to use the switchover command on the other POA if possible.

These commands can also be set with a delay, for the switchover to be performed at some point in the future. Some examples:

RP/0/0/CPU0:ios# mlacp switchover Bundle-Ether <x> in 5 minutes
RP/0/0/CPU0:ios# mlacp switchback Bundle-Ether <x> in 3 hours
RP/0/0/CPU0:ios# mlacp switchover Bundle-Ether <x> at 08:45:30

If it is necessary to cancel a scheduled switchover or switchback operation the following commands can be used. The switchover and switchback variants can be used per bundle, the scheduling variant affects all scheduled bundle actions.

 

RP/0/0/CPU0:ios# clear mlacp switchover Bundle-Ether <x>
RP/0/0/CPU0:ios# clear mlacp switchback Bundle-Ether <x>
RP/0/0/CPU0:ios# clear mlacp scheduling

 

mLACP Synchonization

 

Synchronization between mLACP peer devices occurs on a number of occassions.

  • When  connecting to a peer device mLACP will send a full (unsolicited) sync  of all its configuration and state for the relevant ICCP group. It will  also request a full sync from the peer device.
  • When  inconsistent or incompatible data relating to a foreign port or a  mutual object (i.e. a bundle or system setting) is received from a peer a  full resync is requested. The exception to this is the case where the  data is received as part of a resync in which case it is NAKd. See the  following section for more details.
  • When  incorrect data relating to a local port is received from a peer a full  resync of the local configuration and state is sent back to the peer.
  • When  the local node id is changed a full resync of local configuration and  state is sent to the peer device. This is done because the node id forms  part of the identifier for every port so it is necessary to purge the  old port data and replace it with the new.

The  XR implementation of mLACP always requests a full resync although the  protocol allows for resyncs to be requested for particular objects. This  is done for reliability reasons as well as practicalities of  implementation. Similarly when a resync is request is received, or when  sending an unsolicited sync, the XR implementation always responds with a  full resync of local configuration and state regardless of what was  requested. This is fully compatible with the behavior outlined in the  standard.

 

NAKing mLACP Messages

 

Messages can be NAKd for a number of reasons, including but not limited to

  • Invalid TLV contents (e.g. value out of range)
  • Incompatibility between local and peer device (e.g. clashing node id or different bundle ROID on each device)
  • Object or parent object referenced in the message is not found in the local database

As  explained above, typically when a message is received for which there  is some sort of issue a resync is first requested (or sent). If an issue  is detected within an incoming sync, or for problems such as an  incorrect ROID for a bundle which cannot be resolved with a resync, then  a message is NAKd.

When  a message is NAKd the object referred to the in the message is disabled  in some way. The behavior depends on the type of object.

  • If a NAK for a local port, bundle or system is received it is disabled for LACP operation
  • If a NAK for a remote port, bundle or system is sent it is removed from the local database
  • If a NAK for a remote bundle or system is sent the corresponding local object is disabled for LACP operation

All operations cascade to child objects.

To  re-enable an object after it has been NAKd there must be some change in  its configured state that causes a new Config TLV to be sent. This  resets the NAKd state, although it is possible that the new Config TLV  will be immediately NAKd again if the change is not acceptable either.

 

20. Configuration Changes

 

Certain  elements of mLACP configuration are key, and mismatches or even  configuration changes can have an impact on mLACP operation.

 

mLACP Node ID

 

The value configured for the mlacp node under the ICCP group submode is used in the LACP parameters for bundles  in that ICCP group. This value must be present for bundles in the ICCP  group to be operational, and must differ on each device in the ICCP  group.

When  this value is modified, the LACP session must be renegotiated on each  link of each bundle in the ICCP group. This causes the interface state  of all the bundles to flap.

 

mLACP System ID

 

The mlacp system priority and mlacp system mac values under the ICCP group are also required for bundles in the ICCP  group to be operational. These values must be the same on each router in  the ICCP group.

Once  again, these values are used in LACP negotiations, so changing them  causes all the bundles to flap. (This is why they must be the same on  all devices: Otherwise, when the Active device fails, all the LACP  sessions on the Standby have to be renegotiated using its system ID.)

 

21. Split Brain

 

If  the ICCP link between the POAs goes down but both POAs remain up, a  "split brain" situation is said to have occurred, meaning that the POAs  cannot exchange state information and are not aware of each other's  presence.

In  this scenario, both POAs would attempt to go active: From each one's  perspective, it appears the other has encountered a device-level  failure. This can be protected against in a limited manner in some  cases, using DHD control as described below.

 

22. DHD Control

 

In the recommended configuration,  the switchovers are controlled only by the POAs. The DHD is always  trying to make all links active, and the required set are kept in  standby by the POA.

However, if the DHD supports maximum-active links configuration, this can be used to protect against both POAs going  active in a split brain scenario if all of the following conditions are  met:

  • The DHD has the same number of links to each POA in the bundle.
  • The POAs are configured with minimum-active links <number of links to the DHD>.
  • The DHD is configured with maximum-active links <number of links to each POA>.

Additionally,  if either POA has brought down its links by brute-force (i.e. due to  brute-force behavior being in effect), forwarding on the bundle could  stop (i.e. the bundle could go down on both POAs) when the split brain  event occurs. Therefore dynamic priority management is recommended.

If  a split brain event occurs with this configuration present, the POAs  will continue to operate with the same priorities as they had before and  will both try to go active. However, the DHD will only allow the links  to the POA with higher link priorities to go active due to the  maximum-active links configuration.

NB:  A switchover cannot be coordinated between the POAs while split brain  is in effect. So if the active POA encounters a failure, there is no  guarantee that bundle will switch over to the other POA. (The exact  behavior depends on the number of bundle member links and the manner of  the failure.)

 

 

 

Syslog messages

 

The following lists all the possible syslog messages from the bundle infra specific to mLACP, along with their meanings. In addition many of the RED_MGMT messages relate to LACP and mLACP and may be relevant. Refer to the documentation for that message in the regular troubleshooting guide.

 

 

MLACP_CANNOT_SWITCHOVER: Could not perform mLACP switchover/switchback requested by user for bundle <name>: <reason>

 

A switchover request was triggered using the mlacp switchover <bundle> CLI, and the Bundle Infra has attempted to perform the switchover. However, the bundle was not in an appropriate state to switch over, so no action was taken. This could be because:

 

  • One of the mLACP peers is down.
  • The bundle is not active on the node being switched from.
  • The switchover behavior in operation is not non-revertive.

 

MLACP_CONNECT_FAILED: Failed to connect to another mLACP device in ICCP Group <id>. Reason: <reason>

 

The connection failed so mLACP will not operate over the specified ICCP group. Most likely this is be because no peer device is configured to run mLACP in this group. Other possibilities include

 

  • Version mismatch between the two devices. The output of show mlacp can be used to check the version the device is using.

  • There are more ICCP connections than can be supported currently set up; this can be checked in configuration and corrected if required. To retry the connection, remove and re-add the mlacp node configuration under the ICCP group.

 

MLACP_CONNECT: Connected to <LDP ID>

 

A connection to the specified mLACP peer device has been established.

 

MLACP_DISCONNECT: Disconnected from <LDP ID>

 

The connection to the peer device identified has been terminated.

 

MLACP_SYSTEM_ID_ARBITRATION: The system ID for ICCP group <id> has been established by arbitration

 

If the system ID was established by arbitration then there is a misconfiguration or unadvisable choice of configuration; a different mlacp system priority or mlacp system mac has been specified for the same ICCP group by different peers. One of the values is chosen for operational purposes, but if the peer who owns that value disconnects, the other device will stop using that value, which could trigger a bundle flap.

 

Correct the misconfiguration to clear the alarm by configuring the same value for those configuration items on each router in the RG.

 

MLACP_BUNDLE_MAC_ARBITRATION: The MAC address for <bundle name> has been selected through arbitration.

 

This is the same as the system ID arbitration alarm but for the mac-address configuration under the bundle interface. This should also be the same for a bundle on all devices operating that bundle.

 

MLACP_UNRESOLVABLE_MISCONFIG_DISCONNECT: Disconnecting from <LDP ID> due to an unresolvable misconfiguration: <reason>
MLACP_RESYNC_INCONSISTENCY_DISCONNECT: Disconnecting from <LDP ID due to an inconsistency in the mLACP data that could not be resolved with a resynchronization. Reason: <reason>

 

The configuration specified is mismatched between the two POAs.

 

To recover from this situation you must correct the configuration mismatch, and then remove and re-add an item of RG mLACP configuration (e.g. the mlacp node under the ICCP group submode) on both POAs.

 

MLACP_DEVICE_MISCONFIGURATION: <reason>

 

Correct the specified misconfiguration to clear this issue.

 

MLACP_ITEM_MISCONFIGURATION: <details>

 

This message indicates a possible issue, but can be safely ignored. If there is an ongoing issue, an MLACP_DEVICE_MISCONFIGURATION message will be emitted shortly afterwards.

 

MLACP_ROID_MISMATCH: The ROID (<value>) received from <LDP ID> for <bundle name>, does not match that expected (<value). Please ensure that the ROID for the bundle is the same on both devices.

 

In 4.0, the ROID of the bundle is generated in a set format from the bundle ID, in both the XR and IOS implementations. However, in future it is possible that the ROID will be configurable. So the current implementation emits this message if it gets an unexpected value for the ROID.

 

MLACP_CORE_ISOLATION: <bundle name> marked as isolated due to not being able to connect to the core.

 

ICCP has declared a core isolation event for the redundancy group. As a result, the bundle infra has declared this bundle as isolated, and will switch over to the standby POA if it is available.

 

MLACP_BUNDLE_PEERING: <bundle name> is peering with <LDP ID>
MLACP_BUNDLE_PEERING: <bundle name> is no longer peering with <LDP ID>

 

This message indicates that the peer device being used for operating mLACP on this bundle (or that there is no longer an mLACP peer for this bundle).

 

MLACP_BUNDLE_ACTIVE: This device is now the active device for <bundle>
MLACP_BUNDLE_ACTIVE: This device is no longer the active device for <bundle>

 

This message indicates that the local device has taken on the Active role for the bundle in question, or has surrendered that role to the peer. This may be an expected event or may indicate that some fault has occured to trigger a switchover. Investigate as appropriate.

 

 

 

Recovering from failures

 

If you've identified a problem and collected all the necessary information, or you've hit a known issue, there are some common steps you might be able to use to recover the testbed without needing to reload:

 

  • Shut/no shut the bundle or member that's causing a problem.
  • Unconfigure/reconfigure the bundle or member.
  • process restart bundlemgr_distrib

  • process restart mpls_ldp

  • Remove then add some mLACP configuration under the ICCP group on both POAs, e.g. the mlacp nodde (to cause a reconnection to the ICCP group by mLACP).

 

23.

Simple quick config blocks

 

The connections in this topology are as follows:

 

   DHD                      POA 1                     POA 2

Gi0/0/0/0 --------------- Gi0/0/0/0
Gi0/0/0/1 --------------- Gi0/0/0/1
Gi0/0/0/2
Gi0/0/0/3 ----------------------------------------- Gi0/0/0/0
Gi0/0/0/4 ----------------------------------------- Gi0/0/0/1
                          Gi0/0/0/2                 Gi0/0/0/2
                          Gi0/0/0/3 --------------- Gi0/0/0/3
                          Gi0/0/0/4 --------------- Gi0/0/0/4

 

 

POA 1
redundancy
 iccp
  group 1
   mlacp node 1
   mlacp system mac 000d.000e.000f
   mlacp system priority 1
   member
    neighbor 5.4.3.2
   !
  !
 !
!
interface Bundle-Ether1
 lacp switchover suppress-flaps 300
 mlacp iccp-group 1
 mac-address 0.deaf.0
 bundle wait-while 100
!
interface Loopback0
 ipv4 address 5.4.3.1 255.255.255.255
!
interface GigabitEthernet0/0/0/0
 description Connected to DHD Gi0/0/0/0
 bundle id 1 mode active
 lacp period short
 no shutdown
!
interface GigabitEthernet0/0/0/3
 description Connected to POA2 Gi0/0/0/3
 ipv4 address 1.2.3.1 255.255.255.0
 proxy-arp
 no shutdown
!
router static
 address-family ipv4 unicast
  5.4.3.2/32 1.2.3.2
 !
!
mpls ldp
 router-id 5.4.3.1
 discovery targeted-hello accept
 log
  neighbor
 !
 interface GigabitEthernet0/0/0/3
 !
!

 


 

 

POA 2
redundancy
 iccp
  group 1
   mlacp node 2
   mlacp system mac 000d.000e.000f
   mlacp system priority 1
   member
    neighbor 5.4.3.1
   !
  !
 !
!
interface Bundle-Ether1
 lacp switchover suppress-flaps 300
 mlacp iccp-group 1
 mac-address 0.deaf.0
 bundle wait-while 100
!
interface Loopback0
 ipv4 address 5.4.3.2 255.255.255.255
!
interface GigabitEthernet0/0/0/0
 description Connected to DHD Gi0/0/0/3
 bundle id 1 mode active
 lacp period short
 no shutdown
!
interface GigabitEthernet0/0/0/3
 description Connected to POA1 Gi0/0/0/3
 ipv4 address 1.2.3.2 255.255.255.0
 proxy-arp
 no shutdown
!
router static
 address-family ipv4 unicast
  5.4.3.1/32 1.2.3.1
 !
!
mpls ldp
 router-id 5.4.3.2
 discovery targeted-hello accept
 log
  neighbor
 !
 interface GigabitEthernet0/0/0/3
 !
!

 

 

 

On the DHD
interface Bundle-Ether1
 lacp switchover suppress-flaps 300
 bundle wait-while 100
!
interface GigabitEthernet0/0/0/0
 description Connected to POA1 Gi0/0/0/0
 bundle id 1 mode active
 lacp period short
 no shutdown
!
interface GigabitEthernet0/0/0/3
 description Connected to POA2 Gi0/0/0/0
 bundle id 1 mode active
 lacp period short
 no shutdown
!

 

 


 

 

 

 

 

 

Xander Thuijs, CCIE #6775

  • Principal Engineer ASR9000
Comments

Thanks a lot Xander for this outstanding document. I read that in XR 4.3.1 there is support to pseudo-MC-LAG (Active/Active). Would it be possible to elaborate on that point as well?

Thanks,

Michel.

xthuijs
Cisco Employee
Cisco Employee

Thanks for the feedback Michel!

yes let me get cranking on this, may take a week or so, but I will update the doc with that info.

it basically comes down to one POA is active for vlans 10-20 and the other one for 30-40 (something like that) vlan loadbalancing if you will.

regards

xander

Carlos A. Silva
Level 3
Level 3

Hi, Xander:

I'm implementing the VPLS with decoupled PWs for a customer (but with BGP/AD).

I have a bunch of ASR9ks where the VPLS PWs take place.

The DHDs are 6500s with a LACP etherchannel that seems to be working appropiately.

While the setup works correctly, the switchover recovery time is at around 30s (both ways).

I'm testing this by pinging from an SVI on one 6k to the other.

On the remote side of the PWs I see how the local side MAC address dissapears on switchover like it is supposed to, but it takes 30s to reappear.

As far as I can tell the aggregated ports everywhere are working just fine.

I have 2 questions: are the 30s of recovery time correct? In case you think they are not, do you have any suggestions?

Thanks in advance!

c.

xthuijs
Cisco Employee
Cisco Employee

Hi Carlos, smells like a STP convergence maybe? although MCLAG would not require that...

Can you find out where and why the traffic is getting dropped?

A rapid ping with timeout 0 gives a defined rate in the np counters and we cna potentially see if the issue is in the 9k or somewhere else/to continue finding the culprit.

These scenarios are probably easier handled by the TAC for the right follow up.

30 seconds seems long, I have seen better.

regards

xander

Carlos A. Silva
Level 3
Level 3

Xander:

You're right it's an STP issue. For some reason the 6500 doesn't see both channel links as part of the bundle, so when one goes down, the port-channel an SVIx go down for a bit, and looks like STP has to reconverge.

The weird  thing is that, though I was OK with the mLACP PoA side to see one of the etherchannel ports as 'standby', I was expecting the DHD side to have both etherchannel ports as up and part of the etherchannel.

I'll keep looking.

Thanks,

c.

Carlos A. Silva
Level 3
Level 3

Quick q: in a VPLS-mLACP scenario, should I be using different lacp port-priorities on the lacp etherchannel (PoA side) or the same?

xthuijs
Cisco Employee
Cisco Employee

I see, yeah that will be an STP issue if the bundle on the dhd goes down.

Here some tricks:

The bundle flapped on the POA

Bundle flaps are sometimes expected on mLACP events, but they are usually the result of misconfigurations. So the first thing to do is check that all your configuration is correct. Please refer to the configuration guidance for details, and specifically take note of the following:

  1. If a protocol flap is seen (e.g. OSPF) but not a flap in the state of the bundle interface itself, please refer to the feature-specific wiki and follow up with the team responsible for the feature.
  2. The mlacp system priority and mlacp system mac configured under redundancy iccp group <x> must be configured to be the same value on both devices in the RG.

  3. The same mac-address should  be configured on the bundle interface on both POAs.

  4. If the DHD supports bundle wait-while 100 (XR 4.0) or lacp fast-switchover (XR up to 3.9 & IOS) or equivalent, this should be configured on the DHD.

  5. If the DHD has bundle wait-while 100 configuration, lacp switchover suppress-flaps 300 should be configured on the POAs. Otherwise, lacp switchover suppress-flaps 2500 is required.

  6. The bundle should be configured with bundle wait-while 100 on the POAs.

  7. Check for any syslogs or bistate alarms that could indicate the cause of the flap.

The bundle flapped on the DHD

Bundle flaps on the DHD are usually expected during switchover events. If an XR 4.0 device is in use, you can configure bundle wait-while 100 and lacp switchover suppress-flaps 300 on the bundle to avoid a flap (assuming the POAs are set up correctly as above). If other DHD software provides similar functionality it can be used, otherwise a flap cannot be avoided.

regards

xander

xthuijs
Cisco Employee
Cisco Employee

Must be the same on the POA's Carlos.

cheers

xander

Carlos A. Silva
Level 3
Level 3

Noted. Thanks.

I read your comments on flaps this morning. Can't find a way to delay the port-channel flap on the 6500 (like you do on the ASR). At the moment, we're thinking portfast on the port-channel on 6500 (DHD side).

xthuijs
Cisco Employee
Cisco Employee

hey carlos,

if you dont have a loop (which you probably dont with mclag, then you can indeed set portfast!

if the dhd is not tunable, we have to live with this port flap which will cause the stp convergence.

I cant do anything about it from the 9k/xr side...

regards

xander

Carlos A. Silva
Level 3
Level 3

I think portfast will do it. Just have to wait for a MW to test it.

Will update here if everything goes well. Thanks for everything, Xander.

e.nieuwstad
Level 1
Level 1

we have this particular setup running with a N7K vpc cluster as DHD. Each N7K has one link to the active POA and one link to the standby POA. Failover works like a charm.

Carlos A. Silva
Level 3
Level 3

N7k, should respond faster.

BTW, portfast did the trick, went down to about 1s failover time.

Sadath Puthiyaveettil
Cisco Employee
Cisco Employee

Thanks for this nice document.

Does the ASR9K (POA's)  needs to be directly connected, or the requirement is just to have IP connectivity between POA's.

xthuijs
Cisco Employee
Cisco Employee

Thank Sadath!

no need to be directly connected, as long as there is ip connectivity indeed.

regards!!

xander

Carlos A. Silva
Level 3
Level 3

Xander:

I'm curious about one thing regarding mLACP: when you configure it between 2 directly connected 9ks (haven't tried any other setup), the LDP adjacency goes from NSR:Y to NSR:N/A. Why is this? Is there functionality lost here?

RP/0/RSP0/CPU0:XXXXX#show mpls ldp neighbor  brief

Tue Feb 18 09:29:19.768 CST

Peer               GR  NSR  Up Time     Discovery  Address  IPv4 Label

-----------------  --  ---  ----------  ---------  -------  ----------

[snip]

X.X.X.X:0    Y   N/A  1w3d                2        6        3779

[snip]

RP/0/RSP0/CPU0:XXXXXX#

Thanks,

c.

xthuijs
Cisco Employee
Cisco Employee

this session for ICCP is a targeted LDP session, depending on your release, there may not be a full supprot for NSR/GR on LDP.

if the flag goes to N/A or N it just means that the iccp sessions is not protected against failover with NSR or GR.

same as with a routing protocol.

cheers!

xander

Carlos A. Silva
Level 3
Level 3

Hi, Xander:

I'm running 4.3.2 on this particular setup. Is this supported on 4.3.4 or 5.x?

Thank you,

c.

2mshah
Level 4
Level 4

Hi Xander,

Can you explain in more detail how these 2 recovery timers work together:

LACP:                                      Operational

    Flap suppression timer:                  300 ms

<snip>

  mLACP:                                     Operational

<snip>

    Recovery delay:                          300 s

We are running 3750s as the DHDs connecting to ASR9ks, therefore we don't have access to any of the lacp switchover commands on the DHDs.  On the ASRs we have:

lacp switchover suppress-flaps 300

  mlacp switchover type revertive

mlacp switchover maximize links

bundle wait-while 100

In a situation where we had near constant flapping of the links, we found the switchovers were no longer working as expected and appeared to lose sync. What is a recommended timer config to avoid this issue if we don't have those commands on the DHDs?

Hi xander ,

 

I do have a question here regarding ICCP and MPLS LDP NSR.

MPLS LDP NSR stays operational when iccp group is present as below.

 

Peer LDP Identifier: 192.101.0.232:0
  TCP connection: 192.101.0.232:55427 - 192.101.0.230:646; MD5 on
  Graceful Restart: Yes (Reconnect Timeout: 120 sec, Recovery: 180 sec)
  Session Holdtime: 180 sec
  State: Oper; Msgs sent/rcvd: 3262/3250; Downstream-Unsolicited
  Up time: 01:47:31
  LDP Discovery Sources:
      TenGigE0/1/0/0
      Targeted Hello (192.101.0.230 -> 192.101.0.232, active/passive)
  Addresses bound to this peer:
      172.25.14.90   192.101.0.232  199.212.160.82  199.213.160.10  
  Peer holdtime: 180 sec; KA interval: 60 sec; Peer state: Estab
  NSR: DisabledClients: Session Protection, AToM, ICCP <<<<< ICCP present

 

When iccp group is removed, we see the below.

Peer LDP Identifier: 192.101.0.231:0
  TCP connection: 192.101.0.231:32200 - 192.101.0.230:646; MD5 on
  Graceful Restart: Yes (Reconnect Timeout: 120 sec, Recovery: 180 sec)
  Session Holdtime: 180 sec
  State: Oper; Msgs sent/rcvd: 92/93; Downstream-Unsolicited
  Up time: 00:00:33
  LDP Discovery Sources:
      TenGigE0/0/1/0
      Targeted Hello (192.101.0.230 -> 192.101.0.231, active/passive)
  Addresses bound to this peer:
      172.25.14.81   192.101.0.231  199.213.160.6  
  Peer holdtime: 180 sec; KA interval: 60 sec; Peer state: Estab
  NSR: Operational
  Clients: Session Protection, AToM

 

Now, does that mean ICCP & MPLS LDP NSR cannot work together?

 

Thanks

 

 

nmihaylov
Level 1
Level 1

Hi Xander,

this post is really helpful, however i am also interesting in pseudo-MC-LAG option, as it is not very clear in the Configuration guide.

Thanks in advance.

Kind regards,

Nikola

xthuijs
Cisco Employee
Cisco Employee

Ah thanks for the idea!! yeah let us document that out also, always looking for new ideas as to what to cover next :)

give me a few and we'll get it out there!

cheers!

xander

k_abuasal
Level 1
Level 1

Hi Xander,

Thanks alot for your post, but I appreciate if you can elaborate more about the supported L2VPN and L3VPN services topologies that can be deployed using MCLAG, I read in some Cisco documents that coupled mode is not supported in ASR9K, unlike what's posted here? and can you please provide me with the configuration required for these supported topologies?

Many Thanks

 

 

 

xthuijs
Cisco Employee
Cisco Employee

hi there,

yeah that is why I write my own documentation to make sure the right support and capabilities are reflected :) (not ideal situation i know, we're working on that).

I think you have a great question, and I think I have some detail on that. One of the things on my to do list is to make use-cases. I realize that most of the doc that I have out there just describe the technology and don't have actual use cases whereby you can take a design or topology with a config and details. I recognize the need, but it is a lot of work. Let me get on that.

For the time being what type of design are you looking at and I can let you know how to set that up, rahter then me providing you use cases today that you may not need or find useful.

My personal pref is not using MCLAG to be honest, I find the cluster technology with active active bundles way more appealing, but that is something personal, some folks swear by MCLAG :)

So any case let me know what your need is and I'll detail it out for you as much as possible.

cheers!

xander

 

Carlos A. Silva
Level 3
Level 3

 

 

 

 

 

I agree with Xander, although a cluster setup has its own quirks. The thing with MCLAG is that, since it is MPLS dependent, at times of network instability it can do some funky stuff, specially from DHD perpective. Cluster-based bundles seem to be much more stable. Just my opinion.

 

 

 

 

k_abuasal
Level 1
Level 1

Hi xander,

Thanks alot for your reply and interest to help, I'm currently supporting a big ISP to figure out the best redundancy setup at the aggregation layer PE-CE, and MCLAG is one of the proposed solutions as the customer has many 7600 platforms deployed in the network besides of new ASR9Ks.

what we need is to provide a redundancy solution for P2P PW, VPLS, and L3VPN ccts terminated on the PE boxes, I found some presentations on Ciscolive365 site demonstrating different topologies besides some configuration examples in the configuration guides, but there are some points are still vague for me, and I think things will be be more clear if I can get a use case for the P2P PW, VPLS, and L3VPN cases with the configuratio needed in all cases.

Many Thanks

k_abuasal
Level 1
Level 1

Hi xander,

 

do you have any update on the question below?

 

Regards

Carlos A. Silva
Level 3
Level 3

Hi, Xander:

 

Is an mLACP scenario supported (by TAC) when the POA links are on nv sats?

 

Thanks!

c.

xthuijs
Cisco Employee
Cisco Employee

hey carlos,

I dont see a technical reason why this would not work, but I am quite sure that this is a scenario that we haven't tested specifically, I'll verify.

I would give it a try and see how it works, if there is an issue, I would open a tac case to have this corrected in case so, because I think there is value to do this.

cheers!

xander

atif_hafeez
Level 1
Level 1

Hi,

Can you please confirm if coupled and decoupled refers to the usage of P2P group or bridge group respectively?

 

Best regards!

xthuijs
Cisco Employee
Cisco Employee

hi atif, the reference to coupled vs decoupled basically means that the state of the PW (up/down) is coupled (or not) to the MCLAG status of the bundle interface (active/standby).

p2p PW (vpws) or bridge domains (vpls) can use either coupled or decoupled mode(s).

cheers

xander
 

atif_hafeez
Level 1
Level 1

Hi,

In case of P2P xconnect, the default behavior is coupled mode; and i am not able to find the command to change the mode. Similarly, in case of bridge domain, the default behavior is decoupled mode; and i am not able to find the command to change the mode.

In IOS, the default behavior is always coupled mode; and i found an optional command to change the mode "status decoupled" under l2 vfi config mode.

Can you please provide configuration examples for coupled and decoupled mode in case of p2p xconnect and bridge-domain?

 

Regards!

atif_hafeez
Level 1
Level 1

Hi xander,

Update:

I found a command "coupled-mode" under bridge-domain in order to enable couple mode under bridge-domain which is by default disabled. However, as per my test results, it is not functioning in coupled mode. Moreover this command is not documented in command reference of ASR9K IOS XR 5.x.x. Can you please advise if it is supported? 

 

And after a lot of research, i couldn't find any command to disable the couple mode in xconnect p2p. Can you please confirm if it is supported?

 

Regards!

xthuijs
Cisco Employee
Cisco Employee

hi atif, yup it is supported since XR 4.2.1.

for Xcon it is not configurable, by default it is a coupled mode there.

Here is some extra detail :

 

VPLS coupled-mode implementation (signal down PW when access circuit is down)

Introduction

In a point-to-point cross connect (VPWS), the access circuit (AC) and the PW are coupled. So, if the AC goes down, the L2VPN PE signals via LDP to the remote PE that the PW status should be down. This triggers convergence when PW redundancy is configured.   The AC and PW are not coupled in a LAN service (VPLS) by default.  So if the AC is down, the PW will be up regardless of the status of the AC because it is in the decouple mode.

 

Prerequisites

In 4.2.1 the XR software team made the coupled-mode implementation for VPLS configurable (CSCtr24794).

 

Configure

l2vpn

 bridge group xxxx

  bridge-domain xxxx

   coupled-mode

   interface TenGigEx/x/x/x

   !

   interface TenGigEy/y/y/y

   !

   vfi vfi-XX

    neighbor x.x.x.x pw-id xxx

 

Verify

 

Couple-mode disabled (not Configured) - PW will be up

RP/0/RSP0/CPU0:9010-A-SVE1#show l2vpn bridge-domain bd-name accedian_data2 detail
Legend: pp = Partially Programmed.
Bridge group: accedian, bridge-domain: accedian_data2, id: 12, state: up, ShgId: 0, MSTi: 0
Coupled state: disabled

 

Coupled-mode enabled (AC circuits up) - PW will be up

RP/0/RSP0/CPU0:9010-A-SVE1#sh l2vpn bridge-domain bd-name accedian_data1 detail
Legend: pp = Partially Programmed.
Bridge group: accedian, bridge-domain: accedian_data1, id: 4, state: up, ShgId: 0, MSTi: 0
Coupled state: up

 

Coupled-mode enabled (AC circuits down) - PW will be signaled down

RP/0/RSP0/CPU0:9010-A-SVE1#sh l2vpn bridge-domain bd-name accedian_data1 detail
Legend: pp = Partially Programmed.
Bridge group: accedian, bridge-domain: accedian_data1, id: 4, state: up, ShgId: 0, MSTi: 0
Coupled state: down

 

Troubleshoot

This feature will not work with a VPWS setup with multiple interfaces in a bridge domain.  The work around is to setup a VPLS as a point-to-point circuit with only one PW neighbor endpoint.

atif_hafeez
Level 1
Level 1

Hi,

Thanks a lot. I will test it and will come back to you if any issue pops up.

 

regards!

kalfasamuel
Level 1
Level 1

Hi Xander,

 

I use MC-LAG with VPLS decoupled mode on ASR9k.

My DHD are Nexus with vPC.

 

I try to determine the theorical switchover time from the active POA to the standby POA.

Can you help me ?

xthuijs
Cisco Employee
Cisco Employee

L2 service: Convergence should be sub-second, for most of the asr9k related failures

Convergence also depends on the access DHD switch. For most
of the time, the slow convergence come from the access switch, not asr9k. For
example, most of the access switch LAG convergence depend on the VLAN scale

 

Testing result for the DCI deployment: N7K vPC + asr9k MC-LAG with 500 VLANs, it can get sub-second convergence. But with 1200 VLANs, it can't.

 

failure detection for failover, depends on the speed of the iccp signaling and detection time.

 

cheers

xander

kalfasamuel
Level 1
Level 1

Hi Xander,

 

I have less than 100 vlans and with the Nexus 5k in vPC, I don't have LAG convergence due to spanning-tree for example.

So I will consider that the switchover is sub-second.

 

Thank you so much.

 

Best regards,

Samuel

Jon Berres
Level 4
Level 4

Hey Xander,

 

Great article on MC-LAG, thanks for all the detail! In testing Pseudo-MC-LAG (ICCP-SM) I've run into a few caveats:

  • If l2vpn iccp-sm config is added before the BE and bridge-domain are setup it remains in an 'unconfigured' state until at least one bridge-domain is built with an EFP from the BE attached and the iccp-sm config is removed and re-added. ICCP-SM seems very sensitive to the order the router is configured.

    Interface Name: Bundle-Ether3
      MAC flushing: MVRP
    Recovery Delay: 180 (Timer not running)
       Local State: Unconfigured
      Remote State: Unknown

     
  • Additionally when adding or changing primay/secondary vlans to the iccp-sm config the routers will go into a mis-configured state until both sides match. Is there any way to have the router wait a certain period before going into mis-configured. It appears that any vlan changes are service impacting at this point. Only work around for us at this point is to hit commit on both routers at the same time to limit outage to sub-second hit.

 

Any input you have is appreciated, thanks!

 

 

 

xthuijs
Cisco Employee
Cisco Employee

hey jon!!

ah the first one I know, pain in the you know what indeed! ddts for that is CSCuo63884, fixed in XR53. (doesnt seem to be externally visible via toolkit however).

the wait for time, hmm need to think about that how to resolve that, I dont have an answer for you on that right now.

cheers!

xander

Jon Berres
Level 4
Level 4

Thanks for the input, good to know about the fix for the iccp-sm configuration order.

For the VLAN mis-configure problem, i'm sure this is expected behavior, kind of like a port-channel switchport trunk vlan mismatch. But if default behavior could just block the 'new' VLAN and keep forwarding the existing VLANs until the config is sync'd up that would allow us to make changes without impacting live traffic. This is probably a feature request, but sure would be nice if it were possible.

 

Regards,

Jon

 

Xander - great article!  We've used this config to connect many Calix devices via Layer 2 into a bridge-domain on each 9k (with VFIs connecting the BDs across each 9k and to the other side of the network).

We have now been tasked with coming up with a solution to terminate the Bundles at Layer 3 on each 9k instead of landing them into BDs.  Can we do the same Layer 3 IPv4 address on the bundle on each 9k?  This seems like it may not work because we need to also get that address into OSPF and may cause a duplicate IP in the routing table.  The other solution seems  to be to do HSRP between BVIs that are also attached to the BDs.

What are your thoughts on doing Layer 3 on the 9k?  Maybe we are missing something obvious...

-ben

Hello Xander,

Although some people here have mentioned interoperability between N7K vPC and ASR9K MC-LAG, I'd like to ask if we should take special care of something. vPC on one hand is active/active, MC-LAG on the other hand is active/standby.. Would you recommend such a setup or is it too complicated?

Considering the system priority, both vPC and MC-LAG guides recommend setting higher system priority than the peer. I guess in this case MC-LAG should have higher priority and DHD control should be configured on N7K, correct?

Thanks,

George

Aleksandar Vidakovic
Cisco Employee
Cisco Employee

If the peer is in active/active and the asr9k in active/standby, you may see undesirable behaviour. The asr9k will never send a data frame down the standby link, but the LAG link state is not checked on received frames. We are evaluating a change in the code that will drop ingress packets on a standby link. We are threading carefully because of possible performance impact.

hth,

Aleksandar

Hi Aleksandar,

I had the impression that if MC-LAG cluster had higher priority than the vPC cluster, the vPC would also suspend the links connected to the standby POA.  If it doesn't, I don't see how MC-LAG could interoperate with vPC..

 

Aleksandar Vidakovic
Cisco Employee
Cisco Employee

Hi Georgios,

your expectation is correct. If you set the asr9k system priority to be higher (lower value) than the N7k, the N7k will follow the link assignment set by the asr9k.
 

Christian Vesth
Level 1
Level 1

Great article! thanks.

To my understanding regarding ICCP Groups, it will always go down, if all the backbone interfaces goes down, and thereby also force the bundles to go down?

If so, that would mean that a ICCP Group and bundle will always go down on POA1, if POA2 power off, and they only have single direct link in between them. Because i looses all connectivity to network core and goes into isolation.


A solution to this would then be, to have additional backbone interfaces configured, which is not directly connected between POA1 and POA2. Then we would see a split-brain situation, but the ICCP group will stay up on POA1, because it still have an active backbone interface, when POA2 is off.

Can you verify that?

Thanks.

Christian

xthuijs
Cisco Employee
Cisco Employee

cheers christian! takes long time to write things like this so great to see it is worth the effort!! :)

if your ICCP session takes the path over the core interfaces then yes you are correct.

generally, there is a link of some sort directly betwene the POA's hence if the core ifs go down, it is not necessarily detected by ICCP.

If the POA's can't see each other anymore, it is somewhat tricky, because it might lead to split brain (aka both POA's assume the other is not there and both being active).

it is best to have a "dedicated" channel between both POA's, and use some event detection to identify the core going down and trigger a POA failover with a priority change or some forced switchover in that facinity. EEM can help here or ojbect tracking can also facilitate here.

cheers!

xander

hengkycung
Level 1
Level 1

Hi Xander,

i have a question, here is our current topology and already operational.

We implement MC-LAG from our P1 and P2 connect to PE02. Named Bundle-Ether 25, contain l2vpn traffic on it.

P1 (BE25)------PE02 (BE25)

P2 (BE25)------PE02 (BE25)

My question, is that possible if we create a new MC-LAG on the same chassis to the same destination? The future topology should be look like :

P1 (BE25)------PE02 (BE25) for l2vpn traffic

P2 (BE25)------PE02 (BE25) for l2vpn traffic

P1 (BE30)------PE02 (BE30)

P2 (BE30)------PE02 (BE30)

Thanks.

Aleksandar Vidakovic
Cisco Employee
Cisco Employee

ICCP peering is established between LDP router-IDs. Since you can't have multiple LDP router-IDs on a single router, the configuration that you are considering is not supported. However, if you deploy pseudo-MC-LAG, you would have an active/active scenario in which you can deploy per-VLAN load balancing.

hope this helps,

Aleksandar

Aleksandar Vidakovic
Cisco Employee
Cisco Employee

Hi Hengky,

i actually forgot to ask you whether you wanted to have the two bundles completely separate (i.e. separate ICCP groups) or you wanted them to be in the same ICCP group. The former is not supported because ICCP peering is between LDP router-IDs, but the latter is. You can have multiple bundle interfaces within the same ICCP group.

I hope this clarifies.

Aleksandar

hengkycung
Level 1
Level 1

Hi Aleksandar,

Thanks for your information, just to be clear.

We wanted to create a new bundle with different group from current group.

Below is current configuration :

RP/0/RSP0/CPU0:P1#sh run int be 25
Fri Mar 18 16:10:14.059 WIB
interface Bundle-Ether25
description To PE02 Bundle-Ether25
mtu 9014
lacp switchover suppress-flaps 300
mlacp iccp-group 1
mlacp port-priority 100
mac-address aaaa.bbbb.cccc
bundle wait-while 100
!

RP/0/RSP0/CPU0:P1#sh run redundancy
Fri Mar 18 16:10:19.079 WIB
redundancy
iccp
group 1
mlacp node 1
mlacp system mac 000d.000e.000f
mlacp system priority 1
member
neighbor 192.168.88.6
!
backbone
interface Bundle-Ether11
interface Bundle-Ether12
interface Bundle-Ether13
interface Bundle-Ether101
interface Bundle-Ether103
!
!
!
!

Is that possible if we create new logical interface, lets say Bundle-Ether30, with group 2, with the same neighbor with current configuration?

Thanks.

Aleksandar Vidakovic
Cisco Employee
Cisco Employee

We support only one ICCP group between the same pair of routers.

What is the goal that you are trying to achieve with two ICCP groups?

/Aleksandar

hengkycung
Level 1
Level 1

Oh okay, thanks for your clarification Alek.

The reason we wanted to build a new MC-LAG is for full HA.

so the current BE25 is for l2vpn traffic, later if we able to create new MCLAG, that will be use for VOIP traffic. we wanted to split this different traffic.

Thanks.

Aleksandar Vidakovic
Cisco Employee
Cisco Employee

Have you considered Pseudo-MC-LAG? You would still have a single ICCP group, but you would have active/active scenario, with load-balancing per VLAN.

bn.thiyagarajan
Level 1
Level 1

Hi Xander,

I have a question in LAG (with LACP) built between ASR 9k6 and 4500X VSS. Two ports of ASR 9k6 from different line cards connects to different chassis in a 4500X VSS for redundancy. Initially the interfaces in the LAG was in load-balancing mode since there was an issue in configuring QoS we are now configuring Active-Standby in the LAG. 

We have configured the 'bundle max links 1' and whenever the active link 'te1' goes down the standby link 'te2' comes up and whenever the 'te1' link comes up, immediately the 'te2' is removed from the bundle made as standby and the link 'te1' comes into active mode. During this insertion of 'te1' to active mode there is protocol flaps configured in the bundle.

Is the bundle max-links is the only choice for doing Active-Standby?.

Do we have any mechanism to stop preemption in lag.

Warm Regards,

Thiyagarajan B

xthuijs
Cisco Employee
Cisco Employee
bundle mac links is indeed the right way to do it, but you have two knobs to play with: bundle wait-while and lacp fast-switchover these should help getting the failover done faster, and eliminate the port flapping. with the max-links of 1 we only have one member active, obviously :), but it is the one with the lowest port priority. so there is preemption by nature going on there. cheers xander
bn.thiyagarajan
Level 1
Level 1

Thanks Xander, Is there any way to stop this pre-emption?

xthuijs
Cisco Employee
Cisco Employee

I dont think there is... :( it is part of the way the IEEE defined the member activation in their standards with it pertains to max links and standby links...

however with the tunings of the lacp-fast switchover and the wait while you can make the behavior transparent to sessions running over the bundle, making it opaque really which member is running active.

Even when the port priority is equal, it is the lower port ID then that is the preferred member:

in this case I had port priority of 10 configured on each member.

  Port                  Device           State        Port ID         B/W, kbps

  --------------------  ---------------  -----------  --------------  ----------

  Gi0/0/0/9             Local            Active       0x000a, 0x0006     1000000       Link is Active

  Gi0/0/0/19            Local            Standby      0x000a, 0x0007     1000000       Link is Standby due to maximum-active links configuration

shutting the member connected to port 9:

  Port                  Device           State        Port ID         B/W, kbps

  --------------------  ---------------  -----------  --------------  ----------

  Gi0/0/0/9             Local            Configured   0x000a, 0x0006     1000000       Link is down

  Gi0/0/0/19            Local            Active       0x000a, 0x0007     1000000       Link is Active

bringing back port 9:

  Gi0/0/0/9             Local            Active       0x000a, 0x0006     1000000       Link is Active

  Gi0/0/0/19            Local            Standby      0x000a, 0x0007     1000000      Link is Standby due to maximum-active links configuration

cheers

xander

bn.thiyagarajan
Level 1
Level 1

Thanks  Xander, Working out the wait while timer/lacp fast switchover didn't help. Still trying out if eem can help.

Warm Regards,

Thiyagarajan B

xthuijs
Cisco Employee
Cisco Employee

I ran a few tests to qualify the behavior, albeit I am using an asr9k and a cat3500 switch. What I am noticing is that the switchover happens cleanly, that is when the primary member goes down, the standby immediately takes over in 3 msec, on the return of that primary link, it is brought back up first before the other one is moved to standby again.

I noticed no packet loss during this transition.

What I am thinking of that in your case with VSS that the bundle-E interface itself might be brought down that causes a protocol flap.

You probably want to look into that to see if that is the case, if so, maybe we can mess around with carrier delay (though tricky since this is a logical interface) but if that is the crux that the BE goes down, that is what we need to focus on. 

xander

LC/0/0/CPU0:Apr  7 10:36:39.083 : ifmgr[210]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/0/0/9, changed state to Down

LC/0/0/CPU0:Apr  7 10:36:39.083 : ifmgr[210]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface GigabitEthernet0/0/0/9, changed state to Down

RP/0/RSP0/CPU0:Apr  7 10:36:39.086 : BM-DISTRIB[1159]: %L2-BM-6-ACTIVE : GigabitEthernet0/0/0/9 is no longer Active as part of Bundle-Ether100 (Link is down)

RP/0/RSP0/CPU0:Apr  7 10:36:39.086 : BM-DISTRIB[1159]: %L2-BM-6-ACTIVE : GigabitEthernet0/0/0/19 is Active as part of Bundle-Ether100

LC/0/0/CPU0:Apr  7 10:36:39.741 : vic_0[373]: %PLATFORM-VIC-4-RX_LOS : Interface GigabitEthernet0/0/0/9, Detected Rx Loss of Signal

LC/0/0/CPU0:Apr  7 10:36:44.178 : ifmgr[210]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/0/0/9, changed state to Up

primary member coming back up

LC/0/0/CPU0:Apr  7 10:36:44.180 : ifmgr[210]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface GigabitEthernet0/0/0/9, changed state to Up

2 seconds later the extra member is brought down.

RP/0/RSP0/CPU0:Apr  7 10:36:46.633 : BM-DISTRIB[1159]: %L2-BM-6-ACTIVE : GigabitEthernet0/0/0/19 is no longer Active as part of Bundle-Ether100 (Link is Standby due to maximum-active links configuration)

RP/0/RSP0/CPU0:Apr  7 10:36:48.557 : BM-DISTRIB[1159]: %L2-BM-6-ACTIVE : GigabitEthernet0/0/0/9 is Active as part of Bundle-Ether100

RP/0/RSP0/CPU0:A9K-BNG#

bn.thiyagarajan
Level 1
Level 1

Hi Xander,

There is no problem when the active link goes down, the standby links takes over immediately and there is no be down.

Yes as you mentioned, the be goes down when link with low priority(in value) comes back to up, eventually the protocols are going down

Here's what I find in the router:

1>Active link going down:

RP/0/RSP0/CPU0:LAB-ASR-9006#LC/0/0/CPU0:Apr 7 20:20:27.245 : ifmgr[214]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE0/0/0/0, changed state to Down
LC/0/0/CPU0:Apr 7 20:20:27.245 : ifmgr[214]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE0/0/0/0, changed state to Down
RP/0/RSP0/CPU0:Apr 7 20:20:27.249 : BM-DISTRIB[1163]: %L2-BM-6-ACTIVE : TenGigE0/0/0/0 is no longer Active as part of Bundle-Ether5 (Link is down)
RP/0/RSP0/CPU0:Apr 7 20:20:27.249 : BM-DISTRIB[1163]: %L2-BM-6-ACTIVE : TenGigE0/1/0/0 is Active as part of Bundle-Ether5

RP/0/RSP0/CPU0:LAB-ASR-9006#

2>Bringing back the link:

LC/0/0/CPU0:Apr 7 20:21:29.087 : ifmgr[214]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE0/0/0/0, changed state to Up
LC/0/0/CPU0:Apr 7 20:21:29.089 : ifmgr[214]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE0/0/0/0, changed state to Down
LC/0/0/CPU0:Apr 7 20:21:29.640 : ifmgr[214]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE0/0/0/0, changed state to Up
LC/0/0/CPU0:Apr 7 20:21:29.642 : ifmgr[214]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE0/0/0/0, changed state to Up
RP/0/RSP0/CPU0:Apr 7 20:21:34.485 : BM-DISTRIB[1163]: %L2-BM-6-ACTIVE : TenGigE0/1/0/0 is no longer Active as part of Bundle-Ether5 (Link is Standby due to maximum-active links configuration)
RP/0/RSP0/CPU0:Apr 7 20:21:36.988 : ospf[1014]: %ROUTING-OSPF-5-ADJCHG : Process 10, Nbr 10.0.0.5 on Bundle-Ether5.26 in area 0 from FULL to DOWN, Neighbor Down: interface down or detached, vrf default vrfid 0x60000000
RP/0/RSP0/CPU0:Apr 7 20:21:36.989 : ospf[1014]: %ROUTING-OSPF-5-ADJCHG : Process 10, Nbr 10.0.0.4 on Bundle-Ether5.25 in area 0 from FULL to DOWN, Neighbor Down: interface down or detached, vrf default vrfid 0x60000000
RP/0/RSP0/CPU0:Apr 7 20:21:38.557 : BM-DISTRIB[1163]: %L2-BM-6-ACTIVE : TenGigE0/0/0/0 is Active as part of Bundle-Ether5
RP/0/RSP0/CPU0:Apr 7 20:21:42.481 : ospf[1014]: %ROUTING-OSPF-5-ADJCHG : Process 10, Nbr 10.0.0.5 on Bundle-Ether5.26 in area 0 from LOADING to FULL, Loading Done, vrf default vrfid 0x60000000
RP/0/RSP0/CPU0:Apr 7 20:21:42.482 : bfd[145]: %L2-BFD-6-SESSION_NO_RESOURCES : No resources for session to neighbor 10.26.2.1 on interface Bundle-Ether5.26, interval=100 ms
RP/0/RSP0/CPU0:Apr 7 20:21:43.637 : ospf[1014]: %ROUTING-OSPF-5-ADJCHG : Process 10, Nbr 10.0.0.4 on Bundle-Ether5.25 in area 0 from LOADING to FULL, Loading Done, vrf default vrfid 0x60000000
RP/0/RSP0/CPU0:Apr 7 20:21:43.639 : bfd[145]: %L2-BFD-6-SESSION_NO_RESOURCES : No resources for session to neighbor 10.25.2.1 on interface Bundle-Ether5.25, interval=100 ms
RP/0/RSP0/CPU0:LAB-ASR-9006#

RP/0/RSP0/CPU0:LAB-ASR-9006#show bundle
Thu Apr 7 20:21:38.170 IST

Bundle-Ether5
Status: Down
Local links <active/standby/configured>: 0 / 1 / 2
Local bandwidth <effective/available>: 0 (0) kbps
MAC address (source): 046c.9d53.5be3 (Chassis pool)
Inter-chassis link: No
Minimum active links / bandwidth: 1 / 1 kbps
Maximum active links: 1
Wait while timer: 1 ms
Load balancing: Default
LACP: Operational
Flap suppression timer: 2500 ms
Cisco extensions: Disabled
mLACP: Not configured
IPv4 BFD: Not configured

Port Device State Port ID B/W, kbps
-------------------- --------------- ----------- -------------- ----------
Te0/0/0/0 Local Negotiating 0x0001, 0x0002 10000000
Partner is not Synchronized (Waiting, Standby, or LAG ID mismatch)
Te0/1/0/0 Local Standby 0x0002, 0x0001 10000000
Link is Standby due to maximum-active links configuration
RP/0/RSP0/CPU0:LAB-ASR-9006#

Warm Regards,

Thiyagarajan B

xthuijs
Cisco Employee
Cisco Employee

oh wait, you have BFD running over the OSPF session on a bundle?

are you running BFD directly on OSPF or underneath the bundle.

it could very well be that ospf flaps because of the bfd session needs to be rehoused from one member to the other.

I think you probably are better of here using short lacp period instead of using bfd on ospf on the bundle with maxlinks1.

alternatively since you wanted to use maxlinks1 for qos/loadbal reasons, perhaps we can massage the loadbalancing scheme so that one vlan takes one member preferred, making sure there is qos accuracy, or use a bw percent value so that both members share (half) of the provisioned qos bw with 50/50 lb spread.

xander

bn.thiyagarajan
Level 1
Level 1

Hi Xander,

Done every possible tuning, but still the protocol flap occurs. There is no possible way to stop the bundle link going down.

Can you please brief the way to loadbalance the bundle with attaching the vlan to member.

Warm Regards,

Thiyagarajan B

xthuijs
Cisco Employee
Cisco Employee

it may be best to have a tac case to do a little screen share and some interactive config check/tests etc.

as for the vlan loadbalance:

when you configure the bundle hash <value> under the vlan subinterface, you tie the hash for all packets going out on that vlan to that hash value.

a hash value represents a bundle member.

then if you have 2 vlans and one uses value X and the other Y you can spread the traffic over those different members.

should one member die, then both X and Y will both use the same member.

(Their hash ties then to the single member).

xander

sandybreezebt
Level 1
Level 1

Hi Xander,

I've recently been weighing up the pro's and con's of mLACP vs pseudo-mLACP doing DCI functions.

In one scenario, the mLACP nodes are upstream of a few switches in a VxLAN fabric doing vPC (yes, we're waiting for L2 Gateway VTEP in A9k).  vPC was obviously there only to fool the mLACP cluster its a DHD on the end of it.  Functionally this works alright, though I wasn't overly happy with the bundle failover times when shifting traffic to the backup PoA for a particular bundle, nor that the link wasn't used in the first place.

So then I tried pseudo-mLACP and ripped out the vPC in the switches so the pseudo-mLACP nodes see they're connected to a DHN now, and balanced my VLAN's nicely over the links in the bundle on each PoA's (iccp-sm).  So now I'm active/active in the bundle but forwarding for one VLAN at a time.  The failover is pretty good too here, so seems like a winner.

One problem though, if I want to add or remove a VLAN from my primary / secondary configuration in the L2VPN redundancy, I'm scuppered by both PoA's blocking all VLAN's on the bundle as soon as one of the PoA's gets the config change.  Is there any way I can change this behaviour?  I'm not bothered about manually balancing VLAN's if there is an automatic way of doing it?  Or is there a config push the primary node can do to the secondary?  Or perhaps there is there a delay timer I can configure on config mismatch?

l2vpn
 redundancy
  iccp group 1
   multi-homing node-id 1
   interface Bundle-Ether1
    primary vlan 5,10
    secondary vlan 6,11
    recovery delay 60

l2vpn_mgr[1193]: %L2-L2VPN_ICCP_SM-3-CONFIG_LOCAL_ERROR : L2VPN Redundancy iccp-group 1 Bundle-Ether1 VLAN local configuration mismatch. Primary and Secondary VLANs have been set to blocking.

Great article by the way ;)

Sandy

xthuijs
Cisco Employee
Cisco Employee

hey sandy! thanks :)

yeah that item that you mentioned that when you add a vlan to one POA that the inconsistency is detected and results in a block for everything, until you "resolve" that vlan config on the other node.

I looked at this before on how to resolve this but havent made a lot of progress with it because somehow we need to have a "grace timer" to allow you to configure the other node before we throw this error.

let me revisit this and discuss this with the mclag dev to see what we can do here realistically.

cheers!

xander

xthuijs
Cisco Employee
Cisco Employee

btw track CSCuz23673 for this functionality change I just opened.

hengkycung
Level 1
Level 1

Hi Aleksandar,

So is itpossible if we create new logical interface, lets say Bundle-Ether30, with group 1, with the same neighbor with current configuration?

Thanks.

Aleksandar Vidakovic
Cisco Employee
Cisco Employee

yes, as log as you are using the same group, you can have multiple bundles tied to it.

regards,

/Aleksandar

bn.thiyagarajan
Level 1
Level 1

Hi,

'lacp non-revertive' command is available in 5.3.3. Done it and issue is resolved:)

xthuijs
Cisco Employee
Cisco Employee

very nice and thanks for the update on the discussion with this great find!

xander


Quick question - I believe this is supported based on how one configures the redundancy groups, but I wanted confirmation from the MC-LAG expert.

We have 3 POAs and 2 DHDs.  Can we have DHD-1 connect to POA-1 and POA-2 and DHD-2 connect to POA-2 and POA-3?  Our customer is concerned that once a POA is part of an RG with another POA, it can't be part of a separate RG to another POA.

Additional question - How many RGs are supported on the ASR9k?

Again, great article on MC-LAG.  We are seeing more and more of this in the field for chassis protection.  It works great.

Thanks!

miqbal
Cisco Employee
Cisco Employee

Hi Xander

My ICCP group between POA-1 and POA-2 is not coming up. The only thing different I have is that POA-1 is running 5.3.3 whereas POA-2 is on 4.3.4. Do we need to have matching sw releases ?

Regards

xthuijs
Cisco Employee
Cisco Employee

sw release dont matter for this, though recommended to be the same, just for consistency and similar behavior (regardless of mclag really), but I really need abit more info then the statement it is not working :), some shows, details, configs, debugs anything...

cheers

xander

miqbal
Cisco Employee
Cisco Employee

Thanks Xander. Following are the configs. I have not configured the MCLAG  yet as I except ICCP group to be connected after this config.

ASR9k-1
-------
redundancy
iccp
group 99
member
neighbor 10.5.99.2
!
!
!
!
interface Loopback99
ipv4 address 10.5.99.1/32
!
interface GigabitEthernet0/1/0/10
ipv4 address 10.5.1.1/24
proxy-arp
!
mpls ldp
log
neighbor
!
router-id 10.5.99.1
address-family ipv4
discovery targeted-hello accept
!
interface GigabitEthernet0/1/0/10
!
!
router static
maximum path ipv4 60000
maximum path ipv6 48000
address-family ipv4 unicast
10.5.99.2/32 10.5.1.2
!
!

ASR9k-4
--------
redundancy
iccp
group 99
member
neighbor 10.5.99.1
!
!
!
!
interface Loopback99
ipv4 address 10.5.99.2 255.255.255.255
!
interface GigabitEthernet0/2/0/6
ipv4 address 10.5.1.2 255.255.255.0
proxy-arp
!
mpls ldp
router-id 10.5.99.2
discovery targeted-hello accept
log
neighbor
adjacency
!
interface GigabitEthernet0/2/0/6
!
!
router static
address-family ipv4 unicast
0.0.0.0/0 10.122.96.1
10.5.99.1/32 10.5.1.1
!
!

--------------------------------------------------------------------------------------------------------

RP/0/RSP0/CPU0:ASR9006-01#ping 10.5.99.2
Thu Jun 30 10:53:50.133 PDT
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.5.99.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/11 ms
RP/0/RSP0/CPU0:ASR9006-01#
RP/0/RSP0/CPU0:ASR9006-01#
RP/0/RSP0/CPU0:ASR9006-01#sh iccp group 99
Thu Jun 30 10:54:04.482 PDT
Redundancy Group 99
member ip:10.5.99.2 (Unknown), down (disconnected)
monitor: route-watch (up)
No backbone interfaces.
No applications.
isolation recovery delay timer: 180 s, not running
RP/0/RSP0/CPU0:ASR9006-01#

RP/0/RSP1/CPU0:USSP-ASR9006-04#ping 10.5.99.1
Thu Jun 30 17:56:39.163 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.5.99.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
RP/0/RSP1/CPU0:USSP-ASR9006-04#sh iccp group 99
Thu Jun 30 17:56:44.961 UTC
Redundancy Group 99
member ip:10.5.99.1 (Unknown), init (disconnected)
monitor: route-watch (idle)
No backbone interfaces.
No applications.
isolation recovery delay timer: 180 s, not running
RP/0/RSP1/CPU0:USSP-ASR9006-04#

xthuijs
Cisco Employee
Cisco Employee

it looks like the ldp session is not establishing.

check show mpls ldp neighbor and see if that adj is going fine.

make sure that there is a local label for the loopback also that is seen by the neighbor.

this is a back to back config?

based on the member ip state (init or down/disconnected) itmeans that the ldp session cannot be established .

xander

miqbal
Cisco Employee
Cisco Employee

Thanks Xander. Yes, this back to back in a lab. LDP adj does not seem to be there.

RP/0/RSP0/CPU0:ASR9006-01# sh mpls ldp nei
Thu Jun 30 11:43:39.886 PDT

xthuijs
Cisco Employee
Cisco Employee

ok we're getting somewhere :)

the following debug may be handy to see what the reason is for the ldp not establishing.

RP/0/RSP0/CPU0:A9K-BNG#debug mpls ldp messages re

RP/0/RSP0/CPU0:A9K-BNG#debug mpls ldp messages se

RP/0/RSP0/CPU0:A9K-BNG#debug mpls ldp state

debug mpls ldp discover and error are good to add here too.

possibly a proc restart mpls_ldp may kick something off that may have been stuck.

LPTS is likely not a consideraton since the ldp default is not heavily policed so should be possible to establish.

this base mpls ldp config is correct, there is nothing that can or should be changed there as far as I can see at the moment.

the debugs will liekly clarify what the prob is. I think there is a unidir situation going on (which is why one side has a slightly different state then the other, hence the state debugs mentioned).

regards

xander

miqbal
Cisco Employee
Cisco Employee

On ASR9k-1 POA, it returns nothing even after process restart. But other POA has some hellos going out and some errors messages.

ASR9k-1

---------------

RP/0/RSP0/CPU0:ASR9006-01#process restart mpls_ldp
Thu Jun 30 11:58:39.671 PDT
RP/0/RSP0/CPU0:ASR9006-01#
RP/0/RSP0/CPU0:ASR9006-01#debug mpls ldp messages re
Thu Jun 30 12:02:11.409 PDT
RP/0/RSP0/CPU0:ASR9006-01#debug mpls ldp messages se
Thu Jun 30 12:02:19.229 PDT
RP/0/RSP0/CPU0:ASR9006-01#debug mpls ldp state
^
% Invalid input detected at '^' marker.
RP/0/RSP0/CPU0:ASR9006-01#debug mpls ldp error
Thu Jun 30 12:02:39.135 PDT
RP/0/RSP0/CPU0:ASR9006-01#debug mpls ldp discover hello
Thu Jun 30 12:02:51.567 PDT

On ASR9k-4 POA

------------------------

RP/0/RSP1/CPU0:USSP-ASR9006-04#debug mpls ldp messages re
Thu Jun 30 19:06:16.670 UTC
RP/0/RSP1/CPU0:USSP-ASR9006-04#debug mpls ldp messages se
Thu Jun 30 19:06:21.039 UTC
RP/0/RSP1/CPU0:USSP-ASR9006-04#debug mpls ldp messages seRP/0/RSP1/CPU0:Jun 30 19:06:23.552 UTC: mpls_ldp[1044]: DBG-MsgTx[1], Peer(10.5.99.1:0): Sent 'INIT' msg (size 45)
 
Thu Jun 30 19:06:54.092 UTC
RP/0/RSP1/CPU0:USSP-ASR9006-04#debug mpls ldp state     
                                               ^
% Invalid input detected at '^' marker.
RP/0/RSP1/CPU0:USSP-ASR9006-04#debug mpls ldp discover
% Incomplete command.
RP/0/RSP1/CPU0:USSP-ASR9006-04#debug mpls ldp discover ?
  hello  Discovery hello message
RP/0/RSP1/CPU0:USSP-ASR9006-04#debug mpls ldp discover hello
Thu Jun 30 19:07:43.211 UTC
RP/0/RSP1/CPU0:USSP-ASR9006-04#RP/0/RSP1/CPU0:Jun 30 19:07:43.466 UTC: mpls_ldp[1044]: DBG-Disc-RX[1], Rcvd Link Hello, 10.5.1.1 -> 224.0.0.2, Intf 0x8000240 (GigabitEthernet0_2_0_6), xport=10.5.99.1, ldp_id=10.5.99.1:0, hd_flags=0x4, intf_id=0
RP/0/RSP1/CPU0:Jun 30 19:07:43.466 UTC: mpls_ldp[1044]: DBG-Disc-RX[1], Rcvd Hello from 10.5.1.1 (10.5.99.1:0) to 224.0.0.2, opt 0x4
RP/0/RSP1/CPU0:Jun 30 19:07:43.572 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 192.168.72.4, ifh=0x0, cnt=2055
RP/0/RSP1/CPU0:Jun 30 19:07:44.233 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 1.1.1.1, ifh=0x0, cnt=2058
RP/0/RSP1/CPU0:Jun 30 19:07:44.331 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 192.18.156.71, ifh=0x0, cnt=2061
RP/0/RSP1/CPU0:Jun 30 19:07:44.436 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 24.24.20.27, ifh=0x0, cnt=2057
RP/0/RSP1/CPU0:Jun 30 19:07:44.723 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 192.169.105.10, ifh=0x0, cnt=2059

RP/0/RSP1/CPU0:USSP-ASR9006-04#RP/0/RSP1/CPU0:Jun 30 19:07:45.904 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 96.76.96.76, ifh=0x0, cnt=2057
RP/0/RSP1/CPU0:Jun 30 19:07:46.465 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 10.132.132.20, ifh=0x0, cnt=2057
RP/0/RSP1/CPU0:Jun 30 19:07:47.006 UTC: mpls_ldp[1044]: DBG-Disc-RX[1], Rcvd Tgt Hello, 10.5.99.1 -> 10.5.99.2, Intf 0x8000240 (GigabitEthernet0_2_0_6), xport=10.5.99.1, ldp_id=10.5.99.1:0, hd_flags=0x7, adj=0x10193abc
RP/0/RSP1/CPU0:Jun 30 19:07:47.006 UTC: mpls_ldp[1044]: DBG-Disc-RX[1], Rcvd Hello from 10.5.99.1 (10.5.99.1:0) to 10.5.99.2, opt 0x7
RP/0/RSP1/CPU0:Jun 30 19:07:47.113 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 10.80.3.65, ifh=0x0, cnt=2056
uRP/0/RSP1/CPU0:Jun 30 19:07:47.379 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 50.0.0.30, ifh=0x0, cnt=2061
nRP/0/RSP1/CPU0:Jun 30 19:07:47.813 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Link Hello, ldp_id 10.5.99.2:0, 10.5.1.2-> 224.0.0.2, ifh=0x8000240, cnt=2030
 RP/0/RSP1/CPU0:Jun 30 19:07:48.311 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 192.168.72.4, ifh=0x0, cnt=2056
RP/0/RSP1/CPU0:Jun 30 19:07:48.403 UTC: mpls_ldp[1044]: DBG-Disc-RX[1], Rcvd Link Hello, 10.5.1.1 -> 224.0.0.2, Intf 0x8000240 (GigabitEthernet0_2_0_6), xport=10.5.99.1, ldp_id=10.5.99.1:0, hd_flags=0x4, intf_id=0
RP/0/RSP1/CPU0:Jun 30 19:07:48.403 UTC: mpls_ldp[1044]: DBG-Disc-RX[1], Rcvd Hello from 10.5.1.1 (10.5.99.1:0) to 224.0.0.2, opt 0x4
dRP/0/RSP1/CPU0:Jun 30 19:07:49.133 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 1.1.1.1, ifh=0x0, cnt=2059
RP/0/RSP1/CPU0:Jun 30 19:07:49.187 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 192.169.105.10, ifh=0x0, cnt=2060
eRP/0/RSP1/CPU0:Jun 30 19:07:49.233 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 192.18.156.71, ifh=0x0, cnt=2062
RP/0/RSP1/CPU0:Jun 30 19:07:49.321 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 24.24.20.27, ifh=0x0, cnt=2058
RP/0/RSP1/CPU0:Jun 30 19:07:49.391 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 10.5.99.1, ifh=0x0, cnt=1034
bugRP/0/RSP1/CPU0:Jun 30 19:07:50.253 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 96.76.96.76, ifh=0x0, cnt=2058
 allRP/0/RSP1/CPU0:Jun 30 19:07:51.122 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 10.132.132.20, ifh=0x0, cnt=2058

% Ambiguous command:  "un debug all"
RP/0/RSP1/CPU0:USSP-ASR9006-04#RP/0/RSP1/CPU0:Jun 30 19:07:51.476 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 10.80.3.65, ifh=0x0, cnt=2057
RP/0/RSP1/CPU0:Jun 30 19:07:51.901 UTC: mpls_ldp[1044]: DBG-Disc-RX[1], Rcvd Tgt Hello, 10.5.99.1 -> 10.5.99.2, Intf 0x8000240 (GigabitEthernet0_2_0_6), xport=10.5.99.1, ldp_id=10.5.99.1:0, hd_flags=0x7, adj=0x10193abc
RP/0/RSP1/CPU0:Jun 30 19:07:51.901 UTC: mpls_ldp[1044]: DBG-Disc-RX[1], Rcvd Hello from 10.5.99.1 (10.5.99.1:0) to 10.5.99.2, opt 0x7
RP/0/RSP1/CPU0:Jun 30 19:07:52.140 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 50.0.0.30, ifh=0x0, cnt=2062
RP/0/RSP1/CPU0:Jun 30 19:07:52.327 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Link Hello, ldp_id 10.5.99.2:0, 10.5.1.2-> 224.0.0.2, ifh=0x8000240, cnt=2031
RP/0/RSP1/CPU0:Jun 30 19:07:52.658 UTC: mpls_ldp[1044]: DBG-Disc-RX[1], Rcvd Link Hello, 10.5.1.1 -> 224.0.0.2, Intf 0x8000240 (GigabitEthernet0_2_0_6), xport=10.5.99.1, ldp_id=10.5.99.1:0, hd_flags=0x4, intf_id=0
RP/0/RSP1/CPU0:Jun 30 19:07:52.658 UTC: mpls_ldp[1044]: DBG-Disc-RX[1], Rcvd Hello from 10.5.1.1 (10.5.99.1:0) to 224.0.0.2, opt 0x4
RP/0/RSP1/CPU0:Jun 30 19:07:53.057 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 192.168.72.4, ifh=0x0, cnt=2057
RP/0/RSP1/CPU0:Jun 30 19:07:53.524 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 1.1.1.1, ifh=0x0, cnt=2060
RP/0/RSP1/CPU0:Jun 30 19:07:53.922 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 192.169.105.10, ifh=0x0, cnt=2061
RP/0/RSP1/CPU0:Jun 30 19:07:54.016 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 192.18.156.71, ifh=0x0, cnt=2063
RP/0/RSP1/CPU0:Jun 30 19:07:54.234 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 24.24.20.27, ifh=0x0, cnt=2059
RP/0/RSP1/CPU0:Jun 30 19:07:54.678 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 96.76.96.76, ifh=0x0, cnt=2059
RP/0/RSP1/CPU0:Jun 30 19:07:56.045 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 10.132.132.20, ifh=0x0, cnt=2059
RP/0/RSP1/CPU0:Jun 30 19:07:56.153 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 10.80.3.65, ifh=0x0, cnt=2058
RP/0/RSP1/CPU0:Jun 30 19:07:56.392 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 50.0.0.30, ifh=0x0, cnt=2063
RP/0/RSP1/CPU0:Jun 30 19:07:56.473 UTC: mpls_ldp[1044]: DBG-Disc-RX[1], Rcvd Tgt Hello, 10.5.99.1 -> 10.5.99.2, Intf 0x8000240 (GigabitEthernet0_2_0_6), xport=10.5.99.1, ldp_id=10.5.99.1:0, hd_flags=0x7, adj=0x10193abc
RP/0/RSP1/CPU0:Jun 30 19:07:56.473 UTC: mpls_ldp[1044]: DBG-Disc-RX[1], Rcvd Hello from 10.5.99.1 (10.5.99.1:0) to 10.5.99.2, opt 0x7
RP/0/RSP1/CPU0:Jun 30 19:07:57.297 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Link Hello, ldp_id 10.5.99.2:0, 10.5.1.2-> 224.0.0.2, ifh=0x8000240, cnt=2032
RP/0/RSP1/CPU0:Jun 30 19:07:57.631 UTC: mpls_ldp[1044]: DBG-Disc-RX[1], Rcvd Link Hello, 10.5.1.1 -> 224.0.0.2, Intf 0x8000240 (GigabitEthernet0_2_0_6), xport=10.5.99.1, ldp_id=10.5.99.1:0, hd_flags=0x4, intf_id=0
RP/0/RSP1/CPU0:Jun 30 19:07:57.631 UTC: mpls_ldp[1044]: DBG-Disc-RX[1], Rcvd Hello from 10.5.1.1 (10.5.99.1:0) to 224.0.0.2, opt 0x4
RP/0/RSP1/CPU0:Jun 30 19:07:57.979 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 10.5.99.1, ifh=0x0, cnt=1035
RP/0/RSP1/CPU0:Jun 30 19:07:58.030 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 192.168.72.4, ifh=0x0, cnt=2058
RP/0/RSP1/CPU0:Jun 30 19:07:58.318 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 192.18.156.71, ifh=0x0, cnt=2064
RP/0/RSP1/CPU0:Jun 30 19:07:58.465 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 192.169.105.10, ifh=0x0, cnt=2062
RP/0/RSP1/CPU0:Jun 30 19:07:58.465 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 1.1.1.1, ifh=0x0, cnt=2061
RP/0/RSP1/CPU0:Jun 30 19:07:59.001 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 24.24.20.27, ifh=0x0, cnt=2060
RP/0/RSP1/CPU0:Jun 30 19:07:59.171 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 96.76.96.76, ifh=0x0, cnt=2060
RP/0/RSP1/CPU0:Jun 30 19:08:00.624 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 10.132.132.20, ifh=0x0, cnt=2060
RP/0/RSP1/CPU0:Jun 30 19:08:00.797 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 10.80.3.65, ifh=0x0, cnt=2059
RP/0/RSP1/CPU0:Jun 30 19:08:00.937 UTC: mpls_ldp[1044]: DBG-Disc-RX[1], Rcvd Tgt Hello, 10.5.99.1 -> 10.5.99.2, Intf 0x8000240 (GigabitEthernet0_2_0_6), xport=10.5.99.1, ldp_id=10.5.99.1:0, hd_flags=0x7, adj=0x10193abc
RP/0/RSP1/CPU0:Jun 30 19:08:00.937 UTC: mpls_ldp[1044]: DBG-Disc-RX[1], Rcvd Hello from 10.5.99.1 (10.5.99.1:0) to 10.5.99.2, opt 0x7
RP/0/RSP1/CPU0:Jun 30 19:08:01.120 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 50.0.0.30, ifh=0x0, cnt=2064
RP/0/RSP1/CPU0:Jun 30 19:08:01.773 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Link Hello, ldp_id 10.5.99.2:0, 10.5.1.2-> 224.0.0.2, ifh=0x8000240, cnt=2033
RP/0/RSP1/CPU0:Jun 30 19:08:02.176 UTC: mpls_ldp[1044]: DBG-Disc-RX[1], Rcvd Link Hello, 10.5.1.1 -> 224.0.0.2, Intf 0x8000240 (GigabitEthernet0_2_0_6), xport=10.5.99.1, ldp_id=10.5.99.1:0, hd_flags=0x4, intf_id=0
RP/0/RSP1/CPU0:Jun 30 19:08:02.176 UTC: mpls_ldp[1044]: DBG-Disc-RX[1], Rcvd Hello from 10.5.1.1 (10.5.99.1:0) to 224.0.0.2, opt 0x4
uRP/0/RSP1/CPU0:Jun 30 19:08:02.629 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 192.168.72.4, ifh=0x0, cnt=2059
nRP/0/RSP1/CPU0:Jun 30 19:08:02.822 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 192.18.156.71, ifh=0x0, cnt=2065
RP/0/RSP1/CPU0:Jun 30 19:08:02.861 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 1.1.1.1, ifh=0x0, cnt=2062
deRP/0/RSP1/CPU0:Jun 30 19:08:03.400 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 192.169.105.10, ifh=0x0, cnt=2063
buRP/0/RSP1/CPU0:Jun 30 19:08:03.895 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 96.76.96.76, ifh=0x0, cnt=2061
gRP/0/RSP1/CPU0:Jun 30 19:08:03.968 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 24.24.20.27, ifh=0x0, cnt=2061
 RP/0/RSP1/CPU0:Jun 30 19:08:05.280 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 10.132.132.20, ifh=0x0, cnt=2061
RP/0/RSP1/CPU0:Jun 30 19:08:05.563 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 50.0.0.30, ifh=0x0, cnt=2065
RP/0/RSP1/CPU0:Jun 30 19:08:05.574 UTC: mpls_ldp[1044]: DBG-Disc-RX[1], Rcvd Tgt Hello, 10.5.99.1 -> 10.5.99.2, Intf 0x8000240 (GigabitEthernet0_2_0_6), xport=10.5.99.1, ldp_id=10.5.99.1:0, hd_flags=0x7, adj=0x10193abc
RP/0/RSP1/CPU0:Jun 30 19:08:05.574 UTC: mpls_ldp[1044]: DBG-Disc-RX[1], Rcvd Hello from 10.5.99.1 (10.5.99.1:0) to 10.5.99.2, opt 0x7
RP/0/RSP1/CPU0:Jun 30 19:08:05.734 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Tgt Hello, 10.5.99.2 -> 10.80.3.65, ifh=0x0, cnt=2060
aRP/0/RSP1/CPU0:Jun 30 19:08:06.106 UTC: mpls_ldp[1044]: DBG-Disc-TX[2], Sent Link Hello, ldp_id 10.5.99.2:0, 10.5.1.2-> 224.0.0.2, ifh=0x8000240, cnt=2034
ll
Thu Jun 30 19:08:06.647 UTC
All possible debugging has been turned off
RP/0/RSP1/CPU0:USSP-ASR9006-04#debug mpls ldp error         
Thu Jun 30 19:08:44.805 UTC
RP/0/RSP1/CPU0:USSP-ASR9006-04#RP/0/RSP1/CPU0:Jun 30 19:11:53.582 UTC: mpls_ldp[1044]: ERROR[1]: ldp_tcp_write:215, TCP (10.5.99.1:646): Sendmsg returned -1, error = Broken pipe
RP/0/RSP1/CPU0:Jun 30 19:11:56.111 UTC: mpls_ldp[1044]: ERROR[1]: ldp_tcp_xport_transmit:1636, Peer 10.5.99.1:0, Error writing to socket


RP/0/RSP0/CPU0:ASR9006-01#

xthuijs
Cisco Employee
Cisco Employee

do you have an acl's configured or lpts configuration that blocks tcp packets?

the peer is having a problem accepting tcp messages subsequent to the ones received after sending. the AST9006-01 seems to have a policer or rate limiter applied.

xander

miqbal
Cisco Employee
Cisco Employee

Thank you very much Xander. Removing the residual lpts configs from some previous test engineer helped. Both LDP and ICCP are up.

xthuijs
Cisco Employee
Cisco Employee

nice!! glad we fixed it with that simple trick :)

cheers!

xander

miqbal
Cisco Employee
Cisco Employee

Hi Xander

My customer is trying to carry l2vpn (one dot1q VLAN) over the MC-LAG. The only difference is that the DHD is an ASR903 while both POA are ASR9006. The MC-LAG itself seem to be working fine. However, for some reason, the bundle sub-interface MAC address on ASR9k does not show in show l2vpn bridge-domain bd-name xxxx brief. Only CE facing main interfaces show up. Similar behavior is seen on the Port-Channel side. The two CE's are ASR901 connected to one ASR9k(POA) and other connected on far-side to an ASR903 (DHD)

.                         

ASR901-1 ----ASR9k-1--------------ASR903-1-------ASR901-2

                          |                                |

                          |                                |

                     AR9k-4---------------------

Only LDP enabled is the one between two ASR9k for the ICCP link. PW between two ASR9k is up/up.

My question is do we support such a setup and if so, what could be wrong.

xthuijs
Cisco Employee
Cisco Employee

hi miqbal,

this design doesn't have any problems per-se, but it would require a link from 901-1 to asr9k-4 of some sort, either directly (preferred obviously) or via a trunk connection between 9k-1 and 4.

the bundle mac is not needed in teh BD as it is not switched against. the mac's exist between the l3 endpoints and are the macs that we need to learn. whether it be from the 903-1 routed interface or from the CE 901-2, if the 903-1 is just siwtching between the CE link and the bundle from 903-1 to the 9k1/4.

not very clear as to what precisely is not working here, if 9k4 is active, I can imagine a problem with the comm to 901-1 simply because of the current cabling. May also be best to raise/start a ew discussion on the front page of the xros forum for others to follow and chime in.

when you do, make sure the configs, shows, and detailed problem description is there also to review what needs to be rectified.

cheers!

xander

Crickets, tumbleweeds....

Aleksandar Vidakovic
Cisco Employee
Cisco Employee

Hi Ben,

This will work. You can even have multiple groups between the same pair of POA.

A max of 8 ICCP groups for MCLAG are allowed.

hope this helps,

/Aleksandar

Thanks for the information. This is exactly what we were looking for.

-bb

takao.nakamaru
Level 1
Level 1
Hello Xander and all

I am interested in using mLACP for our ASR9K network. The problem is that in our existing topology, mLACP is already employed at the access network. Specifically in the head-end site, all traffic passes through 2xL2Aggregation devices (that use mLACP) and is forwarded to one ASR9K (the first L3 node). We are looking for a way to add redundancy also for the ASR9k layer.

So for increased redundancy we want to add a second ASR9K in this topology. We are investigating the usage of two ASR9Ks that will also be using mLACP. We are wondering if this is a valid topology:

Continue having two L2_Devices running mLACP (L1 and L2) connected to 2 ASR9K nodes also running a mLACP instance (ARS9k1, ARS9k1) with the connections:

=L1 ====(LACP links)==== ASR9k1===> L3_Uplink
  |                       |
ICCP_L1,2            ICCP_ASR1,2
  |                       |
=L2 ====(LACP links)==== ASR9k2===> L3_Uplink


Is this valid - having two mlacp instances with directly connected members? (The asr9k will be primary used for IP termination)
takao.nakamaru
Level 1
Level 1
Jon Berres
Level 4
Level 4

Hey Xander,

Question on MLACP supported features. Is it possible for the DHD to be a PE and the POA a P router in an MPLS topology? If it is supported do you know what failover looks like? I'm guessing the POA2 would need to re-establish IGP and LDP with with the PE router so there would be a short hit there but would be better than a complete outage.

The scenario here is for an ENNI handoff from another carrier which provides L2 backhaul from a cellsite PE router.

Thanks,

Jon

smailmilak
Level 4
Level 4

Hi Xander,

quick question. Using MC-LAG between to A9K PE's we could use one BNG as DHD?

xthuijs
Cisco Employee
Cisco Employee

hi jon,

I dont think that is the best design for the reason that the PE will assume a label/igp adjacency to tha POA and if it failsover, the peering will have to reset and reestablish all the adj for igp and ldp.

in that design you are probably better off taking 2 ECMP links and not have them as MCLAG.

xander

xthuijs
Cisco Employee
Cisco Employee

better the other way around Smail! :)

use the POA's as BNG with geored and some access device as the DHD allowing failover between the 2 BNG's.

but yeah if the bng is a DHD because it has 2 uplinks to two different devices that is fine also, but these links would then not agg the subs.

x

smailmilak
Level 4
Level 4

In this case we have only one BNG, therefore the idea to use it as DHD. I saw your illustration here under "overview".  So standby PoA will not do any forwarding, only the active one? This is more a active/standby setup?


Does MOD80-SE support PW-HE?

xthuijs
Cisco Employee
Cisco Employee

correct, pMLACP is active/active but for different vlans, so it is really active standby for the same vlan.

so having a BNG that uplinks via mclag there is little value, you might as well use ecmp.

or cost one link out, that is probably faster and easier to deploy.

mclag is more of an access/edge usecase.

cheers!

xander

smailmilak
Level 4
Level 4

Yeah, got it. At first, I thought that ICCP is something like VSS or VPC. 

L3 uplink is not an issue for us, only downstream (broadcast) for PADI should reach the BNG over two possible links from two different routers. I will figure this out, no problem.

Can you please tell me if MOD80-SE has support for PW-HE?

xthuijs
Cisco Employee
Cisco Employee

oh sorry forgot to comment on that smail :) yes it has that support!

xander

Charlene Doherty
Cisco Employee
Cisco Employee

HI Xander,

Great document!

Quick Question, are there any MIBs released to support these MC-LAG functions ?  I can see MIBS for LAG, but none for MC-LAG, ICCP etc.

thanks

Charlene

xthuijs
Cisco Employee
Cisco Employee

hi charlene,

iccp/mclag doesnt have a defined mib for the redundancy operation/communication. so you can use the LAG mib to get member state etc, but there is no specific OID to monitor the ICCP state and details.

cheers

xander

Charlene Doherty
Cisco Employee
Cisco Employee

Thanks a lot Xander! 

samiwehbi
Level 1
Level 1

Dear Xander,

is there a limit on the number of iccp-groups that can be created on ASR9000/IOS-XR, since we hit the limit of 24 & the rgmgr process crashes, already have a case, nothing mentioned in the below guide concerning limitation of iccp groups:

http://www.cisco.com/c/en/us/td/docs/routers/asr9000/software/asr9k_r5-3/interfaces/configuration/guide2/b-interfaces-cg53x-asr9k/Configuring_Link_Bundling.html#ID-1764-00000355          

we found the below while searching online:

Cisco Bug: CSCuv56926 - RGMGR process crash when configuring singleton ICCP group "states a limit of 24 groups"

We need to add more MC-LAG bundles, is there any workaround ? can we use the same iccp group for two different bundles ?

thank you,

Regards

Sami

xthuijs
Cisco Employee
Cisco Employee

hi sami, see response on the mstag doc also, but we wont need multiple groups to manage multiple bundles between the same poa's. we only need one group per router pair. we can than manage multiple bundles together on that same comm channel.

cheers!

xander

Hey Xander.  Does MC-LAG support active/active on 5.3.3 ?  Also, where can I usually find out this information?  I can't seem to locate it on Cisco's site.

Aleksandar Vidakovic
Cisco Employee
Cisco Employee

Look for "Pseudo mLACP" in the configuration guide:

http://www.cisco.com/c/en/us/td/docs/routers/asr9000/software/asr9k_r5-3/interfaces/configuration/guide2/b-interfaces-cg53x-asr9k/Configuring_Link_Bundling.html

/Aleksandar

xthuijs
Cisco Employee
Cisco Employee

yeah it is called pseudo mclag, basically one POA can be active for a set of vlan's and the other POA be active for a *different* set of vlans.

here is some reference material also on that.

xander

Awesome Thank you!

Thank you very much.

Not applicable

Hi Xander,

We had the below scenario attached as pdf for primarily load-balancing and HA between our long haul links on the P-nodes. However with the discontinuation of the NV-Cluster technology we need to find an alternative to this. ICCP-SM was mentioned but I'm not convinced that it will solve the load-balancing issues. 

Any suggestions?

Regards

Aleksandar Vidakovic
Cisco Employee
Cisco Employee

hi Martin,

MC-LAG or Pseudo MC-LAG we recommend on connections towards DHD, not in the core. The bundle between the two P clusters (in the .pdf that you have attached) would be better off as local bundles on each of the P routers (see attached topo1.pdf).

Load-balancing would still work nicely. If on the PE<->P connection you're not using bundles, both uplinks from a given PE are active. So there will be load-balancing at this leg. When the P receives the packet, it will apply the load-balancing algorithm to select the bundle member. Since our load-balancing algorithm has provisioning for avoiding traffic polarisation, you should see good load balancing between bundle members on the P routers.

If the PE<->P connection is a bundle, you could use MC-LAG to achieve redundancy.

There's a very good section on load-balancing in the "BRKSPG-2904 - ASR-9000/IOS-XR Understanding forwarding, troubleshooting the system and XR operations (2017 Las Vegas)" session. This explains in detail how IPv4/6 and MPLS packets are load-balanced in asr9k.

I hope this answers your question.

/Aleksandar

samiwehbi
Level 1
Level 1

Hi Xander,

 

concerning the proxy-arp feature, its only needed in case of static routing on the backbone side ?

 

in case we are using a dynamic routing protocol such as OSPF do we still need proxy-arp on the backbone links ?

 

thank you,

 

Regards

Sami

dhammika_r
Level 1
Level 1

Hi Xander,

 

Can we have two mLACP systems connected back to back as shown in the picture ? platforms of choice system-1 ASR900 and system-2 ASR9000 ?Screen Shot 2018-03-23 at 14.31.04.png

j.hoffmann
Level 1
Level 1

I'm not sure if I have overlooked something....

But what is the right config to use more the two POA => Multi-homed device instead of dual-homed device?

With the singleton CLI command, it is no more necessary to configure the ICCP neighbor explicit.

Using explicit neighbor configuration, only ONE neighbor is possible in a ICCP group. And therefore only DHD topology is possible.

 

Regards,

Jens 

 

ralmeidag1
Level 1
Level 1

Hi Xander

Is there any issue if the mclag is configured on different platforms? Let´s say POA 1 is ASR9904 and POA is ASR9010. Both run the same IOS XR version.

 

Thanks.