What is a Bundle:
- the ability to aggregate multiple interfaces together in an L2 manner, based on the standards defined in 802.1ax/802.1ad
- A group of physical interfaces with the same: speed,media, peer
- Treated as a single interface by higher layer protocol (one MAC, one IP, one set of configuration)
Alternative names used are: EtherChannel, Port-Channel, Link Bundle or LAG.
Advantages
- Increase link availability
- Increase bandwidth
- A single peering between two devices instead of having multiple individual links requiring ECMP (in L3) or STP (L2)
Functionality
- All the features are applied on the bundle interface, but replicated to the NPU's that have members in this bundle.
- Members can be on multiple LCs
- L3 hashing is based on Source IP, Destination IP and Router ID, Src/Dest Ports are included too on TCP/UDP packet
- L2 hashing is based on Source MAC, Destination MAC, and Router ID
for more details on the Loadbalancing architecture see the https://supportforums.cisco.com/docs/DOC-26687
- Global MAC addresses stored in the backplane EEPROM are used as Bundle mac addresses
- Min/Max active member links can be configured on Bundle interface
- Bundle interface and its members can be shut/no shut independently
- There are 3 different modes: LACP (active/passive), or no LACP (ON)
- LACP in active mode will send and receive control packets as opposed to passive mode that will only response to received control packets
- LACP can be configured in short mode (1 sec) or long mode (30 sec) which determines the rate at which control packets are exchanged.
- For each port 28 bytes of information (LAG ID) and is exchanged between peers:
- Local System ID (8 bytes)
- Local Operational Key (2 bytes)
- Local Port ID (4 bytes)
- Local view of peer’s state
- If Bundle interface is outgoing multicast interface, then one bundle line card (BLC) is chosen to forward the multicast traffic and that decision is made by MRIB (In control plane).
- In egress BLC, packet is delivered to all Trident Chips. The member interface selected for a hash (from Trident Packet Header) will be local only to one Trident Chip. The other Trident Chip drops packet as member is not local.
Bundle Configuration
Bundle Config:
config
lacp system priority 40000
lacp system mac 000c.15c0.bd15
interface bundle-ether<x>
bundle minimum-active {links <count> | bandwidth <bw>}
bundle maximum-active links <count> hot-standby
Members Config
config
interface Gig0/1/0/0
bundle id <x> mode {on|active|passive}
bundle port-priority <priority>
lacp period {short | long}
LACP Configuration
- An optional tool for checking that the bundle membership configuration makes sense and that the partner machine is also willing and able to aggregate the same set of links.
- The links in the bundle all terminate on the same partner device.
- The links in the bundle are all configured to be within the same bundle on the partner device.
- The partner device is likely to be ready to receive traffic on the links (state negotiations).
- Each link is assigned a LAGID which is a combination of the LACP system identifier and administrative key for the bundle as specified by both connected devices. For a bundle to be established all links within a bundle must have the same LAGID - any links with a different LAGID are excluded from the bundle.
- Timeout: The default (long) period of a packet every 30 seconds (timeout after 90 seconds), and the "short" period of a packet every second (timeout after 3 seconds).
- Each LACP system has a system identifier which consists of a MAC address and a system priority. The MAC should be unique to the system (if it matches a partner system, LACP negotiations may fail), and the combination of the priority and MAC address are used to determine which is the higher priority system (default 32768)
- Starting from 4.1; the global system priority and MAC address can each be overridden for a specific bundle and configured under the bundle interface config
BFD over Bundle members (BoB or BFDoBM)
- BFD over Bundle members is an enhancement to the Layer 3 fast protection protocol for determining whether a Bundle member can be used to forward traffic
- BFD has two modes
- Async: Each device sends a BFD packet every time period, and listens for the packets sent from the neighboring device. If the local device does not receive a packet within a set time (determined by the user), then the connection is considered to be down
- Echo: A single device sends a BFD packet every time period, and this packet goes through the forwarding path of the neighboring device before being returned to the originator. If the originator does not receive a packet within a set time, then the connection is considered to be down.
- BFD has three external states: Up, Down, Neighbor Unconfigured (Nbr_Unconfig)
- BFD sessions can be shut down in one of two ways:
- Ungraceful: The session is just stopped, and the peer device will probably time out
- Graceful: The session sends an unconfigured message to the peer, which triggers a NBR_UNCONFIG SCN on the peer device.
- BFD sessions are only started on member links once they have reached LACP DISTRIBUTING state. Consequently the status of the BFD protocol does not affect the LACP state of a link in any way
Multichassis Link Aggregation (MC-LAG)
- A new feature for IOS XR release 4.0. MC-LAG involves two devices collaborating to act as a single LAG from the perspective of a (third) connected device
- Only one of the XR devices forwards traffic at any one time, eliminating the risk of forwarding loops. When a failure occurs, the XR devices coordinate to perform a switchover, changing the device on which traffic is being forwarded by manipulating the link LACP states
- The use case for this is at the edge of a provider's network, to allow a "simple" customer edge (CE) device which only supports standard LACP to be connected to two provider edge (PE) devices. Thus the CE device is "dual-homed", providing better L2 redundancy from the provider's side. In the mLACP documentation, the CE device is referred to as a Dual-Homed Device (DHD) and each PE device is known as a Point of Attachment (POA). The POA forwarding traffic for the bundle is the Active device for that bundle with the other POA being Standby
- The two POAs communicate with each other over an LDP link using the Inter-Chassis Communication Protocol (ICCP)
MLACP Configuration
POA-1
redundancy
iccp
group 1
mlacp node 1
mlacp system mac 000d.000e.000f
mlacp system priority 1
member
neighbor 5.4.3.2
interface Bundle-Ether1
lacp switchover suppress-flaps 300
mlacp iccp-group 1
mac-address 0.deaf.0
bundle wait-while 100
interface Loopback0
ipv4 address 5.4.3.1 255.255.255.255
interface GigabitEthernet0/0/0/0
description Connected to DHD Gi0/0/0/0
bundle id 1 mode active
lacp period short
no shutdown
interface GigabitEthernet0/0/0/3
description Connected to POA2 Gi0/0/0/3
ipv4 address 1.2.3.1 255.255.255.0
proxy-arp
no shutdown
router static
address-family ipv4 unicast
5.4.3.2/32 1.2.3.2
mpls ldp
router-id 5.4.3.1
discovery targeted-hello accept
log
neighbor
POA-2
redundancy
iccp
group 1
mlacp node 2
mlacp system mac 000d.000e.000f
mlacp system priority 1
member
neighbor 5.4.3.1
interface Bundle-Ether1
lacp switchover suppress-flaps 300
mlacp iccp-group 1
mac-address 0.deaf.0
bundle wait-while 100
interface Loopback0
ipv4 address 5.4.3.2 255.255.255.255
interface GigabitEthernet0/0/0/0
description Connected to DHD Gi0/0/0/3
bundle id 1 mode active
lacp period short
no shutdown
interface GigabitEthernet0/0/0/3
description Connected to POA1 Gi0/0/0/3
ipv4 address 1.2.3.2 255.255.255.0
proxy-arp
no shutdown
router static
address-family ipv4 unicast
5.4.3.1/32 1.2.3.1
mpls ldp
router-id 5.4.3.2
discovery targeted-hello accept
log
neighbor
DHD
interface Bundle-Ether1
lacp switchover suppress-flaps 300
bundle wait-while 100
interface GigabitEthernet0/0/0/0
description Connected to POA1 Gi0/0/0/0
bundle id 1 mode active
lacp period short
no shutdown
interface GigabitEthernet0/0/0/3
description Connected to POA2 Gi0/0/0/0
bundle id 1 mode active
lacp period short
no shutdown
Troubleshooting Methodology 
Troubleshooting
- show bundle
- show bundle status
- show lacp
- show lacp system-id
- show bundle reasons
- admin show diag chassis eeprom-info
- bundle-hash bundle-ether 10
- show cef <ip dst> hardware egress detail location <x>
- show controller bundle bundle-ether<id> location <x>
- LAG table in the HW for Unicast and Multicast
- How many active members are in the bundle
- What are the “Local” members of the bundle known to each Trident Chip. This is useful in case you see Microcode counter indicating “EGR-LAG-NOT-LOCAL-MEMBER” drop
- Destination SFP (Trident) and port traffic is destined for when sent to that local member
- Ifhandle
- Virtual Port of the bundle
- show bundle trace all/process
- show tech-support bundle file <filesystem:filename>
- admin show tech-support bundle file <filesystem:filename>
- show iccp counters
- show iccp counters
- show iccp trace
- show mlacp inconsistencies
Data Analysis
Common Problems
- When adding a member interface to a bundle ensure NO other configs present under the member interface.
- Eg: No ipv4 address, no subinterfaces etc. If there is any config present then you will get the following error.
RP/0/RSP0/CPU0:RTP-VIKING-L2-3(config)#interface gigabitEthernet 0/6/0/5RP/0/RSP0/CPU0:RTP-VIKING-L2-3(config-if)#bundle id 1 mode active RP/0/RSP0/CPU0:RTP-VIKING-L2-3(config-if)#commit% Failed to commit one or more configuration items during an atomic operation, no changes have been made. Please use 'show configuration failed' to view the errors
Show config failed would show the following reason.
!!% Bundle membership configuration conflicts with underlying physical link configuration (either already existing or also present in the same commit). Please refer to configuration documentation for further assistance.
Remove conflicting configuration to proceed
- Bundle interface doesn’t come up
- Check if bundle interface is in no shut
- Check if member is no shut state
- Check if remote side is up (both bundle and members)
- Check that the port mac address is valid
- Check if LACP parameters are same on both sides If LACP is enabled
- Check LACP status
- show lacp bundle cli
- show lacp counters
- clear lacp counters
- Check if lacp packets are transmitted and received with cli command
- debug bundlemgr local packets port <x>
- Bundle members are not distributing
- Check to make sure the member is up
- show interface <>
- Check why the member is not distributing
- show bundles reasons
- Check if remote side is up
- Check if LACP parameters are same on both sides. If LACP is enabled, check LACP status
- show lacp bundle
- Check if bundle members have the same characteristics
- show running interface <interface name>
- Check if lacp packets are transmitted and received
- debug bundlemgr local packets port <>
- Other useful commands
- show lacp counters
- clear lacp counters
- L3 traffic is not flowing
- Check if arp is resolved on all bundle line cards
- show arp
- Check if mac of the bundle and what is in the arp is same
- show bundle
- show arp
- Verify the bundle mac address is a unicast mac address
- show interface Bundle-Ether <>
- Verify the IP addresses on both sides of the bundle are in the same subnet
- Check if egress uidb shows that it is bundle interface
- show uidb data location <0/x/cpu0> Bundle-Ether <> egress
- Check to make sure ping is working over bundle interface
- Verify there is a route to the destination ip address
- show cef
- Verify the ingress IPv4 forwarding structures in hardware are programmed correctly
- show cef <prefix of next hop> hardware detail ingress location <0/x/cpu0>
- Verify the egress IPv4 forwarding structures in hardware are programmed correctly
- show cef <prefix> hardware detail egress location <0/x/cpu0>"
Make sure the rx_adj field matches the correct lag id. Make sure the "leaf" shows "forwarding" and nrLDI shows RX and TX adjacency as "COMPLETE"
- Check Trident Chip counters to find any drop counters getting incremented
- show controllers stats all location <0/x/cpu0>
- If traffic is still not flowing check the trace to make sure the lag table is programmed
- show bundle trace process adjacency buffer all location <0/x/cpu0>
- Check LAG table setup in the hardware that the table is programmed correctly
Limitations/Restrictions
- All member links should be of the same speed
- Maximum number of Link Bundling interfaces per chassis is 255 (however a few are reserved for Bundle-POS interfaces)
- Maximum number of Link Bundle interface per line card is 64
- Maximum number of members per bundle is 64. User will be allowed to configure more than 8 members and the additional links will be in de-attached state
