Solved: Re: ASR-920 IOS XE - MPLS L2VPN XConnect passing traffic in one direction only

Drew1815 · ‎09-04-2018

Background:
###########

Multicast vlans from an Arista 7150 are trunked to a GE interface on the ASR 920. The ASR 920 GE I/face has been configured for MPLS L2VPN (Xconnect) functionality, where the intention is to extract specific vlan(s) using the "Ethernet Service instance" interface configuration for Layer 2 transport to a remote ISR 4321 (PE) router across the MPLS network. On the remote ISR, a server/host is connected in order to join and receive the multicast stream. The ASR 920 MPLS L2VPN configuration has been adopted from the following Cisco L2VPN Protocol-Based CLIs Guide:

https://www.cisco.com/c/en/us/td/docs/routers/asr920/configuration/guide/mpls/mp-l2-vpns-xe-3s-asr920-book/mp-l2vpn-prot-based.pdf

Arista7150-----L2 Vlan Trunk----ASR920------MPLS Core------ISR4321-----M/Cast Host/PC
Vlan2999 |<=====L2VPN Xconnect====>|

The Issues:
############
1) Vlan traffic is forwarded from the source Arista 7150 device via the L2VPN Xconnect to the ISR port /interface. However it appears that no return traffic is reaching back to the Arista in order respond to PIM hellos & IGMP joins etc.

As shown below, no "input" (return) traffic is seen:

Arista7150#sh interfaces ethernet 44
Ethernet44 is up, line protocol is up (connected)
Hardware is Ethernet, address is 001c.73b5.3143 (bia 001c.73b5.3143)
Description: MCast To MPLS Edge
Ethernet MTU 9214 bytes , BW 1000000 kbit
Full-duplex, 1Gb/s, auto negotiation: on, uni-link: unknown
Up 26 days, 18 hours, 13 minutes, 14 seconds
3 link status changes since last clear
Last clearing of "show interface" counters never
5 minutes input rate 0 bps (0.0% with framing overhead), 0 packets/sec
5 minutes output rate 1.16 kbps (0.0% with framing overhead), 2 packets/sec
     0 packets input, 0 bytes                 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
     Received 0 broadcasts, 0 multicast
     0 runts, 0 giants
     0 input errors, 0 CRC, 0 alignment, 0 symbol, 0 input discards
     0 PAUSE input
     4268424 packets output, 321762584 bytes
     Sent 78122 broadcasts, 4190299 multicast
     0 output errors, 0 collisions
     0 late collision, 0 deferred, 0 output discards
     0 PAUSE output
Arista7150#

2) There are a large number of increasing output drops on the ASR-920 L2VPN interface which may explain why the traffic is not being sent towards the Arista, in additon a "zero" output packets:

ASR-920#sh int gi 0/0/23
GigabitEthernet0/0/23 is up, line protocol is up
Hardware is 24xGE-4x10GE-FIXED-S, address is cc46.d65b.a397 (bia cc46.d65b.a397)
Description: Multicast Vlan
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 1000Mbps, link type is auto, media type is SX
output flow-control is unsupported, input flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 5d21h, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 106870 <<<<<<<<<<<<<<<
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1000 bits/sec, 2 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
     4281163 packets input, 322723106 bytes, 0 no buffer
     Received 78346 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 4202814 multicast, 0 pause input
     0 packets output, 0 bytes, 0 underruns    <<<<<<<<<<<<<<<<<<<<<<<<
     0 output errors, 0 collisions, 3 interface resets
     2670 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
ASR-920#

Assistance Required:
####################

In the "additional information /Captures" section below, traffic appears to be received and sent at both the PW end-points.

However I believe the issue may be between the ASR-920 AC (attached circuit) and the ASR-920 PW end-point. Here are some snippets for easier reference:

ASR-920:
--------
ASR-920#sh mpls l2 vc detail
VC statistics:
    transit packet totals: receive 6658, send 2501
    transit byte totals:   receive 1132735, send 186204
    transit packet drops: receive 0, seq error 0, send 0

ISR-4321:
---------
ISR-4321#sh mpls l2 vc detail
VC statistics:
    transit packet totals: receive 755, send 6760
    transit byte totals:   receive 69802, send 1289478
    transit packet drops: receive 1760, seq error 0, send 0      <<<<<<<<<<<<<<<<<<<<<<<<<<<

*** First question: What are "transit packet drops"???

The AC is the GE interface 0/0/23 and shows an incresing number of output drops and zero packets out:

ASR-920#sh int gi 0/0/23
GigabitEthernet0/0/23 is up, line protocol is up
Hardware is 24xGE-4x10GE-FIXED-S, address is cc46.d65b.a397 (bia cc46.d65b.a397)
Description: Multicast Vlan
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 1000Mbps, link type is auto, media type is SX
output flow-control is unsupported, input flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 5d21h, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 106870 <<<<<<<<<<<<<<<
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1000 bits/sec, 2 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
     4281163 packets input, 322723106 bytes, 0 no buffer
     Received 78346 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 4202814 multicast, 0 pause input
     0 packets output, 0 bytes, 0 underruns                           <<<<<<<<<<<<<<<<<<<<<<<<
     0 output errors, 0 collisions, 3 interface resets
     2670 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
ASR-920#

*** Any hints on how to trouble-shoot what is causing the output drops (have attempted to run a number of debugs but no output (yes 'term mon" was turned on)

*** Has the L2VPN PW been configured correctly? Or is something missing under the AC interface on the ASR-920? (GE0/0/23)

*** Anyone else has seen or familiar with this issue before?

All thoughts and suggestions most welcome!

Thanking you all in advance.

Regards\Drew

#####################################
Additonal Information /Captures etc:
####################################

############
ARISTA7150
############

!
interface Ethernet44
   description MCast To MPLS Edge
   switchport trunk allowed vlan 2900-2999
   switchport mode trunk
   no qos trust
   qos cos 1
!
interface Vlan2999
   description Cust09
   ip address 172.31.9.237/30
   ip igmp version 2
   ip multicast boundary CUST-MULTICAST
   ip pim sparse-mode
   ip pim dr-priority 100
!
ip access-list standard CUST-MULTICAST
   10 remark remark multicast boundary access list
   20 permit <removed>
   30 permit <removed>
   40 permit <removed>
   50 permit <removed>
   60 permit <removed>
   70 remark Allow administratively scoped addresses
   80 permit 224.0.0.0/8
!

=====================================================================================
Arista7150#
Arista7150sh int eth 44
Ethernet44 is up, line protocol is up (connected)
Hardware is Ethernet, address is 001c.73b5.3143 (bia 001c.73b5.3143)
Description: MCast To MPLS Edge
Ethernet MTU 9214 bytes , BW 1000000 kbit
Full-duplex, 1Gb/s, auto negotiation: on, uni-link: unknown
Up 26 days, 20 hours, 16 minutes, 51 seconds
3 link status changes since last clear
Last clearing of "show interface" counters never
5 minutes input rate 0 bps (0.0% with framing overhead), 0 packets/sec
5 minutes output rate 1.14 kbps (0.0% with framing overhead), 2 packets/sec
     0 packets input, 0 bytes
     Received 0 broadcasts, 0 multicast
     0 runts, 0 giants
     0 input errors, 0 CRC, 0 alignment, 0 symbol, 0 input discards
     0 PAUSE input
     4282431 packets output, 322818872 bytes
     Sent 78370 broadcasts, 4204058 multicast
     0 output errors, 0 collisions
     0 late collision, 0 deferred, 0 output discards
     0 PAUSE output
Arista7150#

########
ASR-920:
########
!
interface Loopback0
description CoreRouting
ip address 192.168.234.66 255.255.255.255
ip ospf 18361 area 1
!
interface GigabitEthernet0/0/23
description Multicast
no ip address
negotiation auto
service-policy input EDGE-L2VPN-INGRESS
service instance 666 ethernet
encapsulation dot1q 2999
l2protocol tunnel cdp stp vtp pagp dot1x lldp lacp udld loam esmc elmi ptppd
!
!
l2vpn xconnect context Customer9
member GigabitEthernet0/0/23 service-instance 666
member 192.168.236.3 888 encapsulation mpls
!

======================================================================
ASR-920#sh int gi 0/0/23
GigabitEthernet0/0/23 is up, line protocol is up
Hardware is 24xGE-4x10GE-FIXED-S, address is cc46.d65b.a397 (bia cc46.d65b.a397)
Description: Multicast
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 1000Mbps, link type is auto, media type is SX
output flow-control is unsupported, input flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 5d21h, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 106870
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1000 bits/sec, 2 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
     4281163 packets input, 322723106 bytes, 0 no buffer
     Received 78346 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 4202814 multicast, 0 pause input
     0 packets output, 0 bytes, 0 underruns
     0 output errors, 0 collisions, 3 interface resets
     2670 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
ASR-920#
ASR-920#sh mpls l2transport vc

Local intf     Local circuit              Dest address    VC ID      Status
------------- -------------------------- --------------- ---------- ----------
Gi0/0/23       Eth VLAN 2999              192.168.236.3   888        UP

ASR-920#sh l2vpn atom vc

                                       Service
Interface Peer ID         VC ID      Type   Name                     Status
--------- --------------- ---------- ------ ------------------------ ----------
pw100007 192.168.236.3   888        p2p    Customer9                UP

ASR-920#
ASR-920#sh xconnect interface gi0/0/23 detail
Legend:    XC ST=Xconnect State S1=Segment1 State S2=Segment2 State
UP=Up       DN=Down            AD=Admin Down      IA=Inactive
SB=Standby HS=Hot Standby     RV=Recovering      NH=No Hardware

XC ST Segment 1                         S1 Segment 2                         S2
------+---------------------------------+--+---------------------------------+--
UP pri   ac Gi0/0/23:666(Eth VLAN)       UP mpls 192.168.236.3:888            UP
            Interworking: ethernet               Local VC label 68
                                                 Remote VC label 66

ASR-920#
ASR-920#sh mpls l2 vc detail
Local interface: Gi0/0/23 up, line protocol up, Eth VLAN 2999 up
Interworking type is Ethernet
Destination address: 192.168.236.3, VC ID: 888, VC status: up
    Output interface: Te0/0/24, imposed label stack {22 66}
    Preferred path: not configured
    Default path: active
    Next hop: 10.200.1.10
Create time: 16:48:19, last status change time: 00:08:27
    Last label FSM state change time: 16:48:19
Signaling protocol: LDP, peer 192.168.236.3:0 up
    Targeted Hello: 192.168.234.66(LDP Id) -> 192.168.236.3, LDP is UP
    Graceful restart: not configured and not enabled
    Non stop routing: not configured and not enabled
    Status TLV support (local/remote)   : enabled/supported
      LDP route watch                   : enabled
      Label/status state machine        : established, LruRru
      Last local dataplane   status rcvd: No fault
      Last BFD dataplane     status rcvd: Not sent
      Last BFD peer monitor status rcvd: No fault
      Last local AC circuit status rcvd: No fault
      Last local AC circuit status sent: No fault
      Last local PW i/f circ status rcvd: No fault
      Last local LDP TLV     status sent: No fault
      Last remote LDP TLV    status rcvd: No fault
      Last remote LDP ADJ    status rcvd: No fault
    MPLS VC labels: local 68, remote 66
    Group ID: local 29, remote 0
    MTU: local 1500, remote 1500
    Remote interface description:MultiCast
Sequencing: receive disabled, send disabled
Control Word: On (configured: autosense)
SSO Descriptor: 192.168.236.3/888, local label: 68
Dataplane:
    SSM segment/switch IDs: 12322/8222 (used), PWID: 8
VC statistics:
    transit packet totals: receive 6658, send 2501
    transit byte totals:   receive 1132735, send 186204
    transit packet drops: receive 0, seq error 0, send 0

ASR-920#
ASR-920#sh ver
Cisco IOS XE Software, Version 03.18.04.SP.156-2.SP4-ext
Cisco IOS Software, ASR920 Software (PPC_LINUX_IOSD-UNIVERSALK9_NPE-M), Version 15.6(2)SP4, RELEASE SOFTWARE (fc4)

cisco ASR-920-24SZ-M (Freescale P2020) processor (revision 1.2 GHz) with 900279K/6147K bytes of memory.
Processor board ID CAT1948V05N
24 Gigabit Ethernet interfaces
4 Ten Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
3670016K bytes of physical memory.
1328927K bytes of SD flash at bootflash:.

Configuration register is 0x2102

ASR-920#

########
ISR4321:
#########

!
interface Loopback0
description Core Routing
ip address 192.168.236.3 255.255.255.255
ip ospf 18361 area 80
!
interface GigabitEthernet0/0/1
description MultiCastPort
no ip address
negotiation auto
xconnect 192.168.234.66 888 encapsulation mpls
!

=============================================================================
ISR-4321#sh mpls l2transport vc

Local intf Local circuit Dest address VC ID Status
------------- -------------------------- --------------- ---------- ----------
Gi0/0/1 Ethernet 192.168.234.66 888 UP

ISR-4321#sh l2vpn atom vc

                                       Service
Interface Peer ID         VC ID      Type   Name                     Status
--------- --------------- ---------- ------ ------------------------ ----------
pw100004 192.168.234.66 888        p2p    Gi0/0/1-7                UP

ISR-4321#
ISR-4321#sh mpls l2 vc detail
Local interface: Gi0/0/1 up, line protocol up, Ethernet up
Destination address: 192.168.234.66, VC ID: 888, VC status: up
    Output interface: Gi0/1/0, imposed label stack {35 68}
    Preferred path: not configured
    Default path: active
    Next hop: 10.200.80.17
Create time: 16:58:11, last status change time: 00:14:15
    Last label FSM state change time: 16:54:07
Signaling protocol: LDP, peer 192.168.234.66:0 up
    Targeted Hello: 192.168.236.3(LDP Id) -> 192.168.234.66, LDP is UP
    Graceful restart: not configured and not enabled
    Non stop routing: not configured and not enabled
    Status TLV support (local/remote)   : enabled/supported
      LDP route watch                   : enabled
      Label/status state machine        : established, LruRru
      Last local dataplane   status rcvd: No fault
      Last BFD dataplane     status rcvd: Not sent
      Last BFD peer monitor status rcvd: No fault
      Last local AC circuit status rcvd: No fault
      Last local AC circuit status sent: No fault
      Last local PW i/f circ status rcvd: No fault
      Last local LDP TLV     status sent: No fault
      Last remote LDP TLV    status rcvd: No fault
      Last remote LDP ADJ    status rcvd: No fault
    MPLS VC labels: local 66, remote 68
    Group ID: local n/a, remote 29
    MTU: local 1500, remote 1500
    Remote interface description: Multicast
Sequencing: receive disabled, send disabled
Control Word: On (configured: autosense)
SSO Descriptor: 192.168.234.66/888, local label: 66
Dataplane:
    SSM segment/switch IDs: 8228/4130 (used), PWID: 5
VC statistics:
    transit packet totals: receive 755, send 6760
    transit byte totals:   receive 69802, send 1289478
    transit packet drops: receive 1760, seq error 0, send 0      <<<<<<<<<<<<<<<<<<<<<<<<<<<

ISR-4321#

ISR-4321#sh ver
Cisco IOS XE Software, Version 16.03.05
Cisco IOS Software [Denali], ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.3.5, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Thu 05-Oct-17 01:15 by mcpre

cisco ISR4321/K9 (1RU) processor with 3699024K/6147K bytes of memory.
Processor board ID FDO2116A3EU
3 Virtual Ethernet interfaces
12 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
8388608K bytes of physical memory.
7081983K bytes of flash memory at bootflash:.
0K bytes of at webui:.

Configuration register is 0x2102

ISR-4321#

asiergd · ‎09-05-2018

Hi

Review ASR920 config, perhaps rewrite ingress tag pop 1 symmetric is missing at service instance.

Please share EDGE-L2VPN-INGRESS configuration.

Please indicate what is your wan MTU..., the minimum MTU in the path between PEs.

Regards

View solution in original post

asiergd · ‎09-05-2018

Hi

Review ASR920 config, perhaps rewrite ingress tag pop 1 symmetric is missing at service instance.

Please share EDGE-L2VPN-INGRESS configuration.

Please indicate what is your wan MTU..., the minimum MTU in the path between PEs.

Regards

Drew1815 · ‎09-05-2018

Hi,

Yep - spot on!

I actually spotted this just after posting to the community when I re-reviewed the user guide again.

The rewrite is of course required to add the vlan tag back on after it egresses from the PW in the return direction - hum funny that it appears to have automatically perform this one way without the cmd.

Thank you for your prompt reply and advice - much appreciated.

Regards Drew