L2TPv3 between ISR4K routers sending packets but not receiving

[james.brunner]

Hi all,

I'm having an issue setting up an L2TPv3 tunnel between two routers. The routers are both 4431-SEC/K9's running 17.9.4a and have AppX licences loaded and in use:

ISR_4400_Application (ISR_4400_Application):
Description: AppX License for Cisco ISR 4400 Series
Status: IN USE

I have confirmed we can ping a 1500byte packet with DF set between the routers' Loopback interfaces without issue:

Type escape sequence to abort.
Sending 10, 1500-byte ICMP Echos to aaa.bbb.ccc.22, timeout is 2 seconds:
Packet sent with a source address of aaa.bbb.ccc.32
Packet sent with the DF bit set
!!!!!!!!!!
Success rate is 100 percent (10/10), round-trip min/avg/max = 2/2/3 ms

We're running the vlan up to the routers at both ends over a dot1q trunk from the local site's switch and have a sub-interface defined on each router for that vlan as shown below.

The PWs are created using the loopback interfaces and the configuration is the same on the other router (with the IPs reversed obviously!)

interface Loopback0
ip address aaa.bbb.ccc.32 255.255.255.255

l2tp-class p2p-dc-lc
authentication
password 7 <blahblahblah>

pseudowire-class p2p-dc-pw
encapsulation l2tpv3
protocol l2tpv3 p2p-dc-lc
ip local interface Loopback0
ip pmtu

interface GigabitEthernet0/0/3.40
encapsulation dot1Q 40
xconnect aaa.bbb.ccc.22 40 encapsulation l2tpv3 pw-class p2p-dc-pw

Checking the l2tp state shows the tunnel up and 1 session established (we see the same on the other router)

L2TP Tunnel and Session Information Total tunnels 1 sessions 1

LocTunID RemTunID Remote Name State Remote Address Sessn L2TP Class/
Count VPDN Group
1388630880 833449175 abcdefghij01 est aaa.bbb.ccc.22 1 p2p-dc-lc

LocID RemID TunID Username, Intf/ State Last Chg Uniq ID
Vcid, Circuit
2647075326 2593176190 1388630880 40, Gi0/0/3.40:40 est 10:59:15 0

Checking the xconnect state shows that all are UP (we see the same on the other router)

Legend: XC ST=Xconnect State S1=Segment1 State S2=Segment2 State
UP=Up DN=Down AD=Admin Down IA=Inactive
SB=Standby HS=Hot Standby RV=Recovering NH=No Hardware

XC ST Segment 1 S1 Segment 2 S2
------+---------------------------------+--+---------------------------------+--
UP pri ac Gi0/0/3.40:40(Eth VLAN) UP l2tp aaa.bbb.ccc.22:40 UP

But looking at the session packet data, it seems to be receiving packets from vlan40 on g0/0/3.40 and sending them out across the PW but not receiving anything from the other end. These counters are the same at the other end, it's not receiving these packets but it is sending across the PW its own that are received from vlan40 on its g0/0/3.40 interface.

L2TP Session Information Total tunnels 1 sessions 1

LocID RemID TunID Pkts-In Pkts-Out Bytes-In Bytes-Out
2647075326 2593176190 1388630880 0 492472 0 112487154

The tunnel stats show the same thing, packets sent but none received...

L2TP Tunnel Information Total tunnels 1 sessions 1

Tunnel id 1388630880 is up, remote id is 833449175, 4 active sessions
Remotely initiated tunnel
Tunnel state is established, time since change 12:43:12
Tunnel transport is IP (115)
Remote tunnel name is abcdefghij01
Internet Address aaa.bbb.ccc.22, port 0
Local tunnel name is qrstuvwxyz01
Internet Address aaa.bbb.ccc.32, port 0
L2TP class for tunnel is p2p-dc-lc
Counters, taking last clear into account:
3429708 packets sent, 0 received
409975135 bytes sent, 0 received
Last clearing of counters never
Counters, ignoring last clear:
3429708 packets sent, 0 received
409975135 bytes sent, 0 received
Control Ns 10, Nr 776
Local RWS 1024 (default), Remote RWS 1024
Control channel Congestion Control is disabled
Tunnel PMTU checking enabled
Retransmission time 1, max 1 seconds
Unsent queuesize 0, max 0
Resend queuesize 0, max 1
Total resends 0, ZLB ACKs sent 771
Total out-of-order dropped pkts 0
Total out-of-order reorder pkts 0
Total peer authentication failures 0
Current no session pak queue check 0 of 5
Retransmit time distribution: 0 0 0 0 0 0 0 0 0
Control message authentication is disabled

The detailed session information shows the same sent but not received...

Session id 2647075326 is up, logical session id 65655, tunnel id 1388630880
Remote session id is 2593176190, remote tunnel id 833449175
Remotely initiated session
Unique ID is 0
Session Layer 2 circuit, type is Ethernet Vlan, name is GigabitEthernet0/0/3.40:40
Session vcid is 40
Circuit state is UP
Local circuit state is UP
Remote circuit state is UP
Call serial number is 79700002
Remote tunnel name is abcdefghij01
Internet address is aaa.bbb.ccc.22
Local tunnel name is qrstuvwxyz01
Internet address is aaa.bbb.ccc.32
IP protocol 115
Session is L2TP signaled
Session state is established, time since change 12:46:22
513107 Packets sent, 0 received
113839389 Bytes sent, 0 received
Last clearing of counters never
Counters, ignoring last clear:
513107 Packets sent, 0 received
113839389 Bytes sent, 0 received
Receive packets dropped:
out-of-order: 0
other: 0
total: 0
Send packets dropped:
exceeded session MTU: 0
other: 0
total: 0
DF bit off, ToS reflect disabled, ToS value 0, TTL value 255
Sending UDP checksums are disabled
Received UDP checksums are verified
Session PMTU enabled, path MTU is not known
No session cookie information available
FS cached header information:
encap size = 24 bytes
45000014 00000000 ff733bee c0a8ff20
c0a8ff16 9a90ba7e
Sequencing is off
Conditional debugging is disabled
SSM switch id is 4106, SSM segment id is 16505

And looking at the PW interface, it says it hasn't sent anything at all... but we know Cisco code can be buggy!

pseudowire100001 is up
MTU 1500 bytes, BW not configured
Encapsulation l2tpv3
Peer IP aaa.bbb.ccc.22, VC ID 40
RX
0 packets 0 bytes 0 drops
TX
0 packets 0 bytes 0 drops

If I Wireshark the "transport" interfaces between the two routers, I see the L2TPv3 control packets bounce back and forth but when pinging down the PW from a host in vlan40 on one end to a host in vlan40 on the other we see nothing.

Looking at the vlan40 "input" interface on the routers g0/0/3.40, a Wireshark shows the ARP request come in from one host looking for the other, but then nothing more.

I think I've missed something blindingly obvious but I can think what!

Any help gratefully received!

JB.

[james.brunner]

The L2TP/Xconnect commands are enabled with the AppX - As Rich said we also have the Sec licence loaded and in-use:

ISR_4400_Security (ISR_4400_Security):
Description: Security License for Cisco ISR 4400 Series
Status: IN USE

ISR_4400_Application (ISR_4400_Application):
Description: AppX License for Cisco ISR 4400 Series
Status: IN USE

ISR_4431_1G_Performance (ISR_4431_1G_Performance):
Description: Performance on Demand License for 4430 Series
Status: IN USE

ISR_4400_Hsec (ISR_4400_Hsec):
Description: U.S. Export Restriction Compliance license for 4400 series
Status: IN USE

As for the tunnel's counters, rather than the session counters, they look like this - no drops:

abcdefghij01#sh l2tp count tunnel all
Summary listing of per-tunnel statistics:

LocTunID    RemTunID    Remote IP       Total     Total     Total     Total
                                        XMIT      RE-XMIT   RCVD      DROP
754821136   2283258120  192.168.255.32  6993      0         6991      0

Number of tunnels with per-tunnel stats: 1

[Rich R]

Time to use datapath packet trace:
https://www.cisco.com/c/en/us/support/docs/content-networking/adaptive-session-redundancy-asr/117858-technote-asr-00.html
That follows the packet through each step in QFP till it apparently gets dispatched or dropped.  If it's dropped (which yours seem to be) you can tell where and why.

Also - "show platform hardware qfp active statistics drop" command (and there are some more options after that).

[MHM Cisco World]

do you have any FW or ACL deny udp port 1701 ?

maybe that the reason

MHM

[Georg Pauwen]

Hello,

I might have missed that, but can you post the full running configs (sh run) of both routers, as well as the output of 'show l2tp session all' of both routers ?

[james.brunner · Accepted solution]

Ok, we have success and partly it's my fault by omission...

Thanks to MHM for mentioning Firewalls and Rich, "show platform hardware qfp active statistics drop" was my best friend...

-------------------------------------------------------------------------
Global Drop Stats                         Packets                  Octets
-------------------------------------------------------------------------
FirewallInvalidZone                      37759090              4747564781

Which was the clue I needed, as I forgot to mention the L2TP transport interface had a ZBFW zone attached; let's call it 'Steve'. My loopback interface for the source of the L2TP was also in the same Steve zone, and we had a zone-pair permitting source Steve destination Steve and a class-map that defaults to pass, so the L2TP tunnel came up fine (for the control traffic).

BUT... the xconnect'ed sub-interface was not in any zone - I assumed that as this was a pure L2 tunnel it didn't matter. L2 Packet arrives on sub-interface, gets encapsulated with RouterA loopback as source and RouterB loopback as destination, RouterA to RouterB permitted by Steve and off we go...

Turns out this isn't the case. Once I added the sub-interface into the same Steve zone it began working. It makes some kind of sense but I'm still not sold on why it works: My packet arrives on the sub-interface in my Steve zone but the destination zone doesn't really exist as it should be tunneled. The L2TP doesn't have a zone unlike say a GRE tunnel interface that can take a zone, so it kinda makes it up from the transport/output interface's zone?

Anyone got a better idea?

But for now, just a big thank you to MHM and Rich for keeping me sane!

[MHM Cisco World]

You are so so welcome'

For zone I will share with you the zone use in tunnel later today 

MHM

[Rich R]

Glad you worked it out <smile>
Certainly good to know for future.