cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
802
Views
9
Helpful
21
Replies

L2TPv3 between ISR4K routers sending packets but not receiving

james.brunner
Level 1
Level 1

Hi all,

I'm having an issue setting up an L2TPv3 tunnel between two routers. The routers are both 4431-SEC/K9's running 17.9.4a and have AppX licences loaded and in use:

ISR_4400_Application (ISR_4400_Application):
Description: AppX License for Cisco ISR 4400 Series
Status: IN USE

 

I have confirmed we can ping a 1500byte packet with DF set between the routers' Loopback interfaces without issue:

Type escape sequence to abort.
Sending 10, 1500-byte ICMP Echos to aaa.bbb.ccc.22, timeout is 2 seconds:
Packet sent with a source address of aaa.bbb.ccc.32
Packet sent with the DF bit set
!!!!!!!!!!
Success rate is 100 percent (10/10), round-trip min/avg/max = 2/2/3 ms

 

We're running the vlan up to the routers at both ends over a dot1q trunk from the local site's switch and have a sub-interface defined on each router for that vlan as shown below.

The PWs are created using the loopback interfaces and the configuration is the same on the other router (with the IPs reversed obviously!)

interface Loopback0
ip address aaa.bbb.ccc.32 255.255.255.255

l2tp-class p2p-dc-lc
authentication
password 7 <blahblahblah>

pseudowire-class p2p-dc-pw
encapsulation l2tpv3
protocol l2tpv3 p2p-dc-lc
ip local interface Loopback0
ip pmtu

interface GigabitEthernet0/0/3.40
encapsulation dot1Q 40
xconnect aaa.bbb.ccc.22 40 encapsulation l2tpv3 pw-class p2p-dc-pw

 

Checking the l2tp state shows the tunnel up and 1 session established (we see the same on the other router)

L2TP Tunnel and Session Information Total tunnels 1 sessions 1

LocTunID RemTunID Remote Name State Remote Address Sessn L2TP Class/
Count VPDN Group
1388630880 833449175 abcdefghij01 est aaa.bbb.ccc.22 1 p2p-dc-lc

LocID RemID TunID Username, Intf/ State Last Chg Uniq ID
Vcid, Circuit
2647075326 2593176190 1388630880 40, Gi0/0/3.40:40 est 10:59:15 0

 

Checking the xconnect state shows that all are UP (we see the same on the other router)

Legend: XC ST=Xconnect State S1=Segment1 State S2=Segment2 State
UP=Up DN=Down AD=Admin Down IA=Inactive
SB=Standby HS=Hot Standby RV=Recovering NH=No Hardware

XC ST Segment 1 S1 Segment 2 S2
------+---------------------------------+--+---------------------------------+--
UP pri ac Gi0/0/3.40:40(Eth VLAN) UP l2tp aaa.bbb.ccc.22:40 UP

 

But looking at the session packet data, it seems to be receiving packets from vlan40 on g0/0/3.40 and sending them out across the PW but not receiving anything from the other end. These counters are the same at the other end, it's not receiving these packets but it is sending across the PW its own that are received from vlan40 on its g0/0/3.40 interface.

L2TP Session Information Total tunnels 1 sessions 1

LocID RemID TunID Pkts-In Pkts-Out Bytes-In Bytes-Out
2647075326 2593176190 1388630880 0 492472 0 112487154

 

The tunnel stats show the same thing, packets sent but none received...

L2TP Tunnel Information Total tunnels 1 sessions 1

Tunnel id 1388630880 is up, remote id is 833449175, 4 active sessions
Remotely initiated tunnel
Tunnel state is established, time since change 12:43:12
Tunnel transport is IP (115)
Remote tunnel name is abcdefghij01
Internet Address aaa.bbb.ccc.22, port 0
Local tunnel name is qrstuvwxyz01
Internet Address aaa.bbb.ccc.32, port 0
L2TP class for tunnel is p2p-dc-lc
Counters, taking last clear into account:
3429708 packets sent, 0 received
409975135 bytes sent, 0 received
Last clearing of counters never
Counters, ignoring last clear:
3429708 packets sent, 0 received
409975135 bytes sent, 0 received
Control Ns 10, Nr 776
Local RWS 1024 (default), Remote RWS 1024
Control channel Congestion Control is disabled
Tunnel PMTU checking enabled
Retransmission time 1, max 1 seconds
Unsent queuesize 0, max 0
Resend queuesize 0, max 1
Total resends 0, ZLB ACKs sent 771
Total out-of-order dropped pkts 0
Total out-of-order reorder pkts 0
Total peer authentication failures 0
Current no session pak queue check 0 of 5
Retransmit time distribution: 0 0 0 0 0 0 0 0 0
Control message authentication is disabled


The detailed session information shows the same sent but not received...

Session id 2647075326 is up, logical session id 65655, tunnel id 1388630880
Remote session id is 2593176190, remote tunnel id 833449175
Remotely initiated session
Unique ID is 0
Session Layer 2 circuit, type is Ethernet Vlan, name is GigabitEthernet0/0/3.40:40
Session vcid is 40
Circuit state is UP
Local circuit state is UP
Remote circuit state is UP
Call serial number is 79700002
Remote tunnel name is abcdefghij01
Internet address is aaa.bbb.ccc.22
Local tunnel name is qrstuvwxyz01
Internet address is aaa.bbb.ccc.32
IP protocol 115
Session is L2TP signaled
Session state is established, time since change 12:46:22
513107 Packets sent, 0 received
113839389 Bytes sent, 0 received
Last clearing of counters never
Counters, ignoring last clear:
513107 Packets sent, 0 received
113839389 Bytes sent, 0 received
Receive packets dropped:
out-of-order: 0
other: 0
total: 0
Send packets dropped:
exceeded session MTU: 0
other: 0
total: 0
DF bit off, ToS reflect disabled, ToS value 0, TTL value 255
Sending UDP checksums are disabled
Received UDP checksums are verified
Session PMTU enabled, path MTU is not known
No session cookie information available
FS cached header information:
encap size = 24 bytes
45000014 00000000 ff733bee c0a8ff20
c0a8ff16 9a90ba7e
Sequencing is off
Conditional debugging is disabled
SSM switch id is 4106, SSM segment id is 16505


And looking at the PW interface, it says it hasn't sent anything at all... but we know Cisco code can be buggy!

pseudowire100001 is up
MTU 1500 bytes, BW not configured
Encapsulation l2tpv3
Peer IP aaa.bbb.ccc.22, VC ID 40
RX
0 packets 0 bytes 0 drops
TX
0 packets 0 bytes 0 drops


If I Wireshark the "transport" interfaces between the two routers, I see the L2TPv3 control packets bounce back and forth but when pinging down the PW from a host in vlan40 on one end to a host in vlan40 on the other we see nothing.

Looking at the vlan40 "input" interface on the routers g0/0/3.40, a Wireshark shows the ARP request come in from one host looking for the other, but then nothing more.

I think I've missed something blindingly obvious but I can think what!

Any help gratefully received!

JB.

1 Accepted Solution

Accepted Solutions

james.brunner
Level 1
Level 1

Ok, we have success and partly it's my fault by omission...

Thanks to MHM for mentioning Firewalls and Rich, "show platform hardware qfp active statistics drop" was my best friend...

-------------------------------------------------------------------------
Global Drop Stats                         Packets                  Octets
-------------------------------------------------------------------------
FirewallInvalidZone                      37759090              4747564781

 

Which was the clue I needed, as I forgot to mention the L2TP transport interface had a ZBFW zone attached; let's call it 'Steve'. My loopback interface for the source of the L2TP was also in the same Steve zone, and we had a zone-pair permitting source Steve destination Steve and a class-map that defaults to pass, so the L2TP tunnel came up fine (for the control traffic).

BUT... the xconnect'ed sub-interface was not in any zone - I assumed that as this was a pure L2 tunnel it didn't matter. L2 Packet arrives on sub-interface, gets encapsulated with RouterA loopback as source and RouterB loopback as destination, RouterA to RouterB permitted by Steve and off we go...

Turns out this isn't the case. Once I added the sub-interface into the same Steve zone it began working. It makes some kind of sense but I'm still not sold on why it works: My packet arrives on the sub-interface in my Steve zone but the destination zone doesn't really exist as it should be tunneled. The L2TP doesn't have a zone unlike say a GRE tunnel interface that can take a zone, so it kinda makes it up from the transport/output interface's zone?

Anyone got a better idea?

But for now, just a big thank you to MHM and Rich for keeping me sane!

View solution in original post

21 Replies 21

Believe me' 

Bgp l2tpv3 gre etc. All have pmtu and all face issue with mtu and the recommendation is remove pmtu and hardcoded mtu.

You can ping from LO to LO but that ping not include the additional header of l2tp.

Friend under pw-class remove pmtu and hardcoded mtu to value 1400 and check.

MHM

james.brunner
Level 1
Level 1

Hi MHM,

Unfortunately it made no difference. We still have sent packets but none received. Even small 64 byte packets don't make it.

RouterA:

abcdefghij01#sh l2tp sess pack

L2TP Session Information Total tunnels 1 sessions 1

LocID RemID TunID Pkts-In Pkts-Out Bytes-In Bytes-Out
3695631231 2206957931 833449175 0 25 0 1600

RouterB:

qrstuvwxyz01#sh l2tp sess pack

L2TP Session Information Total tunnels 1 sessions 1

LocID RemID TunID Pkts-In Pkts-Out Bytes-In Bytes-Out
2206957931 3695631231 1388630880 0 1208 0 122610

JB.

Ok' 

You use subinterface' so the packet is tag with vlan'

Under pw-class add

Interworking vlan 

Notice you need to add this in both ends

MHM

james.brunner
Level 1
Level 1

Still no luck. The "sh xconnect interface g0/0/3.40 detail" shows the Interworking is set to vlan but still no traffic even after a "clear l2tp all" on both ends....

abcdefghij01#sh xcon int g0/0/3.40 det
Legend:    XC ST=Xconnect State  S1=Segment1 State  S2=Segment2 State
  UP=Up       DN=Down            AD=Admin Down      IA=Inactive
  SB=Standby  HS=Hot Standby     RV=Recovering      NH=No Hardware

XC ST  Segment 1                         S1 Segment 2                         S2
------+---------------------------------+--+---------------------------------+--
UP pri   ac Gi0/0/3.40:40(Eth VLAN)      UP l2tp aaa.bbb.ccc.32:40            UP
            Interworking: vlan                   Session ID: 701580992
                                                 Tunnel ID: 754821136
                                                 Peer name: qrstuvwxyz01
                                                 Protocol State: UP
                                                 Remote Circuit State: UP
                                                 pw-class: p2p-dc-pw
abcdefghij01#sh l2tp sess pack

L2TP Session Information Total tunnels 1 sessions 1

LocID      RemID      TunID      Pkts-In    Pkts-Out   Bytes-In   Bytes-Out
2966719755 3824011990 754821136  0          33         0          2112

JB.

the main interface is down that why the traffic is down 
I run lab for you and face same issue 
check the status of interface (along the path)

MHM

Screenshot (428).pngScreenshot (429).png

Screenshot (430).png

james.brunner
Level 1
Level 1

So to ensure the sub-interfaces connected to each vlan are UP and working properly and the hosts are live, I removed the xconnect and added an ip address (being the other end's host IP) to each sub-interface. Both routers can ping their hosts fine on their vlan, so the parent g0/0/3, sub-interface and encapsulation are working fine.

After removing the IPs and replacing the xconnects, we back to where we were - 10.1.1.1 still can't ping 10.1.1.254 across the L2TP.

I think I'm at the point of a TAC case as this should just work...

RouterA:

abcdefghij01#
abcdefghij01#conf t
Configuration session is locked. The lock will be cleared once you exit out of configuration mode.
Enter configuration commands, one per line. End with CNTL/Z.
abcdefghij01(config)#int g0/0/3.40
abcdefghij01(config-subif)#no xconnect aaa.bbb.ccc.32 40 encapsulation l2tpv3 pw-class p2p-dc-pw
abcdefghij01(config-subif)#ip add 10.1.1.1 255.255.255.0
abcdefghij01(config-subif)#^Z
abcdefghij01#ping 10.1.1.254 so g0/0/3.40
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.1.254, timeout is 2 seconds:
Packet sent with a source address of 10.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
abcdefghij01#
abcdefghij01#conf t
Configuration session is locked. The lock will be cleared once you exit out of configuration mode.
Enter configuration commands, one per line. End with CNTL/Z.
abcdefghij01(config)#int g0/0/3.40
abcdefghij01(config-subif)#no ip add 10.1.1.1 255.255.255.0
abcdefghij01(config-subif)#xconnect aaa.bbb.ccc.32 40 encapsulation l2tpv3 pw-class p2p-dc-pw
abcdefghij01(config-subif-xconn)#^Z
abcdefghij01#sh l2tp sess L2TP Session Information Total tunnels 1 sessions 1 LocID RemID TunID Username, Intf/ State Last Chg Uniq ID Vcid, Circuit 2890740845 218530236 754821136 40, Gi0/0/3.40:40 est 00:00:12 0 abcdefghij01# abcdefghij01#sh l2tp sess pack L2TP Session Information Total tunnels 1 sessions 4 LocID RemID TunID Pkts-In Pkts-Out Bytes-In Bytes-Out 2890740845 218530236 754821136 0 8 0 512 abcdefghij01#

RouterB:

qrstuvwxyz01#
qrstuvwxyz01#conf t
Configuration session is locked. The lock will be cleared once you exit out of configuration mode.
Enter configuration commands, one per line. End with CNTL/Z.
qrstuvwxyz01(config)#interface GigabitEthernet0/0/3.40
qrstuvwxyz01(config-subif)#no xconnect aaa.bbb.ccc.22 40 encapsulation l2tpv3 pw-class p2p-dc-pw
qrstuvwxyz01(config-subif)#ip add 10.1.1.254 255.255.255.0
qrstuvwxyz01(config-subif)#^Z
qrstuvwxyz01#ping 10.1.1.1 so g0/0/3.40
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.1.1, timeout is 2 seconds:
Packet sent with a source address of 10.1.1.254
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
qrstuvwxyz01#
qrstuvwxyz01#conf t
Configuration session is locked. The lock will be cleared once you exit out of configuration mode.
Enter configuration commands, one per line. End with CNTL/Z.
qrstuvwxyz01(config)#interface GigabitEthernet0/0/3.40
qrstuvwxyz01(config-subif)#no ip add 10.1.1.254 255.255.255.0
qrstuvwxyz01(config-subif)#xconnect aaa.bbb.ccc.22 40 encapsulation l2tpv3 pw-class p2p-dc-pw
qrstuvwxyz01(config-subif-xconn)#^Z
qrstuvwxyz01#sh l2tp sess L2TP Session Information Total tunnels 1 sessions 1 LocID RemID TunID Username, Intf/ State Last Chg Uniq ID Vcid, Circuit 218530236 2890740845 754821136 40, Gi0/0/3.40:40 est 00:00:16 0 qrstuvwxyz01# qrstuvwxyz01#sh l2tp sess pack L2TP Session Information Total tunnels 1 sessions 1 LocID RemID TunID Pkts-In Pkts-Out Bytes-In Bytes-Out 218530236 2890740845 754821136 0 10 0 640 qrstuvwxyz01#

 

Show ip interface brief 

In both ebd of l2tpv3 and on devices connect to these subinterface

Are all UP

MHM

james.brunner
Level 1
Level 1

Yes, all interfaces are up. It looks like the xconnect just isn't moving the traffic.

I'm going to GNS3 the setup with the same configuration cut'n'paste on 8000v's running the same 17.9.4a code (nearest I can get to a 4431 in GNS3) and if it works then I think I'll open a TAC case.

try use CSR1000v this similar to 4431, the GNS3 7000 image is run old IOS 
also I dont try in my lab L2tp auth, I read somewhere that the auth also effect traffic, remove the auth and check in real device 

MHM

james.brunner
Level 1
Level 1

Still no luck - tried removing the auth and got the send/no receive again.

I'm heading to TAC...

I will check how we can debug this traffic' 

Also one more Q what is license in both router?

MHM

show l2tp counters tunnel all <<- can you check this command see if there is any packet drop ?

thanks 

MHM

He said:

The routers are both 4431-SEC/K9's running 17.9.4a and have AppX licences loaded and in use:

ISR_4400_Application (ISR_4400_Application):
Description: AppX License for Cisco ISR 4400 Series
Status: IN USE

the license he need is security K9

MHM

Review Cisco Networking for a $25 gift card