C9300X GRE over IPsec and fragmentations

[dijix1990]

Somebody use this function over WAN? I'm interested how We can use it if almost WAN Provider use mtu 1500 but IPSEC VTI for c9300x doesn't fragment packets

Restrictions for IPsec Virtual Tunnel Interfaces

• Fragmentation of encrypted packets and reassembling of encrypted fragments is not supported. SVTI's MTU needs to be set smaller than physical interface. Fragmentation can be done before encryption or after decryption.

For my remote branch I want to stretch my networks and I'm doing research which type of device can do it

[dijix1990]

Cisco suggests these switches as device which can do security interconnection between Dc's for c9300x it has special asic with 200g performance ipsec. It was said when they presented devices like that c9300x / c9400x / c9500x / c9600x. But I don't think they should say that we can use it over Internet. I use c9300x with l2vpn and channel mtu 9100 for this situation it's good, for Internet it has limitation because of fragmentation

PS in the cisco link there's said - A company can build a secure virtual private network over the Internet or over a public WAN.

[dijix1990]

BTW it's so stupid that cisco recommend it as device for IPSEC connection through Internet)

From security guide (C9300)

IPsec provides the capability to secure communications across a LAN, across private and public WANs, and across the Internet. Examples of its use include:

• Secure branch office connectivity over the Internet: A company can build a secure virtual private network over the Internet or over a public WAN. This enables a business to rely heavily on the Internet and reduce its need for private networks, saving costs and network management overhead.

and the same time:

Restrictions for IPsec Virtual Tunnel Interfaces

• Fragmentation of encrypted packets and reassembling of encrypted fragments is not supported. SVTI's MTU needs to be set smaller than physical interface. Fragmentation can be done before encryption or after decryption.

so if you think that it can replace router for doing ipsec connections via Internet with L2 stretching it's wrong only via channel with MTU above than 1612

For my situation I decided to use C8500L instead

[Torbjørn]

I know you have decided on the C8500L, but this is an interesting issue.

It should work fine as long as you reduce the MTU. Have you tried configuring a reduced MTU on nve1?

[dijix1990]

Yeah.

I think it's because of GRE on the 9300x (catalyst doesn't do fragmentation for tunnel interfaces) because if I do this (on the physical interface) fragmentation is working

For C9300X-01

interface Loopback1
 ip address 10.10.10.1 255.255.255.255

interface TwentyFiveGigE1/0/24
 description -E- ### Link to C9300X-02 - Twe1/0/24
 no switchport
 mtu 1500
 ip address 192.168.100.1 255.255.255.252

ip route 10.10.10.2 255.255.255.255 192.168.100.2

For C9300X-02

interface Loopback1
 ip address 10.10.10.2 255.255.255.255

interface TwentyFiveGigE1/0/24
 description -E- ### Link to C9300X-01 - Twe1/0/24
 no switchport
 mtu 1500
 ip address 192.168.100.2 255.255.255.252

ip route 10.10.10.1 255.255.255.255 192.168.100.1

C9300X-01#sh ip traffic | sec Frags
Frags: 20 reassembled, 0 timeouts, 0 couldn't reassemble
75 fragmented, 240 fragments, 269 couldn't fragment
0 invalid hole
C9300X-01#ping 10.10.10.2 so 10.10.10.1 si 1500 df - Because of Twe1/0/24 has mtu 1500
Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 10.10.10.2, timeout is 2 seconds:
Packet sent with a source address of 10.10.10.1
Packet sent with the DF bit set
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
C9300X-01#ping 10.10.10.2 so 10.10.10.1 si 1501 df  - Because of Twe1/0/24 has mtu 1500
Type escape sequence to abort.
Sending 5, 1501-byte ICMP Echos to 10.10.10.2, timeout is 2 seconds:
Packet sent with a source address of 10.10.10.1
Packet sent with the DF bit set
.....
Success rate is 0 percent (0/5)
C9300X-01#ping 10.10.10.2 so 10.10.10.1 si 1501 - it's fragmented
Type escape sequence to abort.
Sending 5, 1501-byte ICMP Echos to 10.10.10.2, timeout is 2 seconds:
Packet sent with a source address of 10.10.10.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
C9300X-01#sh ip traffic | sec Frags
Frags: 20 reassembled, 0 timeouts, 0 couldn't reassemble
80 fragmented, 250 fragments, 274 couldn't fragment
0 invalid hole

I tried to use ip mtu 1360 and ip tcp mss for Tu1 (ipsec) and before changing I could sent packet size 1388  

8300-01#ping 192.168.10.2 size 1388
Type escape sequence to abort.
Sending 5, 1388-byte ICMP Echos to 192.168.10.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
8300-01#ping 192.168.10.2 size 1389
Type escape sequence to abort.
Sending 5, 1389-byte ICMP Echos to 192.168.10.2, timeout is 2 seconds:
..
Success rate is 0 percent (0/2)

After setting ip mtu as 1360 I could sent packets with size 1302 only

8300-01#ping 192.168.10.2 size 1302
Type escape sequence to abort.
Sending 5, 1302-byte ICMP Echos to 192.168.10.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
8300-01#ping 192.168.10.2 size 1303
Type escape sequence to abort.
Sending 5, 1303-byte ICMP Echos to 192.168.10.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

For NVE changing MTU doesn't affect

[robertpatrick]

Notes:

Cisco Catalyst 9300X supports GRE tunnels, or IPSEC tunnels, but didn't support tunnels that combine GRE-over-IPSEC until release 17.11.1. (thanks for the link! dijx1990)  While the newer IOS XE releases add support for GRE-over-IPSEC, there remain significant restrictions (e.g., no VRF, no tunnel keys, no multiple tunnels between the same source and destination, no mGRE, no DMVPN) mainly due to Cisco limiting the Catalyst 9K against competing with Catalyst 8K features.

Fragmentation to allow packets larger than 1500 to cross the Internet is generally not supported and not recommended, regardless of platform. Best to reduce internal MTU for the tunnel to allow carrying encapsulated packets without performing fragmentation. Typically TCP MSS is adjusted on internal router interface to ensure downstream hosts aren't sending packets that exceed the allowed maximum packet size with overhead (e.g., overlay encapsulation plus encryption headers).

There are Cisco documents available that show VXLAN-over-IPSEC on the Catalyst 9300X using IPSEC IP tunnels.

[dijix1990]

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9300/software/release/17-12/configuration_guide/sec/b_1712_sec_9300_cg/configuring_gre_over_ipsec.html