C9300X GRE over IPsec and fragmentations

[dijix1990]

Somebody use this function over WAN? I'm interested how We can use it if almost WAN Provider use mtu 1500 but IPSEC VTI for c9300x doesn't fragment packets

Restrictions for IPsec Virtual Tunnel Interfaces

• Fragmentation of encrypted packets and reassembling of encrypted fragments is not supported. SVTI's MTU needs to be set smaller than physical interface. Fragmentation can be done before encryption or after decryption.

For my remote branch I want to stretch my networks and I'm doing research which type of device can do it

[Torbjørn]

A 9300X should be fully capable of doing this. Just remember to configure an appropriate IP MTU and ip adjust MSS on the tunnel interfaces on both ends to reduce the load on your devices.

[dijix1990]

I tested and for icmp it doesn't work if I set size above than 1388

9300X-05#ping 192.168.10.2 so 192.168.10.1 si 1388
Type escape sequence to abort.
Sending 5, 1388-byte ICMP Echos to 192.168.10.2, timeout is 2 seconds:
Packet sent with a source address of 192.168.10.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
C9300X-05#ping 192.168.10.2 so 192.168.10.1 si 1389
Type escape sequence to abort.
Sending 5, 1389-byte ICMP Echos to 192.168.10.2, timeout is 2 seconds:
Packet sent with a source address of 192.168.10.1
.....
Success rate is 0 percent (0/5)

Because of C9300X can't dor fragmentation 

[MHM Cisco World]

show ip traffic 
if you see fragment drop is increase then cat9000 with GRE can not support fragment 

MHM

[Torbjørn]

It won`t do fragmentation of encrypted packets. But it should be able to fragment before encryption just fine. Can you post relevant configs? 

[dijix1990]

But it L2 VPN through IPSEC over WAN I can't do fragment before encryption

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9300/software/release/17-9/configuration_guide/vxlan/b_179_bgp_evpn_vxlan_9300_cg/configure-bgp-evpn-vxlan-over-ipsec.html

C9300X-01 as VTEP-1

interface TwentyFiveGigE1/0/24
 description -E- ### Link to C9300X-02 - Twe1/0/24
 no switchport
 mtu 1500
 ip address 192.168.100.1 255.255.255.252

interface Loopback0
 ip address 192.168.50.1 255.255.255.255
 ip ospf 1 area 0

l2vpn evpn
replication-type static
router-id Loopback0

l2vpn evpn instance 101 vlan-based
encapsulation vxlan

interface nve1
 no ip address
 source-interface Loopback0
 host-reachability protocol bgp
 member vni 10000 ingress-replication

crypto ikev2 keyring ikev2_keyring
 peer mypeer
  address 0.0.0.0 0.0.0.0
  pre-shared-key cisco123

crypto ikev2 profile ikev2_profile
 match identity remote address 192.168.100.2 255.255.255.255 
 authentication remote pre-share
 authentication local pre-share
 keyring local ikev2_keyring
 dpd 10 2 periodic

crypto ipsec transform-set tfs esp-gcm 
 esn
 mode tunnel

crypto ipsec profile ipsec_profile
 set transform-set tfs 
 set ikev2-profile ikev2_profile

interface Tunnel1
 ip address 192.168.200.1 255.255.255.252
 ip mtu 1380
 ip tcp adjust-mss 1320
 tunnel source Twe1/0/24
 tunnel mode ipsec ipv4
 tunnel destination 192.168.100.2
 tunnel protection ipsec profile ipsec_profile
 
router bgp 1
 bgp router-id interface Loopback0
 bgp log-neighbor-changes
 bgp update-delay 1
 bgp graceful-restart
 no bgp default ipv4-unicast
 neighbor 192.168.50.2 remote-as 1
 neighbor 192.168.50.2 update-source Loopback0

 address-family ipv4
  redistribute connected
  neighbor 192.168.50.2 activate
  neighbor 192.168.50.2 prefix-list PL-CONNECTED out
 exit-address-family

 address-family l2vpn evpn
  neighbor 192.168.50.2 activate
  neighbor 192.168.50.2 send-community both
 exit-address-family

ip prefix-list PL-CONNECTED seq 5 permit 192.168.50.1/32

C9300X-02 as VTEP-2

interface TwentyFiveGigE1/0/24
 description -E- ### Link to C9300X-01 - Twe1/0/24
 no switchport
 mtu 1500
 ip address 192.168.100.2 255.255.255.252

interface Loopback0
 ip address 192.168.50.2 255.255.255.255
 ip ospf 1 area 0

l2vpn evpn
replication-type static
router-id Loopback0

l2vpn evpn instance 101 vlan-based
encapsulation vxlan

interface nve1
 no ip address
 source-interface Loopback0
 host-reachability protocol bgp
 member vni 10000 ingress-replication

crypto ikev2 keyring ikev2_keyring
 peer mypeer
  address 0.0.0.0 0.0.0.0
  pre-shared-key cisco123

crypto ikev2 profile ikev2_profile
 match identity remote address 192.168.100.1 255.255.255.255 
 authentication remote pre-share
 authentication local pre-share
 keyring local ikev2_keyring
 dpd 10 2 periodic

crypto ipsec transform-set tfs esp-gcm 
 esn
 mode tunnel

crypto ipsec profile ipsec_profile
 set transform-set tfs 
 set ikev2-profile ikev2_profile

interface Tunnel1
 ip address 192.168.200.2 255.255.255.252
 ip mtu 1380
 ip tcp adjust-mss 1320
 tunnel source Twe1/0/24
 tunnel mode ipsec ipv4
 tunnel destination 192.168.100.1
 tunnel protection ipsec profile ipsec_profile
 
router bgp 1
 bgp router-id interface Loopback0
 bgp log-neighbor-changes
 bgp update-delay 1
 bgp graceful-restart
 no bgp default ipv4-unicast
 neighbor 192.168.50.1 remote-as 1
 neighbor 192.168.50.1 update-source Loopback0

 address-family ipv4
  redistribute connected
  neighbor 192.168.50.1 activate
  neighbor 192.168.50.1 prefix-list PL-CONNECTED out
 exit-address-family

 address-family l2vpn evpn
  neighbor 192.168.50.1 activate
  neighbor 192.168.50.1 send-community both
 exit-address-family

ip prefix-list PL-CONNECTED seq 5 permit 192.168.50.2/32

[MHM Cisco World]

you interconnect two DC via GRE to run BGP for VXLAN

What I will suggest is so advanced so if yoh have Q for my suggestion you can ask anytime 

Now 

You have GRE tunnel run over WAN 

Try separate control traffic than data traffic

This done by 

Run bgp normally over the gre tunnel and make bgp advertise l2evpn types route this control traffic 

Config bgp to change the next-hop for any route it receives to be WAN IP this for data traffic 

This way the gre use only for control traffic and it sure less than 1500

The daya traffic will use WAN directly and WAN can sure fragment the traffic.

Try above hope it solve your issue

MHM

[dijix1990]

For what? I need to encrypt all traffic and I remember that cisco suggested this for connection to branch over WAN, but there's big limitation, c9300x with gre with or without ipsec can't fragment traffic as router

There is special article about evpn vxlan + ipsec for wan

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9300/software/release/17-9/configuration_guide/vxlan/b_179_bgp_evpn_vxlan_9300_cg/configure-bgp-evpn-vxlan-over-ipsec.html

[MHM Cisco World]

The bgp IP is different than nve IP' 

Nve is vxlan IP use to interconnect both DC and forwarding data traffic

The secure IPsec vti use only for control traffic bgp.

The link you share is exactly my suggestion.

Show bgp l2vpn evpn advertise 

Ypu see the next-hop appear is NVI which is use clear text not encrypt via lo1 use under nve1.

Make review link carefully if you have Q ask me.

MHM

[dijix1990]

What do you mean? Can you see my config and cisco example? It's almost the same,

1. Interface which connect to provider twe1/0/24

2. Ipsec (tu1) builds session between two 9300x and do secure underlay

3. Ospf (instead of static routes, becouse I have two providers) builds neighbourhood for bgp (loopback 0 routes via ospf) 

4. Bgp uses loopback 0 for establishing session for evpn vxlan enxap

Every packet from vlan 100 is encrypted. I seen packet capture between 9300x and there was ESP only

[MHM Cisco World]

In link 

Nve interface use lo1 and bgp use lo0

So the IP is different' this separate underlying than overlying.

Nve lo1 advertise over ospf.

Only make review link and see how cisco separate lo for each layer

MHM

[dijix1990]

But so on, ospf and bgp work through ipsec and my config works perfectly on the c8500L because packet fragmentation works for ipsec

[MHM Cisco World]

Gre or ipsec in SW ios xe is done in HA not in Software there is so limitations' I explain what Cisco meaning in doc. You share.

MHM

[dijix1990]

You said that CP goes via ipsec, DP goes via WAN directly, but it's wrong they are both going via ipsec. without ipsec it can do fragmentation, but it's clear text and I don't think it's good idea use it via internet

[MHM Cisco World]

Correct for secuirty it must encrypt by ipsec but you use SW not edge router to interconnect two DC sites 

So there is limitations

And link you share is use SP not ISP' i.e. and that why the LO IP is private not public' if it internet then it must be public IP.

MHM