We have tried to replace an ISR G2 by ISR4331. We have found out that ISR4331 breaks IPv6 PMTU. The result is that affected TCPv6 connections stop working and are reset. All sorts of errors and problems occur as soon as the protocol tries to use larger packets. For example robocopy often reports ERROR 64 (0x00000040) The specified network name is no longer available. https connections are not established at all.
There are two problems with the MTU implementation in the ISR4331:
1.
There are three zones configured: INSIDE, OUTSIDE and self.
Between INSIDE and OUTSIDE there are policy maps that include the following class maps in both directions:
class-map type inspect match-all CMAPFW-tcp445-l7
match protocol microsoft-ds
class-map type inspect match-all CMAPFW-tcp-l7
match protocol tcp
class-map type inspect match-all CMAPFW-ping-l7
match protocol ping
On the OUTSIDE there is an IPsec in tunnel mode configured:
crypto ipsec transform-set phase2_ts esp-aes 256 esp-sha-hmac
mode tunnel
Now a host from the other side of the IPsec tunnel (OUTSIDE) executes a ping to the IPv6 host on the INSIDE interface of the ISR4331:
ping -6 -l 2006 ipv6hostaddress
The ASA or an 1921 that are on the other side of the IPsec report back to the sending host using ICMPv6 that the packet is too large and that the maximum MTU (over IPsec) is 1406. The sender fragments the ping request packet which then passes without any problem to the destination (SrcHost -> ASA/1921 -> IPsecTunnel -> ISR4331 -> DstHost).
The destination host replies with a ping reply ICMPv6 packet larger than 1500 bytes. ISR4331 notices that the packet is larger than the allowed MTU and tries to send out ICMPv6 packet informing that the packet was too large. However that ICMPv6 packet never reaches the host that has sent too large packet.
On ISR4331 the following is logged:
Nov 1 17:19:40.635: %IOSXE-6-PLATFORM: SIP1: cpp_cp: QFP:0.0 Thread:001 TS:00000669231556612298 %FW-6-DROP_PKT: Dropping icmp pkt from internal0/0/recycle:0 xxxx::xxxx:2 => xxxx::xxxx:0(target:class)-(OUT-to-IN:CMAPFW-ping-l7) due to INT ERR:Undefined direction with ip ident 0
When the protocol that sends a large packet is TCPv6, the log is a bit different:
Nov 1 15:05:49.660: %IOSXE-6-PLATFORM: SIP1: cpp_cp: QFP:0.0 Thread:001 TS:00000661200444739265 %FW-6-DROP_PKT: Dropping icmp pkt from internal0/0/recycle:0 xxxx::xxxx:2 => xxxx::xxxx:0(target:class)-(OUT-to-IN:CMAPFW-ping-l7) due to INT ERR:Undefined direction with ip ident 0
So the large packet never gets to its destination as the sending host does not receive a message that the MTU available is smaller than 1500. The TCP connection is then reset either by one of the hosts or by the ISR4331.
2.
Same setup as in the first point, just we do the ping in the other direction. A host on the INSIDE interface of the ISR4331 executes a ping to the IPv6 host on other side of the IPsec tunnel (OUTSIDE):
ping -6 -l 2006 ipv6hostaddress
Now ISR4331 replies with the ICMPv6 packet too big. However it indicates the MTU of 1422, which ist not correct.
For IPv6 the packet size after IPsec ESP/AES256/SHA-1 tunnel mode encapsulation calculates like:
40 (IP header) + 4 (SPI) + 4 (Seq) + 16 (IV) + 1422 (original packet) + 16 (padding for AES256) + 1 (PadLen) + 1 (NxtHdr) + 12 (SHA1) = 1516
So when the host sends a packet with the size 1422 which has just been indicated by ISR4331 as the new MTU, the resulting packet with IPsec overhead that would have to be sent out from ISR4331 would be 1516 bytes large. ISR notes that and replies back with the ICMPv6 packet too big. However it again indicates the MTU of 1422, which ist not correct.
The host keeps sending out packets of size 1422 and the packet never gets through. The connection fails in a similar way as when initiated in the other direction, however because of a bit different reason.
The largest packet size which after adding the IPsec overhead does not result in a size larger than 1500 is 1406:
40 (IP header) + 4 (SPI) + 4 (Seq) + 16 (IV) + 1406 (original packet) + 0 (padding for AES256) + 1 (PadLen) + 1 (NxtHdr) + 12 (SHA1) = 1484
That is the size that is indicated by ASA/1921 in ICMPv6 packet too big and also in sh crypto ipsec sa:
plaintext mtu 1406, path mtu 1500, ipv6 mtu 1500, ipv6 mtu idb Dialer1
However the ISR4331 indicates 1422 instead of 1406 as MTU in ICMPv6 packet too big and also in sh crypto ipsec sa:
IPv6: plaintext mtu 1422, path mtu 1500, ipv6 mtu 1500, ipv6 mtu idb GigabitEthernet0/0/0
Can others see the same problems? Dropped "ICMPv6 too big" packets in the firewall and wrong value for MTU on IPv6 IPsec connections (sh crypto ipsec sa)?
