Hello,
I am running Firepower Virtual appliances in AWS. They are behind a GWLB and all part of a target group. The appliances were running 7.2.8 and we updated to 7.4.2. We removed an appliance from the target group, updated the software, and then put it back in the Target group and it would show up healthy. After the updates, most traffic flowing through these appliances was failing. Packet captures (on endpoints having issues) revealed full successful TCP handshakes but payloads being dropped. This led me to think it could be an MTU issue.
When originally enabling VTEP / GENEVE on these appliances, it automatically updated the data interface MTU to 1806 that is connected to the GLWB. The VNI then in turn has an MTU of 1500. This makes sense per the below info from a Cisco doc:
"For AWS with GWLB, the data interface uses Geneve encapsulation. In this case, the entire Ethernet datagram is being encapsulated, so the new packet is larger and requires a larger MTU. You should set the source interface MTU to be the network MTU + 306 bytes. So for the standard 1500 MTU network path, the source interface MTU should be 1806."
After the update during troubleshooting, we saw the MTU on the VNI interface was 1480. You can imagine this would cause huge issues. The MTU on the data interface was still 1806. We had to update the MTU on the data interface to 1826 to fix the issue and increase the MTU on the VNI interface to 1500.
Has anyone seen anything like this before? This caused a huge outage.
