esw_mrvl_soutput: Packet size too big <1663>

[corycandia]

Experts,

Found a syslog message that not even Google has heard of, and nothing comes up on the forum, so maybe I'm the only human being to have this error?  Can't even find it documented anywhere.

"417355: Sep 19 00:21:16.441: esw_mrvl_soutput: Packet size too big <1663>"

All I know is it has to do with the switch module.  Anyone have any ideas?

Hardware:

Cisco IOS Software, C2900 Software (C2900-UNIVERSALK9-M), Version 15.7(3)M7, RELEASE SOFTWARE (fc1)

NAME: "CISCO2921/K9", DESCR: "CISCO2921/K9 chassis, Hw Serial#: FJC1943A1QE, Hw Revision: 1.0"
PID: CISCO2921/K9 , VID: V08 , SN: FJC1943A1QE

NAME: "9 Port FE Switch on Slot 0 SubSlot 1", DESCR: "9 Port FE Switch"
PID: HWIC-D-9ESW , VID: V01 , SN: FOC124455CZ

NAME: "WIC/VIC/HWIC 1 Power Daughter Card", DESCR: "9-Port HWIC-ESW Power Daughter Card"
PID: ILPM-8 , VID: V01 , SN: FOC12443U41

NAME: "PVDM3 DSP DIMM with 32 Channels on Slot 0 SubSlot 4", DESCR: "PVDM3 DSP DIMM with 32 Channels"
PID: PVDM3-32 , VID: V01 , SN: FOC19385G27

NAME: "C2921/C2951 AC Power Supply", DESCR: "C2921/C2951 AC Power Supply"
PID: PWR-2921-51-AC , VID: V03 , SN: QCS19230ST0

[Peter Paluch]

Hello Cory,

It seems that one of the internal drivers to send out packets in process-switched path got to handle a packet whose size in bytes (1663) exceeds the egress interface's MTU even after allowing some extra overhead bytes. The driver had to log that message and drop the packet.

Since this is a process-switched path, it is likely that it is a packet that was generated by the router itself. It is quite unlikely that it is a packet that was received and only being routed.

Please let me ask you:

1. How often do these logging messages appear?
2. Does the packet size indicated in those messages stay the same, or does it change?
3. Did you explicitly modify an MTU of any interface on this router?
4. In the current running-config, is there any setting of any subsystem on this router (such as IP SLA probes or NetFlow exports) that would be enforcing unusual and excessive packet sizes for its own operation?
5. Do you see the operation of the router in any way impacted, or do these logging messages appear to be harmless?

Thank you!

Best regards,
Peter

[corycandia]

Thanks for your reply.

1. How often do these logging messages appear?
   1. (A) I have 9 hits in the last 24hr period, but on the 18th of Sep, we had over 13,600.
2. Does the packet size indicated in those messages stay the same, or does it change?
   1. (A) It changes, between 1536 - 1674
3. Did you explicitly modify an MTU of any interface on this router?
   1. (A) yes, it has multiple VPN tunnels, some to Microsoft Azure, and some DMVPN.  This router is the hub.
4. In the current running-config, is there any setting of any subsystem on this router (such as IP SLA probes or NetFlow exports) that would be enforcing unusual and excessive packet sizes for its own operation?
   1. (A) Yes, there is a traffic-export being used for network IDS.  The export is on the internal interface heading to the local data center.
5. Do you see the operation of the router in any way impacted, or do these logging messages appear to be harmless?
   1. (A) There was recently a memory issue where buffering was misconfigured and not capped, so we determined the buffer log was growing and consuming all the memory.  Since the logs are sent to a syslog server for analysis (which is how we found this issue), we think we don't need a buffer.

I am attaching the running config.

[Peter Paluch]

Hello Cory,

Thank you for the responses - and thank you for sharing your running-config. It gave me a better picture.

I do not see any serious outstanding issues with your configuration. Most certainly, I don't see any excessive MTU configured anywhere. However, I do see a few tunnels that don't have their MTU configured, and they are the primary suspects that with all the overhead, they may generate overly large packets.

This is the summary of my observations:

1. Interfaces Tunnel1 and Tunnel11 have no ip mtu configured. Based on the existing MSS adjustment there, it would make sense to configure them with ip mtu 1430 and also ensure that the other tunnel endpoints use the same MTU. These two tunnels are my primary suspects for causing the logging messages.
2. Interface Tunnel12 has no MSS adjustment configured. Based on the MTU adjustment there, it would make sense to configure it with ip tcp adjust-mss 1310.
3. Interface Tunnel50 has no MSS adjustment configured. Based on the MTU adjusment there, it would make sense to configure it with ip tcp adjust-mss 1210.
4. Interfaces Virtual-Template1 and Virtual-Template5 have no MTU and MSS adjustments configured. Since they are both related to VPNs, I would recommend using ip mtu 1400 and ip tcp adjust-mss 1360 on both of them.
5. The ip default-gateway command is useless if the router is operating in routed mode (which it is). It is safe to remove it.
6. The ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/2 210 command is dangerous the way it is configured - because it does not specify the IP address of the next hop, only the egress interface, it relies on the Proxy ARP feature on the next device and results in excessively large ARP table. I know that the Gi0/2 is using DHCP; therefore, this static route should be removed and replaced with ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/2 dhcp 210.
7. I understand that you had an issue with excess logging buffer size but deactivating the buffered logging entirely is risky since it is not always guaranteed that the syslog server will always be reachable, and you may lose important information about system events. I'm sure that your 2921 router has hundreds of MB of RAM; dedicating 10 KiB to the logging buffer is really an insignificant fraction. I would therefore recommend configuring logging buffered 10240 debugging.

Do you think you could implement these changes - especially the ones related to the MTU and TCP MSS adjustments? I would be keen to see if they bring any improvement.

Best regards,
Peter

[corycandia]

Peter,

Thank you for talking the time to recommend a solution, and looking over the rest of the configuration.  All your suggestions are appreciated, and I will be starting to incorporate them soon.  I'll try to report back to this topic after implementing the changes.

Thanks again.