3650 switch packets consumed

[figura.jan]

Hello all,

I am facing a strange issue on newly installed 3650 switch - this acts as a core switch and is connected to a 2811 router which is terminating multiple IPsec tunnels on one side (ip mtu set to 1400) and to 2 Dell 8024 10Gig switched on another side. After installation of this new switch we have problems with emails coming from ESA (sitting behind one of the tunnels on 2811) and mail gate (VM connected to the Dell switch).

symptoms: RDP & ping to the gate works fine from everywhere. However emails are not transferred from ESA to gate. After installing wireshark on the gate I found lot of TCP retransmits and TCP resets. Then ran a debug ip packet on the core switch and found the following messages in the log:

pak 6686CC78 consumed in input feature , packet consumed, MCI Check(63), rtype 0, forus FALSE, sendself FALSE - 38756

As far as I understand this message the 3650 "consumed" the packet for some reason but I was not able to find further details on this anywhere. There is no ACL / NAT configured on 3650, it should virtually just take the packet from one interface and send it to another one. I raised MTU on all interfaces to 1900, no luck, then tried 9000 but still nothing.

Searched the bug tool but did not find anything that would match this behavior, also upgraded the image to 03.07.03 but did not help.

Any help on this would be greatly appreciated.

[Austin Sabio]

Check ESA logs on the ESA and from the workstation. 

http://www.cisco.com/c/en/us/support/docs/security/email-security-appliance/118552-qa-esa-00.html

http://www.cisco.com/c/en/us/support/docs/security-vpn/secure-shell-ssh/117988-qa-logs-00.pdf

Please rate, if helpful. 

Thank you.

[figura.jan]

This is not ESA problem, communication worked to a different mail gate server. We saw queue of emails building up on ESA and that was it. As I described above, packets from ESA to gate were seen in wireshark on the gate server.

[Austin Sabio]

Please post show version. 

[figura.jan]

Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.07.03.E RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Wed 13-Jan-16 23:40 by prod_rel_team

...

ROM: IOS-XE ROMMON
BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 3.56, RELEASE SOFTWARE (P)

ATVIEDC01C uptime is 3 days, 13 hours, 11 minutes
Uptime for this control processor is 3 days, 13 hours, 14 minutes
System returned to ROM by reload at 14:49:18 UTC Sat Sep 24 2016
System restarted at 17:40:36 UTC Sat Sep 24 2016
System image file is "flash:cat3k_caa-universalk9.SPA.03.07.03.E.152-3.E3.bin"
Last reload reason: Reload command

...

cisco WS-C3650-24TD (MIPS) processor with 4194304K bytes of physical memory.
Processor board ID FDO2021E1JJ
4 Virtual Ethernet interfaces
26 Gigabit Ethernet interfaces
2 Ten Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
250456K bytes of Crash Files at crashinfo:.
1609272K bytes of Flash at flash:.
0K bytes of Dummy USB Flash at usbflash0:.
0K bytes of  at webui:.

Base Ethernet MAC Address          : 00:f6:63:20:96:80
Motherboard Assembly Number        : 73-15895-05
Motherboard Serial Number          : FDO20220D6Y
Model Revision Number              : K0
Motherboard Revision Number        : B0
Model Number                       : WS-C3650-24TD
System Serial Number               : FDO2021E1JJ


Switch Ports Model              SW Version        SW Image              Mode   
------ ----- -----              ----------        ----------            ----   
*    1 28    WS-C3650-24TD      03.07.03.E        cat3k_caa-universalk9 BUNDLE

[rasmus.elmholt]

I dont think this is a switch issue.

And debug ip packets wont show packets in the data plane, only punted packets to the CPU of the switch.

[figura.jan]

Packets were sent out the router but were not seen anywhere behind switch (which is right behind the router) so it tells me the problem is somewhere in the switch :) The debug was bound to an ACL (no "log" keyword):

permit ip host 10.251.0.1 host 10.1.0.75

interestingly packets were seen in the debugs so they must have been hitting CPU, the question is why? I can post full switch config if it helps...

[Austin Sabio]

Bug CSCtx12810 explains the unnecessary output portion. 

CSCtx12810:debug ip packet with ACL displays unnecessary output

Description

Symptom: "debug ip packet" shows extra information, showing all consumed packets though ACL does not permit these packets

Conditions: Running "debug ip packet "

Workaround: none

Please rate, if helpful. 

Thanks!

[figura.jan]

Hello,

actually the switch is running 03.07.03 which is not in the list of affected releases. In addition description does not match the behavior - debugs were showing only IP packets matched by the defined ACL