catalyst 4500x 03.11.* debug ip packet issue

Andrii Oliinyk · ‎12-13-2022

Hi Guys
yesterday i've met something i cannot explain so far. Topology was quite simple: couple of access-switches with mgmt IPs 
(TACACS, RADIUS, SNMP, SSH, etc) use core 4500x VSS as default gateway (mgmt IPs live in the subnets directly connected to core).
in turns, VSS runs OSPF with another switch . VSS receives 0.0.0.0 from that switch & announces all its local subnets to peer.
Access-Switch(VLAN X IP1)----(VLAN X IP2)VSS(VLAN Y IP3)----(VLAN Y IP4)WAN-SWITCH
with captures on either VSS<>WAN-switch or Access-Switch<>VSS interconnects we could see packets from infrastructure services
delivered to VSS-Switch or mgmt IPs of the Access-Switches correspondingly. In reverse direction (from mgmt IPs of Access-Switches)
we only were able to catch traffic from the Access-Switches mgmt IPs on the interconnects Access-Switch<>VSS. 
No mgmt traffic from access-switches were leaving VSS (!). then we turned "debug ip packets 199 detailed" (199 was ACL matching 
traffic of interest) on VSS. & SURPISE we didnt have any logs (!).
i've tried to google if there r any restrictions for "debug ip packet" on C4500X platforms w/o success.
Can anyone be of help here?
thanks in advance

balaji.bandi · ‎12-13-2022

As i understand it correctly here - from WAN Switch to Access switch there is no access ? (is this broke recently or never worked ?)

Access switch are just Layer2 as per the your explanation.

is the VLAN X range in routing domain ?

can you post show ip route and ospf config bit

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Andrii Oliinyk · ‎12-13-2022

Hi 
yes, mgmt IPs of the access-switches were receiving traffic from infrastructure servers located behind WAN-switch.
so, routing info about VSS's connected subnets is propagated & reaching infrastructure assets w/o any blames. for the 
mgmt IPs in turns there is only default route with next-hops represented by IPs on the VSS side each in its VLAN X (yup, 
there is no separate mgmt VLAN , but several - each access-switch has its own one - dont ask me about this design pls :0).
on the VSS there is default route received by OSPF from WAN-switch. so picture was like this
1) traffic destined from servers to mgmt IPs enters VSS from WAN-switch
2) VSS forwards it to mgmt IPs just via attached VLANs X
3) mgmt responds & responses enter VSS back via VLANs X & that's it. responses got lost
& this is what my Q about: with "debug ip packet" we expected to check what's going on with responses after they enter VSS
but no logs from debug at all. Why? ACL 199 was comprised of entries like this: permit ip host <mgmtIP1> host <infraserverN>
& with capture made on VSS on the downlinks we saw mgmt IPs responses matched ACL entering VSS.
unfortunately i cant provide any output here as i was caught on shift with this incident w/o any direct access to environment.
i was just using remote control over shared desktop in Teams. so none output was kept locally

MHM Cisco World · ‎12-13-2022

Bitmelody: Packet Capture - VSS

I dont fully know but, if Acc-SW connect to two SW (VSS) then there is chance that the hash make traffic forward via SW that you not run capture on it.
please check link above for more detail.

Andrii Oliinyk · ‎12-13-2022

thanks. not sure we had problem with capture though. There were a portchannels downward to access-switches from VSS & capturing on it worked just fine (we saw ingress traffic of interest) - moni cap TS interfa po2 bo
between VSS & WAN-switch we had single Te1/1/13 with switch 1 being Active in VSS. again we caught ingress traffic from servers but no egress from affected mgmt IPs....

but "debug ip packet" didnt show anything.

but u will laugh - just now i have similar (almost the same) situation with another 4500X VSS!
will update tread soon

MHM Cisco World · ‎12-13-2022

where you run capture ?
I think you must capture in standby and check the traffic

Andrii Oliinyk · ‎12-13-2022

within 1st case i ran capture on the Te1/1/13 which was L2 interface with transit VLAN Y between VSS & WAN-switch. chassis 1 was Active. also i ran capture on the PoXs which were LAGs to AccessSwitches.

Andrii Oliinyk · ‎12-13-2022

btw this is what i have trying to configure capture on standby:

VSS-standby-console# monitor capture I0707769 interface Port-channel10 both

EPC configuration not permitted on standby

VSS-standby-console#show ethercha 10 summ | i ^10_

10 Po10(SU) LACP Te1/2/7(P) Te2/2/7(P)

VSS-standby-console# monitor capture I0707769 interface Te2/2/7 bo

EPC configuration not permitted on standby

Andrii Oliinyk · ‎12-13-2022

ok. here is reproduction.
affected subnet 10.39.84.0/24 lives behind L3 distri switch in turn connected (with Po19) to VSS-core switch in turn connected to
FW (Po10). from VSS to distri - OSPF, all prefixes learned from distri & local external routes redistributed back (inc. static 
toward proxy 10.45.65.2 living behind FW). users of affected subnets cannot TCP/8080 to proxy. below is output of t/s with short
summary: TCP SYNs egress to VSS via Po19, but dont leave Po10. "debug ip packet 199 deta" shows nothing.

vss#show redun
Redundant System Information :
------------------------------
Available system uptime = 3 years, 39 weeks, 2 days, 11 hours, 17 minutes
Switchovers system experienced = 10
Standby failures = 1
Last switchover reason = user_forced

Hardware Mode = Duplex
Configured Redundancy Mode = Stateful Switchover
Operating Redundancy Mode = Stateful Switchover
Maintenance Mode = Disabled
Communications = Up
Current Processor Information :
------------------------------
Active Location = slot 1/1
Current Software state = ACTIVE
Uptime in current state = 1 year, 28 weeks, 1 day, 18 hours, 45 minutes
Image Version = Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.08.10.E RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2020 by Cisco Systems, Inc.
Compiled Tue 31-Mar-20 20:25 by prod
BOOT = bootflash:cat4500e-universalk9.SPA.03.08.10.E.152-4.E10.bin,12;bootflash:cat4500e-universalk9.SPA.03.08.09.E.152-4.E9.bin,12;
Configuration register = 0x2102
Peer Processor Information :
------------------------------
Standby Location = slot 2/1
Current Software state = STANDBY HOT
Uptime in current state = 1 year, 28 weeks, 1 day, 18 hours, 18 minutes
Image Version = Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.08.10.E RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2020 by Cisco Systems, Inc.
Compiled Tue 31-Mar-20 20:25 by p
BOOT = bootflash:cat4500e-universalk9.SPA.03.08.10.E.152-4.E10.bin,12;bootflash:cat4500e-universalk9.SPA.03.08.09.E.152-4.E9.bin,12;
Configuration register = 0x2102
VSS#sho access-li 199
Extended IP access list 199
10 permit tcp 10.39.84.0 0.0.0.255 host 10.45.65.2 eq 8080
20 permit tcp host 10.45.65.2 eq 8080 10.39.84.0 0.0.0.255
VSS#sho access-li TRAFFTS
Extended IP access list TRAFFTS
30 permit tcp 10.39.84.0 0.0.0.255 host 10.45.65.2 eq 8080
40 permit tcp host 10.45.65.2 eq 8080 10.39.84.0 0.0.0.255
VSS#debug ip packe 199 det
IP packet debugging is on (detailed) for access list 199
VSS#sho moni cap TRAFFTS
Status Information for Capture TRAFFTS
Target Type:
Interface: Port-channel10, Direction: both
Interface: Port-channel19, Direction: both
Status : Active
Filter Details:
Access-list: TRAFFTS
Buffer Details:
Buffer Type: LINEAR (default)
File Details:
Associated file name: bootflash:TRAFFTS.pcap
Size of buffer(in MB): 1
Limit Details:
Number of Packets to capture: 0 (no limit)
Packet Capture duration: 0 (no limit)
Packet Size to capture: 0 (no limit)
Packets per second: 0 (no limit)
Packet sampling rate: 0 (no sampling)
VSS#show ip route 10.39.84.1
Routing entry for 10.39.84.0/24
Known via "ospf 1", distance 110, metric 2, type intra area
Last update from 10.38.88.151 on Vlan1529, 7w0d ago
Routing Descriptor Blocks:
* 10.38.88.151, from 10.38.88.201, 7w0d ago, via Vlan1529
Route metric is 2, traffic share count is 1
VSS#sho run int Vlan1529
interface Vlan1529
description tn-distri
ip address 10.38.88.150 255.255.255.254
ip ospf mtu-ignore
end
VSS#sho ip arp | i 10.38.88.151
Internet 10.38.88.151 7 70db.987c.54bf ARPA Vlan1529
VSS#show mac ad vlan 98 | i 001c.7f00.4869
98 001c.7f00.4869 dynamic ip,ipx,assigned,other Port-channel10
VSS#show ip route 10.45.65.2
Routing entry for 10.45.65.2/30
Known via "static", distance 1, metric 0
Redistributing via ospf 1
Advertised by ospf 1 metric 1000 metric-type 1 subnets
Routing Descriptor Blocks:
* 10.42.4.69
Route metric is 0, traffic share count is 1
VSS#show run int Vlan98
interface Vlan98
description tn-firewall
ip address 10.42.4.66 255.255.255.240
standby version 2
standby 98 ip 10.42.4.65
standby 98 priority 200
standby 98 preempt delay reload 240
end
VSS#sho ip arp | i 10.42.4.69
Internet 10.42.4.69 0 001c.7f00.4869 ARPA Vlan98
VSS#show mac ad | i 70db.987c.54bf
1529 70db.987c.54bf dynamic ip,ipx,assigned,other Port-channel19
VSS#dir bootflash:*pcap
Directory of bootflash:/*pcap
31689 -rw- 22884 Dec 13 2022 15:45:52 +01:00 TRAFFTS.pcap
1659666432 bytes total (502149120 bytes free)
VSS#dir bootflash:*pcap
Directory of bootflash:/*pcap
31689 -rw- 23154 Dec 13 2022 15:46:00 +01:00 TRAFFTS.pcap
1659666432 bytes total (502149120 bytes free)
VSS#show loggi
Syslog logging: enabled (0 messages dropped, 94 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)
No Active Message Discriminator.
No Inactive Message Discriminator.
Console logging: disabled
Monitor logging: level debugging, 0 messages logged, xml disabled,
filtering disabled
Buffer logging: level debugging, 6 messages logged, xml disabled,
filtering disabled
Exception Logging: size (8192 bytes)
Count and timestamp logging messages: disabled
Persistent logging: disabled
No active filter modules.
Trap logging: level informational, 14366 message lines logged
Logging Source-Interface: VRF Name:
Log Buffer (4096000 bytes):
016047: Dec 13 15:22:44.856 CET: %SYS-5-LOG_CONFIG_CHANGE: Buffer logging: level debugging, xml disabled, filtering disabled, size (4096000)
016048: 007500: Dec 13 15:22:44.862 CET: %SYS-5-LOG_CONFIG_CHANGE: STANDBY:Buffer logging: level debugging, xml disabled, filtering disabled, size (4096000)
016049: Dec 13 15:22:46.866 CET: %SYS-5-CONFIG_I: Configured from console by ADMIN on vty0 (192.168.12.22)
016050: Dec 13 15:24:26.010 CET: %SYS-5-CONFIG_I: Configured from console by ADMIN on vty0 (192.168.12.22)
016051: Dec 13 15:41:31.525 CET: %BUFCAP-6-ENABLE: Capture Point TRAFFTS enabled.
016052: 007501: Dec 13 15:41:31.559 CET: %BUFCAP-6-ENABLE: STANDBY:Capture Point TRAFFTS enabled.
VSS#show debug
Generic IP:
IP packet debugging is on (detailed) for access list 199
VSS#moni cap TRAFFTS stop
VSS#sho moni cap file bootflash:IM10707769.pcap
1 0.000000 10.39.84.55 -> 10.45.65.2 TCP 61924 > webcache [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=8 SACK_PERM=1
2 0.537026 10.39.84.55 -> 10.45.65.2 TCP 61925 > webcache [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=8 SACK_PERM=1
3 0.722014 10.39.84.55 -> 10.45.65.2 TCP 61923 > webcache [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=8 SACK_PERM=1
4 0.992004 10.39.84.55 -> 10.45.65.2 TCP 61924 > webcache [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=8 SACK_PERM=1
5 1.546028 10.39.84.55 -> 10.45.65.2 TCP 61925 > webcache [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=8 SACK_PERM=1