Packet-trace on 9500 Switch

[rasmus.elmholt]

I have a 9500 series switch in VSL stack and trying to do a packet-trace on it to see how a packet is handled in the FED.

I have made an Embedded Packet Capture on a port on the switch and have captured some packets in a PCAP file.

When I want to "replay" the packet through the FED I get an error and I cannot figure out why.

SW10#show monitor capture file flash:DNS.pcapng packet-number 22
Starting the packet display ........ Press Ctrl + Shift + 6 to exit

 22   0.138655  10.49.16.65 -> 10.1.5.5     DNS 83 Standard query 0xb5a1 NULL _ta-4f66 OPT

SW10#show platform hardware fed switch 2 forward interface tw2/0/38 pcap flash:DNS.pcapng number 22
023145: Sep  4 13:18:29.152: %SHFWD-6-PACKET_TRACE_FAIL: Chassis 2 R0/0: fed: Show fwd is failed at Unable to trace the packet in hardware.
SW10#show version
Cisco IOS XE Software, Version 16.12.04

SW10#show platform software trace message fed switch active 
This command is being deprecated. Please use 'show logging process' command.
executing cmd on chassis 1 ...
Collecting files on current[1] chassis.
# of files collected = 2

2024/09/04 11:06:02.650 {fed_F0-0}{1}: [ipc] [20953]: (ERR): fed-fed conn service get failed error:22 Invalid argument
2024/09/04 11:06:02.650 {fed_F0-0}{1}: [oir] [20953]: (note): chassis_type:51
2024/09/04 11:06:02.650 {fed_F0-0}{1}: [oir] [20953]: (note): chassis_type:51
2024/09/04 11:05:57.649 {fed_F0-0}{1}: [ipc] [20953]: (ERR): fed-fed conn service get failed error:22 Invalid argument
2024/09/04 11:05:57.649 {fed_F0-0}{1}: [oir] [20953]: (note): chassis_type:51
2024/09/04 11:05:57.649 {fed_F0-0}{1}: [oir] [20953]: (note): chassis_type:51
2024/09/04 11:05:52.648 {fed_F0-0}{1}: [ipc] [20953]: (ERR): fed-fed conn service get failed error:22 Invalid argument
2024/09/04 11:05:52.648 {fed_F0-0}{1}: [oir] [20953]: (note): chassis_type:51
2024/09/04 11:05:52.648 {fed_F0-0}{1}: [oir] [20953]: (note): chassis_type:51

Interface configuration:

interface TwentyFiveGigE1/0/38
 description Firewal-on-a-stick - interface X1                     Link 1
 switchport mode trunk
 channel-group 38 mode active
end

I have done this is the past on other switches without any issues. But on this one I get the errors above without any useful information(useful to me).

Any help on how to troubleshoot this further is appreciated.

[marce1000]

             - FYI : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwe99115
   I would advise to compare the issue against the latest advisory software version for the particular 9500 model , 

 M.

[rasmus.elmholt]

HI @marce1000 

I don't think this bug is relevant to me as I am on another IOS version and the BOARD_ID does not match the bug.

SW10#show romvar
Switch 1
ROMMON variables:
 BOARDID="2E"

[marce1000]

    - Good to know , but always compare against latest advisory release (if desired) , 

 M.

[gtrejoor]

Hi rasmus.elmholt 

I see, you tried to understand how the switch processes the packets in FED (Forwarding Engine Driver), the error is because in your capture you are using the interface Two 2/0/38 into the FED CLI command but the interface where you expected to reply is on Sw1 Twenty 1/0/38.

Unfortunately, the FED process is complex and you need to understand other things such as incoming/outgoing  Index interfaces, etc. 
I would suggest you take a look at the following document to understand better, Doc says C9300 however the FED command is the same on the C9500 platform

https://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/ios-xe-gibraltar-16121/216746-configure-punt-inject-fed-packet-capture.html 

Regards, 

[rasmus.elmholt]

@gtrejoor Maybe I didn't mention it but it is an etherchannel on both ports and I capture on both and have tried to replay the packet on both, with the samme error. It does not matter if I use Twe1/0/38 or Twe2/0/38, the error is the same.

[gtrejoor]

Hi Rasmus

How are you did capture it? Can you confirm that the packet is received on one of the interfaces?

The FED capture should be handle using the RX packet to know the forward decision for the egress decision, also consider those error logs are not related to the FED packet process.

Regards,

[rasmus.elmholt]

Hi @gtrejoor 

The packet 22 as shown is going out the interface and hitting the firewall on that interface, but is routed back as packet 23.

I have tried replaying packet 23 as well, as this packet is rx on the interface. But I get the same error.

I will try it again on Tuesday, and confirm that it is in fact an ingress packet I am trying to replay.

[rasmus.elmholt]

Hi @gtrejoor 

I have now tired the ingress packet from the pcap, but it is still not working.

SW10#show monitor capture file flash:AAE-DNS.pcapng packet-number 23
Starting the packet display ........ Press Ctrl + Shift + 6 to exit

 23   0.138711  10.49.16.65 -> 10.1.5.5     DNS 83 Standard query 0xb5a1 NULL _ta-4f66 OPT

SW10#show monitor capture file flash:AAE-DNS.pcapng packet-number 23 detailed
Starting the packet display ........ Press Ctrl + Shift + 6 to exit

Frame 23: 83 bytes on wire (664 bits), 83 bytes captured (664 bits) on interface 0
    Interface id: 0 (/tmp/epc_ws/wif_to_ts_pipe)
    Encapsulation type: Ethernet (1)
    Arrival Time: Sep  4, 2024 09:45:01.530025000 CEST
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1725435901.530025000 seconds
    [Time delta from previous captured frame: 0.000056000 seconds]
    [Time delta from previous displayed frame: 0.000000000 seconds]
    [Time since reference or first frame: 0.138711000 seconds]
    Frame Number: 23
    Frame Length: 83 bytes (664 bits)
    Capture Length: 83 bytes (664 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:vlan:ethertype:ip:udp:dns]
Ethernet II, Src: 00:09:0f:09:00:1a (00:09:0f:09:00:1a), Dst: 00:a3:8e:b7:4b:c3 (00:a3:8e:b7:4b:c3)
    Destination: 00:a3:8e:b7:4b:c3 (00:a3:8e:b7:4b:c3)
        Address: 00:a3:8e:b7:4b:c3 (00:a3:8e:b7:4b:c3)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Source: 00:09:0f:09:00:1a (00:09:0f:09:00:1a)
        Address: 00:09:0f:09:00:1a (00:09:0f:09:00:1a)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: 802.1Q Virtual LAN (0x8100)
802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 1223
    000. .... .... .... = Priority: Best Effort (default) (0)
    ...0 .... .... .... = CFI: Canonical (0)
    .... 0100 1100 0111 = ID: 1223
    Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 10.49.16.65, Dst: 10.1.5.5
SW10#$e fed switch 1 forward interface tw1/0/38 pcap flash:AAE-DNS.pcapng number 23
Show forward is running in the background. After completion, syslog will be generated.

SW10#
023246: Sep 10 09:55:22.804: %SHFWD-6-PACKET_TRACE_FAIL: Chassis 1 R0/0: fed: Show fwd is failed at Unable to trace the packet in hardware.

[MHM Cisco World]

the packet trace if it pass through CPU if pass through TCAM then I dont think you can do that 
to push it to CPU try clear arp disable IP CEF in ingress interface

MHM

[rasmus.elmholt]

Hi @MHM Cisco World 

This is a L2 switch I don't have any arp to clear.

I think the point of the capture is that the traffic can be captured directly on the port and not on the CPU.

According to BRKTRS-2811 EPC should be able to do Data-Plane Captures on Cat9k

https://www.ciscolive.com/c/dam/r/ciscolive/apjc/docs/2023/pdf/BRKTRS-2811.pdf

[gtrejoor]

Hi rasmus.elmholt 

You should open a case with TAC because that message may or may not be expected depending on how the ASIC forwards the packet. Also, Do you have a problem with the box or just want to learn how to use the tool?

Regards, 

[rasmus.elmholt]

Hi @gtrejoor 

I think step one would be to update the switch to a newer firmware.

I have issues with a firewall on the ports that does not forward the packets, and I am just using the EPC to troubleshoot the firewall.

[MHM Cisco World]

forget the EPC what is issue with FW, maybe I can help you to solve issue without EPC 
and also did you  try use EPC with VLAN instead of specify the interface ?

MHM

[rasmus.elmholt]

Hi @MHM Cisco World 

I am not sure what you mean by trying EPC with VLAN. Could you give an example on how you would configure this?