N9K doesn't forward some traffic

[from88]

Hello,

We're using :

Cisco Nexus9000 C9372PX chassis 
Intel(R) Core(TM) i3- CPU @ 2.50GHz with 16402012 kB of memory.
Processor Board ID SAL2002WLL2 System version: 7.0(3)I2(1a)

Recently we created new vrf and even the routing is good some packets seems doesn't pass through N9K box.

When on incoming layer 3 VLAN interface i add IP access-list in with logging and statistics per entry. it LOGS the traffic but it didn't show the matches.

IP access list test-pvpn
statistics per-entry 
10 permit ip 10.3.47.103/32 10.2.99.0/24 log [match=0] 
20 permit ip any any [match=61392]

but the logging shows the packets:

JAYCORE01# sho logging ip access-list cache 
Source IP Destination IP S-Port D-Port Interface Protocol Hits
----------------------------------------------------------------------------------------
10.3.47.103 10.2.99.132 0 0 port-channel139 (1)ICMP 30

Number of cache entries: 1
----------------------------------------------------------------------------------------
JAYCORE01#

We have another box which is acting as redundant one -so other one in same scenario shows matches even in show ip access-list command.

So after using comparing-method I could guess that this is a sign that this is somehow related to the underlying problem. Seems like this is a HW Forwarding problem. Could you provide any help ?

Thanks.

[balaji.bandi]

At this stage it is hard to say what is the problem, it would be nice if you can post the configuration and your ping logs from where to where you pinging.

how is these devices connected, network topo to understand the setup.

[from88]

The configuration is quite simple:

packets from: 10.3.47.103/32 to 10.2.99.0/24 come to one of these layer 3 interfaces

interface Vlan480
description JAYCORE01-NTGCORE02_vrf_pvpn
no shutdown
ip access-group test-pvpn in
vrf member pvpn
ip address 10.251.7.20/31
ip ospf message-digest-key 1 md5 7 05585209254A681B1F171137392B32250
ip ospf cost 10
no ip ospf passive-interface
ip router ospf 3 area 0.0.0.0

JAYCORE01# show running-config interface vlan 479

interface Vlan479
description JAYCORE01-NTGCORE01_vrf_pvpn
no shutdown
ip access-group test-pvpn in
vrf member pvpn
ip address 10.251.7.18/31
ip ospf message-digest-key 1 md5 7 05585209254A681B1F171137392B32250
ip ospf cost 10
no ip ospf passive-interface
ip router ospf 3 area 0.0.0.0

And they should be catched by this ACL:

JAYCORE01# show access-lists test-pvpn
IP access list test-pvpn
statistics per-entry 
10 permit ip 10.3.47.103/32 10.2.99.0/24 log [match=0] 
20 permit ip any any [match=80918] 

But as we see: it not matches the traffic by these: 10.3.47.103/32 10.2.99.0/24.

But "logging" feature shows the hits it:

JAYCORE01# show logging ip access-list cache 
Source IP Destination IP S-Port D-Port Interface Protocol Hits
----------------------------------------------------------------------------------------
10.3.47.103 10.2.99.132 0 0 port-channel139 (1)ICMP 57

Number of cache entries: 1
----------------------------------------------------------------------------------------

So seems there're some HW problems in this Switch as im understand both commands should show the same result (MAtches and hits)

[Georg Pauwen]

Hello,

the IP addresses specified in your access list do not seem to be directly connected to any of the VLAN interfaces. Try to apply the access list outbound and check if you see any matches then:

ip access-group test-pvpn out

[from88]

Hello,

Yes, the IP addresses in ACL are not directly connected ones. When the traffic with mentioned source <> dst addresses reaches that N9k through that Layer 3 Vlan interfaces (ECMP) it's somehow not forwarded further. The different matches/hit counts displays that there's some kind of problem.

[Georg Pauwen]

What if you apply the access list outbound as suggested ?

[from88]

i added it on OUT direction on outbound vlan interface - and still no matches.

[from88]

just FYI.

Reboot of the BOX helped.

statistics per-entry

10 permit ip 10.3.47.103/32 10.2.99.0/24 log [match=168]
20 permit ip any any [match=3514]