c9300 stack intermittently can't ping connected devices

[nick.chalker]

Hi,

We have a strange intermittent issue (not all devices) with connected devices (mainly PC's).

We have a 5 * c9300-48U stack running Version 16.09.03 with network-advantage enabled.

The issue:

A PC gets a DHCP address from the Windows DHCP server (The PC MAC address registered is correct)

The switch port is up and connected to the correct VLAN.

The MAC address on the stack for the PC associated to the switch port is correct.

The ARP entry on the switch for the IP address and MAC address is correct.

The MAC address on the PC is correct

The ARP entry on the PC for the switch matches the IP address and MAC address for the VLAN

The route on the PC is correct.

If the PC is rebooted the issue still persists.

The PC cannot connect to anything outside the broadcast domain.

The switch can't ping the IP address of the PC, but devices in the broadcast domain can ping the PC, the switch can ping other PC's.

To resolve the issue we have to shutdown the port and manually clear the associated ARP entry, when the entry has gone we then re-enable the port, check the MAC address and ARP table (the PC gets the same IP address), the switch can now ping the PC.

I have been scratching my head on this and was wondering if it could be related to sticky arp or proxy arp.

Standard port configuration:

interface GigabitEthernet1/0/17
description QoS Softphone
switchport access vlan 5
switchport mode access
switchport port-security
auto qos trust dscp
spanning-tree portfast
spanning-tree bpduguard enable
service-policy input AutoQos-4.0-Trust-Dscp-Input-Policy
service-policy output AutoQos-4.0-Output-Policy

VLAN Configuration:

interface Vlan5
description LAN
ip address x.x.x.x 255.255.254.0
no ip redirects
no ip route-cache

Any assistance will be great

Thank you

Nick

[vb10]

Hi Nick,

From what you have described, I understood that you have verified almost all the usual things for such situation. If everything from "show " outputs looks correct, problem is intermittent, and is fixed by shut/no shut and clearing entries, usually it's indication of possible bug. 

I can suggest following:

1. I believe, we can be sure, that port is not "blocked" by any feature, configured on port (port-security, etc.), otherwise PC would not be able to ping other PCs in domain, right?

2. When PC is in "broken" state, you can narrow down a bit the issue and verify, in which exactly direction traffic is lost. Whether switch doesn't send ICMP requests out of the PC's port, or switch doesn't process incoming replies from PC. You can do this by setting up packet capture on PC, or on switch itself. Or clear/check packet counters on interface.

3. What could be possible trigger, how does the problem start again, after it's fixed? Do you see any strange logs on the switch?

4. Do you see anything strange in interface statistics? (drops, errors)

4. Do problematic PCs conncted to stack members only, or also to the master? If to members only, is the problem fixed, if you try to reconnect affected PC to master? Without clearing MAC/ARP.

[nick.chalker]

Hi, Thank you for the quick response and my apologies for delayed reply.

1: Correct the port is not being blocked, no err-disabled or anything to note

2: I will need to check the next time it happens, I believe the PC can't ping the switch or the switch the PC but I also remember that the PC can ping other devices and other devices can ping the PC in the same VLAN. I will see about a packet capture the next time it happens.

3:Not sure what triggers the issue but nothing obvious in the switch logs to indicate an issue.

4:From memory, no errors etc. but will need to check next time it happens.

5:I have seen issues on both but recently more from members, I think that re-patching may fix the issue.

Thank you again

Cheers

Nick

[balaji.bandi]

can post from PC "ipconfig /all" output here.

is the DHCP Server located in same VLAN 5 ?

[nick.chalker]

Hi, Thank you for the quick response and my apologies for delayed reply.

Yes the DHCP server is in the same VLAN, ip helper used for other VLAN's

I will need to get the full ipconfig next time it happens as I only have the short version as well as the arp cache and routes, all of which appear to be correct.

Thank You

Cheers

Nick