ARP issue across same VLAN but different stacks.

[stonent01]

This is an odd issue.

Computer A is connected to a port of a 3 switch stack of 48 Port 3650 switches. We will call that Stack A.

Computer B is connected to a port of a 4 switch stack of 48 Port 3650 switches. We will call that Stack B.

Both computers are on the same VLAN.

Computer B cannot ping Computer A and no ARP entry exists on that PC for Computer A's IP.

If I ping Computer B from Computer A, Computer A can now ping Computer B. (ARP cache gets updated)

If I reboot the Computer B, it will no longer be able to ping Computer A.

About 80% of the computers on Stack A on that same VLAN have that same issue.

I've determined this is not a Windows issue, because I can boot both computers from a Linux Live USB stick and recreate the issue. Also, if I replace Computer A with another PC, the issue now occurs on the replacement PC.

If I move any of those computers to another switch, but the same VLAN, the problem goes away.  If I move them back again, the problem returns.

If I move a computer on Stack A to a different VLAN than Computer B, the problem goes away.

When I did wireshark captures, I noticed that ARP requests coming from Computer B for information on Computer A were not getting responses.

So I put a static ARP entry on Computer B with the information from Computer A.

netsh interface ipv4 add neighbors "Local Area Network" 10.69.137.199 18-66-DA-19-34-07

Now pinging works fine.

I stop the pinging and delete the static arp entry from Computer B.

arp -d 10.69.137.199

Pinging now stops working again.

Computer B is also not alone, we have several other "Computer B's"

Computer B represents the majority of the IT department that is on that VLAN.

According to our local network engineer, all the switches get the same base configuration, all trunk back to the same Nexus core in the server room.

The base configuration gets the stack a static IP enables SSH, disables Telnet, and sets the trunk.

Then each port is configured for what VLAN it needs to be on.

So what would make ARP requests from Computer B on Stack B for information on Computer A on Stack A on the same VLAN get ignored?

Both stacks have been rebooted and updated to the latest Cisco IOS, no change.

[Deepak Kumar]

Hi,

I can say that communication between both Switches (Stacks) not happening. I recommend verifying the Trunk port configuration on both stacks. Have you allowed this VLAN on trunk ports? Is there any VLAN mismatch issue?

Share below command output:

1. Show CDP neighbor

2. Show interface trunk

Regards,

Deepak Kumar 

[stonent01]

Communication is happening because other subnets work and some of the ones on that particular VLAN are fine as well.

Both switch stacks have been there for a few years, it just seems within the last year this issue started happening.

The trunk ports on both sides were checked and correctly set to allow this vlan (and all the others we use).

Switches are specifically WS-C3650-48PD.

They are running: Cisco IOS Software [Everest], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.6.5, RELEASE SOFTWARE (fc3)

[paul driver]

Hello

The fact that this only occurs on a specific switch within the stack warrants a check of that switch 

Session xx(switch number)

sh logging 

sh switch detail

sh switch stack-ring 

sh vlan 

[Deepak Kumar]

Hi,

Thanks for the confirmation that you had verified VLAN and trunk ports. Is this issue occurring on a stack or cross the stock?

1. Have you checked all Stack cables on switches?  "sho switch detail",  "sho switch stack-ring"
2. Have you verified that Switch is learning system MAC-address on the interface where you connected the PC? 
3. "Show mac address-table address <mac-address-of-PC>"
4. "Show mac address-table interface GIgX/x/X"
5. Did you check logs of switches? Any log for this mac-address/IP address?

Regards,

Deepak Kumar