Cisco 9300: Problems with devices dissappearing from the network

[ds_aditec]

We are manufacturer of controllers for automation and we have a touchpanel with Windows Embedded Compact 7 running on it.

We have 6 devices installed at a customer (same hardware, same os-image, same software) and they are running without any problem for about two years. Since about january 21 this year 4 of this 6 devices are having a problem with availability on the network. If you switch them on you can reach (ping) them for about 5 minutes. After this time you can't reach them in the network any more. Our devices have two ethernet cards. No matter which you use, it's the same behaviour. We exchanged two devices for new ones, the same problem. One of them is here on my desk for testing, no problem, I can reach it over weeks without any interruption.Our customer is using a Cisco 9300 Series switch with software version 16.06.04 and says he did not change anything in his network. The use of duplicated IP-addresses can be excluded. For me it looks like there is a problem with renewing the mac address (~5 minutes -> 300 seconds mac address timeout). We logged communication on that port and after about 5 Minutes ping stopped. Attached you will find a screenshot of this. Does anyone has an idea what the problem is?

Thanks for your replys

Dirk.

[marce1000]

 - Some hints , 1) have a look at the logs of the 9300 when this happens 2) Look at the interfaces with show int x/y and have a look at the error related counters, check if they are increasing, if observed 3) Configure the involved interfaces with logging event link-status, to get more elaborated info's in the logs later on.

 M.

[ds_aditec]

Thank you for the hints,

I will try to get it done at the customer. But it will take a few days....

... Ok sometimes things go faster....

When this problem happens, no error counters increase. The logging event link-status is configured and no link is lost. The port has a link the whole time wheter you can reach the device or not, but in this errorstate there's no mac-address visible behind the port.

[Leo Laohoo]

Let me guess:  If you reboot the switch everything works.

[ds_aditec]

No, I never reboot the switch, after rebooting the device everything works for 5 minutes.

[Leo Laohoo]

It is not a downstream issue, it is the switch.

[ds_aditec]

This is what I think, too. I guess that it is something with renewing the mac address because of the lost of connection after 300 seconds. I suggested to change the mac-address aging-time to 400 s to check if the device is now reachable for a longer time, but the customers network admin will not change this because of downtime and other issues he apprehends. I don't know how I can prove this.

[Leo Laohoo]

---

@ds_aditec wrote:

I don't know how I can prove this.

---

Move the affected downstream clients to a different switch with lower uptime.

[ds_aditec]

We will try another Switch, but it's difficult because this device is in production environment and we have to wait until production stops for a while.

I made another suggestions to them: I woud add the mac-address of the device as static mac-address to the switch, to see if there's a problem with mac-aging and refresh. Hope the admin will do this.

[ds_aditec]

Our cutomer connected the device to a Nexus Series switch and got no link to it. After reconnecting it to 9300 switch he got the link and the problem of no connection after 5 minutes, again. I think this was not very helpful, don't know what they did and why he got no link to the nexus switch.

[ds_aditec]

Ok, yesterday our customer connected the device again to a Nexus switch after they found out what the was las time. On the Nexus we don't have this problem, everything is working fine. On the 9300 the problems appears again. I suggested to restart the switch. Now I'm waiting for response if they will do it or not.

[ds_aditec]

Last weekend they restarted the switch, the problem stays the same.

[marce1000]

 - Have them use a syslog-server. for continuous log monitoring. Review logs during normal periods too because sometimes it can lead to detection of a deeper problem or issue. Also configure user-interfaces with logging event link-status.

 M.

[Leo Laohoo]

Post the complete output to the following command: 

sh interface <PORT>

[ds_aditec · Accepted solution]

I found out how to disable IPv6 in our devices and tried it and now it works. Now I have to find out what's the problem there.

Thank you for your help.