CBS350 I2C Failed to Read Errors and Port Issues

[djtech2k]

I unboxed 2 new CBS350 switches and installed them.  First thing I updated to the newest firmware, which is 3.3.0.16.  On one of the switches, early in the process of setting up the config, it started with repeated errors of the I2C failed to read errors with components ADT7476 Fan controller, thermal sensor, and component CPLD.  When that started happening, I also noticed that Cisco copper SFP's would not be recognized in the system status, the port lights on those ports would never light, and the admin portal would constantly freeze up, which resulted in it rebooting itself.  The 2nd switch has worked fine with no issues.  So after hours of trying to resolve the issues on Switch1, I pulled it and replaced it with a brand new CBS350.

On the new switch3, I did the same process.  I updated the firmware to 3.3.0.16 and started loading just some common things like VLAN's, interfaces, and trunk ports.  Everything worked fine on day one.  Today I went back to this same switch and joined ports to the proper VLAN's and started to create an ACL and some ACE's.  Somewhere during this process this switch starting doing the same behavior as the original switch1.  The errors are persisting and not the copper SFP port's do not light up and the admin portal freezes.  Its essentially now acting just like the original switch1 that this switch replaced.

I have tried all different, brand new cables, SFP's, and so on.  The switch2 has continued to work fine with a very similar config with no issues.

All of this equipment is brand new out of the box and was receiver around 6-2022, so it was in the box but still brand new.  I do not know what the issue might be.  I thought the first switch might have been bad out of the box.  Now that switch3 is doing the same thing, its hard to think that 2 out of 3 new switches would be bad out of the box.  I am running the newest available download for firmware, which was released in April 2023.  The config I have been doing is very simple and is working fine in switch2 so I can't think it is a config issue.  

This is in a lab environment that needs to run properly but running this gear is new to me so has anyone else has this kind of issue or does anyone have any suggestions?

[djtech2k]

Here are some examples of the errors that are repeating.

%Box-W-I2C-FAILURE-READ: I2C failed to read information from device at address 0x70 on bus 0 offset 0x08, component CPLD

%Box-W-I2C-FAILURE-READ: I2C failed to read information from device at address 0x2E on bus 0 offset 0x28, component ADT7476 Fan controller and thermal sensor

[atish]

I would suggest opening a TAC Case with Cisco. 

I've been having issues with the 3.3.0.16 firmware on the CBS switches as well, specifically with PoE delivery. TAC support has acknowledged that the latest firmware version "is running through some technical glitch and our developer team is working on it." They recommended downgrading to a previous "stable" version while they work on a fix.

Try opening a TAC case, and running version 3.2.0.89 instead. The 3.2.0.89 has a security vulnerability, however, it should be fine since this is in a lab environment. I would ensure these switches have no internet connectivity for the time being so minimize any security risks.

Atish

[djtech2k]

Thanks so much for the reply.

We have configured 4 new switches and 2 of those have these issues.  Its so strange that it is happening this way.  2 of the switches have been running without errors for almost 2 weeks so I think we will continue to monitor those.

We will probably try the older firmware on the 2 broken switches to test it on the side.

Does anyone know when Cisco might have a fix?

[djtech2k]

I just rack mounted this last switch where its home will be since its been running our config for 2 weeks without errors.  Once it powered back on, the errors came back.  No idea why, but it did.  So as a test, I moved it back to sitting on a shelf in the rack that it ran for 2 weeks without errors.  When it powered back on in that rack, the errors are gone.  How is that possible?

Why would the rack its mounted in cause those errors?  The fans on the sides are unblocked in both racks.  The rack that the errors happen in does have a few 1U devices surrounding the switch, but it is not hot in any way.  The errors happened on this switch even when it was just sitting on top of a 1U server, which was just running idle.

I am baffled as to why the rack its in is making a difference with the errors and functionality of the switch.  Could this be an issue with the firmware?

[atish]

This definitely sounds like an issue with the firmware. Try firmware downgrading one of the switches if possible. Please go through the Cisco Release Notes first, as there are some versions of the CBS switches that cannot be downgraded to certain previous firmware versions

If that fails, your best bet would be to open a support case. There was a similar issue with the SG Series switches (different family of switches, however, I personally think their code base has a number of similarities). 
https://community.cisco.com/t5/switching/read-i2c-failed-to-read-information-from-device-at-address-0x70/td-p/4021589
This was resolved in a newer firmware. 

The issue on the SG series switch was marked as a bug by Cisco, and fixed in a firmware patch.

https://www.cisco.com/c/dam/en/us/td/docs/switches/lan/csbms/350xseries/2_5_9/Release-note/b_tesla-release-2_5_9_16.pdf

Bug ID: CSCvz00101
Symptom: DevicedisplaysI2CrelatederrorswhencertaintypesofGBICsareplugged in.

As I mentioned before, I would do a firmware downgrade as a temporary workaround, and open a TAC support case for a patched firmware for the CBS switches.

Atish

[ntenck]

I think the correct Bug ID for the CBS350 models is CSCwi16752 

[djtech2k]

I was working with the switches again yesterday and saw a couple things.  When the errors happen, they do not stop until it eventually reboots itself.  When it is happening, if I go to the diagnostics page to check the ports/SFP's, it takes a long time and shows "information unavailable".  The admin portal usually freezes and the switch reboots itself.  These things seem to happen together and not independently.

I was trying to figure out how to consistently predict when this behavior happens but it was inconsistent.  It happens a lot, but sometimes the behavior would stop and come back.  Yesterday I found that it seems to happen a lot more often when I insert Cisco Copper SFP's (SFP-10G-T-X).  On many occasions on multiple switches I was able to insert that SFP and it would immediately start the behavior, resulting in a reboot.  There were times that it did not happen, but it seems like the copper SFP triggers it somehow.

All of our switches are running 3.3.0.16 firmware so we took 1 switch and downgraded it to 3.2.0.89.  The same condition occurred.  When we inserted the Cisco copper SFP, more than 50%+ of the time, the condition happened.  We have fiber SFP's and adding/removing any of those did not seem to cause any issues.  Only the copper SFP's seemed to have an effect.  We did try multiple different SFP's just to rule out a bad SFP or something.

[mis]

Same issues here. The CBS350 switches are so far the worst product I've seen from Cisco. If you're considering them, take your money and run. At this point any other brand would be better for small businesses. HPE or even FS.com would be better.

The previous generation of CBS switches were a lot more stable in comparison. The portal and CLI are slower. The portal and the CLI freeze with simple operations such as when shutting a couple of ports, then you have wait 10 minutes. Also the Mgmt port stops responding to ping for hours intermittently. - LACP channels don't work with 10G links.