1.3.0.62 renders SG300 unusable

Dirk Dittert · ‎06-23-2013

Hi,

I just upgrade to 1.3.0.62 on an SG300-52 (from 1.2.7.76). This renders the switch completely unusable:

the switch seems to continue to operate as configured but
the web interface is not reachable at all
when logged in through ssh everything seems normal but stops working after 20-30 sec. Stops working means that the switch doesn't accept any more input and doesn't output anything on the console anymore. The same happens when I log in through the serial console. Sometimes opening another ssh session would allow me to access the admin console for another 20-30sec
sometimes it would not be pingable on the management ip at all

By power cycling the switch a couple of times and using the serial console I was able to switch back to the 1.2.7.76 boot image. It now seems to be fine again. Deleting startup configuration and restoring the config backup from before updating did not help (looking at the backup file -- there isn't anything complicated in the config. Mostly VLAN stuff). Uploading the old configuration said "copying completed with errors" but when I do a diff between the then running configuration and the configuration stored in the backup file, it basically only shows different keys (ssh, certificate).

Is this expected behavior?

Thanks

gimper48 · ‎06-23-2013

Same thing here. really frustrating.

Tom Watts · ‎06-23-2013

Hi, try to reload the 1.3.0.62 firmware, factory default the switch and do not reload the config file. What happens then?

-Tom
Please mark answered for helpful posts

-Tom Please mark answered for helpful posts http://blogs.cisco.com/smallbusiness/

Dirk Dittert · ‎06-24-2013

I already reloaded the firmware. That did not help. I'll try factory default and then not reloading the configuration/deleting boot config.

Any commands you want me to run on the serial console to capture log output?

Dirk Dittert · ‎06-24-2013

Paul,

I reloaded the 1.3.0.62 image (MD5 22347bc1ad4f7f75526896a565dc71c5), reset to factory default and removed startup-config. It seems to run in that state.

Then, I started reloading my previous running config on the serial console, a couple of commands at a time. I did not manage to restore the full config that was running before. The web interface became unreachable and the serial console did not respond anymore in every try. Unfortunately, I wasn't able to find out which commands triggered this because the timing of the freeze was not consistent.

Bottom line: it seems to require at least some configuration for it to happen but I can't tell what the root cause is. It could also be something in the environment (e.g. different handling of IPv6 in 1.3.062 vs. 1.2.7.76).

Frustrating.

Li Zhu · ‎06-25-2013

Hi Dirk,

Sorry for your inconvenient, on 1.3.0.62 add some features like IPv6 routing, DHCP server, also change some CLI, so old configuration is not totally compatible.

My suggestion is factory default the switch then configure it manually refer to your old configuration, don't copy old configuration file directly.

Dirk Dittert · ‎06-25-2013

I do appreciate that new features are still being added to this switch, especially as IPv6 is still a new thing here. After all, this was one of the reasons for buying this product. But seriously: You guys must be kidding me, right? This is not a home product with just two options to configure...

Is there a way to pass on my config for you guys to find out which parts are not working with the new version? It's not a lot that I'm doing in there, so I guess it must be affecting a larger part of the user base. Basically it's just defining vlans, assigning ports to vlans, settting sntp, enabling ssh server, defining ssh keys and ssl certificates. I would understand if the new software version said something like "invalid config statement xy - syntax has changed" but freezing the whole admin web interface/console without telling any reason is a little harsh and should be considered a prio 1 bug.

Li Zhu · ‎06-25-2013

Sure, if you think it's a bug, you could reach out to the SBSC and open a Service Request for it.

Please find the SBSC contact information below:

http://www.cisco.com/en/US/support/tsd_cisco_small_business_support_center_contacts.html

Reuben Farrelly · ‎06-26-2013

By chance do you have an IPv6 reachable NTP server in your config? IPv6 NTP to a Global Address is currently broken in 1.3.0.59/1.3.0.62 - it'll lock the switch management interface up hard once the switch contacts the NTP server. In previous versions it didn't seem to be an issue.

Been there done that, after a month or so trying to convince TAC support they finally accepted and were able to reproduce the problem.

Fix is slated for a later maintenance release this year. I don't have a DDTS bug ID though, sorry.

Dirk Dittert · ‎06-26-2013

You mean something like:

clock timezone " " 1

clock source sntp

sntp unicast client enable

sntp unicast client poll

sntp server 2a01:4f8:x:x::x poll

That would explain why I wasn't able to reproduce it consistently. If it really happens when the ntp client tries to connect the server the timing would be inconsistent.

Do you know if something like

!

interface vlan 1

ipv6 enable

ipv6 address 2a01:4f8:x:x::x/64

!

could also be cauing problems?

Reuben Farrelly · ‎06-26-2013

Yes. That sntp server with a global address (as you have put in there) will cause it.

It's not the act of adding the command that is the problem, if you have no IPv6 connectivity the command won't cause the lockup. The problem occurs when the IPv6 NTP server responds to the SG's NTP request. It seems to be the return packet that causes the SG to die.

I have an open SR for this: 625456299 so if you do go ahead an open a case yourself it may be a good idea to link it to this so that Cisco can be aware of the customer impact this bug is having. Not sure how a bug like this got through QA, but it did

Dirk Dittert · ‎06-26-2013

Thanks for the advice and saving me so much time trying to figure out what might be causing this. I'll try to contact support for that problem as well.

Regarding QA: You're wondering how that kind of bug managed to sneak into a release that advertises IPv6 compatibility? That's indeed a little unfortunate.

puneelam · ‎06-26-2013

Hi

After looking at your problem, I tried on SG300-52 PoE switch and I was successfully upgraded the firmware to 1.3.0.62 and I did not notice any issues while working.

Try to check the ports and hardware of the switch and upgrade the software.

The following link will be useful while upgrading :

http://sbkb.cisco.com/CiscoSB/Loginr.aspx?login=1&pid=4&app=search&vw=1&articleid=947

Thanks and Regards

Punith Kumar Neelam

SBCD Engineer

Dirk Dittert · ‎06-26-2013

Hi Punith,

can you please retry with a IPv6 reachable SNTP server configured to find out if this is really causing the problems? In my config is something like this: