1.3.0.59 firmware issue

bhackbarth · ‎03-29-2013

I installed the 1.3.0.59 on a couple of SG300-28Ps. On the Status and Statistics page, the PoE indicators no longer lit. Physically, on the front of the switch, they did still light. I didn't yet reboot to factory defaults to see if that clears it, because I don't feel like entering the config again this early in the morning. But I am willing to test that, if needbe.

When reverting the image back to 1.2.9.44 (tested) or possibly earlier, the ip default gateway must be re-entered (if it was configured). Even if ip-default gateway x.x.x.x shows up correctly on a 'show run', the switch will not obey it, and on the IPv4 settings page it will report the operational default gateway as blank. This came as a surprise becauset he switch suddenly wouldn't talk to the VPN anymore. Logging onto the switch locally, going to the IPv4 settings, ticking the radio button back on User Defined and typing it back in cleared that up. It appears this cropped up because the syntax for specifying the default gateway changed in 1.3.x but it's still odd that the config shows correctly in console but not in the gui.

Reuben Farrelly · ‎03-30-2013

I too experienced major problems with this new firmware.

I have an SG300-8, and after loading the code on and rebooting, the switch booted up, but after exactly 3 pings succeeding, it then lost all management access and could not recover.

I restarted and the same thing happened again. Unfortunately I wasn't able to console in to see what was going on (USB->Serial didn't seem to work for some odd and likely unrelated reason).

I then did a factory reset, and then set the switch config back up, line by line identical to the old config, saved it and rebooted - at that stage everything worked, so I saved the config and rebooted. 3 pings after it came back and I lost access again.

Traffic to/from ports on the switch was OK including those in different VLANs - just management access that stopped.

After much messing around I was able to get in and revert to the old image and reload, and things have been up and running again.

I'll get my console access sorted out and reattempt the upgrade again.

bhackbarth · ‎03-30-2013

Reuben, thanks for your comments. I just got done trying this firmware on my only SF302-08P and I had identical symptoms. Even though I didn't do a copy run start after switching the image and rebooting, the default gateway moved itself to None (radio button) after reverting the image to 1.2.9.44 even though if I did a show run, the default gateway would show up in the config. The only way to correct this is to go into the web gui, move the radio button there and retype the gateway, because in the text config the line is already there (but the switch does not obey it). This results in the switch not being manageable anywhere but when plugged directly into it. Also, the port lights on the Status and Statistics page didn't match the lights physically on the front of the switch either. I have been able to replicate this on three models in the 300 line so my conclusion is that this release isn't stable.

I haven't yet seen your specific symptoms because as soon as I discovered oddities with it, I reverted immediately. Once back on 1.2.9.44 and after re-typing my default gateway, my troubles are gone. For me this is an isolated case of "if it ain't broke...don't fix it"

Reuben Farrelly · ‎03-30-2013

Hi Brayton

I'm now testing with console plugged in. With the new release I'm getting full hard lockups of the unit, including at the console. This more or less reflects what I was seeing from outside the switch, ie management plane completely dies but the switch keeps forwarding. Fortunately this one is in the spare room with a printer and AP hanging off so not too critical.

I have a second one here so I might load the same config onto that to confirm it's not hardware. I don't think it is (all is well if I revert to 1.2.9 on this unit) but it might be a useful data point.

I manage mine mostly via the CLI (I'm used to the classic IOS CLI, the CLI on these switches is close to it).

Release notes don't indicate any major issues like this. It's mildly annoying if there are major issues like these and they're documented, but quite a lot more of a problem if they exist but don't get picked up in the QA process :-(

Reuben

Reuben Farrelly · ‎03-30-2013

Confirming this problem is not specific to one switch. I have a second identical switch and have put the same config on it and it also locks up hard in the same way at about the same timeframes.

Often running 'show log' will kill the units dead and they require power cycling to bring them back to life.

Can someone from the CS or DE team let me know what we need to do to get this investigated? This new release is completely useless for me and I suspect many others.

Going back to version 1.2.9 now.

Nagaraja Thanthry · ‎04-01-2013

Hello Reuben,

Could you please reach out to the SBSC and open a Service Request for this issue? We would like to investigate this issue further and see what could be causing these lockups. We have not seen these lockups in our environment (I have one switch running in my Network without any issues). We would like to capture your configuration and some additional information from the switch when the lockups happen so we can investigate further.

Please find the SBSC contact information below:

http://www.cisco.com/en/US/support/tsd_cisco_small_business_support_center_contacts.html

Thanks,

Nagaraja

Reuben Farrelly · ‎04-03-2013

Hi Nagaraja

I've tried to reach out - four times now over the last 3 days - and it seems no one is monitoring the Online Chat/Webex. Which brings me back to the forum here because at least I get a response :-)

Each time after an hour or so it terminates with "Sorry, no one is available. Please return to the support community web site, and try again during normal business hours."

1pm Australia time (and 4pm yesterday and the day before) is normal business hours, I though. I have a support contract even.

Somewhat off topic for this thread I know, but I'm not sure where else to post this.

Reuben

Nagaraja Thanthry · ‎04-03-2013

Hello Reuben,

Can you call the SBSC at 1 800 605 731 (Australia) or appropriate SBSC number and log the service request?

Thanks,

Nagaraja

t-markcollins · ‎03-30-2013

Agreed. This is a wider problem. I have applied the update to a SG200-26P and traffic through the switch looks normal. However, admin access hangs for at least a couple of minutes before the login screen appears. I am getting continuous ping response. Nothing useful in the logs.

Update: Ran a few tests. I use Safari and the web interface seems to take about 2 min 45 sec to show. Tried with Firefox and it came up right away.

Update 2: There may be a clue in the attached Wireshark capture. This was from a Bonjour web access to the switch, but direct IP aaccess was also had the same long delay.

- switch is at 192.168.1.9

- my computer is 192.1.100

Update 3: Now it seems to be working. Frustrating to say the least.

Update 4: Bonjour had stopped advertising, and I had no trouble accessing via IP. Rebooted the switch. Bonjour was now advertising again, yet accessing the switch via Bonjour had the 2 minute + delay in getting the login page to show. Now the delayed access is back via IP. And from within Firefox now too. With a request initiated from both Safari and Firefox, they both got the delayed login screen at the same time. The switch is the issue here.

Reuben Farrelly · ‎03-31-2013

Unfortunately disabling bonjour didn't fix the lockup problems for me. I think the problem you have is different to mine, which sounds like different again to what Brayton is experiencing. This thread is turning out to be a good place to share (bad) experiences about this version of code though

Reuben Farrelly · ‎04-21-2013

Following up - the main problem I had turns out to be because I had an IPv6 address specified in the config for the 'sntp server'. This was killing the switch after it attempted to start the SNTP sync. Without that line in the config the switch has been up for a few days now.

Currently waiting for a ddts bug ID and hopefully (in time) a fix to be integrated in.

Apart from that the new release seems to be OK.

bhackbarth · ‎04-21-2013

Reuben, do you have any PoE devices connected to the switch, and if so can you look at the Status and Statistics -> System Summary page and see if you get any orange PoE lights on the webpage? The PoE lights go away on the status page in all switches I tested this image so far. It just shows a double green light like you see on the non PoE models.

Reuben Farrelly · ‎04-21-2013

Unfortunately I don't have the PoE version of the switch so I can't confirm. All my lights are green/on. Sorry Brayton..

t-markcollins · ‎04-21-2013

On the PoE ports of the physical switch they are amber. On the web interface they are green.

vashkenazy · ‎05-19-2013

BTW with this firmware I have an issue in DHCP Server of my SG300-28. Static reservations are no longer working.