SG300-10MP keeps going down!!!

[codystokes]

Hello,

We are in an organization of 80 users with 5 Sx300 series switches with the latest firmware (1.2.7.76). The network is mostly flat with all switches feeding into a Cisco ASA5505 acing as the primary gateway for Internet Access. Only our two 48 port POE switches have an additional vlan configured for the VoIP phones.

We have one SG300-10MP that is powering 5 Cisco Aironet 1024N's that keeps crashing. A couple weeks ago it went down twice in one day. I updated the firmware and it did not have any hiccups unitl today. This issue has been consistent since we installed the switch in April. Originally the issue seemed to be caused by too much bonjour traffic causing a buffer overflow (we have over 100 Apple products). Since I updated the switch I have not seen any bonjour buffer overflow errors. Instead I get the following error when it crashes:

2147424534 2012-Sep-05 15:12:28 Emergency %SYSLOG-F-OSFATAL: FATAL ERROR: tExcTask: ABORT DATA exception ***** FATAL ERROR ***** SW Version : 1.2.7.76 Version Date: 19-Jul-2012 Version Time: 17:54:43 Instruction 0x18 Exception vector 0x10 Program state register 0x80000092 %SNMP-I-CDBITEMSNUM: Number of startup configur ation items loaded: 133 %LINK-W-Down: gi1 %LINK-W-Down: gi2 %LINK-W-Down: gi3 %LINK-W-Down: gi4 %LINK-W-Down: gi5 %LINK-W-Down: gi6 %LINK-W-Down: gi7 %LINK-W-Down: gi8 %LINK-W-Down: gi9 %LINK-W-Down: gi10 %LINK-I-Up: gi9 %LINK-I-Up: Vlan 1 %LINK-I-Up: gi1 %STP-W-PORTSTATUS: gi9: STP status Fo

The CPU and PoE supply are far from being overloaded. Can anyone tell me what this is and what it means? Can't seem to find anyone else with the same problem and it is extremely frusturating! Thanks in advance.

[Tom Watts]

Cody, I will post a reply once I receive an answer.

Thanks.

-Tom
Please rate helpful posts

[codystokes]

Thanks, Tom!

[Tom Watts]

Cody, can you please provide me your telephone number?

Please email me at towatts@cisco.com

Thanks.

-Tom
Please rate helpful posts

[cfarinella]

I am having the same exact problem on 3 of the 5 SF300-24 switches we have here.  Do we have a solution for this?

Sep 10 19:27:51 192.168.24.10 %SYSLOG-F-OSFATAL:      FATAL ERROR: tExcTask: ABORT DATA exception     ***** FATAL ERROR *****     SW Version  :  1.1.2.0  Version Date:  12-Nov-2011  Version Time:  23:34:26    Instruction            0x18  Exception vector       0x10  Program state register 0x80000092

[Tom Watts]

Hi Charles, this is being researched. Once I have an answer it will be updated on this topic.

-Tom
Please rate helpful posts

[cfarinella]

Thank you very much, I appreciate the response.

[Tom Watts]

Cody and Charles,

Unfortunately, with the topology, switch config and details about the network, it was not enough to recreate the error.

Below are some additional requests if any/all of this information can be provided, it would be great so we may attempt to localize the error conditions to generate on demand.

The reboot and error message were  much more frequent with version 1.1.2.0. Therefore I would like to focus  on this version to try to get more details on recreation scenario:

1)  Customer mentioned he say the bonjour overflow notification message in  previous version. Is this message related to reboot? Did customer a  reboot following this message?

2) What does customer mean by “This error was very frequent and resulted in flapping but was also received with the error initially stated.”

            i)  Is customer relating here to the abort data message?

ii) Does flapping mean reboot?

iii)Does this mean reboot message came at the same time as bonjour message?

3) Can customer send full log file from previous version reboot and from current?

4) In terms of configuration – is there any difference between the device that has the reboot, to those that do not?

Customer  mentions they are on a flay L2 network – but I would like to verify if  there is a certain VLAN segregation between them.

5)  I see in the forum that some additional customers are reporting issue.  Is it possible to get additional information from them regarding the  reboot? Main points needed:

a) Config file

b) Setup of switch and network

c) Does this customer have Bonjour agents in his network?

d) Any consistency or pattern in reboot?

i) How frequent are the reboots?

ii) Do they happen at specific times or intervals?

iii) Do they happen on all his switches at the same time?

e) Any occurrences or (configuration and other) steps taken in network at time of reboot.

f) Did additional customer also see any notification (like the bonjour one) prior to reboot? Any other message

g) Full log file(s)

-Tom
Please rate helpful posts

[codystokes]

The bonjour overlow error was related to the switch rebooting, but that has not happened since updating to 1.2.7.76. We are only getting the occasional message about FATAL ERROR: tExcTask. So far the switch has been up for almost 8 days with no issues. Fingers crossed this does not go down again.

I did not have to reboot the switch myself, it would automatically reboot upon crashing. I assume the flapping was reported by Charles so I will disregard that question.

I can certainly get logs before and after updating. What is the best way to do this? I imagine copy and pasting the logs from the browser will get messy.

In regards to question 4. There is not much difference in the configuration nor are there any additional vlans configured. Shouldn't the config file I gave you tell them what the vlan config is like?

[codystokes]

Hey Tom, what's the best way to export my log files so I can submit them for review?

[Tom Watts]

Cody, for me the easiest way is to set up hypterterminal , console the switch then do "show log".

You can set up hyperterminal to send your cli output to a text document.

When you opn the HT on the top click Transfer then click "Transfer Text" then browse the text file you want the output to go to. After that, anything you do on the console session will log to the switch.  (So it is a great way to export show tech support or show log). Don't forget to hit the space bar several times to ensure all the log events will load.

P.S. Charles sent me a lot of documentation which I have forwarded as well. Hopefully the developemnt can recreate for you guys

-Tom
Please rate helpful posts

[codystokes]

Tom,

I can actually do this through telnet as well. I went ahead and ran the "show log" command and it did display log data but i believe it's only showing the log in the RAM memory and not the flash memory so we will only see what's been logged since the last reboot. How do we see the log from flash memory?

[Tom Watts]

Try "show logging file"

-Tom
Please rate helpful posts

[codystokes]

I have the log file ready. Should I paste the results in this window or should I just put it in a text doc and email it to you? I do not see an option to attach files in this forum.

[Tom Watts]

email please

-Tom
Please rate helpful posts