ACE 20 (ACE 30) versus 4710 appliance reliability....

will · ‎06-22-2012

In 2008-2010 timeframe, I used the ace 4710 appliances at one customer and kind of liked them. The deployment was not too SSL intensive and B/W requirements were low, but I configured a few HA pairs and that worked well. The configuration was pretty comparable to other Cisco devices; so easy to learn/pick-up.

Fast forward to 2011: stepped into an environment, where customer purchased 3 - ACE 20 modules (before I got here), and had multiple issues with them. I found 4 documented TAC cases, and 1 was still open. I started working from December 2011 on getting Cisco to own-up WRT modules but customer by that time had had enough.

The most serious issue was a random reboot, hang or lockup. I wasn’t here to work with them to verify, but that’s eventually what the deal breaker was. Around the February 2012 timeframe, talking to Cisco SE, he revealed Cisco had an independent lab in Switzerland verify that some hardware component on the device had a terminal defect, in which a bit would flip, and force the device to lock or reboot - subject ot radioactive decay or interference.

Cisco and the lab attributed this to improper shielding, coupled with defective material in the electronic component; hence the device was highly susceptible to radiation-type errors. This is the kind of stuff you read in doomsday reports! As a result, Cisco was EOL-ing the ACE-20 module. I am trying to get Cisco to replace the ACE-20 modules with something else, but they haven’t been too cooperative. They have also limited their SE/Salseperson presence where I work (Pacific Northwest); and are not too responsive.

I have gotten a verbal agreement to get a credit on prior purchases for the amount this customer spent on the ACE-20 modules. However, the credit is only a few points off their normal discounting model. And Cisco will not go into loss on new product sales. Using example, $100 product would cost me $55 with standard Cisco discounting. Cisco’s cost might be $45 so I will only get another $10 credit on this new purchase.

The 3 Cisco ACE-20’s originally cost customer about $100K, so to dwindle this credit down, we would need to purchase about $1-$2 million of new hardware - that's a lot of new gear! And I don’t have any real way of knowing that Cisco is applying the credit honestly, and they won’t put anything in writing. This entire issue has really dampened customer’s impression of Cisco. They had smartnet on the ACE-20’s for 2+ years, but then dumped that after losing faith in the product. Now I am trying to resurrect smartnet to see if Cisco will give us an alternate product.

And to cap it all off, the original Cisco salesperson (who sold customer the ACE’s), has left and went to work for F5! And yes, he has been calling on customer to try to sell some big-IP's! At least there is some humor in all of this. So... Has anyone else had bad experience with ACE-20 module? How about ACE 4710? Any ideas on how to get a reliable working ACE module from Cisco? Does anyone know about this bug report for the radioactive decay issue?

Jorge Bejarano · ‎06-23-2012

Hi Will,

Hardware designers and developers in general have identified this issue related to SRAM memory which might be triggered by environmental conditions. The way how SRAM memory works makes it susceptible to suffer these issues, Cisco is highly focused on this currently and we are working on that.It is being seen that this behavior may be also linked to some software defects but if you have experienced this issue before and you are running at A2 2.3 then the recommendation is to proceed with a replacement since the device hardware might be affected at that moment. This issue occur with a low very frequency.

In case you have suffered these crashes many times, basically you may talk to your Cisco Account Manager or Partner directly and let them know your specific situation.

ACE30 and ACE 4710 do not suffer this kind of issues since they have a different hardware and internal architecture.

Hope this helps

Jorge

will · ‎06-23-2012

thx jorge, glad to hear cisco is focusing on this bug. Do you have the bug ID for this? Also, do you know if a device under smartnet will be eligible to swap for ACE-30 or possibly ACE 4710, or will it swap with another ACE-20. Customer has already swapped ACE-20's once; and has kind of lost faith in the swap and the smartnet process in general.

Customer is pretty displeased at the moment if I even bring up the word "ACE-20," but my experience is that cisco usually follows through with good customer service and maintains a reputation for quality. Which is why I am trying to reach a better resolution. Unforntuantely, our Cisco account manager appears to be too busy, annoyed at this problem, or unable to solve the issue for us. That is why I am reaching out to this forum.

thx again,

Will

Jorge Bejarano · ‎06-23-2012

Hi Will,

There are several bugs related to this behavior.

If you want to know which specific bug impacted you then it will be good to collect a #show tech-support and take a look of the #dir core:. Usually when you have a crash the ACE module generates some useful core dumps which CISCO TAC can decode and provide you with the details of the bugs.

You can check also under #show version, you may find the last reboot reason under the command which can help as well.

Here you have a link about the #dir core: command and how you can get the core dumps out of the box: http://docwiki.cisco.com/wiki/Cisco_Application_Control_Engine_%28ACE%29_Troubleshooting_Guide_--_Overview_of_ACE_Troubleshooting#Copying_Core_Dumps

As you can see under #show version, it shows the last reboot reason:

ACE_module5/Admin# show version

Cisco Application Control Software (ACSW)

TAC support: http://www.cisco.com/tac

The copyrights to certain works contained herein are owned by

other third parties and are used and distributed under license.

Some parts of this software are covered under the GNU Public

License. A copy of the license is available at

http://www.gnu.org/licenses/gpl.html.

Software

loader: Version 12.2[121]

system: Version A2(2.0) [build 3.0(0)A2(2.0)]

system image file: [LCP] disk0:c6ace-t1k9-mzg.A2_2_0.bin

installed license: no feature license is installed

Hardware

Cisco ACE (slot: 5)

cpu info:

number of cpu(s): 2

cpu type: SiByte

cpu: 0, model: SiByte SB1 V0.2, speed: 700 MHz

cpu: 1, model: SiByte SB1 V0.2, speed: 700 MHz

memory info:

total: 955396 kB, free: 289704 kB

shared: 0 kB, buffers: 2336 kB, cached 0 kB

cf info:

filesystem: /dev/cf

total: 1000000 kB, used: 494912 kB, available: 505088 kB

last boot reason: NP 1 Failed : NP ME Hung ----> here

configuration register: 0x1

ACE_module5 kernel uptime is 4 days 22 hours 42 minute(s) 41 second(s)

Then, at this point as TAC the recommendations which I can provide are:
-Open a TAC case after you got the #show tech-support and core dumps.
-If this is 2nd or 3d time with the same issue, you may ask for a RMA
-It is recommended to have one of the latest software versions
-Monitor the behavior and get core dumps and #show tech-support

In regards to the other question. ACE30 is the hardware upgrade of the ACE20
and the ACE 4710 is the upgrade of the CSS, then you cannot ask for an ACE 4710 if
you have an ACE 20 but again that´s something with you may discuss with your
Cisco Account Manager and/or Cisco SE engineer based on your specific situation and
they can tell you a final word about it

Hope this helps.

Jorge

will · ‎07-10-2012

hi jorge, thx for taking time to answer my ace-20 question regarding the radioactive decay problem. Is there an ace-20 part that is not afflicted with this problem? Will cisco provide me with another ace-20? Customer has gotten one bad ace-20 already and doesn't want another rma of the same part. is it likely that cisco would provide 4710 or ace-30 in rma? that is only scenario under which customer woudl accept rma of the old ace-20 at this point. thx again, Will

Jorge Bejarano · ‎07-26-2012

Hey Will,

Did everything go well?

Jorge

Jorge Bejarano · ‎07-10-2012

Hello Will,

As I mentioned before, since your customer has experienced this behavior once again, then you should open a TAC case where you let the engineer in charge about the current situation and that you have had this behavior several times already, after that the owner of the TAC will evaluate the situation and engage the Cisco Account Manager and/or SE assigned for your company to determine how you need to proceed.

ACE30 is the hardware upgrade of the ACE20/ACE10 then under any condition you will get an ACE4710 instead.

Then, here you have my recommendations:

-Open a TAC case

-Upgrade to the version a2.3.3 or higher.

-Get the current #show tech-support of the ACE module

-Get the core dumps

-Have the engineer of the case to check internally with your Cisco Account Manager or an alternate manager which can assist you

-If you consider necessary, also you may consider to ask your TAC engineer for an E.F.A ( failure analysis)

Every case is handled isolated but since you have experienced this event many time, then you should discuss with a TAC engineer.

If you find these details useful, please mark it

Hope this helps.

Jorge