Cisco ASA IPS Detect service card failure Service card in other unit has failed

teatrodelsogno · ‎03-04-2015

Hi Guys,

I have a strange problem in a part connected to one cisco asa bug, but is not belonging to my version.

I have one cluster active/standby failover and in random time this is the behavior that is happening:

From State To State Reason
==========================================================================
12:48:10 CEST Mar 4 2015
Just Active Active Drain Service card in other unit has failed

12:48:10 CEST Mar 4 2015
Active Drain Active Applying Config Service card in other unit has failed

12:48:10 CEST Mar 4 2015
Active Applying Config Active Config Applied Service card in other unit has failed

12:48:10 CEST Mar 4 2015
Active Config Applied Active Service card in other unit has failed

12:59:04 CEST Mar 4 2015
Active Standby Ready Other unit wants me Standby

12:59:05 CEST Mar 4 2015
Standby Ready Failed Detect service card failure

12:59:11 CEST Mar 4 2015
Failed Standby Ready My service card is as good as peer

12:59:11 CEST Mar 4 2015
Standby Ready Just Active Service card in other unit has failed

12:59:11 CEST Mar 4 2015
Just Active Active Drain Service card in other unit has failed

12:59:11 CEST Mar 4 2015
Active Drain Active Applying Config Service card in other unit has failed

12:59:11 CEST Mar 4 2015
Active Applying Config Active Config Applied Service card in other unit has failed

12:59:11 CEST Mar 4 2015
Active Config Applied Active Service card in other unit has failed

13:03:07 CEST Mar 4 2015
Active Standby Ready Set by the config command

13:03:55 CEST Mar 4 2015
Standby Ready Failed Detect service card failure

13:06:38 CEST Mar 4 2015
Failed Standby Ready My service card is as good as peer

13:10:15 CEST Mar 4 2015
Standby Ready Just Active Other unit wants me Active

13:10:15 CEST Mar 4 2015
Just Active Active Drain Other unit wants me Active

13:10:15 CEST Mar 4 2015
Active Drain Active Applying Config Other unit wants me Active

13:10:15 CEST Mar 4 2015
Active Applying Config Active Config Applied Other unit wants me Active

13:10:15 CEST Mar 4 2015
Active Config Applied Active Other unit wants me Active

I try to manually reset the IPS module with hw-module module 1 reset but I'm receiving the same messages.

Some other information:

Cisco Adaptive Security Appliance Software Version 8.2(1)

Mod Card Type                                    Model              Serial No.
--- -------------------------------------------- ------------------ -----------
0 ASA 5520 Adaptive Security Appliance         ASA5520            JMX1414L0X3
1 ASA 5500 Series Security Services Module-10 ASA-SSM-10         JAB10070GJP

Mod MAC Address Range                 Hw Version   Fw Version   Sw Version
--- --------------------------------- ------------ ------------ ---------------
0 0026.99c0.c0df to 0026.99c0.c0e3 2.0          1.0(11)2     8.2(1)
1 0015.c6fa.3b31 to 0015.c6fa.3b31 1.0          1.0(10)0     7.1(8)E4

Mod SSM Application Name Status SSM Application Version
--- ------------------------------ ---------------- --------------------------
1 IPS Up 7.1(8)E4

Mod Status             Data Plane Status     Compatibility
--- ------------------ --------------------- -------------
0 Up Sys             Not Applicable
1 Up                 Up

------------------ show memory ------------------

Free memory:      1722778808 bytes (80%)
Used memory:       424704840 bytes (20%)
-------------     ----------------
Total memory:     2147483648 bytes (100%)

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Failover On
Failover unit Primary
Failover LAN Interface: faillink Management0/0 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 2 of 250 maximum
Version: Ours 8.2(1), Mate 8.2(1)
Last Failover at: 13:10:15 CEST Mar 4 2015
   This host: Primary - Active
       Active time: 665238 (sec)
       slot 0: ASA5520 hw/sw rev (2.0/8.2(1)) status (Up Sys)
          Interface outside (x.x.x.x): Normal
          Interface inside (10.254.0.3): Normal
          Interface WIFI_no (10.254.14.1): Normal (Not-Monitored)
          Interface DMZ_dsoi (172.16.1.1): Normal (Not-Monitored)
          Interface Adfafd (x.x.x.x): No Link (Not-Monitored)
       slot 1: ASA-SSM-10 hw/sw rev (1.0/7.1(8)E4) status (Up/Up)
          IPS, 7.1(8)E4, Up
   Other host: Secondary - Standby Ready
       Active time: 70225 (sec)
       slot 0: ASA5520 hw/sw rev (1.1/8.2(1)) status (Up Sys)
          Interface outside (81.208.53.221): Normal
          Interface inside (10.254.0.4): Normal
          Interface WIFI_no (0.0.0.0): Normal (Not-Monitored)
          Interface DMZ_dsoi (172.16.1.3): Normal (Not-Monitored)
          Interface Adfafd (85.20.9.11): Normal (Not-Monitored)
       slot 1: ASA-SSM-10 hw/sw rev (1.0/7.1(8)E4) status (Up/Up)
          IPS, 7.1(8)E4, Up

Stateful Failover Logical Update Statistics
   Link : faillink Management0/0 (up)
   Stateful Obj    xmit       xerr       rcv        rerr
   General       483879055 0          14089818   10606
   sys cmd     97977      0          97977      0
   up time     0          0          0          0
   RPC services     0          0          0          0
   TCP conn    97196877   0          4327486    1514
   UDP conn    386494404 0          9650250    9092
   ARP tbl     45801      0          4968       0
   Xlate_Timeout     0          0          0          0
   VPN IKE upd    17191      0          1784       0
   VPN IPSEC upd    26805      0          7353       0
   VPN CTCP upd    0          0          0          0
   VPN SDI upd    0          0          0          0
   VPN DHCP upd    0          0          0          0
   SIP Session    0          0          0          0

   Logical Update Queue Information
         Cur    Max    Total
   Recv Q:    0    25    16395021
   Xmit Q:    0    111    505815152

Could you please send me some suggestion or share past experience about it?

Many regards

Matteo

Tushar Bangia · ‎03-04-2015

There are few know bugs:

https://tools.cisco.com/bugsearch/bug/CSCun81616/?reffering_site=dumpcr

https://tools.cisco.com/bugsearch/bug/CSCun82492/?reffering_site=dumpcr

https://tools.cisco.com/bugsearch/bug/CSCun81616/?reffering_site=dumpcr

Analyzing logs would help to identify the cause of the issue.

Regards,

Tushar Bangia

Note - Please do rate post if you find it helpful!!

teatrodelsogno · ‎03-05-2015

Hi Tushar,

correct, I found also these bugs but as you can see, are different affected release version than mine.

BTW, shall we consider that also other version could be affected? (like mine --> 8.2(1))

And the workaround proposed by bugs, is little bit "crazy". Cause the workaround is only collect logs from syslog?! In which way the collection of the logs could solve the problem? :-)

Any council update from 8.2 could be "flagged" like stable with IPS module installed in the ASA?

Regards

Matteo

Tushar Bangia · ‎03-05-2015

Hi,

Such issues are generally reported because of AIP SSM failure. Please check the sanity of module via "sh module x details".
The issue can be caused because of oversubscription of the sensor module hence advise you to create an ACL to limit the inspected traffic.

Please try below and see if the module comes up as expected:

hw-module module 1 shutdown and then hw-module module 1 reset.

Alternatively, you can raise a case with Cisco TAC for replacement of module.

Regards,

Tushar Bangia

teatrodelsogno · ‎03-05-2015

thanks..

yes, this I forgot to attach, but also the module seems in a good health state:

show module 1 details

Getting details from the Service Module, please wait...

ASA 5500 Series Security Services Module-10

Model: ASA-SSM-10

Hardware version: 1.0

Serial Number: JAB10070GJP

Firmware version: 1.0(10)0

Software version: 7.1(8)E4

MAC Address Range: 0015.c6fa.3b31 to 0015.c6fa.3b31

App. name: IPS

App. Status: Up

App. Status Desc: Normal Operation

App. version: 7.1(8)E4

Data plane Status: Up

Status: Up

Mgmt IP addr: xx.x.x.x

Mgmt Network mask: 255.255.255.0

Mgmt Gateway: x.x.x.x

Mgmt Access List: 1x.x.x.x

Mgmt Access List: 10.x.x.x.x

Mgmt Access List: 10x.x.x.x

Mgmt Access List: 10.x.x.x.x

Mgmt Access List: 1x.x.x.x

Mgmt Access List: 10.x.x.x.x

Mgmt web ports: 443

Mgmt TLS enabled: true

I already tried module reset...

yes, the only way maybe is the TAC.

Thanks the same, I will send an update also here or in case in the while, let me know if you have some other ideas.

Regards

Marvin Rhoads · ‎03-05-2015

While your version 8.2(1) might not be specifically listed in the BugIDs, it is very old and there are many many updates released since then. At a minimum I would upgrade to the last 8.2(x) release - 8.2(5) maintenance release or 8.2(5.13) interim.

teatrodelsogno · ‎03-05-2015

Hi Marvin,

yes, you are right.

I have to follow some other particularly steps to upgrade the firewall with SSM (IPS) module on board?

Or steps to follow are the same like the "normal" upgrading on the ASA appliance?

Many regards in advance.

Matteo

Vibhor Amrodia · ‎03-05-2015

Hi,

I think it is clear from the ASA Fail-over history that the ASA IPS module is causing the fail-over events on the ASA HA pair.

That can be due to various reasons on the IPS module.

I would recommend you open a TAC to find the root cause of this issue on the IPS.

Thanks and Regards,

Vibhor Amrodia

teatrodelsogno · ‎03-05-2015

I Vibhor,

thanks for your answer.

Yes exactly. In the meantime I opened one TAC, but I was curious to receive in any case some feedback from the community about this problem.

Regards

teatrodelsogno · ‎03-05-2015

Ok,

TAC Support answered

As Marvin said. One upgrade of the asa is high suggested.

BTW in other, in one my troubleshooting, I found these different:

NOTE: Primary is the one with problem

Primary:

5520

ips FW version 1.0(10)

Secondary:

5520-k8

ips FW version 1.0(11)

I think that is not connected to my problem ( k8 is just the different belonging to the license) and FW version could be different on the same model? ...

BTW, do you have some other feedback about this?

Regards

Marvin Rhoads · ‎03-05-2015

The "ips FW" (Firmware) versions do not need to be identical. That is most likely due to different manufacturing dates.

Let's see if the issue persists after you have upgraded your ASA software.

Vibhor Amrodia · ‎03-05-2015

Hi,

If you are planning for an upgrade for the ASA code , I would recommend an upgrade to ASA 8.2.5 latest interim to pick up the fix for this Defect as well:- CSCts98806

Thanks and Regards,

Vibhor Amrodia

teatrodelsogno · ‎03-06-2015

Hi all,

thanks for the support once again.

I will proceed with the upgrade.

Anyway I checked the list of interim software and I found also asa825-55-k8.bin version. (55).

There is any particularly reason why you suggested (33)?

Thanks and regards

Matteo

Vibhor Amrodia · ‎03-09-2015

Hi,

No , Actually i just wanted to recommend a Code which fixes the defect that i listed earlier.

Thanks and Regards,

Vibhor Amrodia

teatrodelsogno · ‎03-16-2015

Hi ALL,

guys, confirmed.

The update to 825(55) solve this problem!

In case of similar problem to other devices or version, please try to follow this street first!

Bye!

Rate me if this post has been usefully.