WISM2 goes down for 6 minutes every 12 hours

[Vinay Sharma]

• Introduction
• Problem Category
• Problem Subcategory
• Software Version
• Hardware
• Problem Description
• Logs
• Cause
• Open caveats in the 7.3.112.0 controller software release
• Resolution
• Duplicate Bug Information
• Update

Introduction

In the document Cisco HTTS Wireless engineer "Victor Vasantha Kumar" has explained issue about "WISM2 goes down for 6 minutes every 12 hours".

Problem Category

Wireless

Problem Subcategory

Wireless Services Module 2 (WS-SVC-WISM2)

Software Version

version 7.4.100.60

Hardware

Wireless LAN - WS-SVC-WISM2

Problem Description

Since a few months, we have 4 WISM2 running in AP SSO Mode pairs. On both pairs we have the problem, that the secondary WISM goes down for 6 minutes every 12 hours.

Logs

003252: Sep 16 13:44:41.643 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003253: Sep 16 13:51:13.894 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003254: Sep 17 01:45:10.816 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003255: Sep 17 01:51:43.063 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003256: Sep 17 13:45:39.981 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003257: Sep 17 13:52:12.224 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003258: Sep 18 01:46:09.121 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003259: Sep 18 01:52:41.368 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003260: Sep 18 13:46:38.369 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003261: Sep 18 13:53:10.620 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003262: Sep 19 01:47:07.518 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003263: Sep 19 01:53:39.761 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003264: Sep 19 13:47:36.702 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003265: Sep 19 13:54:08.945 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003267: Sep 20 01:48:05.865 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003268: Sep 20 01:54:38.112 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003269: Sep 20 13:48:35.016 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003270: Sep 20 13:55:07.259 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003271: Sep 21 01:49:04.170 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003272: Sep 21 01:55:36.417 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003273: Sep 21 13:49:33.403 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003274: Sep 21 13:56:05.646 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003275: Sep 22 13:50:31.728 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003276: Sep 22 13:57:03.975 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003277: Sep 23 01:51:00.918 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003278: Sep 23 01:57:33.161 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003279: Sep 23 13:51:30.092 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003280: Sep 23 13:58:02.335 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003282: Sep 24 01:51:59.244 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003283: Sep 24 01:58:31.487 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003284: Sep 24 13:52:28.459 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003285: Sep 24 13:59:00.706 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003286: Sep 25 01:52:57.624 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003287: Sep 25 01:59:29.911 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003288: Sep 25 13:53:26.850 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003289: Sep 25 13:59:59.097 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003291: Sep 26 01:53:56.058 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003292: Sep 26 02:00:28.301 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003293: Sep 26 13:54:25.212 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003294: Sep 26 14:00:57.455 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003295: Sep 27 01:54:54.417 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003296: Sep 27 02:01:26.668 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003297: Sep 27 13:55:23.582 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003298: Sep 27 14:01:55.825 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003299: Sep 28 01:55:52.828 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003300: Sep 28 02:02:25.023 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003301: Sep 28 13:56:21.943 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003302: Sep 28 14:02:54.186 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003303: Sep 29 01:56:51.100 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003304: Sep 29 02:03:23.343 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003305: Sep 29 13:57:20.263 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003306: Sep 29 14:03:52.506 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003307: Sep 30 01:57:49.441 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003308: Sep 30 02:04:21.684 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003309: Sep 30 13:58:18.596 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003310: Sep 30 14:04:50.847 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003311: Oct  1 01:58:47.745 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003312: Oct  1 02:05:19.988 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003313: Oct  1 13:59:16.897 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003314: Oct  1 14:05:49.148 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003315: Oct  2 01:59:46.114 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003316: Oct  2 02:06:18.361 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up

003317: Oct  2 14:00:15.297 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003318: Oct  2 14:06:47.540 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003319: Oct  3 02:00:44.455 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003320: Oct  3 02:07:16.706 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003321: Oct  3 14:01:13.658 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003322: Oct  3 14:07:45.905 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003323: Oct  4 02:01:42.803 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003324: Oct  4 02:08:15.046 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up

Cause

WISM goes down for 6 minutes every 12 hours.  ( SW : 7.4.100.60 )

003252: Sep 16 13:44:41.643 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003253: Sep 16 13:51:13.894 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003254: Sep 17 01:45:10.816 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003255: Sep 17 01:51:43.063 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003256: Sep 17 13:45:39.981 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003257: Sep 17 13:52:12.224 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up
003258: Sep 18 01:46:09.121 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Down
003259: Sep 18 01:52:41.368 ncc: %WiSM-5-STATE: Controller 1 in slot 2 is Oper-Up

Open caveats in the 7.3.112.0 controller software release

CSCuc52952

Symptom: Error messages indicating duplicate IP addresses of the service port for WiSM2 in an HA setup. Conditions: WiSM-SW1-5-DUP_SRVC_IP Service IP 10.6.1.4 of Controller 35/1 is same as Controller 19/1 WiSM-SW1-5-DUP_SRVC_IP Service IP 10.6.1.4 of Controller 35/1 is same as Controller 19/1. Workaround: None.

Resolution

BUG ID : CSCuc52952

Resolution : Upgrade to 7.4.110.0

75Dup service ip add error messages on 6500 for WiSM2 in HA setup

Symptom:
Error messages on the 6500 indicating dup ip addresses of the service port for wism2's in HA setup

Conditions:
WiSM-SW1-5-DUP_SRVC_IP Service IP 10.6.1.4 of Controller 35/1 is same as Controller 19/1
WiSM-SW1-5-DUP_SRVC_IP Service IP 10.6.1.4 of Controller 35/1 is same as Controller 19/1

Workaround:
n/a

More Info:
Although this bug is marked as Fixed-in 7.4.100.60, in fact its symptoms still occur. Track CSCuf30551 for the real fix.

Known Affected Releases:

• 7.3(101.0)
• 7.5(1.14)
• 7.5(1.64)
• 7.4(100.60)

Known Fixed Releases:

• 8.0(50.33)
• 8.0(50.34)
• 8.0(50.32)
• 7.4(100.1)
• 7.4(100.60)
• 7.5(90.10)
• 7.6(0.4)
• 7.4(100.201)
• 7.4(100.115)
• 7.6(1.42)
• 7.5(102.0)
• 7.4(110.0)
• 7.5(90.152)
• 7.4(100.204)

Duplicate Bug Information

HA:WiSM-5-DUP_SRVC_IP seen on SUP when Serv IP is not updated correctly.

CSCuf30551

Symptom:
It is observed that when the service port IP address is not updated in SUP, the supervisor keeps giving the error:

*Mar 14 08:34:41.966: %WiSM-5-DUP_SRVC_IP: Service IP a.b.c.d of Controller 33/1 is same as Controller 17/1
*Mar 14 08:35:21.966: %WiSM-5-DUP_SRVC_IP: Service IP a.b.c.d of Controller 33/1 is same as Controller 17/1

This error is seen when the service IP of one of the module is not updated

In the WISMs the service port IP addresses are different

Conditions:
Unknown yet

Workaround:
n/a

Update

According to Cisco 5500 Series Wireless Controllers release Notes for Cisco Wireless LAN Controllers and Lightweight Access Points for Release 7.4.100.60
Resolved Caveats
CSCuc52952 - 75 DUP service IP address error messages seen on 6500 for WiSM2 in H setup

Lists the caveats that are resolved in the 7.4.100.60 controller software release.