bugID CSCth56224 Clients With a Static IP Address May Get Stuck in DHCP_REQD State

[ewood2624]

We got hit with this bug where our statically assigned mobile carts would not transition to the run state in version 7.0.98.0.  Has anyone else seen this?  Here is what we are seeing on our controller debugs.  We cannot change these to DHCP because they are legacy devices incapable of using DHCP. 

*Dot1x_NW_MsgTask_0: Oct 05 17:00:51.270: 00:40:17:8d:61:5f Received EAPOL-key in PTKINITNEGOTIATING state (message 4) from mobile 00:40:17:8d:61:5f

*Dot1x_NW_MsgTask_0: Oct 05 17:00:51.270: 00:40:17:8d:61:5f apfMs1xStateInc

*Dot1x_NW_MsgTask_0: Oct 05 17:00:51.270: 00:40:17:8d:61:5f 0.0.0.0 8021X_REQD (3) Change state to L2AUTHCOMPLETE (4) last state L2AUTHCOMPLETE (4)

*Dot1x_NW_MsgTask_0: Oct 05 17:00:51.271: 00:40:17:8d:61:5f 0.0.0.0 L2AUTHCOMPLETE (4) Plumbed mobile LWAPP rule on AP 00:19:2f:dc:10:f0 vapId 2 apVapId 2

*Dot1x_NW_MsgTask_0: Oct 05 17:00:51.271: 00:40:17:8d:61:5f 0.0.0.0 L2AUTHCOMPLETE (4) Change state to DHCP_REQD (7) last state DHCP_REQD (7)

*Dot1x_NW_MsgTask_0: Oct 05 17:00:51.271: 00:40:17:8d:61:5f 0.0.0.0 DHCP_REQD (7) pemAdvanceState2 4501, Adding TMP rule

*Dot1x_NW_MsgTask_0: Oct 05 05:38:11.271: 00:40:17:8d:61:5f 0.0.0.0 DHCP_REQD (7) Adding Fast Path rule

  type = Airespace AP - Learn IP address

  on AP 00:19:2f:dc:10:f0, slot 0, interface = 29, QOS = 1

  ACL Id = 255, Jumbo F

*Dot1x_NW_MsgTask_0: Oct 05 17:00:51.271: 00:40:17:8d:61:5f 0.0.0.0 DHCP_REQD (7) Fast Path rule (contd...) 802.1P = 0, DSCP = 0, TokenID = 5006  IPv6 Vlan = 216, IPv6 intf id = 11

*Dot1x_NW_MsgTask_0: Oct 05 17:00:51.272: 00:40:17:8d:61:5f 0.0.0.0 DHCP_REQD (7) Successfully plumbed mobile rule (ACL ID 255)

*Dot1x_NW_MsgTask_0: Oct 05 17:00:51.272: 00:40:17:8d:61:5f Stopping retransmission timer for mobile 00:40:17:8d:61:5f

*Dot1x_NW_MsgTask_0: Oct 05 17:00:51.272: 00:40:17:8d:61:5f Key exchange done, data packets from mobile 00:40:17:8d:61:5f should be forwarded shortly

*Dot1x_NW_MsgTask_0: Oct 05 17:00:51.272: 00:40:17:8d:61:5f Sending EAPOL-Key Message to mobile 00:40:17:8d:61:5f

   state PTKINITDONE (message 5 - group), replay counter 00.00.00.00.00.00.00.02

*spamReceiveTask: Oct 05 17:00:51.273: 00:40:17:8d:61:5f Sent EAPOL-Key M5 for mobile 00:40:17:8d:61:5f

*pemReceiveTask: Oct 05 17:00:51.278: 00:40:17:8d:61:5f 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0

*pemReceiveTask: Oct 05 17:00:51.278: 00:40:17:8d:61:5f Sent an XID frame

*apfReceiveTask: Oct 05 17:00:53.137: 00:40:17:8d:61:5f 0.0.0.0 DHCP_REQD (7) State Update from Mobility-Incomplete to Mobility-Complete, mobility role=Local, client state=APF_MS_STATE_ASSOCIATED

*apfReceiveTask: Oct 05 17:00:53.138: 00:40:17:8d:61:5f 0.0.0.0 DHCP_REQD (7) pemAdvanceState2 4182, Adding TMP rule

*apfReceiveTask: Oct 05 17:00:53.006: 00:40:17:8d:61:5f 0.0.0.0 DHCP_REQD (7) Replacing Fast Path rule

  type = Airespace AP - Learn IP address

  on AP 00:19:2f:dc:10:f0, slot 0, interface = 29, QOS = 1

  ACL Id = 255, Jumb

*apfReceiveTask: Oct 05 17:00:53.138: 00:40:17:8d:61:5f 0.0.0.0 DHCP_REQD (7) Fast Path rule (contd...) 802.1P = 0, DSCP = 0, TokenID = 5006  IPv6 Vlan = 216, IPv6 intf id = 11

*apfReceiveTask: Oct 05 17:00:53.138: 00:40:17:8d:61:5f 0.0.0.0 DHCP_REQD (7) Successfully plumbed mobile rule (ACL ID 255)

*pemReceiveTask: Oct 05 17:00:53.143: 00:40:17:8d:61:5f 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0

*pemReceiveTask: Oct 05 17:00:53.143: 00:40:17:8d:61:5f Sent an XID frame

*osapiBsnTimer: Oct 05 17:00:56.137: 00:40:17:8d:61:5f 802.1x 'timeoutEvt' Timer expired for station 00:40:17:8d:61:5f

*dot1xMsgTask: Oct 05 17:00:56.137: 00:40:17:8d:61:5f Retransmit 1 of EAPOL-Key M5 (length 131) for mobile 00:40:17:8d:61:5f

*Dot1x_NW_MsgTask_0: Oct 05 17:00:56.144: 00:40:17:8d:61:5f Received EAPOL-Key from mobile 00:40:17:8d:61:5f

*Dot1x_NW_MsgTask_0: Oct 05 17:00:56.144: 00:40:17:8d:61:5f Received EAPOL-key in REKEYNEGOTIATING state (message 6) from mobile 00:40:17:8d:61:5f

*Dot1x_NW_MsgTask_0: Oct 05 17:00:56.144: 00:40:17:8d:61:5f Stopping retransmission timer for mobile 00:40:17:8d:61:5f

*apfReceiveTask: Oct 05 17:02:51.138: 00:40:17:8d:61:5f 0.0.0.0 DHCP_REQD (7) DHCP Policy timeout

*apfReceiveTask: Oct 05 17:02:51.138: 00:40:17:8d:61:5f 0.0.0.0 DHCP_REQD (7) Pem timed out, Try to delete client in 10 secs.

*apfReceiveTask: Oct 05 17:02:51.138: 00:40:17:8d:61:5f Scheduling deletion of Mobile Station:  (callerId: 12) in 10 seconds

*osapiBsnTimer: Oct 05 17:03:01.139: 00:40:17:8d:61:5f apfMsExpireCallback (apf_ms.c:599) Expiring Mobile!

*apfReceiveTask: Oct 05 17:03:01.139: 00:40:17:8d:61:5f apfMsExpireMobileStation (apf_ms.c:4888) Changing state for mobile 00:40:17:8d:61:5f on AP 00:19:2f:dc:10:f0 from Associated to Disassociated

[Surendra BG]

The workaround is to downgrade the WLC software to 6.0.199.4.. the 6.0.199.4 Release notes does not specify this bug.

Regards

Surendra

[ewood2624]

Is there any word on the next 7.0 code release?

[Surendra BG]

we are working on it... u can open up a ticket with us (TAC) and one of the TAC engineer and the developemt team may help you out..

Regards

Surendra

[ewood2624]

We have a tac case open right now and our tac engineer didn't find this bug. 

[Surendra BG]

Then no problem at all.. the engineer will help you out in resolving the issue to the core...

Regards

Surendra

[Kayle Miller]

This is a known bug and I encountered it at 3 customer sites in the last month, all 3 sites were medical facilities, although I have only seen it with certain devices it is an issue that is in the wild. I worked with a TAC Engineer and they didn't make any progress with the cause of the issue. We did find an alternative work-around that for my clients was acceptable and worked well.

     Our work-around/resolution was to stay on the 7.0.98.0 code release but make the end client devices DHCP and we created a new DHCP scope and created reservations for the client devices so that they always get the same IP address. Pretty much the same as static but just a little more server side work, but it has been running for them for almost 2 weeks now with no issues.

     The only other thing I have to offer would be have you left a client experiencing this powered off for like 2 days and then attempted to reconnect?? My experience has been that this allows the client to work again, but then in a few days it goes off-line again..  The other odd thing with this was that it seemed to only happen to their sites that were running WiSM based controllers, we couldn't recreate it on 4400's or 5508's

You aren't by chance using Silex SX-500 Serial to wireless terminal servers are you?

Hope this helps... Please rate useful posts.

Thanks,

Kayle

[ewood2624]

That is the exact same issue with the exact same type of device Silex SX500 for an EKG cart.  We've rolled back our test wism to 6.0.199.4 and haven't seen a problem and are probably going to do the same with the other controllers.  Maybe since it only seems an issue with the Silex SX500, there might be some sort of drivers or firmware that may need to be updated?

[Kayle Miller · Accepted solution]

in terms of the firmware the Silex SX-500 device does have a newer firmware available for it but it wasn't certified by GE Medical for their EKG carts therefore running it would have further reaching impacts with the GE support group. If rolling back to 6.0.199.4 is doable in your environment then that is the most supported solution, but in my cases it was too great an impact to roll-back so the decision was made to do the DHCP option instead.

Please rate useful posts.

Thanks,

Kayle

[mdallum]

We are having the same issues with some medical deivces we have as well. We use  a WISM but i have a 4404 that is used to anchor devices from remote locations. I was able to anchor this WLAN to the 4404 and everything seems to be working now. I would like to move them back to the WISM asap. I opened a TAC case this morning to see if they have any updates on this issue.

[ewood2624]

I had heard that there is a patch to 7.0.98.0 developed, but no word on the release date.

[mdallum]

I have an engineer release of code from TAC I will install

tonight. I will update after I test it for a few days.

[Lhausklsg]

Did anybody see this bug behavior on the current 8.2.100?

I have a client, that works fine when I configure it in DHCP mode, but when I assign a static address it gets stuck in DHCP_REQD state...