Duplicated IP error on some clients

[valeriy.nebogin]

Hello,  Dear All.

We have typical deployment with WLC550x (7.0.116.0) and 16 APs (AIR-LAP1242G-E-K9) placed on same site. WLC connected to 3560 (with LAG , and dhcp relay)  , and all wireless clients( Motorolla MC3100 handled PC ) work with same WPA2-PSK SSID. All APs configured as HREAP group and SSID has local switching and auth settings. DHCP server for clients work on Windows 2008r2 failover cluster, APs give addresses from WLC builtin server.

Almost all works great.  But sometimes some clients go insane . After wake up,  they show duplicate IP error and wont connect to nework.  On DHCP server  this IP shown as leased to client mac(without any errors and so).

Client reboot wont resolve this issue.

After reboot client try another dhcp address (after marking dchp decline message) but also without luck, with same error and another IP.

All this looks like client side problem . But when i try debug arp on root switch 3560  i get following situation.

After client wake-up

*Apr 10 18:44:32.773: IP ARP: rcvd req src 10.116.51.59 0023.68cb.a8fc, dst 10.116.51.59 Vlan51

*Apr 10 18:44:32.782: IP ARP: rcvd req src 10.116.51.59 0023.68cb.a812, dst 10.116.51.59 Vlan51

After reboot

*Apr 10 19:16:40.123: IP ARP: rcvd req src 10.116.51.24 0023.68cb.a8fc, dst 10.116.51.24 Vlan51

*Apr 10 19:16:40.131: IP ARP: rcvd req src 10.116.51.24 0023.68c9.a29b, dst 10.116.51.24 Vlan51

*Apr 10 19:16:40.459: IP ARP: rcvd req src 10.116.51.27 0023.68cb.a8fc, dst 10.116.51.27 Vlan51

*Apr 10 19:16:40.467: IP ARP: rcvd req src 10.116.51.27 0023.68cb.a9b6, dst 10.116.51.27 Vlan51

Where  0023.68cb.a8fc problem clent mac and   0023.68cb.a812,

0023.68cb.a9b6, 0023.68c9.a29b    - another full working clients  MACs(with another ip address).

Looks like another client (or ??  ap or controller) send ARP reqest with same IP right after problem client. How this possible ?

I'll be

appreciate for any opinions

and comments!

[valeriy.nebogin]

Looks like problem related with controler  because after wlc reboot problem temporary solved.

*Apr 10 20:43:40.924: IP ARP: rcvd req src 10.116.51.27 0023.68cb.a8fc, dst 10.116.51.27 Vlan51

*Apr 10 20:43:40.949: IP ARP: rcvd req src 10.116.51.27 0023.68cb.a8fc, dst 10.116.51.27 Vlan51

*Apr 10 20:43:41.956: IP ARP: rcvd req src 10.116.51.27 0023.68cb.a8fc, dst 10.116.51.27 Vlan51

And client sucessufuly receive ip 10.116.51.27 .

[Scott Fella · Accepted solution]

I had a client that had the same issues on certain laptops and they had to tweak the power save setting on the device.  Now I'm not a big fan of upgrading code, but 7.0.230.0 has worked better in my deployments than the other previous 7.0.x versions.  One thing you can also try is to enable passive mode in the wlan to see if that helps with your devices.

So to undersand your setup, you have the WLC and AP's in the same site, but instead of running the ap's in local mode, you have them in h-reap.  And all AP's are in one h-reap group and you are not using 802.1x?

[valeriy.nebogin]

Thank for advice.  I will try upgrade wlc.

Yes, we use HREAP localy for survivability (when\if controller died). We dont use 8021x now , but plan it for another service with dedicated SSID (for laptops acess to corporate network).

[Scott Fella]

Just note that h-reap groups only benefit if your doing 802.1x and cckm/okc.

Sent from Cisco Technical Support iPhone App

[valeriy.nebogin]

For some reason, the problem was postponed until now. All this time, the system has worked successfully in the h-reap mode , with a disabled controller. Now while waiting our service contract delivery (to try to update the controller), I began to investigate  the issue in more detail.

In syslog I have following indiication of problem.

Oct 24 12:55:55 10.116.50.11 idp16.wlc5502: *dtlArpTask: Oct 24 12:55:58.492: %APF-6-MOBILE_NOT_EXCLUDED: apf_ms.c:4344 Could not exclude the mobile 00:23:68:cb:a7:f1.

Oct 24 12:55:55 10.116.50.11 idp16.wlc5502: *dtlArpTask: Oct 24 12:55:58.492: %APF-4-REGISTER_IPADD_ON_MSCB_FAILED: apf_foreignap.c:1487 Could not Register IP Add on MSCB. Identity theft alert for IP address. Address:00:23:68:cb:a7:f1

All clients exclussion options disabled on WLC . Learning IP Option also disabled for  WLAN.

Example of  "debug client" for one,    in attach.

        http://pastebin.com/hQAbtWJa

[valeriy.nebogin]

Upgrade WLC to 7.0.235.3 code wont help me to  resolve problem.

After 24 hour of work clients again display warning about duplicated ip`s.

And WLC log contains following error again:

Dec 11 13:49:46 10.116.50.11 idp16.wlc5502: *dtlArpTask: Dec 11 13:49:48.301: %APF-4-REGISTER_IPADD_ON_MSCB_FAILED: apf_foreignap.c:1504 Could not Register IP Add on MSCB. Identity theft alert for IP address. Address:00:23:68:cb:a9:87

I don know what to do next.

[Scott Fella]

You have passive mode enabled on the WLAN. Maybe also disable the session timeout on the wlan. Also increase the the idle timeout to 14400 and see if helps. What is your dhcp lease time on these devices?

Sent from Cisco Technical Support iPhone App

[valeriy.nebogin]

Thanks for suggestions  , Scott

DHCP lease time = 8 hours.

Session timeout already disabled in WLAN - Advanced settings . All exclussion policies also disabled  globaly. 

I will try to increase  Controller > User Idle Timeout (seconds) form default 300 to 14400.

I cant enable "passive client " feature because it requre Multicast mode  , but only Unicat mode  supported with H-REAP

[valeriy.nebogin]

Changing  User Idle Timeout wont change anything. Same problem.

[Scott Fella]

I'm thinking its a client side issue then. I don't think there are any other changes you can make to stabilize the issue.

Sent from Cisco Technical Support iPhone App

[valeriy.nebogin]

Why you think so ?

Reboot of controller resolve problem for some time.

Clients work flawlessly when controller disconnected from network and APs in H-REAP mode. Also same clients (handles PC models) works  in online mode with wlc 4400 without problems.

There are two difference in this deployment:

1. wlc 5500 and 7.0 major release

2. APs in  H-REAP mode and WLAN configured as localy switched

From my point of view problem related with controller  incorrect  behavior(or I think so).

Process of obtaining  ip addresses interrupted by  controller   for some unclear  reasons.

1.Client obtain IP address from DHCP without problem(via dhcp relay on 3750)

2. Client check that no one use it by sending  ARP request "ARP Who has address (leased from DHCP address here)"

3. No response mean that address free.

4. Controller interrupt process. By sending ARP resposne  "ARP  (leased from DHCP address here) used by (MAC of one of  client what used it in past)   "

5. After this client show duplicated ip error.

6. Controller log message

%APF-4-REGISTER_IPADD_ON_MSCB_FAILED:apf_foreignap.c:1504 Could not Register IP Add on MSCB. Identity theft alert for IP address. Address:00:23:68:cb:a9:87

I think this mean controller failed to update some internal table  with new ip-mac pair and prevent client work.

But how disable such controller behavior I dont know.

I disabled controller DHCP proxy , I disabled client ip learning in WLAN settings , I disabled any exclusion policies.

[valeriy.nebogin]

At first I will try to disable  Local switching for WLAN and look to result.

[Scott Fella]

What if you disable the WLAN and enable the WLAN? Does that fix your issues. If you set a static address in the devices does that help?

Sent from Cisco Technical Support iPhone App

[Scott Fella]

You ever find the issue... if not, what do you have your idle timeout set at and how long is your dhcp lease?

Thanks,

Scott

Help out other by using the rating system and marking answered questions as "Answered"