This original post is edited to correct some factual errors, but the error remains and I still need help with my quaestion at the end!!
=======
Hello!
I have (and my customer) a rather "cute" problem ... Two months ago I installed a WLC5508 with sw r7.0.116.0 and we installed 8 AP1142 as a start. Everything came up and running and we were happy....
10 days the customer calls and says that the wlc logs error messages that says there is an IP-conflict with the management-interface. The conflict source is its own ip-adress!!! He reboots the WLC but the error message keeps coming. After a short while he notices that there are No APs on the WLC...!!!...
After some discussion I suggested that he should disconnect one of the two gig-ports of the WLC (LAG was enabled and in use). He did so and rebooted the WLC for good measure ... The IP-conflict disappeared but still no APs.....
In the DHCP-server (MS Win2003 server), in the AP-scope we also see that the "leased adresses" fills up with "BAD_IP_ADDRESS" (or the sort ... )
The setup is like this:
- WLC5508 running r7.0.116.0
- WLC has LAG activated and is connected to two different Cat3750G that are stacked
- The management if of the WLC is on the same subnet as the AP1142s, no other hosts on this subnet
- WLC management interface on own subnet as the only host (+ def gwy...)
- APs on their own subnet, no other hosts
- When the APs gets a IP-adress it is possible to ping them from the WLC!
- All VLANs/interfaces on the WLC is tagged, ie a "pure" trunk between WLC and 3750-stack
- Option 43 configured on the AP-scope
- All APs is connected via Power-injector (PWR-INJ4)
WLC5508 => LAG => 2x Cat3750G => TRUNK => Cat2960 with 1x AP1142 => TRUNK => Cat2960 with 6x AP1142
What we have done so far:
- Made sure the port-channel on the 3750-stack was configured: port-channel load-balance src-dest-ip
- Physically disonnected on port of the WLC
- Do "shutdown"on port gig-ports on the 3750-stack and "no shut" on only the one that is still physically connected to WLC
- Do "shutdown" on all AP-ports in the 2960-switches. Clear the lease-pool of the DHCP-server, Deactivate scope, Reactivate Scope and finally "no shut" on all AP-ports
Nothing helped, still no APs on the WLC.
I the connected my PC to the consloe of an AP that was easily accessible and powercycled it. The output can be seen below, the most peculiar was the line : " *Mar 1 00:15:53.351: %CAPWAP-3-ERRORLOG: Invalid event 38 & state 2 combination"
Powering up and booting of AP1142:
=============================
using eeprom values
WRDTR,CLKTR: 0x88000800 0x40000000
RQDC ,RFDC : 0x80000037 0x00000211
ddr init done
Running Normal Memtest...
Passed.
IOS Bootloader - Starting system.
FLASH CHIP: Numonyx P33
Checking for Over Erased blocks
......................................................................................................................................................................................................................................................
Xmodem file system is available.
DDR values used from system serial eeprom.
WRDTR,CLKTR: 0x88000800, 0x40000000
RQDC, RFDC : 0x80000037, 0x00000211
PCIE0: link is up.
PCIE0: VC0 is active
PCIE1: link is up.
PCIE1: VC0 is active
PCIEx: initialization done
flashfs[0]: 28 files, 8 directories
flashfs[0]: 0 orphaned files, 0 orphaned directories
flashfs[0]: Total bytes: 32385024
flashfs[0]: Bytes used: 7346176
flashfs[0]: Bytes available: 25038848
flashfs[0]: flashfs fsck took 20 seconds.
Reading cookie from system serial eeprom...Done
Base Ethernet MAC address: e8:b7:48:25:b7:92
Ethernet speed is 1000 Mb - FULL duplex
Loading "flash:/c1140-k9w8-mx.124-23c.JA2/c1140-k9w8-mx.124-23c.JA2"...################################################################################################################################################################################################################################################################################################################################################################################################################################################################
File "flash:/c1140-k9w8-mx.124-23c.JA2/c1140-k9w8-mx.124-23c.JA2" uncompressed and installed, entry point: 0x4000
executing...
enet halted
Restricted Rights Legend
Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.
cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706
Cisco IOS Software, C1140 Software (C1140-K9W8-M), Version 12.4(23c)JA2, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Wed 13-Apr-11 12:50 by prod_rel_team
Proceeding with system init
Proceeding to unmask interrupts
Initializing flashfs...
FLASH CHIP: Numonyx P33
Checking for Over Erased blocks
......................................................................................................................................................................................................................................................
flashfs[1]: 28 files, 8 directories
flashfs[1]: 0 orphaned files, 0 orphaned directories
flashfs[1]: Total bytes: 32126976
flashfs[1]: Bytes used: 7346176
flashfs[1]: Bytes available: 24780800
flashfs[1]: flashfs fsck took 6 seconds.
flashfs[1]: Initialization complete.
flashfs[2]: 0 files, 1 directories
flashfs[2]: 0 orphaned files, 0 orphaned directories
flashfs[2]: Total bytes: 11999232
flashfs[2]: Bytes used: 1024
flashfs[2]: Bytes available: 11998208
flashfs[2]: flashfs fsck took 0 seconds.
flashfs[2]: Initialization complete....done Initializing flashfs.
Ethernet speed is 1000 Mb - FULL duplex
Radio0 present 8363 8000 90020000 0 90030000 B
Radio1 present 8363 8000 98020000 0 98030000 0
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
export@cisco.com.
cisco AIR-LAP1142N-E-K9 (PowerPC405ex) processor (revision A0) with 98294K/32768K bytes of memory.
Processor board ID FCZ1519W2YU
PowerPC405ex CPU at 586Mhz, revision number 0x147E
Last reset from watchdog timer expired
LWAPP image version 7.0.116.0
1 Gigabit Ethernet interface
2 802.11 Radio(s)
32K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address: E8:B7:48:25:B7:92
Part Number : 73-12836-01
PCA Assembly Number : 800-33767-01
PCA Revision Number : A0
PCB Serial Number : FOC15161W2S
Top Assembly Part Number : 800-33775-01
Top Assembly Serial Number : FCZ1519W2YU
Top Revision Number : A0
Product/Model Number : AIR-LAP1142N-E-K9
% Please define a domain-name first.
Press RETURN to get started!
*Mar 1 00:00:08.186: %SOAP_FIPS-2-SELF_TEST_IOS_SUCCESS: IOS crypto FIPS self test passed
*Mar 1 00:00:08.196: *** CRASH_LOG = YES
Security Core found.
Base Ethernet MAC address: E8:B7:48:25:B7:92
*Mar 1 00:00:09.753: %SOAP_FIPS-2-SELF_TEST_RAD_SUCCESS: RADIO crypto FIPS self test passed on interface Dot11Radio 0
*Mar 1 00:00:10.322: %SOAP_FIPS-2-SELF_TEST_RAD_SUCCESS: RADIO crypto FIPS self test passed on interface Dot11Radio 1
*Mar 1 00:00:10.351: %LWAPP-3-CLIENTEVENTLOG: Read and initialized AP event log (contains, 8 messages)
*Mar 1 00:00:10.374: status of voice_diag_test from WLC is false
*Mar 1 00:00:11.417: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to up
*Mar 1 00:00:12.513: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to up
*Mar 1 00:00:12.549: %SYS-5-RESTART: System restarted --
Cisco IOS Software, C1140 Software (C1140-K9W8-M), Version 12.4(23c)JA2, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Wed 13-Apr-11 12:50 by prod_rel_team
*Mar 1 00:00:12.549: %SNMP-5-COLDSTART: SNMP agent on host APe8b7.4825.b792 is undergoing a cold start
*Mar 1 00:00:12.558: %CDP_PD-4-POWER_OK: Full power - AC_ADAPTOR inline power source
*Mar 1 00:13:24.043: %CAPWAP-5-CHANGED: CAPWAP changed state to DISCOVERY
*Mar 1 00:13:24.113: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset
*Mar 1 00:13:24.121: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
*Mar 1 00:13:24.129: %LINK-3-UPDOWN: Interface Dot11Radio0, changed state to up
*Mar 1 00:13:24.145: %LINK-3-UPDOWN: Interface Dot11Radio0, changed state to down
*Mar 1 00:13:25.113: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down
*Mar 1 00:13:25.121: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
*Mar 1 00:13:25.635: %DOT11-2-TX_PWR_TIMO: Interface Dot11Radio1, Timed out waiting for fw to clear txp_pndg flag
*Mar 1 00:13:25.833: %SSH-5-ENABLED: SSH 2.0 has been enabled
*Mar 1 00:13:25.966: Interface Dot11Radio1 transmit power indices set/txp_pndg cleared
*Mar 1 00:13:25.989: %LINK-3-UPDOWN: Interface Dot11Radio1, changed state to up
*Mar 1 00:13:26.843: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up
*Mar 1 00:13:32.229: %DHCP-6-ADDRESS_ASSIGN: Interface GigabitEthernet0 assigned DHCP address 172.22.130.126, mask 255.255.255.0, hostname APe8b7.4825.b792
*Mar 1 00:13:42.900: status of voice_diag_test from WLC is false
*Mar 1 00:13:42.957: Logging LWAPP message to 255.255.255.255.
User Access Verification
Username:
Translating "CISCO-CAPWAP-CONTROLLER.sodra.com"...domain server (172.21.51.23) [OK]
*Mar 1 00:13:52.990: %CAPWAP-5-DHCP_OPTION_43: Controller address 172.21.135.30 obtained through DHCP
% Username: timeout expired!
APe8b7.4825.b792 con0 is now available
Press RETURN to get started.
User Access Verification
Username:
*Mar 1 00:14:43.992: %CAPWAP-3-DHCP_RENEW: Could not discover WLC using DHCP IP. Renewing DHCP IP.
*Mar 1 00:14:44.992: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 255.255.255.255 started - CLI initiated
Username:
Username:
Not in Bound state.Username:
APe8b7.4825.b792 con0 is now available
Press RETURN to get started.
User Access Verification
Username: Cisc
*Mar 1 00:14:56.855: %CAPWAP-3-ERRORLOG: Invalid event 38 & state 2 combination.o
*Mar 1 00:14:56.875: %DHCP-6-ADDRESS_ASSIGN: Interface GigabitEthernet0 assigned DHCP address 172.22.130.126, mask 255.255.255.0, hostname APe8b7.4825.b792
Translating "CISCO-CAPWAP-CONTROLLER.sodra.com"...domain server (172.21.51.23) [OK]
*Mar 1 00:14:59.498: %CAPWAP-5-DHCP_OPTION_43: Controller address 172.21.135.30 obtained through DHCP
% Username: timeout expired!
APe8b7.4825.b792 con0 is now available
Press RETURN to get started.
User Access Verification
Username: Cisco
Password:
APe8b7.4825.b792>en
Password:
*Mar 1 00:15:40.498: %CAPWAP-3-DHCP_RENEW: Could not discover WLC using DHCP IP. Renewing DHCP IP.
Not in Bound state.
APe8b7.4825.b792#
APe8b7.4825.b792#
APe8b7.4825.b792#
APe8b7.4825.b792#
*Mar 1 00:15:53.351: %CAPWAP-3-ERRORLOG: Invalid event 38 & state 2 combination.
*Mar 1 00:15:53.376: %DHCP-6-ADDRESS_ASSIGN: Interface GigabitEthernet0 assigned DHCP address 172.22.130.126, mask 255.255.255.0, hostname APe8b7.4825.b792
Translating "CISCO-CAPWAP-CONTROLLER.sodra.com"...domain server (172.21.51.23) [OK]
*Mar 1 00:15:56.000: %CAPWAP-5-DHCP_OPTION_43: Controller address 172.21.135.30 obtained through DHCP
APe8b7.4825.b792#
APe8b7.4825.b792#
APe8b7.4825.b792#
===============================
My apologies for the rather massive amount of text.........
Questions:
1. It worked initially but after the ip-address confilct and reboot of WLC no APs associate to the WLC. Why??
2. The error message on the AP console: "*Mar 1 00:15:53.351: %CAPWAP-3-ERRORLOG: Invalid event 38 & state 2 combination" What does this mean?
3. Since the AP gets an IP-address an it is possible to ping WLC -> AP, is there some freakish ogre in the Cat2960 that eats up the CAPWAP-packets??
Anybody who has a solution?????
Best Regards
Göran Blomqvist
TDC, Sweden
