Introduction
The WAAS system consists of a set of devices called wide area application engines (WAEs) that work together to optimize TCP traffic over your network. The WAEs examine the traffic and use built-in application policies to determine whether to optimize the traffic or allow it to pass through your network unoptimized. WAAS accelerates encrypted Secure Sockets Layer (SSL) and Transport Layer Security (TLS) traffic. The SSL accelerator provides traffic encryption and decryption within WAAS to enable end-to-end traffic optimization. The SSL accelerator also provides secure management of the encryption certificates and keys.
Problem
Device is showing multiple certificate related alarms, also the SSL service is showing inactive. The device may go offline as shown below
WAAS-WC#sh cms info
Device registration information :
Device Id = 3582
Device registered as = WAAS Application Engine
Current WAAS Central Manager = 192.168.1.14
Registered with WAAS Central Manager = 192.168.1.14
Status = Offline
Time of last config-sync = Tue Nov 01 20:19:24 2011
Following alarms can be seen:
Major Alarms:
-------------
Alarm ID Module/Submodule Instance
--------------- -------------------- ---------------
1 cert_near_expiration sslao/CA/VeriSign cert_near_expiration
Nov 01 20:30:14.116 EDT, Processing Error Alarm, #000003, 26000:26005
Certificate 'VeriSign.ca' is near expiration. It is configured in CA 'VeriSign'
1 rtr_unreachable WCCP/svc061/rtr10.15.13.12
Nov 01 21:20:53.153 EDT, Communication Alarm, #000001, 17000:17002
WCCP router 10.15.13.12 unreachable for service id: 61.
Critical Alarms:
----------------
Alarm ID Module/Submodule Instance
--------------- -------------------- ---------------
1 mstore_key_retrieval cms ssl_mstore_key
Nov 01 23:27:53.046 EDT, Processing Error Alarm, #000173, 3000:700008
Unable to generate and/or retrieve SSL managed store encryption key from the Key Manager
2 mstore_key_failure sslao mstore_key_failure
Nov 01 20:39:41.391 EDT, Processing Error Alarm, #000004, 26000:26002
Failed to open SSL store due to failure in getting key from Central Manager
Critical Alarms:
----------------
Alarm ID Module/Submodule Instance
--------------- -------------------- ---------------
1 peering_svc_inactive sslao peering_svc_inactive
Nov 01 00:27:27.460 EDT, Processing Error Alarm, #000238, 26000:26003
SSL AO: peering service is inactive.
WAAS and SSL Certificates
When running in an SSL-protected session, the server and client can authenticate one another and negotiate an encryption algorithm and cryptographic keys before the application protocol transmits or receives its first byte of data. Client certificates provide an additional way to authenticate a client to a server using SSL.
Cisco WAAS supports client authentication and can verify the client before allowing the SSL session with the server to proceed. Client certificate authentication is commonly deployed in highly secure environments, in which message-layer authentication mechanisms using user IDs and passwords, or tokens, are not considered sufficient from a security standpoint.
Description
The above shown situation happens because of some issue with device crypto keys. The device (WAAS) usually has a self-signed certificate and a key. The device key is not exposed to the end user. This key is generated when the device is configured for the very first time. The key is required under the global-setting section. Any problem with the key, like the key getting deleted or overwritten, will cause a number of issues with SSL and device certificates. The "keystore" alarm is due to ssl certificates, it might be expired. The "rtr unreachable" alarm is because the waas can't see the router you have specified. Go the wccp settings of the waas and see whether you have the correct router configured and have the same secret in the waas and the router.
Resolution
Follow the steps to resolve the issue:
1) Backup the running config of your WAE and verify. Make sure you have the details regarding wccp router IP/Static routes etc.(If WCCP is used)
2) Disable WCCP either on the WAE or on the router, either way is fine.
(WAE command : no wccp version 2 (config mode command) Router Command : (no ip wccp 61 and no ip wccp 62))
3) Restore the device to factory default.
* eg: core-wa# restore factory-default preserve basic-config
* The device will have to be reloaded after that.
* Once the device reboots proceed to step 4
4) Configure primary interface
* eg: core-wa(config)#primary-interface GigabitEthernet1/0
* Wait for couple of minutes
5) Enable Enterprise license
* eg: core-wa#license add Enterprise
6) Log into the CM Web GUI. Delete the WAAS from the Central manager.
7) Set the central manager IP address
* eg: core-wa(config)#central-manager address 192.168.1.14
8) Execute cms deregister force on the WAAS.
* eg: core-wa#cms deregister force
9) Enable cms on the WAAS
* eg: core-wa(config)#cms enable
10) Wait for couple of minutes. Execute "sh alarms" to verify if the alarm went away.
11) Enable WCCP. Check if the connections are optimized. Execute "sh stat conn".
Note that step 4 requires the device to be rebooted, so make sure it is done during a maintenance window.
Related Information
Cisco WAAS: Error certificate expired Error system is degraded
WCCP Missing Assignment alarm on Cisco WAAS
Cisco WAAS: device_mgr and actastor_watchdog service has been disabled
