CSCwi78109 | issue with WLC running 17.9.5 ? | feature or system busy

[M.Voortmans]

CLI agent: Error while processing NVGEN command: ...

There is an impact to the management GUI + we don't know if running configuration is saved, due to the face each save action results in "CLI agent: Error while processing NVGEN command" warnings. How sure are we, that our configuration is intact and correct saved!?

When saving configurations, we see following messages:

% Command 'location algorithm' unsuccessful: feature or system busy
% Command 'location expiry' unsuccessful: feature or system busy
% Command 'location notify-threshold' unsuccessful: feature or system busy

....

Also messages in GUI:
example:
Jul 2 12:26:52.996: %CLI_AGENT-1-NVGEN_ERR: Error while processing NVGEN command: cloud-services
Jul 2 12:26:52.997: %CLI_AGENT-1-NVGEN_TRACEBACK: Traceback: 1#b30f30cb912860428c70b1e72fb10927 ....

[marce1000]

  - You could try to reboot the controller and check if the problem goes away ; (if production permits)
     As far as verifying a saved configuration , a trick could be by showing the running config through putty with logging enabled , 
     then save it and copy the configuration from flash to an external repository and compare both , 

 M.

[Leo Laohoo]

---

@M.Voortmans wrote:
CLI agent: Error while processing NVGEN command

---

This error message occurs when the WLC is not adhering to the "80% rule" and is overloaded with AP.

C9800 design is no different and, generally, Cisco recommends limiting the load to around 80% of the AP and client scale.

The 80% scale is just a recommendation to start planning the design and deployment of a catalyst wireless network as this is tested and validated number.

For C9800-80, for example, this means 4800 APs and/or around 50k clients. Does this mean that you cannot have six thousand APs on a single C9800-80? No, not really; Cisco has a lot of successful deployments at maximum scale. The 80% scale is just a recommendation to start planning the design and deployment of a catalyst wireless network.

---

I've had this error message (and subsequent Tracebacks) when our 9800-80 had over 5400 APs.  This same error message has also appeared on our 9800-L with 210 APs.  Uptime for either WLC is >45 days and we've seen this in 17.9.4, 17.9.5 & 17.12.3. 

Do not load the 9800-80 more than 50% or risk a memory leak like below: 

9800-80, 17.12.3, 3050 x APs, Uptime: 6 weeks

[M.Voortmans]

Hi,
Our WLC is not overloaded with AP; We have just 2 WLC’s in HA-State (SSO) and just a limit of AP’s: 15 AP’s, running 17.9.5 : Uptime 6 Weeks, 21 Days.
[cid:image001.png@01DACD24.223EAAE0]

In our situation, it looks like it was started when Smart Licensing was enabled (cloud access).
Lot of alert notifications when opening GUI Dashboard:
[cid:image008.png@01DACD25.85883070]

[Leo Laohoo]

Is this a 9800-L?

Post the complete output to the following commands: 

sh platform resource
sh platform software status con brief

[M.Voortmans]

 

[M.Voortmans]

Hi Leo,

See output, below.

wgtsnlve01#sh platform resource

**State Acronym: H - Healthy, W - Warning, C - Critical

Resource                 Usage                 Max             Warning         Critical        State

----------------------------------------------------------------------------------------------------

RP0 (ok, active)                                                                               H

Control Processor       11.91%                100%            80%             90%             H

  DRAM                   4729MB(30%)           15467MB         88%             93%             H

ESP0(ok, active)                                                                               H

QFP                                                                                           H

  DRAM                   174021KB(9%)          1835008KB       85%             95%             H

  IRAM                   207KB(10%)            2048KB          85%             95%             H

  CPU Utilization        0.00%                 100%            90%             95%             H

wgtsnlve01# sh platform software status con brief

Load Average

Slot  Status  1-Min  5-Min 15-Min

1-RP0 Healthy   1.14   1.18   1.23

2-RP0 Healthy   1.16   1.15   1.16

Memory (kB)

Slot  Status    Total     Used (Pct)     Free (Pct) Committed (Pct)

1-RP0 Healthy 15838740  4843796 (31%) 10994944 (69%)   4922096 (31%)

2-RP0 Healthy 15838740  4500504 (28%) 11338236 (72%)   4790256 (30%)

CPU Utilization

Slot  CPU   User System   Nice   Idle    IRQ   SIRQ IOwait

1-RP0    0   2.40   0.80   0.00  96.79   0.00   0.00   0.00

         1   4.00   1.40   0.00  94.58   0.00   0.00   0.00

         2   2.59   0.79   0.00  96.60   0.00   0.00   0.00

         3   0.70   0.80   0.00  98.50   0.00   0.00   0.00

         4   0.60   0.70   0.00  98.70   0.00   0.00   0.00

         5   3.30   0.80   0.00  95.90   0.00   0.00   0.00

         6   0.50   0.60   0.00  98.90   0.00   0.00   0.00

         7  99.80   0.09   0.09   0.00   0.00   0.00   0.00

         8   0.00   0.00   0.00 100.00   0.00   0.00   0.00

         9   0.10   0.00   0.00  99.89   0.00   0.00   0.00

        10   0.19   0.09   0.00  99.70   0.00   0.00   0.00

2-RP0    0   2.20   3.70   0.00  94.10   0.00   0.00   0.00

         1   2.50   3.50   0.00  93.90   0.00   0.10   0.00

         2   2.30   3.60   0.00  94.10   0.00   0.00   0.00

         3   0.50   0.70   0.00  98.79   0.00   0.00   0.00

         4   0.30   0.50   0.00  99.20   0.00   0.00   0.00

         5   3.20   0.70   0.00  96.10   0.00   0.00   0.00

         6   0.60   0.60   0.00  98.80   0.00   0.00   0.00

         7  99.80   0.20   0.00   0.00   0.00   0.00   0.00

         8   0.10   0.00   0.00  99.90   0.00   0.00   0.00

         9   0.10   0.10   0.00  99.80   0.00   0.00   0.00

        10   0.00   0.00   0.00 100.00   0.00   0.00   0.00

On this site location, we have very small HA-deployment with 2x C9800-L-C-K9 and 15 AP’s (types: CW9164I-E) .

Best regards,

Marc