NGNIX issue on an APIC

[Luca.M]

Hi all,

We have a cluster of three controllers Cisco APIC L2 (software version 2.2(1n)) in our datacenter. One of these devices became unreachable via browser and was not displayed as active and available in the 'controllers' section. We noticed that a fault code “F0321 - data layer partially diverged” was generated. We also tried to log via CLI (OOB management and console) but the system refused the connection giving a response of incorrect user/password . Connecting via console we saw furthermore the message /mgmt/usr/bin/loginshell: Too many open files in system. So we decided to reboot the apic. The connectivity problems is (temporary) solved, and we can easily connect to the device via browser and CLI.
We noticed that a log process due to ngnix was running at 100% and occupying 50G

/dev/mapper/vg_ifc0-dmecores      50G   50G     0 100% /var/log/dme/core

We check the rotate frequency and it is the same as the other APICs.
Do you have any suggests?
thanks

Luca

[Manuel Velasco]

Hi Moretti,

There a couple known issues regarding the auto rotate of the logs like the one below.  In your case I would recommend to open a case with Cisco TAC to take a look at this partition and see what type of files are filling it up.

Files Named *.log Do Not Autorotate Properly

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvd36006/?reffering_site=dumpcr

[Luca.M]

Hi Velasco,

many thanks for your reply, I was not able to find a related bug.

Have a nice day

Luca