Logging Problem N7K

[N3t W0rK3r]

One of our N7K's has stopped logging to logfile.  When I checked /var/log is showed to be at 100%.  So I cleared the file and soon after, it stops logging.

We've had these 7Ks up and running for 4.3 years (v6.2(14)) and this just started happening recently (last week) on only one chassis of the pair.

Currently, show logging reveals this:

ogging console: enabled (Severity: critical)
Logging monitor: enabled (Severity: debugging)
Logging linecard: enabled (Severity: notifications)
Logging timestamp: Seconds
Logging loopback : disabled
Logging server: enabled
{10.72.45.10}
server severity: debugging
server facility: local7
server VRF: management
Logging origin_id : enabled (Hostname: N7KACORE)
Logging logflash: enabled (Severity: errors)
Logging logfile: enabled
Name - LOGFILE: Severity - errors Size - 4194304

Any idea why this would have started suddenly and how can I fix this?

Thanks.

[Sergiu.Daniluk]

Hi @N3t W0rK3r

How do you test the logging?

Do you see logs with severity 3 being sent to the syslog server, but not on the logfile?

Regards,

Sergiu

[N3t W0rK3r]

Hi Sergiu,

I simply do a show logging logfile and see the last log entry is from yesterday.  There should be current log entries showing but they do not.  This works correctly on the other 7K in the pair.

[Sergiu.Daniluk]

Hi @N3t W0rK3r 

Check if the /var/log directory is used up to 100%:

Nexus# show system internal flash
Mount-on                  1K-blocks      Used   Available   Use%  Filesystem
/                            409600     61104      348496     15   /dev/root
/proc                             0         0           0      0   proc
/sys                              0         0           0      0   none
/isan                        716800    315088      401712     44   none
/var                          51200       612       50588      2   none
/etc                           5120      1616        3504     32   none
/nxos/tmp                     40960         4       40956      1   none
/var/log                      51200     51200           0    100   none

if yes, then you can look for the file which is occupying most of the space:

Nexus# show system internal dir /var/log/
Nexus# show system internal dir /var/log/external/

One of the file which usually gets high in size is libdt_helper.log.

You can delete it:

Nexus# delete log:libdt_helper.log

Also, what you can do is to clear the logging file:

1. Back up the old messages in the local logging buffer to a file on bootflash with this command:

Nexus# show logging log > bootflash:oldlogs.txt

2. Clear the logging buffer with this command:

Nexus# clear logging logfile
Nexus#

3. You should see a "Logging logfile cleared by user" log generated in logging buffer.

You can then test the logging with this trick:

Nexus# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
Nexus(config)# end
Nexus# show logging log
Nexus %VSHD-5-VSHD_SYSLOG_CONFIG_I:
 Configured from vty by admin on .....

Stay safe,

Sergiu

[N3t W0rK3r]

Thank you Sergiu.

I can backup the old messages and clear the log with no problem, but the log fills up again very quickly and I'm not sure why. I really need to get to the bottom of this.

Also, these commands do NOT work for me for some reason:

Nexus# show system internal dir /var/log/
Nexus# show system internal dir /var/log/external/

It doesn't like the dir argument.

Thanks.

John

[Sergiu.Daniluk]

Hello,

"but the log fills up again very quickly" -> this means most likely that the filesystem is full.

"these commands do NOT work for me " -> make sure you run them from default VDC. The command does not work in user VDCs.

Best regards,

Sergiu

[N3t W0rK3r]

OK that makes sense now Sergiu.

So from the admin VDC I ran those commands and this is what it came back with.  What files are safe to delete??

NEXUS# show system internal dir /var/log/
./ 400
../ 240
m2rib.2.24505.log 0
m2rib.2.8181.log 0
rem_boot_images 115
local_boot_images 93
cmond/ 60
isan.log@ 34
boot.log@ 34
current_isan_src.path 34
current_isan.path 38
current_isan.uri 34
current_ks_src.path 40
current_ks.path 36
current_ks.uri 40
mdstat.boot 48
log_flash_node 9
obfl_flash_node 9
boot_node 9
external/ 400

NEXUS# show system internal dir /var/log/external/
./ 400
../ 400
libfipf.31672 0
libfipf.24483 0
libfipf.24274 0
vdc_2/ 260
libfipf.8158 0
libfipf.7950 0
eobc_port_test_result 3
mgmt_port_test_result 3
bootup_test.log 4536
libfipf.6811 0
libfipf.6805 0
snmp_log 193
libfipf.6266 0
libfipf.6224 0
syslogd_ha_debug 2076
messages 14642
startupdebug 4194304
dmesg@ 31

NEXUS# show system internal dir /var/log/external/vdc_2/
./ 260
../ 400
LOGFILE 0
DHCP 0
syslogd_debugs 35889152
JOHN1 1841357
syslogd_tmpfile.24258 0
spantree-RCP-debug 0
spantree-KIRKWOOD-debug 71624
spantree-debug 1803713
snmp_log 196
messages 4194304
startupdebug 4194304

[Sergiu.Daniluk]

Can you verify if you have any running debugs?

Looks like the files which are eating up the space are:

startupdebug 4194304
syslogd_debugs 35889152
startupdebug 4194304

Can you do a "show debug" both in default vdc and user vdc?

Also you can delete the debug files without a problem. 

dir log:
delete log:<filename>
dir log:/vdc_2/
delete log:/vdc_2/<filename>

Stay safe,

Sergiu

[N3t W0rK3r]

OK... so the show debug command reveals the same thing on all 7Ks and all VDCs:

IP QOS MGR Debug Level:
Set to Minor(1)
default for new sessions logging level: 3

Also, I noticed that the log:startupdebug and log:vdc_2/syslog_debugs have current date/time stamps on them, so something is clearly writing to those file on an ongoing basis.  As such I am concerned about deleting them. Should I be??

As for log:vdc2/startupdebug it has a much older date/time stamp so I'm not concerend abotu deleting that one.

Thanks.

John

[Sergiu.Daniluk]

You can delete the debug files without a problem. I just tested in my lab and I do not see any issues with the files being deleted.

If there is something which writes to it, it will create the file again.

Let me know if the logging messages appear after you clear the debug files.

Regards,

Sergiu