05-20-2020 08:01 AM
One of our N7K's has stopped logging to logfile. When I checked /var/log is showed to be at 100%. So I cleared the file and soon after, it stops logging.
We've had these 7Ks up and running for 4.3 years (v6.2(14)) and this just started happening recently (last week) on only one chassis of the pair.
Currently, show logging reveals this:
ogging console: enabled (Severity: critical)
Logging monitor: enabled (Severity: debugging)
Logging linecard: enabled (Severity: notifications)
Logging timestamp: Seconds
Logging loopback : disabled
Logging server: enabled
{10.72.45.10}
server severity: debugging
server facility: local7
server VRF: management
Logging origin_id : enabled (Hostname: N7KACORE)
Logging logflash: enabled (Severity: errors)
Logging logfile: enabled
Name - LOGFILE: Severity - errors Size - 4194304
Any idea why this would have started suddenly and how can I fix this?
Thanks.
05-20-2020 08:20 AM
Hi @N3t W0rK3r
How do you test the logging?
Do you see logs with severity 3 being sent to the syslog server, but not on the logfile?
Regards,
Sergiu
05-20-2020 08:26 AM
Hi Sergiu,
I simply do a show logging logfile and see the last log entry is from yesterday. There should be current log entries showing but they do not. This works correctly on the other 7K in the pair.
05-23-2020 11:37 PM
Hi @N3t W0rK3r
Check if the /var/log directory is used up to 100%:
Nexus# show system internal flash Mount-on 1K-blocks Used Available Use% Filesystem / 409600 61104 348496 15 /dev/root /proc 0 0 0 0 proc /sys 0 0 0 0 none /isan 716800 315088 401712 44 none /var 51200 612 50588 2 none /etc 5120 1616 3504 32 none /nxos/tmp 40960 4 40956 1 none /var/log 51200 51200 0 100 none
if yes, then you can look for the file which is occupying most of the space:
Nexus# show system internal dir /var/log/ Nexus# show system internal dir /var/log/external/
One of the file which usually gets high in size is libdt_helper.log.
You can delete it:
Nexus# delete log:libdt_helper.log
Also, what you can do is to clear the logging file:
1. Back up the old messages in the local logging buffer to a file on bootflash with this command: Nexus# show logging log > bootflash:oldlogs.txt 2. Clear the logging buffer with this command: Nexus# clear logging logfile Nexus# 3. You should see a "Logging logfile cleared by user" log generated in logging buffer.
You can then test the logging with this trick:
Nexus# conf t Enter configuration commands, one per line. End with CNTL/Z. Nexus(config)# end Nexus# show logging log Nexus %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on .....
Stay safe,
Sergiu
05-25-2020 07:51 AM
Thank you Sergiu.
I can backup the old messages and clear the log with no problem, but the log fills up again very quickly and I'm not sure why. I really need to get to the bottom of this.
Also, these commands do NOT work for me for some reason:
Nexus# show system internal dir /var/log/ Nexus# show system internal dir /var/log/external/
It doesn't like the dir argument.
Thanks.
John
05-25-2020 08:11 AM
Hello,
"but the log fills up again very quickly" -> this means most likely that the filesystem is full.
"these commands do NOT work for me " -> make sure you run them from default VDC. The command does not work in user VDCs.
Best regards,
Sergiu
05-25-2020 08:28 AM
OK that makes sense now Sergiu.
So from the admin VDC I ran those commands and this is what it came back with. What files are safe to delete??
NEXUS# show system internal dir /var/log/
./ 400
../ 240
m2rib.2.24505.log 0
m2rib.2.8181.log 0
rem_boot_images 115
local_boot_images 93
cmond/ 60
isan.log@ 34
boot.log@ 34
current_isan_src.path 34
current_isan.path 38
current_isan.uri 34
current_ks_src.path 40
current_ks.path 36
current_ks.uri 40
mdstat.boot 48
log_flash_node 9
obfl_flash_node 9
boot_node 9
external/ 400
NEXUS# show system internal dir /var/log/external/
./ 400
../ 400
libfipf.31672 0
libfipf.24483 0
libfipf.24274 0
vdc_2/ 260
libfipf.8158 0
libfipf.7950 0
eobc_port_test_result 3
mgmt_port_test_result 3
bootup_test.log 4536
libfipf.6811 0
libfipf.6805 0
snmp_log 193
libfipf.6266 0
libfipf.6224 0
syslogd_ha_debug 2076
messages 14642
startupdebug 4194304
dmesg@ 31
NEXUS# show system internal dir /var/log/external/vdc_2/
./ 260
../ 400
LOGFILE 0
DHCP 0
syslogd_debugs 35889152
JOHN1 1841357
syslogd_tmpfile.24258 0
spantree-RCP-debug 0
spantree-KIRKWOOD-debug 71624
spantree-debug 1803713
snmp_log 196
messages 4194304
startupdebug 4194304
05-25-2020 09:31 AM
Can you verify if you have any running debugs?
Looks like the files which are eating up the space are:
startupdebug 4194304 syslogd_debugs 35889152 startupdebug 4194304
Can you do a "show debug" both in default vdc and user vdc?
Also you can delete the debug files without a problem.
dir log: delete log:<filename> dir log:/vdc_2/ delete log:/vdc_2/<filename>
Stay safe,
Sergiu
05-25-2020 10:51 AM
OK... so the show debug command reveals the same thing on all 7Ks and all VDCs:
IP QOS MGR Debug Level:
Set to Minor(1)
default for new sessions logging level: 3
Also, I noticed that the log:startupdebug and log:vdc_2/syslog_debugs have current date/time stamps on them, so something is clearly writing to those file on an ongoing basis. As such I am concerned about deleting them. Should I be??
As for log:vdc2/startupdebug it has a much older date/time stamp so I'm not concerend abotu deleting that one.
Thanks.
John
05-27-2020 12:00 AM
You can delete the debug files without a problem. I just tested in my lab and I do not see any issues with the files being deleted.
If there is something which writes to it, it will create the file again.
Let me know if the logging messages appear after you clear the debug files.
Regards,
Sergiu
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide