cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
186
Views
20
Helpful
9
Replies
Highlighted
Participant

Logging Problem N7K

One of our N7K's has stopped logging to logfile.  When I checked /var/log is showed to be at 100%.  So I cleared the file and soon after, it stops logging.

We've had these 7Ks up and running for 4.3 years (v6.2(14)) and this just started happening recently (last week) on only one chassis of the pair.

Currently, show logging reveals this:

ogging console: enabled (Severity: critical)
Logging monitor: enabled (Severity: debugging)
Logging linecard: enabled (Severity: notifications)
Logging timestamp: Seconds
Logging loopback : disabled
Logging server: enabled
{10.72.45.10}
server severity: debugging
server facility: local7
server VRF: management
Logging origin_id : enabled (Hostname: N7KACORE)
Logging logflash: enabled (Severity: errors)
Logging logfile: enabled
Name - LOGFILE: Severity - errors Size - 4194304

 

Any idea why this would have started suddenly and how can I fix this?

 

Thanks.

9 REPLIES 9
Highlighted
Rising star

Re: Logging Problem N7K

Hi @N3t W0rK3r

How do you test the logging?

Do you see logs with severity 3 being sent to the syslog server, but not on the logfile?

 

Regards,

Sergiu

Highlighted
Participant

Re: Logging Problem N7K

Hi Sergiu,

 

I simply do a show logging logfile and see the last log entry is from yesterday.  There should be current log entries showing but they do not.  This works correctly on the other 7K in the pair.

Highlighted
Rising star

Re: Logging Problem N7K

Hi @N3t W0rK3r 

Check if the /var/log directory is used up to 100%:

Nexus# show system internal flash
Mount-on                  1K-blocks      Used   Available   Use%  Filesystem
/                            409600     61104      348496     15   /dev/root
/proc                             0         0           0      0   proc
/sys                              0         0           0      0   none
/isan                        716800    315088      401712     44   none
/var                          51200       612       50588      2   none
/etc                           5120      1616        3504     32   none
/nxos/tmp                     40960         4       40956      1   none
/var/log                      51200     51200           0    100   none

if yes, then you can look for the file which is occupying most of the space:

Nexus# show system internal dir /var/log/
Nexus# show system internal dir /var/log/external/

One of the file which usually gets high in size is libdt_helper.log.

You can delete it:

Nexus# delete log:libdt_helper.log

Also, what you can do is to clear the logging file:

1. Back up the old messages in the local logging buffer to a file on bootflash with this command:

Nexus# show logging log > bootflash:oldlogs.txt

2. Clear the logging buffer with this command:

Nexus# clear logging logfile
Nexus#

3. You should see a "Logging logfile cleared by user" log generated in logging buffer.

You can then test the logging with this trick:

Nexus# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
Nexus(config)# end
Nexus# show logging log
Nexus %VSHD-5-VSHD_SYSLOG_CONFIG_I:
 Configured from vty by admin on .....

Stay safe,

Sergiu

Highlighted
Participant

Re: Logging Problem N7K

Thank you Sergiu.

I can backup the old messages and clear the log with no problem, but the log fills up again very quickly and I'm not sure why. I really need to get to the bottom of this.

 

Also, these commands do NOT work for me for some reason:

Nexus# show system internal dir /var/log/
Nexus# show system internal dir /var/log/external/

It doesn't like the dir argument.

 

Thanks.

 

John

Highlighted
Rising star

Re: Logging Problem N7K

Hello,

"but the log fills up again very quickly" -> this means most likely that the filesystem is full.

"these commands do NOT work for me " -> make sure you run them from default VDC. The command does not work in user VDCs.

 

Best regards,

Sergiu

 

Highlighted
Participant

Re: Logging Problem N7K

OK that makes sense now Sergiu.

So from the admin VDC I ran those commands and this is what it came back with.  What files are safe to delete??

 

NEXUS# show system internal dir /var/log/
./ 400
../ 240
m2rib.2.24505.log 0
m2rib.2.8181.log 0
rem_boot_images 115
local_boot_images 93
cmond/ 60
isan.log@ 34
boot.log@ 34
current_isan_src.path 34
current_isan.path 38
current_isan.uri 34
current_ks_src.path 40
current_ks.path 36
current_ks.uri 40
mdstat.boot 48
log_flash_node 9
obfl_flash_node 9
boot_node 9
external/ 400

 

NEXUS# show system internal dir /var/log/external/
./ 400
../ 400
libfipf.31672 0
libfipf.24483 0
libfipf.24274 0
vdc_2/ 260
libfipf.8158 0
libfipf.7950 0
eobc_port_test_result 3
mgmt_port_test_result 3
bootup_test.log 4536
libfipf.6811 0
libfipf.6805 0
snmp_log 193
libfipf.6266 0
libfipf.6224 0
syslogd_ha_debug 2076
messages 14642
startupdebug 4194304
dmesg@ 31

 

NEXUS# show system internal dir /var/log/external/vdc_2/
./ 260
../ 400
LOGFILE 0
DHCP 0
syslogd_debugs 35889152
JOHN1 1841357
syslogd_tmpfile.24258 0
spantree-RCP-debug 0
spantree-KIRKWOOD-debug 71624
spantree-debug 1803713
snmp_log 196
messages 4194304
startupdebug 4194304

 

Highlighted
Rising star

Re: Logging Problem N7K

Can you verify if you have any running debugs?

Looks like the files which are eating up the space are:

startupdebug 4194304
syslogd_debugs 35889152
startupdebug 4194304

Can you do a "show debug" both in default vdc and user vdc?

Also you can delete the debug files without a problem. 

 

dir log:
delete log:<filename>
dir log:/vdc_2/
delete log:/vdc_2/<filename>

Stay safe,

Sergiu

Highlighted
Participant

Re: Logging Problem N7K

OK... so the show debug command reveals the same thing on all 7Ks and all VDCs:


IP QOS MGR Debug Level:
Set to Minor(1)
default for new sessions logging level: 3

 

Also, I noticed that the log:startupdebug and log:vdc_2/syslog_debugs have current date/time stamps on them, so something is clearly writing to those file on an ongoing basis.  As such I am concerned about deleting them. Should I be??


As for log:vdc2/startupdebug it has a much older date/time stamp so I'm not concerend abotu deleting that one.

 

Thanks.

 

John

Highlighted
Rising star

Re: Logging Problem N7K

You can delete the debug files without a problem. I just tested in my lab and I do not see any issues with the files being deleted.

If there is something which writes to it, it will create the file again.

 

Let me know if the logging messages appear after you clear the debug files.

 

Regards,

Sergiu

CreatePlease to create content
Content for Community-Ad

Cisco COVID-19 Survey