Can't edit/add Syslog server on Nexus 3k

[Andrew White]

Hello,

I need to remove an old syslog server and add a new one to 2 our our Cisco Nexus 3k switches, but it won't do it.

sh run | inc logging
logging logfile messages 7 size 100000
logging distribute
logging server 192.168.18.9 6 use-vrf Unknown
logging server 172.28.14.9 6 use-vrf Unknown

Here you see me add the new one and then check and it's not there?

 logging server 10.18.18.18 use-vrf Management
x(config)# sh run | inc logging
logging logfile messages 7 size 100000
logging distribute
logging server 192.168.18.9 6 use-vrf Unknown
logging server 172.28.14.9 6 use-vrf Unknown

What am I doing wrong?

[Christopher Hart]

Hi Andrew!

I have a few clarifying questions on this:

1. Can you confirm the exact model of Nexus 3000 switch that you're working with? show module will be helpful here.
2. Can you clarify the NX-OS software release both Nexus 3000 switches are currently running? show module will again be helpful here.
3. Can you verify the most recent NX-OS upgrade path these switches have taken? show system reset-reason may be helpful here.
4. Can you show us the VRFs currently configured on your switch? show vrf will be helpful here.

Typically, when we see issues with adding or removing configuration from Nexus 3000/9000 series switches running modern (later 7.x) NX-OS software releases, the root cause stems from a software defect causing an inconsistency within the DME component of NX-OS. DME stands for Data Management Engine, and it is essentially an abstracted data model that houses NX-OS configuration and operational data structures. Once I know the specific Nexus hardware, NX-OS software, and NX-OS upgrade path, I can see about reproducing this issue internally.

Thank you!

-Christopher

[Andrew White]

Here you go:

show system reset-reason
----- reset reason for module 1 (from Supervisor in slot 1) ---
1) At 49503 usecs after Tue Jan 21 20:10:56 2020
    Reason: Reset Requested by CLI command reload
    Service:
    Version: 7.0(3)I7(6)
2) At 607184 usecs after Tue Oct  1 20:19:33 2019
    Reason: Reset Requested by CLI command reload
    Service:
    Version: 7.0(3)I7(6)
3) At 44591 usecs after Tue Oct  1 20:08:18 2019
    Reason: Reset Requested by CLI command reload
    Service:
    Version: 7.0(3)I2(2b)

 show vrf
VRF-Name                           VRF-ID State   Reason
DMZ1                                    3 Up      --
default                                 1 Up      --
management                              2 Up      --
vPC-Keepalive                           4 Up      --

show module
Mod Ports             Module-Type                       Model          Status
--- ----- ------------------------------------- --------------------- ---------
1    54   48x10GT + 6x40G Supervisor            N3K-C3172TQ-10GT      active *

Mod  Sw                Hw     Slot
---  ----------------  ------ ----
1    7.0(3)I7(6)       1.2    NA

[Christopher Hart]

Hi Andrew!

I appreciate you providing that output! Would you also mind sharing the output of the below two commands?

show running-config | include logging
show system internal dme running-config dn sys/syslog

I know you've already provided the running config a handful of times, but I'd like to see a clean version of it once more just in case.

Thank you!

-Christopher

[Andrew White]

Sure and thanks for you help!

show system internal dme running-config dn sys/syslog
### DME is not enabled on this platform  ###

and

 show running-config | include logging
logging logfile messages 7 size 100000
logging distribute
logging server 192.168.18.9 6 use-vrf Unknown
logging server 172.28.14.9 6 use-vrf Unknown
logging commit
logging source-interface Vlan154
logging timestamp milliseconds
logging monitor 6
logging console 0

[Christopher Hart]

Hi Andrew,

I appreciate the output!

Interestingly, it doesn't appear that the "Unknown" VRF currently exists on the switch. I would be willing to bet that this is playing a role in this issue.

Let me perform some additional testing internally, and I'll try to get back to you in the next few hours.

Thank you!

-Christopher

[Andrew White]

Thanks I did try and add the new logging server with the vrf of management and it accepted it, but when I do a show run it's not there, just the same 2 logging servers.  Does this on the other Nexus too, which is the same version/firmware etc.

[Andrew White]

Hello Christopher, did you have any luck on finding anything in your lab?

[Christopher Hart]

Hi Andrew,

Unfortunately, I have not been able to reproduce this precise behavior in the lab.

I suspect that if you were to reload this switch using the reload ascii command, this behavior would be resolved and you would be able to edit syslog server configuration on these switches once more. A reload ascii is similar to a normal reload except that as the switch is coming up, an ASCII version of the switch's configuration is replayed instead of the optimized binary configuration. This can clear out memory corruption issues (including issues with PSS [Persistent Storage Service] databases and DME [Data Management Engine] data structures), which are typically caused by software defects.

If you have the time, I would like you to open a support case with Cisco TAC on this issue so that we can investigate further and identify the root cause of this issue. Please feel free to PM me the number of the support case you open so that I can keep an eye on it.

Thank you!

-Christopher

[Andrew White]

Thanks I will go ahead and do that later on.