Hi Mario!
I performed some similar testing in my lab and was able to reproduce similar latency. To summarize, the CLI Python library that NX-OS uses introduces some additional overhead when executing a command that is not present when the same command is executed through a TCL script or through the CLI.
First, let's baseline about how long it takes to execute the command show interface Ethernet1/1 counters errors through the CLI. We can do this relatively well by prepending and appending the show clock command on either side. This isn't perfect (it doesn't account for the overhead of the show clock command itself) but it is close, as shown below.
N9K# show clock ; show interface Ethernet1/1 counters errors ; show clock
14:31:48.555 UTC Thu Aug 20 2020
Time source is NTP
--------------------------------------------------------------------------------
Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards
--------------------------------------------------------------------------------
Eth1/1 0 0 0 0 0 0
--------------------------------------------------------------------------------
Port Single-Col Multi-Col Late-Col Exces-Col Carri-Sen Runts
--------------------------------------------------------------------------------
Eth1/1 0 0 0 0 0 0
--------------------------------------------------------------------------------
Port Giants SQETest-Err Deferred-Tx IntMacTx-Er IntMacRx-Er Symbol-Err
--------------------------------------------------------------------------------
Eth1/1 0 -- 0 0 0 0
--------------------------------------------------------------------------------
Port InDiscards
--------------------------------------------------------------------------------
Eth1/1 0
14:31:48.639 UTC Thu Aug 20 2020
Time source is NTP
All three commands took about 84 milliseconds to execute. Through my own testing (I can expand on this if you'd like, but I don't want to crowd this post with too much unrelated detail), I determined that the show clock command takes about 10 milliseconds to execute, so that means the show interface Ethernet1/1 counters errors command takes about 64 milliseconds to execute through the CLI.
Next, let's inspect the code for the cli Python function that you're using. We can find the location of the file with the find_module() function from the imp module in the standard library.
N9K# python
Warning: Python 2.7 is End of Support, and future NXOS software will deprecate
python 2.7 support. It is recommended for new scripts to use 'python3' instead.
Type "python3" to use the new shell.
Python 2.7.11 (default, Jun 4 2020, 09:48:24)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import imp
>>> imp.find_module("cli")
(<open file '/isan/python/scripts/cli.py', mode 'U' at 0xf39deee8>, '/isan/python/scripts/cli.py', ('.py', 'U', 1))
We can use the Bash shell to view the contents of this file as of NX-OS 9.3(5). This module contains more functions than just the cli function, but I've purposefully cut it down to just the cli function we're using.
def cli(cmd):
'''
Execute CLI commands. Takes CLI command string and returns show command output in a plain string form.
Arguments:
cmd: Single CLI command or a Batch of CLI commands. Delimiter for mutlple CLI
commands is space + semi-colon. Configuration commands need to be in a
fully qualified form.
For example, configure terminal ; interface ethernet1/1 ; no shutdown
Returns:
string: CLI output string for show commands and an empty string for configuration
commands.
Raises:
cli_syntax_error: CLI command is not a valid NXOS command.
cmd_exec_error: Execution of CLI command is not successful.
'''
args = ["/isan/bin/vsh", "-N", "-c", cmd]
p = subprocess.Popen(args, env=getenv(cmd), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output,error = p.communicate()
output = convToStr(output)
error = convToStr(error)
if p.returncode == 0:
return output
else:
msg = "{0}{1}".format(output, error)
if p.returncode == 16:
raise cli_syntax_error(msg)
else:
raise cmd_exec_error(msg)
This function is relatively straightforward - it executes a given command by passing it into a vsh binary at /isan/bin/vsh as a parameter. This binary is then executed by opening up a subprocess, the results of which are fetched with the communicate() function.
This additional overhead is introducing the latency you're observing in your Python script. We can verify this with the timeit module from the standard library through the below Python statements:
N9K# python
Warning: Python 2.7 is End of Support, and future NXOS software will deprecate
python 2.7 support. It is recommended for new scripts to use 'python3' instead.
Type "python3" to use the new shell.
Python 2.7.11 (default, Jun 4 2020, 09:48:24)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import timeit
>>> timeit.timeit('output, error = subprocess.Popen(["/isan/bin/vsh", "-N", "-c", "show interface Ethernet1/1 counters errors"], stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()', "import subprocess", number=10)
6.059631109237671
>>> 6.059631109237671 / 10
0.6059631109237671
This shows that this specific one-liner (which is essentially what the cli function does) takes about 606 milliseconds to execute on average. This timing corresponds with the latency you're observing in your script.
There is no supported way to speed up the CLI library you're importing. Theoretically, you could write your own library that performs a similar task without opening a subprocess - however, I would theorize that there's a very good reason the developers chose to open a subprocess to execute this command instead of executing it natively in the shell, and you might discover that reason if you choose to refactor this library on your own!
With that being said, let's talk about the use case behind some of your Python scripts. Specifically, I have two questions:
- Are your Python scripts truly latency-sensitive and need to be completed as quickly as possible?
- Do your Python scripts need to be executed on the Nexus switch itself, or can they be executed off-the-box?
I hope this helps - thank you!
-Christopher
