Slow file copy to a Nexus 93180YC-EX

[Network Diver]

Hi,

We have four Nexus 93180YC-EX running NXOS 7.0(3)I4(6) and an uptime of > 2100 days. I need to update them to the latest recommended NXOS version.

Management interface is a VLAN interface. The dedicated management interface is used for vPC peer keep-alive.

Copying a software image of circa 950 MB size takes about 2 hours, no matter whether I copy it via HTTP or SCP or SFTP from a host in the same datacenter. Another copy did not finish after 12 hours. We have a bunch of newer Nexus switches that came with NXOS 9.3.x and there same copy was much faster.

nexus9k# copy http://1.2.3.4/cisco/nexus/93180YC-EX/nxos.7.0.3.I7.10.bin bootflash:
Enter vrf (If no input, current vrf 'default' is considered):
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 16  940M   16  151M    0     0   136k      0  1:57:23  0:18:57  1:38:26  105k

Is it a software bug or due to the long uptime? 

Regards,

Bernd

[marce1000 · Accepted solution]

                   - FYI : https://bst.cisco.com/bugsearch/bug/CSCvh45935

 M.

[Network Diver]

Thanks. With NXOS NXOS 7.0(3)I4(6) the default copp policy is:

nexus9k# show run all | section copp-system-p-policy-strict
policy-map type control-plane copp-system-p-policy-strict
  class copp-system-p-class-l3uc-data
    set cos 1
    police cir 800 kbps bc 32000 bytes conform transmit violate drop
  class copp-system-p-class-critical
    set cos 7
    police cir 36000 kbps bc 1280000 bytes conform transmit violate drop
  class copp-system-p-class-important
    set cos 6
    police cir 2500 kbps bc 1280000 bytes conform transmit violate drop
  class copp-system-p-class-openflow
    set cos 5
    police cir 1000 kbps bc 32000 bytes conform transmit violate drop
  class copp-system-p-class-multicast-router
    set cos 6
    police cir 2600 kbps bc 128000 bytes conform transmit violate drop
  class copp-system-p-class-multicast-host
    set cos 1
    police cir 1000 kbps bc 128000 bytes conform transmit violate drop
  class copp-system-p-class-l3mc-data
    set cos 1
    police cir 2400 kbps bc 32000 bytes conform transmit violate drop
  class copp-system-p-class-normal
    set cos 1
    police cir 1400 kbps bc 32000 bytes conform transmit violate drop
  class copp-system-p-class-ndp
    set cos 6
    police cir 1400 kbps bc 32000 bytes conform transmit violate drop
  class copp-system-p-class-normal-dhcp
    set cos 1
    police cir 1300 kbps bc 32000 bytes conform transmit violate drop
  class copp-system-p-class-normal-dhcp-relay-response
    set cos 1
    police cir 1500 kbps bc 64000 bytes conform transmit violate drop
  class copp-system-p-class-normal-igmp
    set cos 3
    police cir 3000 kbps bc 64000 bytes conform transmit violate drop
  class copp-system-p-class-redirect
    set cos 1
    police cir 280 kbps bc 32000 bytes conform transmit violate drop
  class copp-system-p-class-exception
    set cos 1
    police cir 150 kbps bc 32000 bytes conform transmit violate drop
  class copp-system-p-class-exception-diag
    set cos 1
    police cir 150 kbps bc 32000 bytes conform transmit violate drop
  class copp-system-p-class-management
    set cos 2
    police cir 10000 kbps bc 32000 bytes conform transmit violate drop
  class copp-system-p-class-monitoring
    set cos 1
    police cir 150 kbps bc 128000 bytes conform transmit violate drop
  class copp-system-p-class-l2-unpoliced
    set cos 7
    police cir 50 mbps bc 8192000 bytes conform transmit violate drop
  class copp-system-p-class-undesirable
    set cos 0
    police cir 200 kbps bc 32000 bytes conform transmit violate drop
  class copp-system-p-class-fcoe
    set cos 6
    police cir 1060 kbps bc 128000 bytes conform transmit violate drop
  class copp-system-p-class-nat-flow
    set cos 7
    police cir 800 kbps bc 64000 bytes conform transmit violate drop
  class copp-system-p-class-l2-default
    set cos 0
    police cir 400 kbps bc 32000 bytes conform transmit violate drop
  class class-default
    set cos 0
    police cir 400 kbps bc 32000 bytes conform transmit violate drop
  service-policy input copp-system-p-policy-strict

 The planned recommended upgrade path is NXOS 7.0(3)I4(6) → 7.0(3)I7(3) → 7.0(3)I7(10) → 9.3(11) → 10.2(4)M. From the bug description this is fixed in 7.0(3)I7(3).

[Petr Jaros]

I have similar issue, but policy values seems to be corrected, in compare with bug description. We have NX-OS 9.3(7).

class copp-system-p-class-management
set cos 2
police cir 36000 kbps bc 512000 bytes conform transmit violate drop

SCP transfer starts at 2MB/s, then after few seconds drop to ~60KB/s. Upload image take ages with this speed. Source (windows server) and destination (switch) are in same subnet.

[dgaunt]

You can use the “use-kstack” syntax with the copy command on the nexus switches to increase the speed of the copy commands.  See attached documentation for details.

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/93x/progammability/guide/b-cisco-nexus-9000-series-nx-os-programmability-guide-93x/b-cisco-nexus-9000-series-nx-os-programmability-guide-93x_chapter_0101100.html#id_105304  

[Joseph W. Doherty]

@dgaunt did you note in OP:  ". . . I copy it via HTTP or SCP or SFTP from a host in the same datacenter."

and in your reference:  "This option can be beneficial when copying files from remote servers that are multiple hops from the switch."

So, might not make much of a difference in OP's case.

Also, unclear whether the alternative network stack would bypass CoPP, or the noted bug, again, might not make a difference.

That said, should be easy enough to try, and I suspect, unlikely to do worst or have other adverse effect.

[thatgeekinit]

I'm having this problem too on four N9K-C9336C-FX2 running 10.2.6M via the mg0 port on the management vrf. The file transfers seem to be capped at ~320KB/s per switch. If I run two switches at once, they both hit around 320KB/s so its not any other limiter along the way that I can see.

I've tried http and sftp. It's not a local server but its connected via 1gbps w 1gpbs switching and into a 10gbps internet connection that no one else is using. 

[Joseph W. Doherty]

If the actual file transfer is via the management port, the port might be the issue.  Sometimes such ports are not hardware supported like the data ports.