3850-24XS Dropped Network Large file Transfers - 10Gb Fiber

[metuckness]

Hi,

Using Windows 10 PRO

I have an issue and I am not sure where to start to troubleshoot it. I put in a 10Gbt Fiber network and got it all tied into a 3850-24XS 10Gbt switch. Everything seems to work great except when I try to do a large file from my system to a NAS or my system to another PC. I lose all network connectivity once the massive file transfer begins. I can copy small files, check email, use applications, browse the web fine, but large file copies fail, consistently and I lose all network. I have to reboot the system to get back up and running.

Am I missing something to handle large file copies through the switch?

Here is the config of the switch:

LCCA-F24#sh run
Building configuration...

Current configuration : 7694 bytes
!
! Last configuration change at 00:46:46 UTC Fri Feb 16 2018 by mtuckness
!
version 15.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
service compress-config
!
hostname LCCA-F24
!
boot-start-marker
boot-end-marker
!
!
vrf definition Mgmt-vrf
 !
 address-family ipv4
 exit-address-family
 !
 address-family ipv6
 exit-address-family
!
enable secret 5 $1$ITeG$IAAtxUjLwB7GBgNmZIP6J.
enable password 7 06575F72404B08485544
!
username  privilege 15 password 7 03285E1F0B0A28420F
username  privilege 15 secret 5 $1$0wxz$dPB5nbTJJfOVK0SZ81ylF/
aaa new-model
!
!
!
!
!
!
!
!
aaa session-id common
switch 1 provision ws-c3850-24xs
!
!
!
!
!
coap http enable
!
!
!
!
!
!
ip routing
!
!
!
qos queue-softmax-multiplier 100
!
crypto pki trustpoint TP-self-signed-4287722003
 enrollment selfsigned
 subject-name cn=IOS-Self-Signed-Certificate-4287722003
 revocation-check none
 rsakeypair TP-self-signed-4287722003
!
!
crypto pki certificate chain TP-self-signed-4287722003
 certificate self-signed 02
  3082022B 30820194 A0030201 02020102 300D0609 2A864886 F70D0101 05050030
  31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274
  69666963 6174652D 34323837 37323230 3033301E 170D3137 30393238 32333537
  31345A17 0D323030 31303130 30303030 305A3031 312F302D 06035504 03132649
  4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D34 32383737
  32323030 3330819F 300D0609 2A864886 F70D0101 01050003 818D0030 81890281
  8100DF6B 2A3FE75C 6CE57D14 EC874E2B B77656F3 D862D0E7 3E0569DE BAA305FA
  3AA8AF6D 80348F03 B0E867CA C3A0234B CFB53765 1FF4C71E 250840AE 1AEFFF96
  CC1882F9 0B087F7E F2B35A26 E6F7A479 9205BB30 F086E5AC AE13A512 3D3A1F10
  A111E8E1 52D04E30 A0915F24 5E2D5A68 CB7532C5 B89AE321 40488EE6 3EB76107
  B7B10203 010001A3 53305130 0F060355 1D130101 FF040530 030101FF 301F0603
  551D2304 18301680 14DA8873 74DF0B63 A845B02E 272EBA5C 7F5A497F 9F301D06
  03551D0E 04160414 DA887374 DF0B63A8 45B02E27 2EBA5C7F 5A497F9F 300D0609
  2A864886 F70D0101 05050003 818100DD 6C99228A 20EBCFBB 7DFDF29C 2AC88450
  D3ABCCD6 03801231 6E351FA8 251D3EFC 22ECF0D7 FB0F309A 1FEF5C13 BA01FEF4
  ED3E7223 A8D65100 99F0D3BF EA42D101 6B72553E 229FB38D 0A71CA96 89D3FC91
  A20CD9A6 AA790AA1 4A40DBCC ED1A175C FA0E7201 AE0EF5F9 037C1E7E CBBFC2DE
  4A70C0F0 A346F0A0 12BF91A3 A04858
        quit
diagnostic bootup level minimal
spanning-tree mode rapid-pvst
spanning-tree extend system-id
hw-switch switch 1 logging onboard message level 3
!
redundancy
 mode sso
!
!
!
class-map match-any non-client-nrt-class
!
policy-map port_child_policy
 class non-client-nrt-class
  bandwidth remaining ratio 10
!
!
!
!
!
!
!
!
!
!
!
!
!
!
interface Port-channel1
 description Netapp Cluster Node 3 Pool
 switchport access vlan 113
 switchport trunk allowed vlan 113,116
 switchport mode trunk
 switchport nonegotiate
!
interface Port-channel2
 description Netapp Cluster Node 4 Pool
 switchport access vlan 113
 switchport trunk allowed vlan 113,116
 switchport mode trunk
 switchport nonegotiate
!
interface Port-channel3
 description 20GB LACP to 3750 Stack
 switchport trunk allowed vlan 113,116
 switchport mode trunk
 switchport nonegotiate
!
interface GigabitEthernet0/0
 vrf forwarding Mgmt-vrf
 no ip address
 negotiation auto
!
interface TenGigabitEthernet1/0/1
 switchport access vlan 113
 switchport mode access
!
interface TenGigabitEthernet1/0/2
 switchport access vlan 113
 switchport mode access
!
interface TenGigabitEthernet1/0/3
 description LCCA-VIRT-1
 switchport access vlan 113
 switchport mode access
!
interface TenGigabitEthernet1/0/4
 description LCCA-VIRT-1
 switchport access vlan 113
 switchport mode access
!
interface TenGigabitEthernet1/0/5
 description Team to NetApp
 switchport access vlan 113
 switchport mode access
!
interface TenGigabitEthernet1/0/6
 description Team to NetApp
 switchport access vlan 113
 switchport mode access
!
interface TenGigabitEthernet1/0/7
 description Uplink to 3750 Stack
 switchport trunk allowed vlan 113,116
 switchport mode trunk
 switchport nonegotiate
 channel-group 3 mode desirable non-silent
 spanning-tree portfast trunk
!
interface TenGigabitEthernet1/0/8
 description Uplink to 3750 Stack
 switchport trunk allowed vlan 113,116
 switchport mode trunk
 switchport nonegotiate
 channel-group 3 mode desirable non-silent
 spanning-tree portfast trunk
!
interface TenGigabitEthernet1/0/9
 description NETAPP TEAM 1
 switchport access vlan 113
 switchport trunk allowed vlan 113,116
 switchport mode trunk
 switchport nonegotiate
 channel-group 1 mode active
!
interface TenGigabitEthernet1/0/10
 description NETAPP TEAM 1
 switchport access vlan 113
 switchport trunk allowed vlan 113,116
 switchport mode trunk
 switchport nonegotiate
 channel-group 1 mode active
!
interface TenGigabitEthernet1/0/11
 switchport access vlan 113
 switchport trunk allowed vlan 113,116
 switchport mode trunk
 switchport nonegotiate
 channel-group 2 mode active
!
interface TenGigabitEthernet1/0/12
 switchport access vlan 113
 switchport trunk allowed vlan 113,116
 switchport mode trunk
 switchport nonegotiate
 channel-group 2 mode active
!
interface TenGigabitEthernet1/0/13
 switchport access vlan 113
 switchport mode access
!
interface TenGigabitEthernet1/0/14
 switchport access vlan 113
 switchport mode access
!
interface TenGigabitEthernet1/0/15
 switchport access vlan 113
 switchport mode access
!
interface TenGigabitEthernet1/0/16
 switchport access vlan 113
 switchport mode access
!
interface TenGigabitEthernet1/0/17
 switchport access vlan 113
 switchport mode access
!
interface TenGigabitEthernet1/0/18
 switchport access vlan 113
 switchport mode access
!
interface TenGigabitEthernet1/0/19
 switchport access vlan 113
 switchport mode access
!
interface TenGigabitEthernet1/0/20
 switchport access vlan 113
 switchport mode access
!
interface TenGigabitEthernet1/0/21
 switchport access vlan 113
 switchport mode access
!
interface TenGigabitEthernet1/0/22
 switchport access vlan 116
 switchport mode access
!
interface TenGigabitEthernet1/0/23
 switchport access vlan 113
 switchport mode access
!
interface TenGigabitEthernet1/0/24
 switchport access vlan 113
 switchport mode access
!
interface TenGigabitEthernet1/1/1
!
interface TenGigabitEthernet1/1/2
!
interface TenGigabitEthernet1/1/3
!
interface TenGigabitEthernet1/1/4
!
interface TenGigabitEthernet1/1/5
!
interface TenGigabitEthernet1/1/6
!
interface TenGigabitEthernet1/1/7
!
interface TenGigabitEthernet1/1/8
!
interface FortyGigabitEthernet1/1/1
!
interface FortyGigabitEthernet1/1/2
!
interface Vlan1
 no ip address
 shutdown
!
interface Vlan113
 description Subnet
 ip address 192.168.113.247 255.255.255.0
 ip helper-address 192.168.116.240
!
interface Vlan116
 description Admin Subnet
 ip address 192.168.116.247 255.255.255.0
 ip helper-address 192.168.116.240
!
ip forward-protocol nd
ip http server
ip http authentication local
ip http secure-server
!
ip route 0.0.0.0 0.0.0.0 192.168.113.6
!
!
!
snmp-server community LCCA-RO RO
snmp-server community LCCA-RW RW
!
!
!
!
!
line con 0
 stopbits 1
line aux 0
 stopbits 1
line vty 0 4
 session-timeout 30
 password 7 062A0A35414B001744
line vty 5 15
 password 7 022A014F0603062F0D
!
wsma agent exec
 profile httplistener
 profile httpslistener
!
wsma agent config
 profile httplistener
 profile httpslistener
!
wsma agent filesys
 profile httplistener
 profile httpslistener
!
wsma agent notify
 profile httplistener
 profile httpslistener
!
!
wsma profile listener httplistener
 transport http
!
wsma profile listener httpslistener
 transport https
!
ap group default-group
end

[Reza Sharifi]

Hi,

Since you are dealing with storage, can you enable jumbo frame on the switch globally and test again?

Is jumbo frame enabled on the NAS heads?

HTH

[metuckness]

Jumbo frames are not enabled on the NAS LIF's. They are for the interconnects between the cluster nodes, but not traffic over the network.

Plus, the NAS is connected to a 3750, not this 3850. This 3850 is connected to a backup NAS which doesn't serve files to systems.

If I enable Jumbo frames, will it impact PC's sending traffic through it?

Thanks

[Leo Laohoo]

Post the complete output to the command "sh interface <NAS port>."

[metuckness]

LCCA-3750X#sh interfaces port-channel 1
Port-channel1 is up, line protocol is up (connected)
  Hardware is EtherChannel, address is 7cad.74eb.60b6 (bia 7cad.74eb.60b6)
  Description: Netapp Cluster Node 1 Pool
  MTU 1500 bytes, BW 20000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 10Gb/s, link type is auto, media type is unknown
  input flow-control is off, output flow-control is unsupported
  Members in this channel: Te3/1/1 Te3/1/2
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 24000 bits/sec, 22 packets/sec
     12562984450 packets input, 9823698426347 bytes, 0 no buffer
     Received 8577931 broadcasts (8571388 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 8571388 multicast, 0 pause input
     0 input packets with dribble condition detected
     13842439861 packets output, 18124811170009 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

LCCA-3750X#sh interfaces port-channel 2
Port-channel2 is up, line protocol is up (connected)
  Hardware is EtherChannel, address is ccd8.c1f3.8a36 (bia ccd8.c1f3.8a36)
  Description: Netapp Cluster Node 2 Pool
  MTU 1500 bytes, BW 20000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 10Gb/s, link type is auto, media type is unknown
  input flow-control is off, output flow-control is unsupported
  Members in this channel: Te4/1/1 Te4/1/2
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 1341000 bits/sec, 270 packets/sec
  5 minute output rate 459000 bits/sec, 265 packets/sec
     78641847765 packets input, 64187679504069 bytes, 0 no buffer
     Received 8596678 broadcasts (8577329 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 8577329 multicast, 0 pause input
     0 input packets with dribble condition detected
     85607270769 packets output, 106477382144911 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

[metuckness]

A single interface would be:

LCCA-3750X#sh interfaces te 4/1/1
TenGigabitEthernet4/1/1 is up, line protocol is up (connected)
  Hardware is Ten Gigabit Ethernet, address is ccd8.c1f3.8a35 (bia ccd8.c1f3.8a35)
  MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 10Gb/s, link type is auto, media type is SFP-10GBase-SR
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 8w6d, output 00:00:34, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 1086000 bits/sec, 214 packets/sec
  5 minute output rate 11000 bits/sec, 16 packets/sec
     13736669440 packets input, 20178811796336 bytes, 0 no buffer
     Received 4295233 broadcasts (4288899 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 4288899 multicast, 0 pause input
     0 input packets with dribble condition detected
     9619038205 packets output, 1258644497177 bytes, 0 underruns
     0 output errors, 0 collisions, 2 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

[metuckness]

And, to set the Jumbo Frames on the 3850, what size do you recommend?

[metuckness]

1518 seems like what Cisco recommends, but I would like to confirm.

[Leo Laohoo]

The output to the command "sh interface" did not show any signs of output drops nor line errors. Unless this is a bug with the IOS-XE (shudder), I don't see any issue(s) here.

[metuckness]

Anyone else?

[Reza Sharifi]

Jumbo frame should be 9216. On the 3850, this is a global command and you have to reboot the switch for it to take effect. Also, enabling jumbo frame should not have the effect on any hosts.

HTH

[metuckness]

---

@Reza Sharifi wrote:

Jumbo frame should be 9216. On the 3850, this is a global command and you have to reboot the switch for it to take effect. Also, enabling jumbo frame should not have the effect on any hosts.

HTH

---

The highest I can go is: <1500-9198>  MTU size in bytes

[ITexpert · Accepted solution]

I have some questions before saying anything .

Did you test this scenario by connecting some other PC or laptop to same switch ?

If this happen with all the computers or PCs connected with this switch only but not PCs connected with other switch, that means nothing to do with NAS.

But if this is happening with only your PC but not with remaining devices then try updating network drivers.

Thanks,

Lovejit

[metuckness]

---

@ITexpert wrote:

I have some questions before saying anything .

Did you test this scenario by connecting some other PC or laptop to same switch ?

If this happen with all the computers or PCs connected with this switch only but not PCs connected with other switch, that means nothing to do with NAS.

But if this is happening with only your PC but not with remaining devices then try updating network drivers.

Thanks,

Lovejit

---

Will do. I am doing some testing now to see. the other PC's don't ever really transfer large files so they haven't encountered it yet. But I will do some testing to see and also check f there are new drivers for the 10GB cards.

Will post an update.

[metuckness]

So far it looks like it was a driver issue. I was able to locate a new driver for the card and updated it. So far, every test I have done with considerable transfers has completed without dropping network entirely.

I think that might have solved the issue. It never occurred to me to update the drivers.

Thanks!