Re: Jumbo Frames not Working cisco 9300 or 3650

nishantagrawal0322 · ‎04-19-2023

Hello Friends. We have a situation.

Current Setup:- VMware-->Cisco 3750-->>NetApp iscsi. Jumbo frames enabled mtu 9000 bytes. VMWare Storage Resides on NetApp Via ISCSI using switch 3750.

We would like to replace the cisco 3750 with Catalyst 3650. Exact same configuration. The issue we are running into is when we move our connections from (existing 3750) to new 3650, vmware gets stuck on refreshing the storage. The weird thing is we can ping everything from all devices, it's just the storage cannot be scanned and updated. That is the only reason I'm thinkin the Global MTU on new 3650 should be 9000 (same as existing 3750) or should it be set to max which is 9198 bytes. Any help/assistance would be appreciated.

MHM Cisco World · ‎04-19-2023

same config
Note: Gigabit Ethernet ports are not affected by the system mtu command; 10/100 ports are not affected by the system mtu jumbo command. If you do not configure the system mtu jumbo command, the set of the system mtu command applies to all Gigabit Ethernet interfaces.

nishantagrawal0322 · ‎04-20-2023

Tried it.. Still not passing the Jumbo Frames.

Here is what I have tried until now..

set mtu to 9000/9198 on per port basis and reboot and tested. still same..

Here is existing/new topology

existing topology:

VMware host( VMNIC's configured to MTU 9000)-->> Cisco 3750X Switch (Ports in use configured to MTU9000/9198)-->> NetAPP iscsi Storage (MTU 9000). Everything works fine and vmware host can rescan/see/connect to the storage via iscsi with no issues.

New Topology (with Switch Replacement)

VMware host( VMNIC's configured to MTU 9000)-->> Cisco 9300 Switch (Ports in use configured to MTU9000/9198)-->> NetAPP iscsi Storage (MTU 9000). In this, vmware host cannot rescan/see/connect to the Netapp storage via iscsi..

Any help would be appreciated..

set global system mtu to 9000 or 9198, reboot and tested, still vmware cannot connect to storage via iscsi..

MHM Cisco World · ‎04-21-2023

are the OLD design use Access interface and new design use trunk ? you must keep in mind that there are extra 4 bytes for VLAN if the connection via trunk

nishantagrawal0322 · ‎04-21-2023

No. Exact same setup. Both connections are using Trunk.

Thanks

MHM Cisco World · ‎04-21-2023

show system mtu

show interfaces mtu

show int gige x/x | i MTU

nishantagrawal0322 · ‎04-21-2023

Here you go..

Server-Switch3#show int gi 1/1/1 | i MTU
MTU 9198 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
Server-Switch3#sh int
Server-Switch3#sh interfaces mtu

Port Name MTU
Gi1/0/1 Trunk port to core 9198
Gi1/0/2 new host 9000
Gi1/0/3 Temporary Connecti 9000
Gi1/0/4 New VM HOST 9000
Gi1/0/5 Vmware ESXi 5.5 se 9000
Gi1/0/6 New VM HOST 9000
Gi1/0/7 New VM HOST 9000
Gi1/0/8 Server 9000
Gi1/0/9 New VM HOST 9000
Gi1/0/10 HOST 1 9000
Gi1/0/11 HOST 1 9000
Gi1/0/12 DUAL HOME ESXi VM2 9000
Gi1/0/13 New Host 9000
Gi1/0/14 servers 9000
Gi1/0/15 servers 9000
Gi1/0/16 New Host 3 9198
Gi1/0/17 CV-CBB-Backup-Host 9000
Gi1/0/18 LSECV01-CBB-Physic 9000
Gi1/0/19 New Host 3 9198
Gi1/0/20 Servers 9000
Gi1/0/21 Link to HP VM by 9000

Port Name MTU
Gi1/0/22 Link to HP VM by 9000
Gi1/0/23 DUAL HOME ESXi VM2 9000
Gi1/0/24 DUAL HOME ESXi VM1 9198
Gi1/1/1 TO STORAGE CONTROL 9198
Gi1/1/2 TO STORAGE CONTROL 9000
Gi1/1/3 LSE STORAGE MGMT P 9198
Gi1/1/4 9198
Ap1/0/1 9198
Gi2/0/1 Trunk port to core 9198
Gi2/0/2 Trunk Ports 9000
Gi2/0/3 Vmotion VM3 9000
Gi2/0/4 9000
Gi2/0/5 Vmware ESXi 5.5 se 9000
Gi2/0/6 Vmotion VM1 9000
Gi2/0/7 New Host 9000
Gi2/0/8 Servers Backup Lin 9000
Gi2/0/9 Servers Backup Lin 9000
Gi2/0/10 DUAL HOME ESXi VM2 9000
Gi2/0/11 Servers Backup Lin 9000
Gi2/0/12 Server 9000
Gi2/0/13 Servers Backup Lin 9000
Gi2/0/14 HOST 1 9000
Gi2/0/15 HOST 1 9000

Port Name MTU
Gi2/0/16 New VM HOST 9198
Gi2/0/17 LSEDSO2-nic2 9000
Gi2/0/18 LSEDSO1-nic2 9000
Gi2/0/19 9198
Gi2/0/20 Servers 9000
Gi2/0/21 Link to HP VM by 9000
Gi2/0/22 Link to HP VM by 9000
Gi2/0/23 DUAL HOME ESXi VM2 9000
Gi2/0/24 DUAL HOME ESXi VM1 9198
Gi2/1/1 TO STORAGE CONTROL 9198
Gi2/1/2 TO STORAGE CONTROL 9000
Gi2/1/3 LSE STORAGE MGMT - 9198
Gi2/1/4 9198
Ap2/0/1 9198
Po5 9198
Po6 9000
Po7 9198
Server-Switch3#sh system mtu
Global Ethernet MTU is 9198 bytes.

MHM Cisco World · ‎04-21-2023

just I need one more
show interface x/x <<- interface connect to both vm and Netapp

nishantagrawal0322 · ‎04-21-2023

I have switched the connections back. Do you need the live interface? if yes, I'll have to send it that way. Below are the details:-

Interface connected to VM host:- gi 1/0/16,1/0/19,2/0/16,2/0/19

Interface connected to NetAPP:- gi1/1/1, gi2/1/1

nishantagrawal0322 · ‎04-21-2023

Currently nothing is connected and below are the details you requested.. We can switch the connections and resend them:-

Server-Switch3#sh int gi1/0/16
GigabitEthernet1/0/16 is down, line protocol is down (notconnect)
Hardware is Gigabit Ethernet, address is ac7a.5649.b610 (bia ac7a.5649.b610)
Description: New Host 3
MTU 9198 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Auto-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 21:46:02, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
95793 packets input, 41159430 bytes, 0 no buffer
Received 10 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
575279 packets output, 520367530 bytes, 0 underruns
Output 115609 broadcasts (76405 multicasts)
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
Server-Switch3#sh int gi1/0/19
GigabitEthernet1/0/19 is down, line protocol is down (notconnect)
Hardware is Gigabit Ethernet, address is ac7a.5649.b613 (bia ac7a.5649.b613)
Description: New Host 3
MTU 9198 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Auto-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 21:45:27, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
68044 packets input, 20873338 bytes, 0 no buffer
Received 41 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
369554 packets output, 210256597 bytes, 0 underruns
Output 115848 broadcasts (76653 multicasts)
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
Server-Switch3#sh int gi2/0/16
GigabitEthernet2/0/16 is down, line protocol is down (notconnect)
Hardware is Gigabit Ethernet, address is ac7a.566e.c110 (bia ac7a.566e.c110)
Description: New VM HOST
MTU 9198 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Auto-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 21:45:40, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
114 packets input, 14240 bytes, 0 no buffer
Received 8 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
198705 packets output, 35060010 bytes, 0 underruns
Output 115851 broadcasts (76650 multicasts)
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
Server-Switch3#sh int gi2/0/19
GigabitEthernet2/0/19 is down, line protocol is down (notconnect)
Hardware is Gigabit Ethernet, address is ac7a.566e.c113 (bia ac7a.566e.c113)
MTU 9198 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Auto-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 21:45:32, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
198517 packets output, 34979030 bytes, 0 underruns
Output 115939 broadcasts (76691 multicasts)
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
Server-Switch3#sh int gi1/1/1
GigabitEthernet1/1/1 is down, line protocol is down (notconnect)
Hardware is Gigabit Ethernet, address is ac7a.5649.b619 (bia ac7a.5649.b619)
Description: TO STORAGE CONTROLLER1
MTU 9198 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 10/100/1000BaseTX SFP
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 21:45:53, output 21:45:39, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
1766298 packets input, 1706049756 bytes, 0 no buffer
Received 1006 broadcasts (935 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 935 multicast, 0 pause input
0 input packets with dribble condition detected
105867 packets output, 34570765 bytes, 0 underruns
Output 35484 broadcasts (39972 multicasts)
0 output errors, 0 collisions, 2 interface resets
370 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
Server-Switch3#sh int gi2/1/1
GigabitEthernet2/1/1 is down, line protocol is down (notconnect)
Hardware is Gigabit Ethernet, address is ac7a.566e.c119 (bia ac7a.566e.c119)
Description: TO STORAGE CONTROLLER1
MTU 9198 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 10/100/1000BaseTX SFP
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 21:45:43, output 21:45:41, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
1269658 packets input, 1142643020 bytes, 0 no buffer
Received 1046 broadcasts (939 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 939 multicast, 0 pause input
0 input packets with dribble condition detected
1865691 packets output, 705094171 bytes, 0 underruns
Output 70136 broadcasts (32861 multicasts)
0 output errors, 0 collisions, 2 interface resets
370 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

MHM Cisco World · ‎04-21-2023

just I want to see if giant count increase in these interface no need life since you dont clear the counter it appear oldest count

nishantagrawal0322 · ‎04-21-2023

okok.. Thanks.. lmk how to proceed, what's the issue what can we try..

MHM Cisco World · ‎04-22-2023

1-

C9k#show platform software fed active ifm mappings
Interface                 IF_ID    Inst Asic Core Port SubPort Mac Cntx LPN GPN Type Active
ge x/x                     0xF        1   0   1    20     0      16  4    1   101 NIF  Y 


C9k#show platform software fed active ifm if-id 0xF | inc MTU
                Jumbo MTU ............ [9xxx]

2- check the giant frame when connection is life (if you can)
3- share output of below in both case life (if you can) and no connection

c9k#show controllers ethernet-controller ge x/x

nishantagrawal0322 · ‎04-24-2023

Server-Switch3#show platform software fed switch active ifm mappings

Interface IF_ID Inst Asic Core Port SubPort Mac Cntx LPN GP N Type Active
GigabitEthernet2/0/1 0x33 1 0 1 0 0 26 6 1 9 7 NIF Y
GigabitEthernet2/0/2 0x34 1 0 1 1 0 6 7 2 9 8 NIF Y
GigabitEthernet2/0/3 0x35 1 0 1 2 0 28 8 3 9 9 NIF Y
GigabitEthernet2/0/4 0x36 1 0 1 3 0 27 9 4 1 00 NIF Y
GigabitEthernet2/0/5 0x37 1 0 1 4 0 30 10 5 1 01 NIF Y
GigabitEthernet2/0/6 0x38 1 0 1 5 0 29 11 6 1 02 NIF Y
GigabitEthernet2/0/7 0x39 1 0 1 6 0 32 12 7 1 03 NIF Y
GigabitEthernet2/0/8 0x3a 1 0 1 7 0 31 13 8 1 04 NIF Y
GigabitEthernet2/0/9 0x3b 1 0 1 8 0 19 14 9 1 05 NIF Y
GigabitEthernet2/0/10 0x3c 1 0 1 9 0 5 15 10 1 06 NIF Y
GigabitEthernet2/0/11 0x3d 1 0 1 10 0 21 16 11 1 07 NIF Y
GigabitEthernet2/0/12 0x3e 1 0 1 11 0 20 17 12 108 NIF Y
GigabitEthernet2/0/13 0x3f 1 0 1 12 0 23 18 13 109 NIF Y
GigabitEthernet2/0/14 0x40 1 0 1 13 0 22 19 14 110 NIF Y
GigabitEthernet2/0/15 0x41 1 0 1 14 0 25 20 15 111 NIF Y
GigabitEthernet2/0/16 0x42 1 0 1 15 0 24 21 16 112 NIF Y
GigabitEthernet2/0/17 0x43 0 0 0 16 0 26 6 17 113 NIF Y
GigabitEthernet2/0/18 0x44 0 0 0 17 0 6 7 18 114 NIF Y
GigabitEthernet2/0/19 0x45 0 0 0 18 0 28 8 19 115 NIF Y
GigabitEthernet2/0/20 0x46 0 0 0 19 0 27 9 20 116 NIF Y
GigabitEthernet2/0/21 0x47 0 0 0 20 0 30 10 21 117 NIF Y
GigabitEthernet2/0/22 0x48 0 0 0 21 0 29 11 22 118 NIF Y
GigabitEthernet2/0/23 0x49 0 0 0 22 0 32 12 23 119 NIF Y
GigabitEthernet2/0/24 0x4a 0 0 0 23 0 31 13 24 120 NIF Y
GigabitEthernet2/1/1 0x4b 1 0 1 24 0 3 4 25 121 NIF Y
GigabitEthernet2/1/2 0x4c 1 0 1 25 0 2 5 26 122 NIF Y
GigabitEthernet2/1/3 0x4d 0 0 0 26 0 3 4 27 123 NIF Y
GigabitEthernet2/1/4 0x4e 0 0 0 27 0 2 5 28 124 NIF Y
TenGigabitEthernet2/1/1 0x4f 1 0 1 28 0 0 3 29 125 NIF N
TenGigabitEthernet2/1/2 0x50 1 0 1 29 0 0 2 30 126 NIF N
TenGigabitEthernet2/1/3 0x51 1 0 1 30 0 0 1 31 127 NIF N
TenGigabitEthernet2/1/4 0x52 1 0 1 31 0 0 0 32 128 NIF N
TenGigabitEthernet2/1/5 0x53 0 0 0 32 0 0 3 33 129 NIF N
TenGigabitEthernet2/1/6 0x54 0 0 0 33 0 0 2 34 130 NIF N
TenGigabitEthernet2/1/7 0x55 0 0 0 34 0 0 1 35 131 NIF N
TenGigabitEthernet2/1/8 0x56 0 0 0 35 0 0 0 36 132 NIF N
FortyGigabitEthernet2/1/1 0x57 1 0 1 36 0 0 0 37 133 NIF N
FortyGigabitEthernet2/1/2 0x58 0 0 0 37 0 0 0 38 134 NIF N
TwentyFiveGigE2/1/1 0x59 1 0 1 38 0 0 0 39 135 NIF N
TwentyFiveGigE2/1/2 0x5a 0 0 0 39 0 0 0 40 136 NIF N
AppGigabitEthernet2/0/1 0x5b 1 0 1 40 0 11 22 41 137 NIF Y

nishantagrawal0322 · ‎04-24-2023

Server-Switch3#show platform software fed switch active ifm if-id 0xF | inc MTU

Jumbo MTU ............ [0]