WS-C4948-10GE-S MTU Change

[ungachuck]

We have a 2 ISCSi SANs and 4 ESXi Nodes in a Lenovo Chassis. The Chassis has a Lenovo Flex System EN2092 1Gb Ethernet Switch connected to it. The nodes are connected to this and the EN2092 is connected to WS-C4948-10GE-S through a Port-Channel configuration. See below:

Po3 is connected to the Flex Switch with Gi35-36 members of the port channel.

***EDIT*** Incorrect port information, correct information provided.

NITSNT0007#show int po3

Port-channel3 is up, line protocol is up (connected)

  Hardware is EtherChannel, address is c84c.7599.09a2 (bia c84c.7599.09a2)

  Description: Flex-Bay2

  MTU 9198 bytes, BW 2000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 1000Mb/s, media type is N/A

  input flow-control is off, output flow-control is unsupported

  Members in this channel: Gi1/35 Gi1/36

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 20:42:38, output never, output hang never

  Last clearing of "show interface" counters never

  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 3162000 bits/sec, 425 packets/sec

  5 minute output rate 2098000 bits/sec, 302 packets/sec

     25002501 packets input, 21869407361 bytes, 0 no buffer

     Received 94318 broadcasts (88127 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 input packets with dribble condition detected

     21779631 packets output, 18418258687 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier

     0 output buffer failures, 0 output buffers swapped out

interface Port-channel3

description Flex-Bay2

switchport

mtu 9198

interface GigabitEthernet1/35

description IBM-FLEX-bay2-eth1

mtu 9198

channel-protocol lacp

channel-group 3 mode active

!

interface GigabitEthernet1/36

description IBM-FLEX-bay2-eth2

mtu 9198

channel-protocol lacp

channel-group 3 mode active

We have the majority of the SAN ISCSi ports connected to the EN2092, however we don’t have enough ports on the EN2092 to connect all SAN ports so have connected the remaining to the WS-C4948-10GE-S. Port configuration above.

The ESXi nodes have jumbo frames enabled, and are having issues with connecting to the SAN interfaces that are connected to the C4948-10GE-S. I’ve run ping tests from the ESXi node to all SAN interface IP’s, and regular pings can access the SAN interfaces connected to the C4948-10GE-S, however any jumbo frames sent don’t get a response from the SAN interfaces on C4948-10GE-S. Jumbo frames can reach the SAN interfaces connected to the EN2092. The EN2092 has jumbo frames enabled by default.

As you can see I’ve enabled an MTU on the specific interfaces as per the instructions on this page. https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst4500/12-2/18ew/configuration/guide/config/sw_int.html#wp1049274 However jumbo frames still cannot reach the SAN interfaces connected to the C4948-10GE-S.

Have I missed something in configuring the larger MTU on C4948-10GE-S?

[Reza Sharifi]

Hi,

Does "sh interface GigabitEthernet1/39" or 1/40 show the MTU being 9216?

HTH

[ungachuck]

Hi Reza,

Thank you for your response. It shows the MTU being 9198, this was the maximum available. The MTU is displayed in the show interface command for the all interfaces configured, including the Po3.

Switch(config-if)#mtu ?
<1500-9198> MTU size in bytes

Switch#show int gi1/39
GigabitEthernet1/39 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet Port, address is c84c.7599.09a6 (bia c84c.7599.09a6)
MTU 9198 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is 10/100/1000-TX
input flow-control is on, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 597000 bits/sec, 44 packets/sec
5 minute output rate 21000 bits/sec, 32 packets/sec
5065296 packets input, 8064220330 bytes, 0 no buffer
Received 15 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
4227544 packets output, 362370306 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

Switch#show int gi1/40
GigabitEthernet1/40 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet Port, address is c84c.7599.09a7 (bia c84c.7599.09a7)
MTU 9198 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is 10/100/1000-TX
input flow-control is on, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 113000 bits/sec, 59 packets/sec
5 minute output rate 742000 bits/sec, 98 packets/sec
5138511 packets input, 3700063982 bytes, 0 no buffer
Received 273 broadcasts (228 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
7512810 packets output, 6240333646 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

Switch#show int po3
Port-channel3 is up, line protocol is up (connected)
Hardware is EtherChannel, address is c84c.7599.09a2 (bia c84c.7599.09a2)
Description: Flex-Bay2
MTU 9198 bytes, BW 2000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is N/A
input flow-control is off, output flow-control is unsupported
Members in this channel: Gi1/35 Gi1/36
ARP type: ARPA, ARP Timeout 04:00:00
Last input 20:05:12, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 3291000 bits/sec, 475 packets/sec
5 minute output rate 2856000 bits/sec, 423 packets/sec
24074325 packets input, 21021975939 bytes, 0 no buffer
Received 89855 broadcasts (83930 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
21085778 packets output, 17855296632 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

Switch#show int po3
Port-channel3 is up, line protocol is up (connected)
Hardware is EtherChannel, address is c84c.7599.09a2 (bia c84c.7599.09a2)
Description: Flex-Bay2
MTU 9198 bytes, BW 2000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is N/A
input flow-control is off, output flow-control is unsupported
Members in this channel: Gi1/35 Gi1/36
ARP type: ARPA, ARP Timeout 04:00:00
Last input 20:05:12, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 3291000 bits/sec, 475 packets/sec
5 minute output rate 2856000 bits/sec, 423 packets/sec
24074325 packets input, 21021975939 bytes, 0 no buffer
Received 89855 broadcasts (83930 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
21085778 packets output, 17855296632 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

Switch#show int gi1/35
GigabitEthernet1/35 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet Port, address is c84c.7599.09a2 (bia c84c.7599.09 a2)
Description: IBM-FLEX-bay2-eth1
MTU 9198 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is 10/100/1000-TX
input flow-control is off, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:22, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 12000 bits/sec, 4 packets/sec
5 minute output rate 2634000 bits/sec, 377 packets/sec
599743 packets input, 172977154 bytes, 0 no buffer
Received 52451 broadcasts (48941 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
20107492 packets output, 16488282934 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

Switch#show int gi1/36
GigabitEthernet1/36 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet Port, address is c84c.7599.09a3 (bia c84c.7599.09a3)
Description: IBM-FLEX-bay2-eth2
MTU 9198 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is 10/100/1000-TX
input flow-control is off, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:03, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 2890000 bits/sec, 416 packets/sec
5 minute output rate 70000 bits/sec, 15 packets/sec
23488859 packets input, 20859483816 bytes, 0 no buffer
Received 37422 broadcasts (35003 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
993125 packets output, 1380277055 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
Switch#

Interestingly there is a system mtu command available but the highest MTU available for this is 1552.

Switch(config)#system mtu ?
<1500-1552> MTU size in bytes

Any other thoughts?

[Reza Sharifi]

Hi,

MTU 9198 bytes, BW 1000000 Kbit, DLY 10 usec,

Interfaces and Po all look good with the correct MTU size. I would not worry about the system MTU, as that is mostly for routed ports and applies to the entire switch.

Do the SAN interfaces have the same MTU size as the Cisco switch?

HTH  

[ungachuck]

Hi Reza,

All SAN interfaces have MTU configured for 9000, I have double checked this.

Thanks,

ungachuck

[Reza Sharifi]

Hi,

Can you clarify if the SVI for the storage system is on the 4948 or it is on a different device?

If it is on the 4948, can you ping the storage system (with larger MTU size) from the 4948?

Can you post "sh run" from the switch?

HTH

[ungachuck]

Hi Reza,

A SAN interface is connected to the switch which has an IP address. The Switch is on a different subnet but within VLAN. (This configuration was inherited, trying to move to a VLAN structure).  

show run
Building configuration...

Current configuration : 4807 bytes
!
version 15.0
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
service compress-config
!
hostname Switch
!
boot-start-marker
boot system bootflash:cat4500-entservicesk9-mz.150-2.SG10.bin
boot-end-marker
!
enable secret
!
username ***** privilege 0 secret
username ***** privilege 0 secret
!
!
aaa new-model
!
!
aaa authentication login default local
aaa authorization exec default local
!
!
!
aaa session-id common
clock timezone UTC 1
ip subnet-zero
no ip domain-lookup
ip domain-name XXX.com
ip vrf mgmtVrf
!
!
!
vtp domain XXX.com
vtp mode transparent
!
!
!
power redundancy-mode redundant
port-channel load-balance src-dst-mac
!
!
!
!
spanning-tree mode mst
spanning-tree portfast bpduguard default
spanning-tree extend system-id
!
spanning-tree mst configuration
name Block001
revision 1
instance 1 vlan 220, 230, 235
!
!
vlan internal allocation policy ascending
!
vlan 220
name AllTraffic
private-vlan primary
private-vlan association 230,235
!
vlan 230
name StandAlone
private-vlan isolated
!
vlan 235
name cust001
private-vlan community
!
!
!
interface Port-channel1
description QNAP-THE-NAS-DR-eth1+2
switchport
switchport mode access
spanning-tree portfast
!
interface Port-channel2
description SwitchUplink
switchport
switchport trunk encapsulation dot1q
switchport mode trunk
switchport nonegotiate
!
interface Port-channel3
description Flex-Bay2
switchport
mtu 9198
!
interface FastEthernet1
ip vrf forwarding mgmtVrf
no ip address
speed auto
duplex auto
!
interface GigabitEthernet1/1
!
interface GigabitEthernet1/2
!
interface GigabitEthernet1/3
!
interface GigabitEthernet1/4
!
interface GigabitEthernet1/5
description Sumo-NAS-eth2
spanning-tree portfast
!
interface GigabitEthernet1/6
!
interface GigabitEthernet1/7
!
interface GigabitEthernet1/8
!
interface GigabitEthernet1/9
!
interface GigabitEthernet1/10
mtu 9198
!
interface GigabitEthernet1/11
!
interface GigabitEthernet1/12
!
interface GigabitEthernet1/13
!
interface GigabitEthernet1/14
!
interface GigabitEthernet1/15
!
interface GigabitEthernet1/16
!
interface GigabitEthernet1/17
description NITSSAN01 MGMT Ctrl B
!
interface GigabitEthernet1/18
!
interface GigabitEthernet1/19
description NITSSAN01 Crtl B e0c
mtu 9198
!
interface GigabitEthernet1/20
description NITSSAN01 Crtl A e0d
mtu 9198
!
interface GigabitEthernet1/21
!
interface GigabitEthernet1/22
!
interface GigabitEthernet1/23
!
interface GigabitEthernet1/24
!
interface GigabitEthernet1/25
description NAS-DR-eth1
switchport mode access
channel-protocol lacp
channel-group 1 mode active
!
interface GigabitEthernet1/26
description NAS-DR-eth2
switchport mode access
channel-protocol lacp
channel-group 1 mode active
!
interface GigabitEthernet1/27
!
interface GigabitEthernet1/28
!
interface GigabitEthernet1/29
!
interface GigabitEthernet1/30
!
interface GigabitEthernet1/31
description SS-DR-CB-P1
spanning-tree portfast
!
interface GigabitEthernet1/32
!
interface GigabitEthernet1/33
!
interface GigabitEthernet1/34
!
interface GigabitEthernet1/35
description IBM-FLEX-bay2-eth1
mtu 9198
channel-protocol lacp
channel-group 3 mode active
!
interface GigabitEthernet1/36
description IBM-FLEX-bay2-eth2
mtu 9198
channel-protocol lacp
channel-group 3 mode active
!
interface GigabitEthernet1/37
!
interface GigabitEthernet1/38
!
interface GigabitEthernet1/39
description THE-SAN
mtu 9198
!
interface GigabitEthernet1/40
description THE-SAN
mtu 9198
!
interface GigabitEthernet1/41
!
interface GigabitEthernet1/42
!
interface GigabitEthernet1/43
description 2960
switchport trunk encapsulation dot1q
switchport mode trunk
!
interface GigabitEthernet1/44
description Flex-CMM-1
!
interface GigabitEthernet1/45
spanning-tree portfast
!
interface GigabitEthernet1/46
description MX100-eth4
!
interface GigabitEthernet1/47
description SW1-Gig47
switchport trunk encapsulation dot1q
switchport mode trunk
switchport nonegotiate
channel-protocol pagp
channel-group 2 mode desirable
!
interface GigabitEthernet1/48
description SW1-Gig48
switchport trunk encapsulation dot1q
switchport mode trunk
switchport nonegotiate
channel-protocol pagp
channel-group 2 mode desirable
!
interface TenGigabitEthernet1/49
!
interface TenGigabitEthernet1/50
!
interface Vlan1
ip address x.x.x.x x.x.x.x
!
no ip http server
no ip http secure-server
!
!
!
!
!
snmp-server community ****** 
snmp-server host x.x.x.x version 2c ******
!
control-plane
!
!
line con 0
logging synchronous
stopbits 1
line vty 0 4
logging synchronous
transport input ssh
!
ntp clock-period 17181152
ntp server x.x.x.x prefer
end

Switch#

[Reza Sharifi]

Hi,

interface Port-channel3
description Flex-Bay2
switchport
mtu 9198

interface Vlan1
ip address x.x.x.x x.x.x.x

So, does the storage system live in VLAN-1?

If this is correct, I think because the interfaces to the storage are accepting large MTU from the storage system and then fragment it at layer-3 (SVI). Can you enable jumbo frame on the vlan interface (SVI) and test again?

HTH