Hi,I have configured HSRP

Dragomir · ‎03-18-2014

I have an iscsi appliance attached to the nexus 7k and port on the appliance is set at 9216.

but when i ping from the switch to the appliance with

# ping 10.60.1.106 packet-size 9216 c 20
PING 10.60.1.106 (10.60.1.106): 9216 data bytes
9224 bytes from 10.60.1.106: icmp_seq=0 ttl=63 time=10.943 ms
Request 1 timed out
9224 bytes from 10.60.1.106: icmp_seq=2 ttl=63 time=8.93 ms
Request 3 timed out
9224 bytes from 10.60.1.106: icmp_seq=4 ttl=63 time=17.072 ms
Request 5 timed out
9224 bytes from 10.60.1.106: icmp_seq=6 ttl=63 time=8.977 ms
Request 7 timed out
9224 bytes from 10.60.1.106: icmp_seq=8 ttl=63 time=8.984 ms
Request 9 timed out
9224 bytes from 10.60.1.106: icmp_seq=10 ttl=63 time=9.028 ms
Request 11 timed out
9224 bytes from 10.60.1.106: icmp_seq=12 ttl=63 time=8.827 ms
Request 13 timed out
9224 bytes from 10.60.1.106: icmp_seq=14 ttl=63 time=9.171 ms
Request 15 timed out
9224 bytes from 10.60.1.106: icmp_seq=16 ttl=63 time=9.054 ms
Request 17 timed out
9224 bytes from 10.60.1.106: icmp_seq=18 ttl=63 time=8.76 ms
Request 19 timed out

they time out.

what would be the issue?

Steve Fuller · ‎03-19-2014

Hi,

There are a couple of things to consider here.

The value you specify with the packet-size option on the ping command is actually the data portion of the packet i.e., the value not including the ICMP and IP header. The packet sent by the router in the test you did would have been 9244-bytes (9216 data + 8-bytes ICMP + 20-bytes IP). The value you specify should be 28-bytes less than the MTU size to accommodate this.

Also when testing jumbo frames you should use the df-bit option, otherwise the router will simply fragment the packet that is sent and so you won't properly be able to determine the MTU size.

So if you have an end system with an MTU of 9216-bytes the command you should use to test end to end jumbo frame connectivity would be ping <ip-address> df-bit packet-size 9188. That way when the 8-byte ICMP and 20-byte IP header are added, the packet size will be 9216-bytes.

Just on this point, I would also confirm the MTU value set on the storage is 9216-bytes. Ethernet jumbo frames are not properly specified within any standard and so different vendors use different values. On end systems I've seen and played around with the MTU is generally 9000-bytes.

The other area to watch out for is Control Plane Policing (CoPP) on the Nexus. The Nexus 7000 runs CoPP by default to limit traffic hitting the CPU on switch supervisor. I've seen issues in the past with ICMP as the default value used within the policer quite low, and if you're specifying large packet sizes with the ping it's possible CoPP is dropping the traffic. You can use the show policy-map interface control-plane | in "class-map|violate" command to quickly check if you have drops in any of the classes. If the Nexus is running the default CoPP configuration then the class-map is likely to be copp-system-class-monitoring. Remember also note that you can only check copp from the default VDC.

If CoPP is dropping the ICMP traffic then you can change the police cir <value> within the policy-map, but obviously consider the reason for CoPP and don't increase the value to the point where CoPP is no longer effective.

Regards

Dragomir · ‎03-19-2014

Will not work

Any packet above 1472 gets dropped

--- 10.60.1.106 ping statistics ---

1 packets transmitted, 0 packets received, 100.00% packet loss

# ping 10.60.1.106 df-bit packet-size 1472

PING 10.60.1.106 (10.60.1.106): 1472 data bytes

1480 bytes from 10.60.1.106: icmp_seq=0 ttl=63 time=1.481 ms

1480 bytes from 10.60.1.106: icmp_seq=1 ttl=63 time=0.864 ms

1480 bytes from 10.60.1.106: icmp_seq=2 ttl=63 time=1.187 ms

1480 bytes from 10.60.1.106: icmp_seq=3 ttl=63 time=1.468 ms

1480 bytes from 10.60.1.106: icmp_seq=4 ttl=63 time=1.229 ms

--- 10.60.1.106 ping statistics ---

5 packets transmitted, 5 packets received, 0.00% packet loss

round-trip min/avg/max = 0.864/1.245/1.481 ms

# ping 10.60.1.106 df-bit packet-size 1473

PING 10.60.1.106 (10.60.1.106): 1473 data bytes

^C

--- 10.60.1.106 ping statistics ---

1 packets transmitted, 0 packets received, 100.00% packet loss

WSGRCSA# ping 10.60.1.106 df-bit packet-size 1473

PING 10.60.1.106 (10.60.1.106): 1473 data bytes

Request 0 timed out

Request 1 timed out

Request 2 timed out

Request 3 timed out

Request 4 timed out

----------------------------------------------------------------------------------------------

# show policy-map interface control-plane | in "class-map|violate"
class-map copp-system-p-class-critical (match-any)
violated 0 bytes; action: drop
violated 0 bytes; action: drop
class-map copp-system-p-class-important (match-any)
violated 564980520 bytes; action: drop
violated 16564096880 bytes; action: drop
class-map copp-system-p-class-management (match-any)
violated 0 bytes; action: drop
violated 0 bytes; action: drop
class-map copp-system-p-class-normal (match-any)
violated 38979438 bytes; action: drop
violated 102233820206 bytes; action: drop
class-map copp-system-p-class-normal-dhcp (match-any)
violated 15555618 bytes; action: drop
violated 257350720 bytes; action: drop
class-map copp-system-p-class-normal-dhcp-relay-response (match-any)
violated 0 bytes; action: drop
violated 0 bytes; action: drop
class-map copp-system-p-class-redirect (match-any)
violated 0 bytes; action: drop
violated 0 bytes; action: drop
class-map copp-system-p-class-exception (match-any)
violated 0 bytes; action: drop
violated 0 bytes; action: drop
class-map copp-system-p-class-monitoring (match-any)
violated 278218 bytes; action: drop
violated 274881 bytes; action: drop
class-map copp-system-p-class-l2-unpoliced (match-any)
violated 0 bytes; action: transmit
violated 0 bytes; action: transmit
class-map copp-system-p-class-undesirable (match-any)
violated 8578 bytes; action: drop
violated 20752 bytes; action: drop
class-map copp-system-p-class-l2-default (match-any)
violated 68855 bytes; action: drop
violated 5191298 bytes; action: drop
class-map class-default (match-any)
violated 39535752498 bytes; action: drop
violated 78583238971 bytes; action: drop

i have a lot of dropped or violated packets.

Steve Fuller · ‎03-20-2014

Hi,

Let's forget the Control Plane Policing at the moment then as there's a more fundamental issue in that you don't have jumbo frames configured end-to-end. This can be seen as you're dropping any ICMP packets larger than 1472-bytes with the DF bit set.

Can you provide details of how the storage is connected specifically the following:

Is the iSCSI storage device connected directly to the Nexus 7K and if so is the interface it is connected to configured with mtu 9216?
Does the storage connect via multiple interfaces e.g., port-channel, and if so are all interfaces configured for jumbo frames?
Are you also routing on the Nexus 7K and if so do the VLAN interfaces have mtu 9216 configured ?

Can you post the relevant show interface commands from the Nexus, and if possible an ifconfig (or equivalent command) from the storage?

Regards

Dragomir · ‎03-24-2014

hello

Storage is connected directly to the 7k. Verified that the port is set to mtu 9216.

Also verified on the appliance itself that the port is at mtu 9000.

storage connects via multiple interfaces but not port channel. EAch iscsi interface functiona as individual interfaces.

Are you also routing on the Nexus 7K and if so do the VLAN interfaces have mtu 9216 configured ?

how do I check this?

sh interface ethernet 3/34
Ethernet3/34 is up
admin state is up, Dedicated Interface
Hardware: 1000/10000 Ethernet, address: 6073.5c04.98cd (bia 6073.5c04.98cd)
Description: storesimple data3 primary
MTU 9216 bytes, BW 1000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is access
full-duplex, 1000 Mb/s, media type is 1G
Beacon is turned off
Auto-Negotiation is turned off
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
Last link flapped 5d06h
Last clearing of "show interface" counters never
81 interface resets
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 3320 bits/sec, 1 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 0 bps, 0 pps; output rate 2.70 Kbps, 1 pps
RX
57989568648 unicast packets 188 multicast packets 6254 broadcast packets
57989437082 input packets 94861146540597 bytes
1481562539 jumbo packets 0 storm suppression packets
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
17057991061 unicast packets 42719697 multicast packets 42453871 broadcast packets
17143026620 output packets 19149860787894 bytes
708264551 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause

# sh interface ethernet 3/36
Ethernet3/36 is up
admin state is up, Dedicated Interface
Hardware: 1000/10000 Ethernet, address: 6073.5c04.98cf (bia 6073.5c04.98cf)
Description: storesimple secondary data1
MTU 9216 bytes, BW 1000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is access
full-duplex, 1000 Mb/s, media type is 1G
Beacon is turned off
Auto-Negotiation is turned off
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
Last link flapped 21week(s) 1day(s)
Last clearing of "show interface" counters never
6 interface resets
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 2608 bits/sec, 3 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 0 bps, 0 pps; output rate 1.79 Kbps, 3 pps
RX
12217843366 unicast packets 0 multicast packets 145 broadcast packets
12217705503 input packets 8077006642708 bytes
0 jumbo packets 0 storm suppression packets
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
35911585661 unicast packets 28648470 multicast packets 90374246 broadcast packets
36030470368 output packets 52448125979820 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause

Dragomir · ‎03-25-2014

hi

when i look at my vlan interface

I see this

interface Vlan60
no ip redirects
ip address 10.60.0.1/16
no ipv6 redirects
ip proxy-arp
no hsrp bfd
hsrp version 2
hsrp delay minimum 0 reload 0
no hsrp use-bia
hsrp 60
authentication cisco
name hsrp-Vlan60-60
mac-address 0000.0C9F.F03C
no preempt
priority 100 forwarding-threshold lower 1 upper 100
timers 3 10
ip 10.60.0.5
description SERVERS60
no shutdown
mtu 1500
bandwidth 1000000
delay 1
medium broadcast
snmp trap link-status
carrier-delay msec 100
load-interval counter 1 60
load-interval counter 2 300
no load-interval counter 3
mac-address d867.d907.ffc1
no management

looks like mtu is at 1500.

do I need to change mtu on vlan interfaces as well? According to cisco document they did not mention it.

http://www.cisco.com/c/en/us/support/docs/switches/nexus-5000-series-switches/112080-config-mtu-nexus.html

Steve Fuller · ‎03-25-2014

Hi,

If all your hosts and storage are in the same Layer-2 broadcast domain i.e., no Layer-3 hops between them, then there should be no need to configure mtu 9216 i.e., jumbo frames, on your Nexus 7000 VLAN interfacces.

If there are any layer-3 hops between the hosts and storage, or if you want to be able to ping the storage or hosts with jumbo frames from the Nexus 7000, then you will need to add mtu 9216 to the VLAN interfaces.

That's an interesting document that you link to above, and in my humble opinion, not very well produced. In the first part of the document it is shown that ping 172.16.0.1 packet-size 9216 c 20 fails, but not really explained why. There's no use of the df-bit option, and as seen, this is key when testing jumbo frames. Additionally after jumbo frames are enabled, you are told to verify with the ping -l 9000 command. Why do they not verify using the same ping as originally?

Regards

Mukund M Deshpande · ‎09-28-2016

Hi,

I have configured HSRP with nexus 7000 (primary) and 4500 (Secondary)

I am able to ping hsrp virtual ip from cmd with 15000 byte but not able to ping same ip with 17000 byte for nexus device.

Please help me on this

jumbo frames dropped when pining from nexus 7k to iscsi port appliance