Seeing input errors relating to Giant packets on ASR device connected to Nexus3K

nishikesh · ‎07-17-2019

In our DC we have connection between TWO devices via 10Gbase-SR-S module.

|-------------| |---------|

| ASR |-------------------| Nexus3K |------> Link Going to other devices to reach to internet.

|-------------| |---------|

Sample diagram attached.

ASR = is managed by Customer to which we do not have access.

Nexus3K = is our managed device connected to ASR. To reach to internet traffic/customer will go through our Nexus3K.

Now customer is complaining that he is getting input errors relating to Giant packets on ASR interface connected to Nexus3K.

HTML

Can someone please advise how to troubleshoot & resolve this input error & giant frame issue. Both interface outputs given below:

Interface output from ASR Router:

============================

TenGigabitEthernet0/2/0 is up, line protocol is up
Hardware is SPA-1X10GE-L-V2, address is XXXX.XXXX.XXXX (bia XXXX.XXXX.XXXX)
Description: LINK TO MYDC
Internet address is 1.1.1.1/30
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 8/255
Encapsulation ARPA, loopback not set
Keepalive not supported
Full Duplex, 10000Mbps, link type is force-up, media type is 10GBase-SR/SW
output flow-control is on, input flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:23, output 00:08:18, output hang never
Last clearing of "show interface" counters never
Input queue: 0/375/0/77 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: Class-based queueing
Output queue: 0/40 (size/max)
30 second input rate 323818000 bits/sec, 48155 packets/sec
30 second output rate 77769000 bits/sec, 31904 packets/sec
128878724099 packets input, 104957199191876 bytes, 0 no buffer
Received 58 broadcasts (0 IP multicasts)
0 runts, 131419 giants, 0 throttles
131419 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 1104324 multicast, 0 pause input
77159795011 packets output, 25230501108212 bytes, 0 underruns
0 output errors, 0 collisions, 37 interface resets
521082 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

Interface output from Nexus3K:

============================

Ethernet1/4 is up
admin state is up, Dedicated Interface
Hardware: 100/1000/10000 Ethernet, address: XXXX.XXXX.XXXX (bia XXXX.XXXX.XXXX)
Description: INTERNET ASR
Internet Address is 1.1.1.2/30
MTU 9154 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 11/255, rxload 2/255
Encapsulation ARPA, medium is broadcast
full-duplex, 10 Gb/s, media type is 10G
Beacon is turned off
Auto-Negotiation is turned on FEC mode is Auto
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
Last link flapped 6week(s) 6day(s)
Last clearing of "show interface" counters 23:43:02
0 interface resets
Load-Interval #1: 30 seconds
30 seconds input rate 112207000 bits/sec, 47856 packets/sec
30 seconds output rate 468268584 bits/sec, 66750 packets/sec
input rate 112.21 Mbps, 47.86 Kpps; output rate 468.27 Mbps, 66.75 Kpps
Load-Interval #2: 5 minute (300 seconds)
300 seconds input rate 115136448 bits/sec, 49080 packets/sec
300 seconds output rate 491616600 bits/sec, 69325 packets/sec
input rate 115.14 Mbps, 49.08 Kpps; output rate 491.62 Mbps, 69.32 Kpps
RX
2326472037 unicast packets 141 multicast packets 0 broadcast packets
2326472178 input packets 775986649348 bytes
0 jumbo packets 0 storm suppression packets
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
3695012739 unicast packets 4267 multicast packets 0 broadcast packets
3695017006 output packets 2982950344189 bytes
43444 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause

Giuseppe Larosa · ‎07-17-2019

Hello nishikesk,

you can see that the root cause for this issue is the fact that the two devices are using a different MTU

ASR:

>> MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,

Nexus3K:

>> MTU 9154 bytes, BW 10000000 Kbit, DLY 10 usec

However, the impact of Giant error packets is minimal as we need to compare input errors to total input packets on ASR interface:

error ratio = input errors / (input errors + input packets) = 131419 / ( 131419 + 128878724099) = 1,01 10^-6.

To fix this issue on the Nexus you need to change the MTU.

You need to verify if NX-OS counts the L2 overhead in MTU settings in this case you should set the MTU to 1514 bytes

if NX-OS is like IOS and the MTU is a layer3 concept you should set the MTU to 1500 bytes.

However, there is no impact with an error ratio of 1 10^-6.

So if for any reason you cannot change the MTU on the Nexus interface (if it is set at global node level) you can explain this to the customer.

Hope to help

Giuseppe

nishikesh · ‎07-18-2019

Thanks for the information..

However it was working with same MTU setting and without any issue before 10th July 2019, this input error started from 10th July only. Not sure what went wrong.

Also the MTU 9154 is setup on all our DC devices or globally for this customer. I have shared that information with customer already..

Now not sure how to remediate this input error related issue.

Giuseppe Larosa · ‎07-18-2019

Hello nishikesk,

it means there is at least one host inside the datacenter trying to send packets with IP size greater then 1500 bytes to the Internet ASR router.

Only one packet every million is a giant packet and discarded so there is no real impact for the customer.

Hope to help

Giuseppe

Giuseppe Larosa · ‎07-18-2019

Hello nishikesh,

>> However it was working with same MTU setting and without any issue before 10th July 2019, this input error started from 10th July only. Not sure what went wrong.

As I have already explained you have an MTU mismatch on the interfaces, each end user host in the datacenter could use bigger MTU.

However, you have one error every one million of packets received, so this means that almost all hosts use normal packets in accessing internet.

Look for a new server installation in the DC in day July 10th. Likely you will find the host that tries to use bigger packets.

But from a networking point of view this is not a real problem.

Hope to help

Giuseppe