arp error message on Nexus 7000

[leonmeg123]

hello,

we keep getting the following message and I could notr find any information on cisco website and over the web:

2012 Mar  4 07:10:46.211 MA-B11-N7-DEV-11-DEV-15 %ARP-3-INVAL_HDR_HRD_TYPE:  arp [16329]  Found incorrect hardware type in ARP header: 6

2012 Mar  4 07:11:46.111 MA-B11-N7-DEV-11-DEV-15 last message repeated 2 times

2012 Mar  4 07:12:47.016 MA-B11-N7-DEV-11-DEV-15 last message repeated 2 times

2012 Mar  4 07:13:59.025 MA-B11-N7-DEV-11-DEV-15 last message repeated 3 times

what does it mean?

Thanks

[phiharri]

Greetings,

This means your Nexus 7000 received an ARP packet with the hrd/htype field set to 6 (IEEE 802 Networks) rather than the typical 1 (Ethernet). Reference: http://www.iana.org/assignments/arp-parameters/arp-parameters.xml#hardware-type-rules

You could use Ethanalyser to identify the source of these ARP packets, example:

ethanalyzer local interface inband capture-filter 'arp' limit-captured-frames 0 detail | no-more

When the INVAL_HDR_HRD_TYPE syslog is seen, look for an ARP packet having Hardware Type 6 in the Ethanalyser output and identify the source MAC address. Use Ctrl+C to stop the capture.

Hope this helps,

/Phil

[Marc Lariviere]

Hi Philip,

I have the same issue, "Found incorrect hardware type in ARP header: 6", and it's not possible to identify the source with "ethanalyser".

I use this command to capture packeta for the inband interface:

ethanalyzer local interface inband capture-filter 'arp[1]!=1' detail | no-more

I use this command to capture the packet for the mgmt interface:

ethanalyzer local interface mgmt capture-filter 'arp[1]!=1' detail | no-more

The capture filter is to capture all arp packets with the hardware type different from "1".

I captured nothing. 

As it is not possible with ethanalyser to capture packets from other interfaces than "inband" and "mgmt", do you know another method to identify the source of these messages?

Thanks.

[phiharri]

Hey Marc,

Only just caught your reply here. How frequently are you seeing %ARP-3-INVAL_HDR_HRD_TYPE messages? If you're getting nothing in your captures I'd recommend simplifying the capture-filter to just 'arp' and correlating by timestamp, as I've often seen more specific filter strings (even when valid in a tcpdump sense) producing unexpected results due to how Ethanalyser handles the platform header of internal frames.

Regarding "other interfaces", Ethanalyser is a tool for analysing traffic towards the control-plane of the device and all such traffic (on Nexus 7000) passes either the 'inband' or 'mgmt' capture interfaces. Internally there are two Ethernet PHYs towards the Supervisor CPU which can receive traffic from the network, one ('inband') physically connected to the switching fabric to receive traffic redirected from the data-plane (eg. ARP on a user VLAN, ICMP to an address configured on an SVI), and another ('mgmt') which separately connects only to the mgmt0 front-panel interface for dedicated management purposes. All packets reaching the CPU from outside the switch arrive on one of these two capture interfaces and it's likely that frames causing %ARP-3-INVAL_HDR_HRD_TYPE messages would be received via 'inband'.

Cheers,

/Phil

[udo.konstantin]

Hi,

I also noticed this problem on our N7K switches. 

Is there currently a workaround to stop logging these messages ? And anyone know why only the N7K log this messages ? I mean the C6K VSS doesn't recognize these nor we saw any log message about these packets...

thanks

[kgras · Accepted solution]

Hello guys,

via SR 622442533 I found the reason for ARP packets with hardware type 6 being not collected via following command:

ethanalyzer local interface inband capture-filter 'arp' limit-captured-frames 0 detail | no-more

It is because they are NOT recognized as ARP packets at the capture-filter level, yet they are recognized as ARP packets at the display-filter level. Hence, the right method to capture those packets is to issue following command:

ethanalyzer local interface in display-filter "arp.hw.type !=1" limit-captured-frames 0 det

On the other hand, the keyword "det" is not so helpful here, because to find the source of such messages, we should rather focus on source MAC addresses.

So, the shorter format of the command above would be:

ethanalyzer local interface in display-filter "arp.hw.type !=1" limit-captured-frames 0

***

In the SR mentioned above, I proposed an enhancement to include source MAC address to the %ARP-3-INVAL_HDR_HRD_TYPE message, and I asked, if TAC would investigate this capture-filter behavior.

BR.

Krzysztof Gras.

[Marc Lariviere]

Hello Krzysztof,

Excellent !

With your solution, I found where ARP packets with hardware type 6 originated.

This is an IBM mainframe configured with an LLC protocol on Ethernet II.  Disabling the LLC protocol corrected the proble.

Thank you very mich, your answer is much appreciated.

Marc Larivière

[Ruediger Oertel]

thanks a milliion... finally found this thread.

These ARP-3-INVAL_HDR messages (I'm seeing them on a pair of nexus 5010) were driving me crazy.

With this display-filter I finally got the mac-address (and it's also a IBM mainframe here)

[the thread could possibly be marked as "correctly answered"]

I agree, having the mac in the error message would have been extremely helpful ...

[phiharri]

Hi All,

We have enhancement CSCub10429 "ARP-3-INVAL_HDR_HRD_TYPE error message should reference source MAC." open to include the source address directly in this message.

Cheers,

/Phil

[diepes]

Great post, found same problem on iSeries (AS400).

Default ethernet setup sends 802.3-SNAP and Ethernet 2 arp frames.

Need to delete and recreate ethernet line, setting "ETHSTD(*ETHV2)" to only use Ethernet Version 2.

[Marc Lariviere]

Hi all,

Although I found the source causing "ARP-3-INVAL_HDR_HRD_TYPE error message, how do I configure the Nexus 7000 for not having these messages?

Thanks.

[oboulat]

Hi Marc,

Do you have an update regarding this issue?

I've the same type of messages in my nexus 7000 coming from iseries too.

Thanks.

[andrey.l.cintra]

Marc,

We have the same problem, could you share how to disable LLC protocol on IBM mainframe?

Thanks in advanced.

[Marc Lariviere]

Hi oubulat,

No update yet. 

This is annoying because the log is flooded by these messages. It is therefore not possible to use the log to troubleshoot. I just want to know how to configure the Nexus 7000 that does not take account of these messages, but I do not know how.

Does anybody knows how to configure Nexus 7000 so that it does not take account of these messages ?

Thanks

[Marc Lariviere]

Hi all,

I found a way to get rid of "ARP-3-INVAL_HDR_HRD_TYPE" error messages.

When I use the command "show logging level", the 'arp' facility doesn't showed up.  But using the command "show logging level arp", I see this:

Facility        Default Severity        Current Session Severity

--------        ----------------        ------------------------

arp                     3                       3

So, I used this command:

conf t

logging level arp 2

end

But I will get rid of all "ARP-3-xxxx" messages.

Not elegant, but it works !