Sles11 SP2 on C220-M3 vNIC errors

[Timo Scheller]

Hi,

we installed Sles11 SP2 on our new C220-M3 (1.5.1b), which is connected to our N5k with a VIC 1225 and has the two default vNiCs eth0 and eth1.

What we see with the default enic driver from the Sles11 is a lot of packet loss. The N5k is configured as mode access with one VLAN.

We thougt the packet loss came maybe from the missing VLAN or access-mode in the vNIC eth0.

But ff we try and set the VLAN and the access mode, by the way really great CLI, to the same VLAN an access-mode as on the N5k,

then there can, after a reboot of course, no more TCP/IP connections made.

We can`t find any related documents for such an "normaly easy" setup getting a server with one 10GBit default vNIC up and running properly.

So we can`t find any proper documentation for SLES11, the enic-Driver and the vNIC on a C220 (or related).

Thanks in advance for any hints on this one.

If anyone can shed some lights on this, i am happy to make a documenation on this one. If it is working. ;-)

All the best

Timo

OK, here is a little bit more of a message:

Jun  3 15:21:32 HOSTNAME kernel: [ 2808.923467] enic 0000:08:00.0: eth0: Link DOWN

Jun  3 15:21:32 HOSTNAME kernel: [ 2808.924528] host0: libfc: Link down on port (000000)

Jun  3 15:22:39 HOSTNAME kernel: [ 2875.333162] enic 0000:08:00.0: eth0: Link UP

And this is the moment, when every TCP/IP related traffic is cut off.

[padramas]

Hello Timo,

A few question and system will perform the tricks it is supposed to do :-)

How many vNICs are created on VIC 1225 ?

Have you configured teaming of these vNIC in OS ?

Are the physical ports of VIC connected to single N5K switch ?

Are these individual links or in port-channel ?

N5K hardware model and version ?

Please make sure firmware and drivers are as per HCL

http://www.cisco.com/web/techdoc/ucs/interoperability/matrix/matrix.html

---------------------------------------

Adapter Driver   = 1.5.0.22 ( FNIC )  / 2.1.1.41 ( NIC )

Adapter Firmware = 2.1(1c)

Boot Code / BIOS = 2.1(1c)

-------------------------------------

Padma

[Timo Scheller]

Hi Padma,

thanks for that quick response, ill try to answer:

>How many vNICs are created on VIC 1225 ?

Default, two, eth0 and eth1

>Have you configured teaming of these vNIC in OS ?

No, only eth0 is configured.

>Are the physical ports of VIC connected to single N5K switch ?

No, just one port (port 0) is connected to N5K-Switch (but there is a pair of N5K)

>Are these individual links or in port-channel ?

Individual, just one VLAN on this link

>N5K hardware model and version ?

Sorry, no access to that at the moment

>Adapter Driver   = 1.5.0.22 ( FNIC )  / 2.1.1.41 ( NIC )

FNIC = 1.5.0.2 (with dmesg |grep -i fnic) who came with SLES11

ENIC = 2.1.1.41 with rpm installed from latest Cisco Driver-CD

>Adapter Firmware = 2.1(1c)

Check

>Boot Code / BIOS = 2.1(1c)

Check

[padramas]

Hello Timo,

I did setup a lab with same hardware and software configuration and it was working as expected.

SLES 11 SP 2, VIC 1225, NIC driver 2.1.1.41 , connected to N5548 running 6.0.2 version.

Can you please verify the layer 1 issues ?

On SLES, check the stats via ethtool -S eth0

On N5K, check the counters by

show interface eth x/y

show interface eth x/y counters detailed all

Padma

[Timo Scheller]

Hi Padma,

i sent you a PM with the detailed information to your questions.

I am not allowed to put this information on a public platform.

But if we found what the problem is, i will put here all detailed information.

Thx in advance.

Timo

[Timo Scheller]

Dear all,

we managed, with many thanks to Padma, to find the problem:

http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/system_management/configuration/guide/sm_lldp.html#wp1300592

It has to do with the DCBXP (Data Center Bridging Protocoll) and the (new) Link Layer Discovery Protocol (LLDP) which is integrated in the DCBXP. Per default this protocoll is enabled globally on the Nexus 5k.

So what we did was this on the interface connected to the C220 series.

interface Ethernet1/19

  no lldp transmit

  no lldp receive

After that, no more timeouts, no loss of connections, no more "Link down" entries on /var/log/messages.

There is (some) information on the LLDP. And on SLES11 SP2 you got also an agent for this, called

lldpad (zypper se lldpad) which is part of the core-repos on SP2.

But this is a lot of information on the lldpad to get to, which needs time to do it. So for the time beeing, we just

disabled lldp on this specific port.

That did the trick for us.

HTH,

Timo

[padramas]

Hello Timo,

Thanks for the follow up.

In my setup, I do not have lldpad package installed in SLES.

LLDP is handled by VIC ( CNA ) firmware and is tansparant to OS. I am not clear about the usecase, but you can remove the lldpad package from OS and turn on lldp on the switchport.

If you are planning to connect to FC / FCoE SAN arrays, LLDP is required for FCoE between CNA and switchport.

Padma

[padramas · Accepted solution]

And a few commands to check lldp peers connected to N5K

show lldp nei

show system internal dcbx info interface eth x/y | no-more

ethanalyzer local interface inbound-hi display-filter vlan.etype==0x88cc limit-captured-frames 0

HTH

Padma

[Timo Scheller]

Hi Padma,

we hadn`t installed lldpad nor was it by default installed. After digging around, we tried it. But too complex for a quick setup. If anybody has any more tips and tricks about that, feel free to post some stuff. 

You`re right, LLDP is handled by VIC. But the N5K doesn`t like it, if you configure your port just as an access port for 10GigE without the need of FCoE.

And then, that`s were LLDP comes into the play.

But we have solution and it works perfektly for our need.

Much thanks for the additional information on LLDP. I`ll try this in the next weeks to see, what comes out of it. ;-)

All the best,

Timo