C-200 M2 with P81E to Nexus 5548UP

Burak Atasal · ‎01-23-2012

Hi all,

I'm working on a project which is about to set up a Cisco UCS Lab.

Actually the topology looks simple and easy but there must be some missconfiguration on my side because i can't see any of IDs (P81E's wwpn and mac address info) from the Nexus 5548UP.

I connected my 5548UP to MDS 9124 for providing the FCoE traffic to the C-200 Farm.

Then i configured the MDS with NPIV and converted couple of ethernet ports to the FC ports on 5548UP (which i'm using to connect to the MDS)

Finally i connected my C-200M2's VIC to the 5548UP but i couldn't see any information about server. The ports actually are up. But can't see nothing.

Some help and advises will be appreciated.

padramas · ‎01-23-2012

Hello Burak,

To start with, check out following configuration parameters

Option ROM slot is enabled for P81E card

FIP is enabled on the HBA properties

Make sure the native vlan configured on Nexus switchport matches the default VLAN defined on vHBA properties on the card.Reboot of the server is required for the changes to take effect.

Padma

Burak Atasal · ‎01-24-2012

Hi Padramas thank you for your reply first.

I discoverd that setting while browsing for some additional settings about vHBAs. Then i saw that FIP and NIV mode options. i checked enable and save configuration but when i look back again to that settings it was unchecked. I did some searches on internet but didn't find anything.

I also connected to the CIMC via SSH and i did the same configuration there but again FIP changed to disable again.

If you have any advises about that i'll be glad for it.

Thank you again.

padramas · ‎01-24-2012

Burak,

Do you receive any error message while saving the changes ?

Please provide me the following commands output

scope chassis

show adapter 1 detail

set fip-mode enable

commit

show adapter 1 detail

show host-fc-if detail

Here is adapter config guide

http://www.cisco.com/en/US/docs/unified_computing/ucs/c/sw/cli/config/guide/1.4.1/Cisco_UCS_C-Series_CLI_Configuration_Guide_141_chapter9.html

Also provide us the CIMC version.

Padma

Burak Atasal · ‎01-24-2012

Actually that issue was about the Firmware version. I think i changed it yesterday after i tried 1000 different options.

Now i can save FIP setting.

The main problem is ;

I connected P81E 10GB ports to the N5548UP with Twinax cables. And server ports are up now. But i can not see any mac or wwpn information of server. For example flogi database or fcns databases are empty while server is powered on and ports are up.

Then i discovered that i installed ESXi 4.1 to the local drive of C200 before. When the POST screen appears i can see flogi database entries on N5548 but when the ESXi comes up the flogi database dissapears again.

All i want is running FCoE on the 5548 and providing access to the Storage.

padramas · ‎01-24-2012

Burak,

Can you please share the switchport and vfc configuration ?

Does your storage array allows login from this wwpn ?

Padma

Burak Atasal · ‎01-24-2012

Padma,

Right now i connected the N5548 directly to the storage for that reason you can not see the fc uplink connections on running-configuration.

On Nexus 5548 ;

Ethernet 1/1 and Ethernet 1/2 are the server ports (P81E)

Ethernet 1/15 is the Uplink Ethernet port

Fc 1/31 1/32 is the Storage Ports

Storage has been configured for the wwpns and LUNs.

And here is the configuration :

N5548(config-if)# show running-config

!Command: show running-config

!Time: Tue Jan 24 13:32:53 2012

version 5.0(3)N2(1)

feature fcoe

feature telnet

feature lldp

username admin password 5 $1$XCMHODt3$yEItwOs0Vl.LLYyXBPjZu0 role network-admin

no password strength-check

ip domain-lookup

hostname N5548

class-map type qos class-fcoe

class-map type queuing class-fcoe

match qos-group 1

class-map type queuing class-all-flood

match qos-group 2

class-map type queuing class-ip-multicast

match qos-group 2

class-map type network-qos class-fcoe

match qos-group 1

class-map type network-qos class-all-flood

match qos-group 2

class-map type network-qos class-ip-multicast

match qos-group 2

slot 1

port 1-24 type ethernet

port 25-32 type fc

snmp-server user admin network-admin auth md5 0x29cde3d41c6724dcbec14cf28387824d

priv 0x29cde3d41c6724dcbec14cf28387824d localizedkey

snmp-server enable traps entity fru

vrf context management

vlan 1

fcoe vsan 1

fcdomain fcid database

vsan 1 wwn 50:06:0e:80:10:4d:03:60 fcid 0xa10000 dynamic

vsan 1 wwn 50:06:0e:80:10:4d:03:62 fcid 0xa10001 dynamic

interface vfc1

bind interface Ethernet1/1

no shutdown

interface vfc2

bind interface Ethernet1/2

no shutdown

interface fc1/25

interface fc1/26

interface fc1/27

interface fc1/28

interface fc1/29

interface fc1/30

interface fc1/31

no shutdown

interface fc1/32

no shutdown

interface Ethernet1/1

interface Ethernet1/2

interface Ethernet1/3

interface Ethernet1/4

interface Ethernet1/5

interface Ethernet1/6

interface Ethernet1/7

interface Ethernet1/8

interface Ethernet1/9

interface Ethernet1/10

interface Ethernet1/11

interface Ethernet1/12

interface Ethernet1/13

interface Ethernet1/14

interface Ethernet1/15

switchport mode trunk

speed 1000

interface Ethernet1/16

interface Ethernet1/17

interface Ethernet1/18

interface Ethernet1/19

interface Ethernet1/20

interface Ethernet1/21

interface Ethernet1/22

interface Ethernet1/23

interface Ethernet1/24

interface mgmt0

ip address 192.168.10.210/24

line console

line vty

boot kickstart bootflash:/n5000-uk9-kickstart.5.0.3.N2.1.bin

boot system bootflash:/n5000-uk9.5.0.3.N2.1.bin

interface fc1/25

interface fc1/26

interface fc1/27

interface fc1/28

interface fc1/29

interface fc1/30

interface fc1/31

interface fc1/32

N5548(config-if)#

padramas · ‎01-24-2012

Burak,

We need to make sure that FCoE VLAN on the adapter matches the switch port configuration.

Can you please share following command output

From CIMC CLI,

scope chassis

show adapter 1 detail

show host-fc-if detail

From Nexus switch

show int vfc 1

show int vfc 2

show vsan membership

Padma

Burak Atasal · ‎01-27-2012

Hi Padma first of all sorry for late response. Second is i fixed that issue. It was about VSAN trunking on the vfcs that i created.

Now the system is seems like working but actually not. C200 can go and find Server specific LUN. I can see my ESXi. And i can see the flogi database outputs on the N5548 but after a while (when the host operating system loaded ) the flogi database disappeares and VFCx looses the Vsan traffic.

For me possible problems :

1. ESXi driver problem!

2. we did not create a zoneset configuration on N5548 s this a must!!!!)

3. or the VIC configuration.

because of this issue i can use my ESX OS but cannot see any LUN under the storage tab. Even the OS installed LUN.

Need to find out how can i provide my vHBAs up permanently.

padramas · ‎01-28-2012

Burak,

Do you notice any specific error message ( in ESXiarray, Nexus switch ) around the time when ESXi loses connection to LUN / datastore ?

Padma

Burak Atasal · ‎01-28-2012

Actually i fixed all the problems tonight and now All C-200M2 with Palo Adapter.

Here is the problem and the solution.

First of all i was using VMware ESXi 4.1 Update 1 for the UCS Lab enviroment because original Lab Guide says to use that build.

That build has not included the proper drivers for Palo Adapter.Which is using 1.6.(1c) firmware.Anyway, beacuse of this driver mismacth after the OS Loading the vHBAs disappears.

I figured out this issue by trying to deploy ESXi5.0. But when i googled it this afternon a topic has come in first place says "How to inject P81E driver to ESXi5.0 iso" and i get this must be the driver problem! The adapter needs it's driver to continue operating.

Then i watched the video and applied it to the ESXi5.0.iso. After i tried, it worked!

All the configuration i did on the Nexus5548UP and MDS9124 was correct.

I hope it helps someone else other then me.

Thanks for your pation and help.

BR from Istanbul,

Burak.

padramas · ‎01-29-2012

Burak,

Good work and thanks for sharing the resolution with the community

Padma