Configuration Guide for 6248 and 5548 and VNX 5500

Amit Vyas · ‎04-08-2013

Hi,

Need help here to configure the FCoE Uplink + Ethernet Uplink in Port channel

Following is the connectivity details from 6248, 5548 and VNX5500.

I have configured Port Channel in following way on Both Fabric Interconnect

After Configuring above things I am getting following Faults on Servers

For NetApp3240 ENM source pinning failed, I have allowed VLAN ID 21 in Uplink manager then too this Link state is not showing UP and Same thing go for both FC_vHBA

Kinldy help me to configure the setup

-Amit

padramas · ‎04-08-2013

Hello Amit,

For Ethernet vNIC with ENM source pinning failed,please check whether configured on that vNIC is allowed on respective FI Ethernet uplinks

For vHBAs, please make sure that VSAN to VLAN mapping is configured correctly on FI and N5K.

connect nxos a

show int vfc 768

show int port-channel

Padma

Amit Vyas · ‎04-08-2013

Thanks fory quick revert Padma,

Below is the out for all commands which you asked

Switch-A(nxos)# show interface vfc 768

vfc768 is down (NPV upstream port not available)

Bound interface is Vethernet8960

Port description is server 1/1, VHBA FC_vHBA_1

Hardware is Virtual Fibre Channel

Port WWN is 22:ff:00:2a:6a:09:8c:3f

Admin port mode is F, trunk mode is on

snmp link state traps are enabled

Port vsan is 100

1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

414 frames input, 50796 bytes

1 discards, 0 errors

12478 frames output, 24145448 bytes

0 discards, 0 errors

last clearing of "show interface" counters never

Interface last changed at Mon Apr 8 09:35:05 2013

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

tulip-A(nxos)# show interface port-channel 4

port-channel4 is up

Hardware: Port-Channel, address: 547f.eee2.5b76 (bia 547f.eee2.5b76)

Description: C: FcoeUplink

MTU 1500 bytes, BW 20000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA

Port mode is trunk

full-duplex, 10 Gb/s

Beacon is turned off

Input flow-control is off, output flow-control is off

Switchport monitor is off

EtherType is 0x8100

Members in this channel: Eth2/7, Eth2/8

Last clearing of "show interface" counters never

30 seconds input rate 1400 bits/sec, 175 bytes/sec, 0 packets/sec

30 seconds output rate 280 bits/sec, 35 bytes/sec, 0 packets/sec

Load-Interval #2: 5 minute (300 seconds)

input rate 672 bps, 0 pps; output rate 352 bps, 0 pps

RX

43 unicast packets 234 multicast packets 0 broadcast packets

277 input packets 135320 bytes

43 jumbo packets 0 storm suppression bytes

0 giants 0 input error 0 short frame 0 overrun 0 underrun 0 watchdog 0 if down drop

0 input with dribble 0 input discard

0 Rx pause

TX

0 unicast packets 244 multicast packets 0 broadcast packets

244 output packets 39698 bytes

0 jumbo packets

0 output errors 0 collision 0 deferred 0 late collision

0 lost carrier 0 no carrier 0 babble

0 Tx pause

1 interface resets

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

tulip-A(nxos)# show interface port-channel 5

port-channel5 is up

Hardware: Port-Channel, address: 547f.eee2.5b7b (bia 547f.eee2.5b7b)

Description: C: FcoeUplink

MTU 1500 bytes, BW 20000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA

Port mode is trunk

full-duplex, 10 Gb/s

Beacon is turned off

Input flow-control is off, output flow-control is off

Switchport monitor is off

EtherType is 0x8100

Members in this channel: Eth2/12, Eth2/13

Last clearing of "show interface" counters never

30 seconds input rate 192 bits/sec, 24 bytes/sec, 0 packets/sec

30 seconds output rate 192 bits/sec, 24 bytes/sec, 0 packets/sec

Load-Interval #2: 5 minute (300 seconds)

input rate 384 bps, 0 pps; output rate 368 bps, 0 pps

RX

0 unicast packets 238 multicast packets 0 broadcast packets

238 input packets 42180 bytes

0 jumbo packets 0 storm suppression bytes

0 giants 0 input error 0 short frame 0 overrun 0 underrun 0 watchdog 0 if down drop

0 input with dribble 0 input discard

0 Rx pause

TX

0 unicast packets 243 multicast packets 0 broadcast packets

243 output packets 40252 bytes

0 jumbo packets

0 output errors 0 collision 0 deferred 0 late collision

0 lost carrier 0 no carrier 0 babble

0 Tx pause

1 interface resets

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Do let me know anything else is required from my end

Regarding ENM source pinning issue it got resolved after chaning Port channel setting at N5K end. We made it port channel mode active at N5K end and it got started working. Also wanted to check if we enable "vpc" at N5K end do we need to do any changes at 6248 lelve?

Awaiting for your quick revert

-Amit

padramas · ‎04-08-2013

Amit,

Have you configured any hard pinning for vHBA 768 ( server 1/1 ) ?

Is VSAN 100 / FCoE vlan mapping created on N5K switch ?

Can you disable / enable the FCoE port channel interface ( if it is not in production ) ?

Regarding ethernet port-channel, no additional configuration is needed on FI. If FI is in End host mode, you can configure N5K vpc interface as STP trunk edge.

Padma

Amit Vyas · ‎04-08-2013

No, we haven't configured any hard pinning

Issue got resolved after changing "vsan" & "vfc" setting at N5K end.

Yes, we are running FIs in End Host mode but while configuring "vpc" at N5K end getting following error / issue

----------------------------------------------------------------------------

id Port Status Consistency Reason Active vlans

------ ----------- ------ ----------- -------------------------- -----------

2 Po2 up failed vpc port channel -

mis-config due to vpc

links in the 2 switches

connected to different

partners

Let me know what details you want from N5k, will provide you at earliest

Update: Regarding Major Faults in Admin tag

How can I clear this errors ?

-Amit

Amit Vyas · ‎04-09-2013

Any updated on above questions ?

I am still facing both issues as describe above

1. vPC - error message

2. Major faults on UCSM

Please help

-Amit

padramas · ‎04-09-2013

Hello Amit,

1) UCSM fault - F1083 - VSAN membership down

This is cosmetic error message as we create vfc as a trunk interface but is initialized only certain VSANs.

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCud81857

2) vPC status

Please share the output of following commands from both N5K switches

show vpc

show run int port-channel membership

sh int port-channel

Padma

Amit Vyas · ‎04-10-2013

Below is the ouput for two commands "show run int port-channel membership" and "sh int port-channel "

===========================================================================

primary# sh run int po2 membership

!Command: show running-config interface port-channel2 membership

!Time: Wed Feb 25 07:55:31 2009

version 5.2(1)N1(1b)

interface port-channel2

description ***Link to FI-A in R17***

switchport mode trunk

switchport trunk allowed vlan 12-13,21,101-600

spanning-tree port type edge

spanning-tree bpdufilter enable

interface Ethernet1/3

description ***Link to FI-A in R17***

switchport mode trunk

switchport trunk allowed vlan 12-13,21,101-600

channel-group 2 mode active

interface Ethernet1/4

description ***Link to FI-A in R17***

switchport mode trunk

switchport trunk allowed vlan 12-13,21,101-600

channel-group 2 mode active

interface Ethernet1/5

description ***Link to FI-B in R18***

switchport mode trunk

switchport trunk allowed vlan 12-13,21,101-600

channel-group 2 mode active

===========================================================================

primary# sh run int po3 membership

!Command: show running-config interface port-channel3 membership

!Time: Wed Feb 25 07:57:14 2009

version 5.2(1)N1(1b)

interface port-channel3

description ***Link to FI-B in R18***

switchport mode trunk

switchport trunk allowed vlan 12-13,21,101-600

spanning-tree port type edge

spanning-tree bpdufilter enable

interface Ethernet1/8

description ***Link to FI-B in R18***

switchport mode trunk

switchport trunk allowed vlan 12-13,21,101-600

channel-group 3 mode active

interface Ethernet1/9

description ***Link to FI-B in R18***

switchport mode trunk

switchport trunk allowed vlan 12-13,21,101-600

channel-group 3 mode active

interface Ethernet1/10

description ***Link to FI-B in R18***

switchport mode trunk

switchport trunk allowed vlan 12-13,21,101-600

channel-group 3 mode active

===========================================================================

primary# sh int po2

port-channel2 is up

Hardware: Port-Channel, address: 002a.6a0a.4caa (bia 002a.6a0a.4caa)

Description: ***Link to FI-A in R17***

MTU 1500 bytes, BW 30000000 Kbit, DLY 10 usec

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA

Port mode is trunk

full-duplex, 10 Gb/s

Input flow-control is off, output flow-control is off

Switchport monitor is off

EtherType is 0x8100

Members in this channel: Eth1/3, Eth1/4, Eth1/5

Last clearing of "show interface" counters never

30 seconds input rate 504 bits/sec, 0 packets/sec

30 seconds output rate 2256 bits/sec, 0 packets/sec

Load-Interval #2: 5 minute (300 seconds)

input rate 456 bps, 0 pps; output rate 656 bps, 0 pps

RX

22208045 unicast packets 747374 multicast packets 27634234 broadcast packets

50589653 input packets 49083144072 bytes

21411174 jumbo packets 0 storm suppression bytes

0 runts 0 giants 0 CRC 0 no buffer

0 input error 0 short frame 0 overrun 0 underrun 0 ignored

0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop

0 input with dribble 0 input discard

0 Rx pause

TX

2801196 unicast packets 353162904 multicast packets 549464 broadcast packets

356513564 output packets 25978572978 bytes

101325 jumbo packets

0 output errors 0 collision 0 deferred 0 late collision

0 lost carrier 0 no carrier 0 babble 0 output discard

0 Tx pause

35 interface resets

===========================================================================

primary# sh int po3

port-channel3 is up

Hardware: Port-Channel, address: 002a.6a0a.4caf (bia 002a.6a0a.4caf)

Description: ***Link to FI-B in R18***

MTU 1500 bytes, BW 30000000 Kbit, DLY 10 usec

reliability 48052/255, txload 1/255, rxload 1/255

Encapsulation ARPA

Port mode is trunk

full-duplex, 10 Gb/s

Input flow-control is off, output flow-control is off

Switchport monitor is off

EtherType is 0x8100

Members in this channel: Eth1/8, Eth1/9, Eth1/10

Last clearing of "show interface" counters never

30 seconds input rate 104 bits/sec, 0 packets/sec

30 seconds output rate 1704 bits/sec, 0 packets/sec

Load-Interval #2: 5 minute (300 seconds)

input rate 448 bps, 0 pps; output rate 1.08 Kbps, 0 pps

RX

9070 unicast packets 288523 multicast packets 143322 broadcast packets

440915 input packets 72677230 bytes

7 jumbo packets 0 storm suppression bytes

0 runts 0 giants 36 CRC 0 no buffer

2161727821137838086 input error 0 short frame 0 overrun 0 underrun 0 ignored

0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop

0 input with dribble 0 input discard

0 Rx pause

TX

62914 unicast packets 74425404 multicast packets 14434365 broadcast packets

88922683 output packets 7182753461 bytes

14 jumbo packets

0 output errors 0 collision 0 deferred 0 late collision

0 lost carrier 0 no carrier 0 babble 0 output discard

0 Tx pause

17 interface resets

===========================================================================

secondary# sh run int po2 mem

!Command: show running-config interface port-channel2 membership

!Time: Wed Feb 25 07:56:41 2009

version 5.2(1)N1(1b)

interface port-channel2

description ***Link to FI-A in R17***

switchport mode trunk

switchport trunk allowed vlan 12-13,21,101-600

spanning-tree port type edge

spanning-tree bpdufilter enable

interface Ethernet1/8

description ***Link to FI-B in R18***

switchport mode trunk

switchport trunk allowed vlan 12-13,21,101-600

channel-group 2 mode active

interface Ethernet1/9

description ***Link to FI-B in R18***

switchport mode trunk

switchport trunk allowed vlan 12-13,21,101-600

channel-group 2 mode active

interface Ethernet1/10

description ***Link to FI-B in R18***

switchport mode trunk

switchport trunk allowed vlan 12-13,21,101-600

channel-group 2 mode active

===========================================================================

secondary# sh run int po3 mem

!Command: show running-config interface port-channel3 membership

!Time: Wed Feb 25 07:58:30 2009

version 5.2(1)N1(1b)

interface port-channel3

description ***Link to FI-B in R18***

switchport mode trunk

switchport trunk allowed vlan 12-13,21,101-600

spanning-tree port type edge

spanning-tree bpdufilter enable

interface Ethernet1/3

description ***Link to FI-A in R17***

switchport mode trunk

switchport trunk allowed vlan 12-13,21,101-600

channel-group 3 mode active

interface Ethernet1/4

description ***Link to FI-A in R17***

switchport mode trunk

switchport trunk allowed vlan 12-13,21,101-600

channel-group 3 mode active

interface Ethernet1/5

description ***Link to FI-B in R18***

switchport mode trunk

switchport trunk allowed vlan 12-13,21,101-600

channel-group 3 mode active

===========================================================================

secondary# sh int po2

port-channel2 is up

Hardware: Port-Channel, address: 002a.6a0a.4d6f (bia 002a.6a0a.4d6f)

Description: ***Link to FI-A in R17***

MTU 1500 bytes, BW 30000000 Kbit, DLY 10 usec

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA

Port mode is trunk

full-duplex, 10 Gb/s

Input flow-control is off, output flow-control is off

Switchport monitor is off

EtherType is 0x8100

Members in this channel: Eth1/8, Eth1/9, Eth1/10

Last clearing of "show interface" counters never

30 seconds input rate 1752 bits/sec, 0 packets/sec

30 seconds output rate 1016 bits/sec, 0 packets/sec

Load-Interval #2: 5 minute (300 seconds)

input rate 832 bps, 0 pps; output rate 496 bps, 0 pps

RX

76009142 unicast packets 544440 multicast packets 228999 broadcast packets

76782581 input packets 146179423660 bytes

68566412 jumbo packets 0 storm suppression bytes

0 runts 0 giants 0 CRC 0 no buffer

0 input error 0 short frame 0 overrun 0 underrun 0 ignored

0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop

0 input with dribble 0 input discard

0 Rx pause

TX

192711593 unicast packets 154928956 multicast packets 31420967 broadcast packets

379061516 output packets 385914168860 bytes

179761364 jumbo packets

0 output errors 0 collision 0 deferred 0 late collision

0 lost carrier 0 no carrier 0 babble 0 output discard

0 Tx pause

24 interface resets

===========================================================================

secondary# sh int po3

port-channel3 is up

Hardware: Port-Channel, address: 002a.6a0a.4d6a (bia 002a.6a0a.4d6a)

Description: ***Link to FI-B in R18***

MTU 1500 bytes, BW 30000000 Kbit, DLY 10 usec

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA

Port mode is trunk

full-duplex, 10 Gb/s

Input flow-control is off, output flow-control is off

Switchport monitor is off

EtherType is 0x8100

Members in this channel: Eth1/3, Eth1/4, Eth1/5

Last clearing of "show interface" counters never

30 seconds input rate 456 bits/sec, 0 packets/sec

30 seconds output rate 1784 bits/sec, 0 packets/sec

Load-Interval #2: 5 minute (300 seconds)

input rate 456 bps, 0 pps; output rate 912 bps, 0 pps

RX

75986748 unicast packets 490267 multicast packets 124569 broadcast packets

76601584 input packets 146159548726 bytes

68566413 jumbo packets 0 storm suppression bytes

0 runts 0 giants 0 CRC 0 no buffer

0 input error 0 short frame 0 overrun 0 underrun 0 ignored

0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop

0 input with dribble 0 input discard

0 Rx pause

TX

192585042 unicast packets 190939131 multicast packets 13399201 broadcast packets

396923374 output packets 386206869311 bytes

179761514 jumbo packets

0 output errors 0 collision 0 deferred 0 late collision

0 lost carrier 0 no carrier 0 babble 0 output discard

0 Tx pause

24 interface resets

===========================================================================

We have removed vPC configuration as we are facing same problem as i said in my previous post

----------------------------------------------------------------------------

id Port Status Consistency Reason Active vlans

------ ----------- ------ ----------- -------------------------- -----------

2 Po2 up failed vpc port channel -

mis-config due to vpc

links in the 2 switches

connected to different

partners

Do let me know if anything else is required from my end

-amit

padramas · ‎04-10-2013

vPC - 2

Three ports on N5K-A

Three ports on N5K-B

Are all these 6 ports connected to single FI ?

I assume FI is in Ethernet end host mode.

Amit Vyas · ‎04-10-2013

Yes, FI is in Ethernet end host mode

Following is the connectivity from both FIs to both N5K

Three ports on N5K-A is from - FI-A - Eth1/3,1/4,1/5 - Po2

Three ports on N5K-B is from - FI-A - Eth1/8,1/9,1/10 - Po3

Three ports on N5K-B is from - FI-B - Eth1/3,1/4,1/5 - Po2

Three ports on N5K-A is from - FI-B - Eth1/8,1/9,1/10 - Po3

You want to me create vPC-2 on N5K? if yes then i would like you suggest me which two Port Channel i should take on both N5K

padramas · ‎04-10-2013

This is normal port-channel and you will not have any issues.

If you need a vPC setup where single port channel from FI would be connected to two N5K switches.

Sample topology

FI-A ( 2 ports ) ( PC 2 ) <------ First link to vPC 2 ( N5K A )

<----- Second link to vPC 2 ( N5K B )

You will receive the error if you have connected the second linkt to FI B

Padma

Amit Vyas · ‎04-11-2013

Padma, I am sorry to say but I am failed to understand the example you have given above.

I am attaching the diagram the way I have connected my cables from FI to N5K. Kindly guide me based on this

padramas · ‎04-11-2013

Hello Amit,

The current deployment is not an issue. However, if you convert these port channels into vpc on N5K, it would throw error messages as your Port channel 2 / 3 are connecting to two FIs ( devices ) .

In other words, vpc interfaces of a channel ( say ID 2 ) from both N5K should connect to same device ( FI )

Padma

Amit Vyas · ‎04-12-2013

Hi Padma,

As per suggestion have made changes on N5k but still facing the same error.

Also want to know that while confiuring vpc it is required that on both N5k switches connections should be same? like in our case FI-A-Port 2/9-11 is going to N5k-A-Port-1/3-5 and FI-A-Port 2/14-16 is going to N5k-B-Port-1/8-10. I hope this should not create problem for me

I am attaching updated diagram for your reference and output of command few commands

=================================================================

primary# sh int po2

port-channel2 is up

vPC Status: Up, vPC number: 2

Hardware: Port-Channel, address: 002a.6a0a.4cac (bia 002a.6a0a.4cac)

Description: ***Link to FI-A in R17***

MTU 1500 bytes, BW 30000000 Kbit, DLY 10 usec

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA

Port mode is trunk

full-duplex, 10 Gb/s

Input flow-control is off, output flow-control is off

Switchport monitor is off

EtherType is 0x8100

Members in this channel: Eth1/3, Eth1/4, Eth1/5

Last clearing of "show interface" counters never

30 seconds input rate 1400 bits/sec, 0 packets/sec

30 seconds output rate 848 bits/sec, 0 packets/sec

Load-Interval #2: 5 minute (300 seconds)

=================================================================

secondary# show int po2

port-channel2 is down (No operational members)

vPC Status: Down, vPC number: 2 [packets forwarded via vPC peer-link]

Hardware: Port-Channel, address: 002a.6a0a.4d71 (bia 002a.6a0a.4d71)

Description: ***Link to FI-A in R17***

MTU 1500 bytes, BW 100000 Kbit, DLY 10 usec

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA

Port mode is trunk

auto-duplex, 10 Gb/s

Input flow-control is off, output flow-control is off

Switchport monitor is off

EtherType is 0x8100

Members in this channel: Eth1/8, Eth1/9, Eth1/10

Last clearing of "show interface" counters never

30 seconds input rate 0 bits/sec, 0 packets/sec

30 seconds output rate 0 bits/sec, 0 packets/sec

Load-Interval #2: 5 minute (300 seconds)

=================================================================

primary# sh run int po2 mem

!Command: show running-config interface port-channel2 membership

!Time: Fri Feb 27 07:19:14 2009

version 5.2(1)N1(1b)

interface port-channel2

description ***Link to FI-A in R17***

switchport mode trunk

switchport trunk allowed vlan 12-13,21,101-600

spanning-tree port type edge

spanning-tree bpdufilter enable

vpc 2

interface Ethernet1/3

description ***Link to FI-A in R17***

switchport mode trunk

switchport trunk allowed vlan 12-13,21,101-600

channel-group 2 mode active

interface Ethernet1/4

description ***Link to FI-A in R17***

switchport mode trunk

switchport trunk allowed vlan 12-13,21,101-600

channel-group 2 mode active

interface Ethernet1/5

description ***Link to FI-B in R18***

switchport mode trunk

switchport trunk allowed vlan 12-13,21,101-600

channel-group 2 mode active

=================================================================

secondary# sh run int po2 mem

!Command: show running-config interface port-channel2 membership

!Time: Fri Feb 27 07:19:50 2009

version 5.2(1)N1(1b)

interface port-channel2

description ***Link to FI-A in R17***

switchport mode trunk

switchport trunk allowed vlan 12-13,21,101-600

spanning-tree port type edge

spanning-tree bpdufilter enable

vpc 2

interface Ethernet1/8

description ***Link to FI-B in R18***

switchport mode trunk

switchport trunk allowed vlan 12-13,21,101-600

channel-group 2 mode active

interface Ethernet1/9

description ***Link to FI-B in R18***

switchport mode trunk

switchport trunk allowed vlan 12-13,21,101-600

channel-group 2 mode active

interface Ethernet1/10

description ***Link to FI-B in R18***

switchport mode trunk

switchport trunk allowed vlan 12-13,21,101-600

channel-group 2 mode active

=================================================================

primary# show vpc ====== status as below

vPC status

----------------------------------------------------------------------------

id Port Status Consistency Reason Active vlans

------ ----------- ------ ----------- -------------------------- -----------

2 Po2 up failed vpc port channel 12-13,21,10

mis-config due to vpc 1-600

links in the 2 switches

connected to different

partners

=================================================================

secondary# show vpc ====== status as below

vPC status

----------------------------------------------------------------------------

id Port Status Consistency Reason Active vlans

------ ----------- ------ ----------- -------------------------- -----------

2 Po2 down* failed vpc port channel -

mis-config due to vpc

links in the 2 switches

connected to different

partners

=================================================================

padramas · ‎04-12-2013

Did you make the change on FI as well i.e single port channel interface having 6 member interfaces ?

Flap the interface at both ends and verify the outcome.