Uplink problems from UCS 6128 FI

stankovic.igor · ‎10-23-2012

Hi,

We have one demo UCS system. It was working pretty smoth, we take it to few presentations etc etc.

Few days we cleaned system, and decided to reconfigure and deploy.

Overview :

Cisco UCS blade chasis, 2 b200m2 servers with VIC adapters, 2x6128 FI.

Uplinks are only 1 Gb and GLC-T modules are used for uplinks.

Problem now is that as soon as i configure port as uplink and configure it for 1 Gbps, port changes to admin down, and no connections to upper switches are available.

We decided to upgrade to latest software and now is 2.0(4a). Conection from VIC, through IOM and to server ports is ok, but uplinks are still not working.

Upper switch c2950 has ports that connects to ucs as trunk, everything as usual.

Just to clarfy, this exact setup was active before and working.

Errors from UCS are :

F0479 2012-07-25T13:04:26.635 112798 Virtual interface 704 link state is down

F0283 enm source pinning failed

Output from nxos for uplink port :

======================================================

Hardware: 1000/10000 Ethernet, address: 0005.73cc.4b48 (bia 0005.73cc.4b48)

Description: U: Uplink

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA

Port mode is trunk

auto-duplex, 1000 Mb/s, media type is 10G

Beacon is turned off

Input flow-control is off, output flow-control is off

Rate mode is dedicated

Switchport monitor is off

EtherType is 0x8100

Last link flapped never

Last clearing of "show interface" counters never

30 seconds input rate 0 bits/sec, 0 bytes/sec, 0 packets/sec

30 seconds output rate 0 bits/sec, 0 bytes/sec, 0 packets/sec

Load-Interval #2: 5 minute (300 seconds)

input rate 0 bps, 0 pps; output rate 0 bps, 0 pps

RX

0 unicast packets 0 multicast packets 0 broadcast packets

0 input packets 0 bytes

0 jumbo packets 0 storm suppression packets

0 giants 0 input error 0 short frame 0 overrun 0 underrun 0 watchdog 0 if down drop

0 input with dribble 0 input discard

0 Rx pause

TX

0 unicast packets 0 multicast packets 0 broadcast packets

0 output packets 0 bytes

0 jumbo packets

0 output errors 0 collision 0 deferred 0 late collision

0 lost carrier 0 no carrier 0 babble

0 Tx pause

0 interface resets

=====================================================

Output from UCS cli for vif paths

Server: 1/1

Fabric ID: A

VIF vNIC Link State Overall Status Prot State Prot Role Admin Pin Oper Pin Transport

---------- --------------- ----------- -------------- ------------- ----------- ---------- ---------- ---------

41 Unknown Unknown 0/0 0/0 Unknown

689 eth1 Error Error No Protection Backup 0/0 0/0 Ether

690 eth0 Error Error No Protection Primary 0/0 0/0 Ether

Fabric ID: B

VIF vNIC Link State Overall Status Prot State Prot Role Admin Pin Oper Pin Transport

---------- --------------- ----------- -------------- ------------- ----------- ---------- ---------- ---------

42 Unknown Unknown 0/0 0/0 Unknown

688 eth1 Error Error No Protection Primary 0/0 0/0 Ether

691 eth0 Error Error No Protection Backup 0/0 0/0 Ether

Any idea? Everything is connected as before, it used to work flawless.

abbharga · ‎10-23-2012

Hi Igor,

Two quick questions:

1) The FI is the 6120? You mentioned 6128, we only have 6120 or 6140.

Also what is the port number which you are trying to configure as 1G? On 6120 only the first eight ports can be configured at a 1 G speed.

./Abhinav

stankovic.igor · ‎10-23-2012

6120, yes (i am configuring some 6248, and mix it up ), and i tried few of 1st eight ports, just to check if port is damaged.

One more thing is that when server reboots, uplink status changes to up for 1 second (or less) and then again admin down. When port is unconfigured, port light is green, but as soon as i configure it as uplink, it goes down, no light. We have monitored ports on c2950 that connects to ucs, no traffic at all.

abbharga · ‎10-23-2012

Igor,

The server ports will remain down untill the uplink port comes up, this is due to the pinning of host to network ports mechanism.

Did you verify the SFP you are using is correct? What is the port configuration on the upstream switch?

./Abhinav

stankovic.igor · ‎10-23-2012

well, server ports are up. I know this is uplink ports problem, but really cannot see where the problem is :). Ports on northbound switch that ucs connects to are trunk ports, vlans are defined on that switch.

I also suspect sfp's (cisco glc-t) but it seems pretty strange that 2 sfp modules are malfunctioning. They used to work pretty neat.

Just to mention we have same UCS in production and it is working, config is same, ports are same, nxos outputs for ethernet uplinks are same..

Just to check somenthing i created two hba's (we do have FC expansion card, but ucs is not connected to FC switches), and vif paths for FC are up...seems that this is releated to ethernet problem solely.

stankovic.igor · ‎10-23-2012

just few more things

Proces of server reboot (re-associating service profile, i got hint that thos may help)

1. Virtual circuit for DCE interface is cretaed, state UP. Eth0 and eth1 VC are created status unknown

2. Esx (installed on local disks) begins to boot. VC for DCE is no more showing, eth0 and eth1 are unknown status

3. Drivers are loaded to esx, and eth0 and eth1 are again in error as before.