cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
6945
Views
0
Helpful
6
Replies

Fabric Interconnect B doesn't respond on its Management Network

egha
Level 1
Level 1

I have a pair of Fabric Interconnects running 2.0(3a). They are clusterd and everything from the clustering point of view and the functionality of the UCSM seems fine. However FI-B doesn't repond to ping or ssh on its management even though the IP settings and configs is right and I can get into it from the FI-A. Obvisosely something is not right with FI-B.

I digged in further and found out that FI-B's management is set to 10 Mb/s for speed and that could be the problem. I checked my management network which is a 100Mb switch and the port that FI-B is connected seems fine. I can't seem to find a way to change the speed setting on the management of FI-B to force it for 100 Mb/s.

Another issue that you see below is that fact that it shows my management is down for both FI's. This is not true. I think I read somewhere that this is a known bug. Any help would be appreciated.

FI01-A(nxos)# show interface mgmt 0

mgmt0 is down (Link not connected)

  Hardware: GigabitEthernet, address: 547f.eead.ce40 (bia 547f.eead.ce40)

  Internet Address is 172.24.0.12/24

  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA

  auto-duplex, 100 Mb/s

  EtherType is 0x0000

  1 minute input rate 304 bits/sec, 0 packets/sec

  1 minute output rate 0 bits/sec, 0 packets/sec

  Rx

    53574 input packets 0 unicast packets 53574 multicast packets

    0 broadcast packets 4012306 bytes

  Tx

    0 output packets 0 unicast packets 0 multicast packets

    0 broadcast packets 0 bytes

FI01-B(nxos)# show interface mgmt 0

mgmt0 is down (Link not connected)

  Hardware: GigabitEthernet, address: 547f.eead.c780 (bia 547f.eead.c780)

  Internet Address is 172.24.0.13/24

  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

     reliability 6839/255, txload 1/255, rxload 1/255

  Encapsulation ARPA

  auto-duplex, 10 Mb/s

  EtherType is 0x0000

  1 minute input rate 0 bits/sec, 0 packets/sec

  1 minute output rate 0 bits/sec, 0 packets/sec

  Rx

    0 input packets 0 unicast packets 0 multicast packets

    0 broadcast packets 0 bytes

  Tx

    0 output packets 0 unicast packets 0 multicast packets

    0 broadcast packets 0 bytes

6 Replies 6

Daniel Laden
Level 4
Level 4

This is a bug,

FI01-A(nxos)# show interface mgmt 0

mgmt0 is down (Link not connected)

You are looking at the NXOS running on the FI.  The physical management port is managed by UCSM.  The NXOS code will always show interface MGMT0 as down.

====

Try swapping the network cables between the FI-A and FI-B and see if the problem follows the cable.  Try pinging FI-B from FI-A (or some other device on same VLAN).  Shut the switchport down and confirm the link light goes down on FI management port.  Is there a mac address on the switchport and does it match up with the mac on the interface mgmt0 on the FI.

Thank You,

Dan Laden

Cisco PDI Data Center

Want to know more about how PDI can assist you?

http://www.youtube.com/watch?v=3OAJrkMfN3c

http://www.cisco.com/go/pdihelpdesk

Thanks for your response Dan.

Swapping the cables did not help. FI-A keeps working on either cables/ports and FI-B does not.

I can ping everywhere within the network from FI-A except FI-B.

The switch port that the FI-B is connected doesn't show mac + the switchport goes down immidiately when I plug the FI-B's managenet cable.

Is there a way to manually adjust the speed for the FI-B management? I have no explanation why the speed is 10Mb/s. I beleive that's our problem.

Would upgrading the FI's to latest firmware be a possible solution?

What are the FI's physicaly cabled into?  What model, make & version of switch?

Upgrading the firmware would not help here. This is a Layer1 issue with either the Mgmt Interface itself, the cable or upstream device.

The ports are auto, so you should be able to hard config the upstream switchport to 100MB Full - have you tried this or is the upstream switch unmanaged?

Robert

From each FI, what is the 'show cluster extend-state' output. 

Also, there is a M0 and M1 physical interface.  Ensure you are using M0 as M1 is a non-functioning port. 

Has this install always shown this condition or at some point in the past were you able ping/ssh to the FI.

Thank You,

Dan Laden

Cisco PDI Data Center

Want to know more about how PDI can assist you?

http://www.youtube.com/watch?v=4BebSCuxcQU&list=PL88EB353557455BD7

http://www.cisco.com/go/pdihelpdesk

egha
Level 1
Level 1

Thanks for your prompt responses.

The upstream switch is 100Mb/s auto full.

Management is connected through M0.

Cisco UCS 6200 Series Fabric Interconnect (6248)
version 5.0(3)N2(2.03a)

The FI's has shown this behaviour from the beginning. I did the initial config on FI-A and then on FI-B using CLI. At that point I rememebr both FI's were pingable on their management. Then consoled to FI-A, it prompted that it wants to sync and finalize the cluster. Apparantly at that point forward I lost the management access to FI-B.

The cluster stat shows everything is fine. I can even failover to FI-B, but nothing works (can't get to UCSM).

FI01-A# show cluster extended-state
Cluster Id: 0x1a129962618d11e2-0xb5ee547feeadce44

Start time: Fri Feb 01 11:36:20 2013
Last election time: Fri Feb 01 11:44:04 2013

A: UP, PRIMARY
B: UP, SUBORDINATE

A: memb state UP, lead state PRIMARY, mgmt services state: UP
B: memb state UP, lead state SUBORDINATE, mgmt services state: UP
   heartbeat state PRIMARY_OK

INTERNAL NETWORK INTERFACES:
eth1, UP
eth2, UP

HA READY
Detailed state of the device selected for HA storage:
Chassis 1, serial: FOXxxxxxxx, state: active


Both FI's L1 and L2 is connected and using the link I can get into FI-B from FI-A.

The FI has also two server links to each side of the UCS Chassis. Standard straight cabling.

I also have two appliance port from each FI to NetApp.

Here is one thing. In this setup, I have only one N5K. Not sure if this creates any issues. Each FI has two uplinks to same N5K. The two links from each FI is port channel.

Interesting...

Let go to the L3 gateway for the FIs.  Ping each FI MGMT0 interface and review the IP ARP entries on the L3 interface.  Lets confirm the entries are missing or matching the MAC address of the respective MGMT0 interface.  If another MAC is seen, we may be looking at a duplicate MAC issue.

This would not explain the 10Mbs / 100Mbs mismatch between the FI MGMT0 and upstream switch. They are negotiating with something since you are stating the upstream switch is reporting 100F.  If set for autonegotiation and the other end was hardcoded, negotiation would fail on duplex and Half duplex would be selected.

The uplinks and management ports are independant of each other.  What switch is your M0 interfaces connected to.  Provide the switch details requested by Robert.

In my lab, running 2.04b, I see RX activity on both.  Your FI-B shows 0.  Did you reboot the FI since it had been successful?

  Rx

    0 input packets 0 unicast packets 0 multicast packets

You look to be a Cisco partner eligible to engage with PDI.  If this is preproduction or lab, we can assist with reviewing this in a webex.  If deemed hardware issue, you would need to work with TAC for a RMA.

Thank You,

Dan Laden

Cisco PDI Data Center

Want to know more about how PDI can assist you?

http://www.youtube.com/watch?v=4BebSCuxcQU&list=PL88EB353557455BD7

http://www.cisco.com/go/pdihelpdesk

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: