Nexus 5548up connection issue (Hosts can ping but can't ssh)

[scottchang]

Hi All,

We have a Nexus 5548up IOS 5.0(2)N1(1) with a FEX (2248TP)

5548 has 1 TenG connection to 6509 and also 2 x 1G RJ45 connections to 4948. Both connections are on 5548.

4948 also connect to 6509 so there is loop but block by STP.

Got a weird problem when traffic going thru TenG, all switches connecting to 5548 couldn't be ssh but ping was good. (ssh connection couldn't be established. Telnet switch IP port 22 could get reply but when use putty, it couldn't establish ssh)

Neither was hosts connecting to the switches then to 5548.

But, if the switch connect to FEX 2248 (BPDU filter enable), it's no problem to ping and ssh. So does hosts connecting to switch then to 2248.

If I force traffic to 2 x 1G RJ45 connections to 4948, the problem changed to switches connecting to 2248. (Ping is good but ssh couldn't established)

And the switches connecting to 5548 became no problem.

Is it hardware issue or I need add some command on 5548 ?

Appreciate your response !!

[Reza Sharifi]

Hi,

Can you post a diagram of your network including port assignments and IPs?

[scottchang]

Hi Reza,

Attached the diagram, the problem is :

When traffic go thru LINK 10G, I could ping both Host 1 and Host 2 but couldn't ssh Host 1. (ssh Host 2 is ok)

When traffic change to LINK 2G(shutdown LINK10G), I could ping both Hosts but couldn't ssh Host 2.

I tried to connect a laptop on FEX 2248, I could ping both Hosts but couldn't ssh Host 1 or SW1.

It looks like icmp is good, but tcp handshaking couldn't complete. There is no ACL on Interfaces.

Also 5548 is just layer 2. 

Thank you,

[Reza Sharifi]

Hi Scott,

Host1 is connected to Sw1 and host2 is connected to Sw2.  Are these hosts in the same subnet/vlan?  Where is the default gateway for the hosts, 6500 or 4948?

Is 6500 a layer-2 device?

HTH

[scottchang]

Hi Reza,

They are in the same vlan. Default gateway is 4948-01 but STP root is on 6509. The problem happen when I change STP root to 4948 and traffic redirected to LINK 2G. So I changed back STP root to 6509.

4948 and 6509 are doing HSRP. But now 4948 is the Active one so default gateway is on 4948.

The other test case I did is I connected a laptop on FEX2248 with the same VLAN as SW1 and SW2. I could ping both switches but couldn't ssh SW1. 

Thanks,

[Reza Sharifi]

Hi Scott,

Lets clarify  a few things from your diagram.

1-Not sure why you are connecting a switch to the 2248.  The 2ks are designed for host connectivity and not to other switches. So, you could connect both hosts directly to the 2k and then the 2k has 2 uplinks to the 5k.

2-I am not sure what is the purpose of the 6500 in the middle, as you have 2 4948 switches. You can connect the 5k to both 4948s and they can be your default gateway.  You can also run HSRP between them for fault tolerance and have active HSRP serve as the root and the other 4948 as backup root.

4948 and 6509 are doing HSRP. But now 4948 is the Active one so default gateway is on 4948.

3-Not sure why you are running HSRP between the 6500 and 4948, instead of just using both 4948s.

HTH

[scottchang]

Hi Reza,

The reason connected switch to 2248 is the media type. switches connect to 2248 thru RJ45, connect to 5548 thru fiber.

I understand the confusion for 2 and 3. The reason is I am going to replace 6509 with 2 x 4948. So it's in transition period and almost done once I move STP root from 6509 to 4948.  But the problem happened when I changed STP root to 4948 so I had to roll back in case impact production.

The problem I feel is in 5548 because I couldn't ssh SW1 even I am in the same vlan of SW1 but connected to Fex 2248.

Thanks,

[Reza Sharifi]

Scott,

The reason connected switch to 2248 is the media type. switches connect to 2248 thru RJ45, connect to 5548 thru fiber.

not sure if I understand.  The 2248 comes with 48 100/1000 copper ports and 4 10Gig interfaces (fiber) for uplinks.  As long as your hosts have copper 100/1000 NICs you can directly connect them to the 2k. 

Are sw1 and 2 Cisco switches? if yes, what type and what address you trying to SSH to?

Is the 5k configured with mgmt ip under management vrf?

Can you post sh run form the 5k?

Reza

[scottchang]

Hi Reza,

Because hosts are in different racks so all hosts connected to top cisco switch then one uplink to 2248.

Sw1 and Sw2 are Cisco switches. SW1 is fiber uplink to 5548, SW2 is Copper uplink to 2248. Address is IPv4. (10.16.200.x)

5k is configured with mgmt IP under management vrf.

I will provide show run later. 

Thanks,

[Reza Sharifi]

Hi Scott,

Because hosts are in different racks so all hosts connected to top Cisco switch then one uplink to 2248.

As I mentioned before, you should not be connecting other switches to the 2248. The ports are designed to be host ports and not be used for uplink to another switch.  BPDU guard by default is enabled and will block communication between the 2k and the other switch. The 2ks are not capable of any local switching and the communication between hosts have to go up to the 5k for packet forwarding.

Please have a look at this doc for more info on how the 2ks should be used and connected:

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus2000/sw/configuration/guide/rel_521/Configuring_the_Cisco_Nexus_2000_Series_Fabric_Extender_rel_5_2_chapter1.html

Reza

[scottchang]

Hi Reza,

Thanks for the reply.

I know it's not recommended. But in some situation we have to connect switch to FEX and I did some research as discussion below:

https://supportforums.cisco.com/discussion/11281941/connecting-switch-2232-fex

Actually I have another Nexus 5548 with similar topology (some switch connected to 5548, some are on Fex) and it's working fine. But this one is weird when switches are only working either on 5548 or FEX even the Fex is connecting to 5548.....So I am thinking firmware or hardware problem see if anyone met this problem before.

The problem 5548 firmware is 5.0(2)N1(1)

Thanks again,