Cisco 3750 3650 switches SSH issue

[Adam Hudson]

My network infrastructure is set up with several 3750-Xs as my switch stack. From there I have several satellite 3650 switches that connect back to the core via two fiber pairs set up as a port channel. To manage my switches I have a separate management VLAN.

Here's my issue: while performing on something unrelated I noticed I could no longer SSH to one of my switches from either my linux machine nor my Win 7 machine (neither of these machines have an IP in the management VLAN). This switch trunks back to the core switch stack (like several others) and also has two switches that trunk into it to get back to the switch stack (they're "farther out" so to speak). I can ssh into those just fine, then ssh "back" into the switch I can no longer SSH into from my desktop machines.

At first I couldn't even SSH into the problem switch from other switches "closer" to the core switch stack including the core switch stack itself, then (through no change I made, and I'm the only one who should be working on these switches) suddenly I could.

Troubleshooting this further:
-I can ping all of the management IPs from my desktops besides the problem switch
-I can ping the problem switch's management IP from all of my switches, even when I couldn't connect into it from some of them
-SSH debug shows nothing helpful
-All switches have the same version of SSH
-Checking the allowed VLANs, the management VLAN is allowed on the trunk heading to the problem switch from the core stack
-I keep versioning history on all of my switches and this switch's config hasn't been changed for at least 4 weeks, even then, any of the changes made to the problem switch or the core switch within the last year have had nothing to do SSH communications and I know I've SSH'd into the switch recently with no issue.

This one is a head scratcher for me. Any help is appreciated.

[Jon Marshall]

Adam

Not sure what you mean by any network traffic over the management vlan.

If your switch is only acting as a L2 switch, which it is because the L3 routing between vlans is done on your core switch, then the default gateway is only used by the switch itself ie. for traffic to and from the switch.

Any traffic from clients connected to that switch or clients in vlans on other switches that go through this switch is not affected by the default gateway.

So the only traffic on the management vlan is management traffic.

If I have misunderstood what you are saying please clarify.

Jon

[Adam Hudson]

Jon, maybe I'm misunderstanding because I thought whatever the default gateway is, all traffic gets sent to that address, then sorted out through there. But you're saying the only "traffic" going to the switch, that is the only traffic referring to that switch is referring to it by it's IP address which is in the management VLAN.

If all of that made sense and is correct, I really should change the gateways on all of the switches to the management VLAN gateway. But, if every other switch is working right now, do I really gain anything by switching this?

[Peter Paluch]

Adam,

I hope I won't be too intrusive for answering this one.

I thought whatever the default gateway is, all traffic gets sent to that address, then sorted out through there.

No, that is most certainly not the case, at least not in such a general sense.

The term "default gateway" only has meaning within each network, or VLAN, separately and independently. Assume that you have a router with three interfaces in networks 10.0.1.0/24, 10.0.2.0/24, 10.0.3.0/24, and its own addresses in these networks are 10.0.1.1, 10.0.2.1, and 10.0.3.1, respectively.

Then, the default gateway for hosts in 10.0.1.0/24 is the 10.0.1.1. The default gateway for hosts in 10.0.2.0/24 is 10.0.2.1. The default for hosts in 10.0.3.0/24 is 10.0.3.1. Any other combination is invalid.

A switch behaves like a host in its management VLAN only (the one that has interface Vlan configured and activated). For all other VLANs, a switch is a totally transparent device. It has no impact on what gateway will be used to handle the traffic in those other VLANs. After all, using a gateway is a Layer3 function, and switch is a Layer2 device, so it has does not (and can not) have any impact on Layer3 operations. Only because the switch also has a computer inside, and that computer acts like a host, it is attached to one of the VLANs (the management VLAN) and must use the gateway in that VLAN in order to communicate with stations outside its IP networks.

I really should change the gateways on all of the switches to the management VLAN gateway.

Definitely and absolutely.

But, if every other switch is working right now, do I really gain anything by switching this?

The question you are asking here is, essentially, "What do I gain by having my devices configured correctly?" You surely feel yourself that such a question shouldn't really be asked at all.

Anyway, the reason your other switches work is, most probably, the result of a Proxy ARP mechanism in action. What I suggest testing is looking at the show ip arp output of the switches that work right now, and verifying whether all IP addresses that are recorded together with their MAC addresses on the Vlan200 interface are exclusively from the 20.1.200.0/24 network. If they are not, and I suppose they are not, then that is a proof that those switches rely on the Proxy ARP mechanism to work. In short, Proxy ARP lies in a host asking for the ARP mapping of the packet's destination, instead of asking for the ARP mapping of its default gateway. The default gateway will respond with its own MAC address even though the packet's destination address is different, simply because it knows the way to the packet's destination. Relying on Proxy ARP is disadvantageous because of several reasons:

• Depending on the amount of management traffic destinations outside VLAN 200, the ARP traffic grows proportionally, and so does the number of ARP entries in your ARP caches. We've had a case here on CSC where a router relying on Proxy ARP routing company traffic toward internet reloaded itself intermittently because its ARP cache grew so large that it consumed its entire memory.
• When the Proxy ARP functionality is deactivated on default gateways (and deactivating it is considered best practice), stations that relied on Proxy ARP will become unable to communicate with anyone outside their own network.

Proxy ARP is a very fragile mechanism that should best be not relied upon. Moving over to the use of proper default gateways will make sure you do not rely on the Proxy ARP.

Best regards,
Peter

[Jon Marshall]

Adam

Correct in what you say and it does make sense.

A switch acting purely at L2 only uses it's default gateway for it's own traffic. Traffic from connected devices is simply forwarded based on mac addresses.

A L3 switch on the other hand wouldn't use a default gateway, it would have routes and multiple SVIs which act as the default gateways for clients.

So if you want to make sure then all your clients in all vlans should be using the corresponding SVI IP address on the L3 core switch (switch stack in your network).

Yes you really should change all your other L2 switches as well to have a gateway in the same IP subnet as the SVI IP address.

Again I am assuming the L3 switch stack has the SVI for vlan 200 with the IP 20.1.200.1 and that all routing between vlans is done there so you need to be sure that is the case before making any changes.

The benefits are -

1) it is a standard configuration so would be less confusing to anyone new to the network.

2) one switch is not working but the others are and it is definitely related to the default gateway. The problem may reoccur with another switch especially as we don't really know the exact cause of the problem.

3) the other switches are working because of proxy arp as far as I can tell. Some people disable proxy arp and if at a later date it was disabled without realising the consequences you may well lose connectivity to all your switches.

Like I say you should be sure about what switches are routing for the vlans before making changes but it does sound as though it is all being done on your L3 switch stack.

Any questions, clarifications etc. please feel free to ask.

Jon

[Adam Hudson]

Jon, Peter, thank you for all of your help with this odd issue. I've changed all of my switch's default gateways to the management gateway and everything seems to be working fine.