C9500-48Y4C 16.12.03a is ip routing enabled by default?

[saba1418]

Hello, I am having issues here. Due to hardware failure, I recently had to hastily replace my core with equipment that was still being configured for installation, and was not quite ready. I was still testing certain features to implement on our site, so during the mad dash to get everything back up and working I just had to go with it. Now after installation, although vlan traffic is flowing to the router and users and systems are connected properly, I am not able to connect to the devices on the management vlan (10.200.255.0/25). I am wondering that even though the routing table shows all vlans, as well as the default route set to send all traffic to the router, if the command "ip routing" still needs to be executed. Is it enabled by default? is there a way to know if its already enabled other than in the show config output?

[saba1418]

When you attempt to SSH to the new core switch what happens? Do you get any prompt, or any other type of response, or does it just hang?

If using the V100 address nothing happens, no authentication prompt. If using the V1 int address, it allows for local login - no radius...

I am not clear about a couple of things when you tell us "the vlan 1 interface ip will respond to SSH attempts, although not authenticated through radius."

- am I correct in assuming that your normal SSH access uses the vlan 100 IP address?

yes, we are trying to move away from using vlan 1, so all new devices will be managed on vlan 100

- if the vlan 1 interface responds to SSH attempts why would it not be authenticated through radius?

I think it may be subject to a device name change, although I am not sure. The my teammate that runs the radius server is out with Covid, so I havent been able to ask him.

It also occurs to me to wonder when you replaced the old core switch with the new switch, would the radius server recognize that this is a different device? (would the device name perhaps be different? or is there any other parameter that radius might be using that is different?)

I think this is a yes to all inquires on this topic. I am starting to think the radius may be the main issue to all of this.

My question is, if vlan 100 lives on the firewall/router, wouldnt using the default route of 0.0.0.0 0.0.0.0 10.38.10.1 route all traffic to the router for it to be then routed to its destination within the network regardless of how radius plays in to the equation?

[Richard Burts]

As far as being to access using SSH on vlan 1 and authenticating using local ID this is explained in the partial config that you posted:

aaa authentication login default group radius local

This specifies to attempt authentication with radius and if the radius server does not respond (or responds with some error message) then to use local authentication. This certainly does suggest that there is an issue with radius that is impacting the new core switch. 

I am puzzled that SSH to vlan 1 address does get a response but SSH to vlan 100 address does not get a response. I do not see anything in the partial config that you posted that would explain this. Is the device you are sourcing SSH from perhaps in vlan 1 and so that is a local connection while vlan 100 is remote? Perhaps running debug for SSG and attempting SSH to vlan 100 might shed some light on this?

[saba1418]

I think this issue is due to a hodgepodge of factors that were were abruptly changed and now have to be ironed out one by one. Case in point, now that I have replaced my core, and it is a router, I obviously need to configure a static route from the new core to the interface on our sonicwall where the vlan lives.

I think the route below would work to get traffic from the core to the SW:

ip route [core SVI - V100 ip] 255.255.255.128 [Sonicwall V100 interface ip]

Then once traffic can flow in/out of the vlan, I can deal with the Radius issue, as I am unsure why the vlan 1 interface is not accessible with SSH, but I suspect it is a naming/policy that issue that has been overlooked. 

[Richard Burts]

I have been thinking about a question that you asked

"My question is, if vlan 100 lives on the firewall/router, wouldnt using the default route of 0.0.0.0 0.0.0.0 10.38.10.1 route all traffic to the router for it to be then routed to its destination within the network regardless of how radius plays in to the equation?"

The new core switch would not necessarily route ALL traffic to the router. Any traffic that it considers to be locally connected or for which it has an entry in the switch routing table would be routed directly to that network and would not use the default route. According to the output that you posted the switch has entries for these vlans/subnets

vlan 1 10.38.8.0/21

vlan 25 10.39.1.0/24

vlan 30 172.16.8.0/21

vlan 35 10.39.2.0/24

vlan 100 10.200.255.0/25

according to this vlan 100 lives on the new core switch. Does vlan 100 also live on the router/firewall? Is there a connection for vlan 100 that goes to the router/firewall without going through the new core switch?

And I am still wondering about the difference in behavior between attempting SSH to the vlan 1 address or to the vlan 100 address. Based on the partial config that you posted I would expect the switch to respond the same way to both requests. But you tell us that you do get a prompt (and are able to authenticate locally) when accessing the vlan 1 address but get no response/no prompt when you attempt the vlan 100 address. I wonder if the difference might be based on where the request is coming from and how it gets to the switch. What is the address of the device you are attempting SSH from? Would you post the output from that device for a traceroute to the vlan 1 address and to the vlan 100 address. I would like to see if there is any difference in the path.

[saba1418]

vlan 100 also lives on the Sonicwall - 10.200.255.1 so I think there needs to be a route added between that interface and the new core, which is not allowing communication...

I am tracing from my laptop

tracert 10.38.10.208

Tracing route to core.hq.com [10.38.10.208]
over a maximum of 30 hops:

1 26 ms 7 ms 1 ms core.hq.com [10.38.10.208]

tracert 10.200.255.20

Tracing route to 10.200.255.20 over a maximum of 30 hops

1 2 ms 2 ms 2 ms 10.200.255.20

I will need to talk to the server admins before being able to dig in to the radius issue as I dont have access to their servers. 

[saba1418]

Since vlan 100 lives on both the sonicwall and the core, would this route suffice?

ip route 10.200.255.20 255.255.255.128 10.200.255.1

[Richard Burts]

There are several things to respond to:

- first if there is an interface on the firewall in vlan 100 and an interface on the new core in vlan 100 then everything in vlan 100 is locally connected to both devices. This means that neither device would need any route statement for the subnet of vlan 100.

- your suggestion of ip route 10.200.255.20 255.255.255.128 10.200.255.1 has a couple of issues (besides the fact that I have just said that it is not needed at all).   

* there is a mismatch between the address given of 10.200.255.20 and the mask of 255.255.255.128. With that mask the address should be 10.200.255.0.

* the next hop address 10.200.255.1 is an address within the subnet. That implies that the subnet is locally connected - and locally connected subnets do not need route statements.

- if both the new core switch and the firewall are locally connected in vlan 100 then what is the default gateway for devices in vlan 100?

- was it the case with the old core switch (which was replaced) that vlan 100 was locally connected on both the switch and the firewall?

- on the old core switch (which was replaced) was ip routing enabled? or was it a layer 2 switch with routing on the router/firewall?

- I note that the tracert from your PC to both vlan interfaces of the new core switch get a response indicating that both destinations are 1 hop away. Can you tell us for the PC what is its IP, mask, and gateway?

[saba1418]

- first if there is an interface on the firewall in vlan 100 and an interface on the new core in vlan 100 then everything in vlan 100 is locally connected to both devices. This means that neither device would need any route statement for the subnet of vlan 100.

I should clarify it is a subinterface and its trunked from SW to core, but V100 is trunking

- if both the new core switch and the firewall are locally connected in vlan 100 then what is the default gateway for devices in vlan 100?

The gateway was 10.200.255.1 before the core was replaced. All devices on that vlan use 10.200.255.1 as their GW. After replacement since 10.200.255.20 is the IP of the SVI I assumed it now acts as the GW

- was it the case with the old core switch (which was replaced) that vlan 100 was locally connected on both the switch and the firewall?

no, only on the SW, it was named on the old core as only a tagged vlan...

- on the old core switch (which was replaced) was ip routing enabled? or was it a layer 2 switch with routing on the router/firewall?

it was indeed routing, but not for vlan 100

- I note that the tracert from your PC to both vlan interfaces of the new core switch get a response indicating that both destinations are 1 hop away. Can you tell us for the PC what is its IP, mask, and gateway?

IP: 172.16.11.102

S: 255.255.248.0

G: 172.16.8.1

[Richard Burts]

I made the statement that if vlan 100 has an interface on the firewall/router and has an interface on the new switch that neither device needs a static route for the subnet of vlan 100. In response you say "I should clarify it is a subinterface and its trunked from SW to core, but V100 is trunking". I do not understand what you are telling me. Can you clarify what this means? 

I asked what is the gateway configured on the devices in vlan 100. You gave this answer:

"The gateway was 10.200.255.1 before the core was replaced. All devices on that vlan use 10.200.255.1 as their GW. After replacement since 10.200.255.20 is the IP of the SVI I assumed it now acts as the GW" 

I do not understand what you are telling me here. So let me ask again - what is configured as the gateway on the devices connected in vlan 100?

I asked if ip routing was enabled on the old core switch. You gave this answer "it was indeed routing, but not for vlan 100". I am not clear about this. Can you explain that it was routing but not routing for vlan 100? 

Probably the most important thing in your response was the IP address, mask, and gateway of the PC. Am I correct in understanding that the PC is in vlan 30? If so it makes sense that the tracert for both of the switch vlan addresses should indicate that the destination is one hop away and that the responding device is the new core switch. But that leads to a puzzling situation: if you attempt SSH to 10.38.10.208 then you get a prompt and are able to access using local authentication. But if you attempt SSH to 10.200.255.20 then you get no prompt and are not able to access using SSH. It certainly suggests that something is configured that treats the interfaces differently. It might be something in vlan 1, something in vlan 30 or something in vlan 100. Could you post the configuration of those interfaces?

[saba1418]

I made the statement that if vlan 100 has an interface on the firewall/router and has an interface on the new switch that neither device needs a static route for the subnet of vlan 100. In response you say "I should clarify it is a subinterface and its trunked from SW to core, but V100 is trunking". I do not understand what you are telling me. Can you clarify what this means?

just meant that there is not a dedicated connection only for V100, but its actually trunked with all of the other vlans from the sonicwall to the core.

I asked what is the gateway configured on the devices in vlan 100. You gave this answer:

"The gateway was 10.200.255.1 before the core was replaced. All devices on that vlan use 10.200.255.1 as their GW. After replacement since 10.200.255.20 is the IP of the SVI I assumed it now acts as the GW"

I do not understand what you are telling me here. So let me ask again - what is configured as the gateway on the devices connected in vlan 100?

Sorry for the confusion, the configured gateway for devices in V100 is indeed 10.200.255.1. The part about the SVI's address of 10.200.255.20 was really just poorly timed commentary on my confusion about whether or not the address of the new cores SVI would now take over as the gateway for the entire vlan...

I asked if ip routing was enabled on the old core switch. You gave this answer "it was indeed routing, but not for vlan 100". I am not clear about this. Can you explain that it was routing but not routing for vlan 100?

Yes, the vlan was defined on the switch, but only tagged on PO's and a few trunks, and didnt have an ip address on the vlan interface. So the vlan rode on the trunk from the sonicwall to the core, and was included on trunks to other edge switches and everything always connected without issues, again using 10.200.255.0/25 addressing and a 10.200.255.1 gateway.

Probably the most important thing in your response was the IP address, mask, and gateway of the PC. Am I correct in understanding that the PC is in vlan 30? If so it makes sense that the tracert for both of the switch vlan addresses should indicate that the destination is one hop away and that the responding device is the new core switch. But that leads to a puzzling situation: if you attempt SSH to 10.38.10.208 then you get a prompt and are able to access using local authentication. But if you attempt SSH to 10.200.255.20 then you get no prompt and are not able to access using SSH. It certainly suggests that something is configured that treats the interfaces differently. It might be something in vlan 1, something in vlan 30 or something in vlan 100. Could you post the configuration of those interfaces?

Okay, I apologize, but something changed with the situation, or I had a misconfiguration in my saved session through putty - I can actually connect locally to the new core. I just re-added a session and it is allowing me to locally access the device. I will investigate furthur though, as I am still unable to reach any other devices on vlan 100.

interface Vlan1
ip address 10.38.10.208 255.255.248.0
!
interface Vlan30
description Workstation
ip address 172.16.8.1 255.255.248.0
ip helper-address 10.38.10.61
ip helper-address 10.38.10.62
!
interface Vlan100
description OEC-MGMT
ip address 10.200.255.20 255.255.255.128
no ip redirects

[Richard Burts]

In your response you say:

just meant that there is not a dedicated connection only for V100, but its actually trunked with all of the other vlans from the sonicwall to the core.

It does not matter whether the connection is dedicated or is on the trunk. What is significant is that vlan 100 is locally connected on both devices. So neither device needs a static route for the subnet. (and if you were to configure a static route for the subnet it would be ignored)

You also say:

whether or not the address of the new cores SVI would now take over as the gateway for the entire vlan...

For any device in vlan 100 if it attempts access to any other device in vlan 100 the device would simply arp for the destination and then communicate directly with the destination device - no need for gateway. For any device in vlan 100 if it attempts to access devices in other vlans/other subnets then it will continue to use 10.200.255.1 as the gateway. To the extent that you might want the devices in vlan 100 to easily communicate with other devices inside your network you might think about changing their gateway to be the new core switch (why go through the firewall to get to inside resources?). But to the extent that you might want to keep vlan 100 isolated from the network then gateway on the firewall is appropriate. I do not know enough about your network to know which would be more preferable for you.

Thanks for clarifying that on the old core switch that vlan 100 passed through the core switch but did not have a vlan interface on the switch. So indeed the old core switch did not route for that vlan. Now the new core switch does have a vlan interface and does route for the vlan. That is a change in the network. If that was a conscious decision then it is probably fine. 

So now the issue about SSH access to new core is resolved? But there is still an issue about access to other devices in vlan 100? Is this just SSH access or is this any access at all? Are you able to tracert from your PC to one of those devices? And is the device able to traceroute to your PC? I suspect that if you do that we will find that the path to and from the devices is not symmetrical. Your PC packet to the vlan 100 device would go to the core switch which will forward directly to the destination (would not go through the firewall). But the response from the device would go to the firewall (which is the gateway for the device). Would the firewall security policies think it was a problem to get a response when it did not see the request (which is what stateful inspection on a firewall does).

[saba1418]

Thank you for your help on this issue. I did the traceroutes as you suggested and found the path is not symmetrical. We have also found that after locally logging in to the core we can successfully use it as a starting point to SSH to all devices on V100. So, it is definitely a routing issue between vlan 30 and vlan 100. What would the ip route between those two vlans look like? Would it include the vlan 100 range to vlan 30, or the vlan 30 range to vlan 100?

10.200.255.0 25/ 172.16.8.1

or 

172.16.8.0 21/ 10.200.255.20

or both?

[Richard Burts]

Thanks for confirming that the issue does turn out to be that the paths are not symmetrical. My response has several parts:

- first I do not understand what you are asking when you say "What would the ip route between those two vlans look like? Would it include the vlan 100 range to vlan 30, or the vlan 30 range to vlan 100?". In setting up routing you are looking at destination addresses and how to get to them. Your question about 100 to 30 or 30 to 100 suggests that you are considering both the destination address and the source address. To do that you would need Policy Based Routing.

- it seems to me that the issue here really is about who is doing the routing. If I understand the discussion correctly the original environment provided routing for vlan 100 only on the firewall. When you implemented the core switch you changed that and now have routing for vlan 100 on both the firewall and the core switch. And this produces the paths that are not symmetrical.

- If that change was intentional, then we need to figure a way that things will work. If the change was not intentional then just remove the vlan interface for vlan 100 on the core switch and you should be back to the original environment.

- and that makes me wonder if vlan 100 is the only vlan where you added a vlan interface (and therefore routing for that subnet) on the new core switch? Are there other subnets where you potentially have issues about path not being symmetrical?

- this really has become a discussion about what you want for the architecture for the network. The original architecture provided routing between vlans (or at least between vlan 100 and other vlans) on the firewall. The new architecture provides routing between vlans (or at least between vlan 100 and other vlans) on both the core switch and the firewall. Do you want routing on both? Or want routing on just one? And if on just one then which one? Once we get this resolved we can address how to change things to make them work.

[saba1418]

- first I do not understand what you are asking when you say "What would the ip route between those two vlans look like? Would it include the vlan 100 range to vlan 30, or the vlan 30 range to vlan 100?". In setting up routing you are looking at destination addresses and how to get to them. Your question about 100 to 30 or 30 to 100 suggests that you are considering both the destination address and the source address. To do that you would need Policy Based Routing.

I am just asking what the route would need to be to allow for a workstation on vlan 30 to be able to connect to a device on vlan 100 without having to leave the switch. Please disregard the question about "vlan 100 to vlan 30".

- this really has become a discussion about what you want for the architecture for the network. The original architecture provided routing between vlans (or at least between vlan 100 and other vlans) on the firewall. The new architecture provides routing between vlans (or at least between vlan 100 and other vlans) on both the core switch and the firewall. Do you want routing on both? Or want routing on just one? And if on just one then which one? Once we get this resolved we can address how to change things to make them work.

I considered the change to the network architecture unintentional. I included the vlan 100 SVI because I wanted to be able to manage the new switch on vlan 100 using the 10.200.255.20 management address, but wasn't really intending on changing the routing or anything in the process. Now that the new switch is in place, I just want it to be able to work, and I am okay with leaving the routing on both, unless there is a way to just manage the new core on vlan 100 without it routing as well...

[Richard Burts]

Thanks for confirming that the change was not intentional. I understand the logic of wanting to manage the new switch by an interface in vlan 100. However giving the new core switch an interface in vlan 100 means that the switch will begin to route for vlan 100. It seems to me that you have a couple of alternatives to consider which could resolve your problem.

- you could remove the vlan 100 routing from the firewall. That might be as simple as removing the vlan interface. Or it might require changing some routing logic on the firewall.

- a related question is whether there are any other vlans in your network which have a similar set up where routing for that vlan is on the firewall and not on the core switch?

- you might be able to leave the vlan interface on the firewall and just on every device in vlan 100 change their default gateway from the firewall IP to the switch IP.

- it might be possible to change some security policy on the firewall to permit traffic from devices in vlan 100 to go through the firewall even though its path was not symmetric. This would certainly be the most complicated and least desirable alternative, but it might provide a solution if the other alternatives are not selected.