Purpose of using a Switch as a default gateway as opposed to the Router it’s connected to

[fbeye]

Hello

This is sort of a generic question and don’t really have a good reason as to what my meaning is but…

I have a Router connected to a Switch.. On the Switch I have 3 vlans, all coming from different Routers. 
Ive noticed that any device connected to the Switch can either #1 see each other if they use the Switch gateway (but no internet access) or #2 connect to the internet by using their default routers gateway )but can not connect to other devices in the switch).

I’ve eliminated this issue by directly connect the other 2 routers to 2 interfaces in my default router (bypassing switch) and then their ip routes are in the ASA (router) and everything works fine.

I just don’t want to use 2 of the 8 only Interfaces on the ASA so that everyone can talk.

Would ‘ip routes’ on each Router to each other work instead of having a common gateway kn the switch?

[fbeye]

That makes sense. By no default route each subnet goes through their own Gateway for Internet access. The Catalyst will allow cross subnet traffic as it sees each subnet and with ip routing will allow communication to each network.

Once I verify with this setup as you mentioned the pinging inwards and outwards and each other but deduce that no data comm (unless this indeed allows it) that will lead us to a software /permission issue in the .111.

I will do all of this without a fresh approach for now but have no issues redoing it from scratch for the fun of it. 

[fbeye]

To clarify, with your latest response that is still with the understanding I’d need those routes from each Router to each other l. 

[Richard Burts]

Yes. For this to work each router/ASA will need 2 static routes for the other 2 subnets in the network.

[fbeye]

My mind is blown.

I did everything verbatim. On the ASA and the Router/s. Everything can ping everything else. 
I did not mention this earlier as I felt it as not relative, but maybe it is.

There is a device 10.0.2.111 (NAS) and 10.0.2.126 (NAS). Both are set the same, neither have a permissions / restrictions but I can not access 10.0.2.111 OR 10.0.2.1 but I can 10.0.2.126.

So as far as routing all is correct. But something is blocking me from data!

When I place the interface back to the ASA from the Catalyst then I can access everything everywhere. It simply doesn’t make sense.

Be it Catalyst or ASA I am using the same Interface IP so if some secret or unknown restriction/access were applied it would carry over. On my ASA there are no access lists or anything. 

[Richard Burts]

I am puzzled at this statement " I can not access 10.0.2.111 OR 10.0.2.1 but I can 10.0.2.126." I thought that you have been describing problems in access to both NAS. Is that not the case? You really can access one NAS but not the other?

I am also puzzled about the statement that if you move the connection from Catalyst to ASA that you can access both NAS. When you move the connection to the ASA which interface is the connection using?

[fbeye]

Correct.

  .126 can get Pinged and access data while .111 can get Pinged but no data (when I go through Catalyst).

When through ASA (I think it was 1/3 dedicated to 10.0.2.124) I could access 126 and 111 data and Ping. Right now the ASA has no ports dedicated to the 10.0.2.0 Network. I moved the “incoming” Ethernet from the 10.0.2.0 feed from ASA 1/3 to GE 1/0/11 (vlan 3) on Catalyst and created vlan 3 interface ip address 10.0.2.124 255.255.255.0.

I did not want to complicate the situation in the beginning and assumed if I fixed one, all would be fixed.

So yes this is true; through the Catalyst I can Ping all but only Data on 10.0.2.126. Through the ASA I can ping and access data on all INCLUDING accessing 10.0.2.1 (Router). Through Catalyst I can not access Router GUI or data on .111.

I can understand maybe it being a software issue in one of the NAS as the .126 is an actual OS (Linux based) and I can configure whereas .111 is proprietary OS with primitive limited ability but neither answers the riddle of the 10.0.2.1 GUI being access through ASA and not Catalyst. Especially cause though I am changing devices and interfaces I am not changing IP addresses. 

[Richard Burts]

Am I correct in understanding that you would be accessing .111 from the same device, with the same IP address and that if the connection for .111 is through the ASA you can access data but when the connection is through the Catalyst that you can still ping but can not access data?

If you could not access data and could not ping then we could assume that it was some sort of connectivity issue. But if ping succeeds but data fails then it suggests some sort of security policy somewhere. Perhaps on .111 or maybe somehow on ASA? Is it possible that when the connection was through ASA that it translated the source address but through Catalyst it does not?

I am wondering if perhaps .111 will respond to ICMP ping but not respond to other protocols. tracert from Windows uses ICMP for probes, but if you have some device in your network using *nix or perhaps the Catalyst you could try traceroute and see if .111 responds to that?

[fbeye]

I would be accessing both .111 and .126 from a 192.168.1.x IP Address. The “same” IP address was in reference to the “gateway” IP I had made, for example, on the ASA on Interface 1/3 I made it 10.0.2.124 so that 10.0.2.0 could be routed to through the ASA.

By removing from the ASA (nothing on the ASA has any reference to 10.0.2.0 other than the new ip route inside 10.0.2.0 255.255.255.0 192.168.1.5) and moving to the Catalyst, I made a vlan3 up address 10.0.2.124 and made GE 1/0/11-1/0/20 vlan 3. In case of any unknown ACL type security I wanted to at least keep the same IP that referenced the 10.0.2.0.

And that is correct. When through ASA I can ping AND connect to .111 data and .1 (Router) GUI but through Cat, only Ping. 
The only Linux box that I have is on the same subnet so I can set up another device on diff subnet with Linux and do some testing… My Windows PC on 192.168.1.6 should be able to give some results of tracert.

Last night looking at ASA and Cat, there are no and were no (when it works) NAT or security policies appended. 

It is such a weird issue. What is changing from ASA to Cat that a device who relies on IP address and uses same up address to work and not work. I understand having the 1/3 Interface on the ASA be 10.0.2.124 is one less route as it’s ON the device would be less complicated but then the ‘ip route inside 10.0.2.0 255.255.255.0 192.168.1.5’ would essentially be that same routing reference, in a way.

[nagrajk1969]

Hi

Can you disable "ip redirect" on ALL the routers (Router1/Router2/ASA) and the catalyst switch too?

- on the catalyst i think the "no ip redirect" has to be applied on the vlanX L3-interfaces

Iam thinking that the issue is happening mostly due to one or more of the Routers/Catalyst doing IP-redirect, and therefore only the Ping is working but not the tcp/udp connections....something like that

[fbeye]

Hello

Thank you for that suggestion. After looking into it it looked to be the exact option I needed to disable BUT to no avail. 
This has got to be an OS issue or some embedded Cisco issue but I feel it is not. The .111 and .126 are wired to the same Cat in the same manner meaning they are both connected the same way so there’s no weird obstacle for one and not the other. Even more so confusing is that I also lose ability to connect to the GUI of the 10.0.2.1 Router itself via the Catalyst. 
From my Windows I did a tracert to .111 and it had 2 hops; the Windows IP and the .111 IP and it was 52ms and it even listed .111 as TERASTATION so it sees it all, just won’t access. 
It works fine through the ASA and I was only hoping to eliminate / free up an Interface but that clearly isn’t happening. And I blame my OS.

[fbeye]

I had a quick thought.

I have 3 different subnets from various routers and the only thing in common would be/ is the Catalyst. 

Would I be able to set each device connected to the Catalyst to have its Default Gateway its respective IP of the vlan Interface IP? 
So for example;

vlan 1 IP address 192.168.1.5

   Everything connected to vlan 1 use      

   192.168.1.5 as its Gateway 

vlan 2 IP address 10.0.1.5

   Everything connected to vlan 2 use 10.0.1.5  

   as its Gateway 

vlan 3 IP address 10.0.2.5

   Everything connected to vlan 3 use 10.0.2.5

   as its Gateway. 

This would make their routing default the Catalyst.
And also, vlan 1 use 192.168.1.1 as its INTERNET, vlan 2 use 10.0.1.1 as IT’s Internet and vlan 3 use 10.0.2.5 as IT’s Internet.

So, the devices to all the vlans have their default gateways as their IP interfaces of their vlan interface counterparts but then also use their subnets ROUTER for Internet?  

[Richard Burts]

Early in this discussion I made a suggestion that was essentially what you suggest in your recent response: every device in each subnet/vlan have their default gateway as the Catalyst vlan interface for that vlan as their default gateway. It makes the routing decision as close to the source as possible, and I thought that would be a good thing. But the flaw in that suggestion is that if some device has 192.168.1.5 as its gateway then when it wants to access Internet it will forward the packet to the Catalyst address and not to the router address. So all 3 subnets would be using the Catalyst default route. This solution would need Policy Based Routing to work as you want. The better solution is for each device to have its router address as its default gateway (and for the Catalyst to not have a default route and to do only local inter vlan routing).

[fbeye]

You are correct you did mention this. I apologize to make it sound as if I had had an epiphany. 
So after I tested my theory I made all my my default gateways on my 192.168.1.0 Subnet to be 192.168.1.5 and NOW via Catalyst I can access 10.0.2.111, 10.0.2.126 AND 10.0.2.1 but had no internet so on Cat I did ‘ip route 0.0.0.0 0.0.0.0 192.168.1.1’ and gained back internet access as well. Devices on 10.0.2.0 and 10.0.1.0 still have their 10.0.x.1 as their Gateways so they get their correct ‘outside’ access. 
So, this “does” solve my problem and I am happy to free up an ASA Interface but is it the correct way? Should I have to change all the 192.x devices to 192.168.1.5 Gateway simply to access data on other subnet devices? I am fine with making something work in an unconventional way but I learn nothing this way…. But if it is indeed a correct alternative, then ok.

I just feel every device should be able to have their own default gateways of their origin router and frickin talk to each other and access data via gateway. You guys have all shown me so many variables and I have truly replicated all the advice but nothing makes it work as it should. 
Maybe I’ll just settle for “it works” rather than “it works but not for Cisco standards”. 

[Richard Burts]

Like you, I believe that usually there is a way that it should work and generally I prefer to use that solution. In this situation it should work for each network device to have its default gateway as its router/ASA, and for each router/ASA to have a static route for the 2 remote subnets. But when you implement this something prevents data access to a server that you need to access (while permitting ping access). I am very puzzled about what would cause this symptom. But we have not been able to solve it. And at this point I vote for "it works".

[nagrajk1969]

You should try applying the below on the Hosts

--------------------------------------------

With Manual Config on each of the Hosts/Servers 

-----------------------------------------------

1. On hosts in 192.168.1.0/24 network

a) If its windows:

route add 0.0.0.0 mask 0.0.0.0 192.168.1.1

route add 10.0.1.0 mask 255.255.255.0 192.168.1.5

route add 10.0.2.0 mask 255.255.255.0 192.168.1.5

b) If its a Linux-Host

ip route add default via 192.168.1.1 dev ethX

ip route add 10.0.1.0/24 via 192.168.1.5 dev ethX

ip route add 10.0.2.0/24 via 192.168.1.5 dev ethX

and 

2. On hosts in 10.0.1.0/24 network

a) If its windows:

route add 0.0.0.0 mask 0.0.0.0 10.0.1.1

route add 10.0.2.0 mask 255.255.255.0 10.0.1.5

route add 192.168.1.0 mask 255.255.255.0 10.0.1.5

b) If its a Linux-Host

ip route add default via 10.0.1.1 dev ethX

ip route add 10.0.2.0/24 via 10.0.1.5 dev ethX

ip route add 192.168.1.0/24 via 10.0.1.5 dev ethX

3. And the same steps as above for hosts in 10.0.2.0/24 network

Note: The route add commands for windows host may not be exactly correct, so check it out apply the correct way. 

- So in the result of the above routes would be that, by default all internet traffic will be routed to their respective Router/ASA

- And any traffic to other local-subnets match the specific routes that point to the respective vlan-interface ipaddress on the Catalyst-switch. So the specific routes always get precedence over the default-route (which is the last resort anyways)

--------------------------------------------

With automatic/dynamic config using DHCP

----------------------------------------------

Here on the dhcp-server,

a) in each of dhcp-scope (i.e for each subnet):

- configure the default-gw ipaddress of the respective router/ASA interface (192.168.1.1/10.0.1.1/10.0.2.1)

- and also enable/configure dhcp-option 121(or 249) and add the required static routes

for example in 192.168.1.0 dhcp-scope: the entries for option 121/249 will be

destination-net-1: 10.0.1.0/24 ; Gw: 192.168.1.5

destination-net-2: 10.0.2.0/24 ; Gw: 192.168.1.5

...and so on for other 2 subnets/scopes..

The dhcp-options 121/249 will do the needful of dynamically/automatically pushing the required static-routes (along with the default-route to the respective Router/ASA) to the respective dhcp-clients...

So the above is what implements Richard's proposal and allows you to route to internet via the respective Routers and to route to local-subnets via the catalyst-switch (using inter-vlan routing)

best regards