Multiple interfaces on multiple VLANs, multiple gateways?

[jnojr]

I feel really weird asking this.  It goes against everything I think I know about networking, but the guy telling me this is a CCNP, so I gotta figure either I'm wrong, or I'm completely missing something.  We have VMs with several network interfaces, each on a different VLAN / subnet.  As you can imagine, this is playing merry hell with properly routing stuff.  I'm being told that each subnet needs a default gateway to reach the router for that subnet.  When I ask how on Earth the IP stack is supposed to figure out which of those "default gateways" it should use to, say, send an ACK in response to a SYN from a non-local network, that something is supposed to somehow remember the MAC address of the router on the interface that the packet came in from and use that.

I've always been under the impression that the IP layer doesn't know jack except for the source and destination IPs, and that it has no way to know or care what interface a packet came in on, and the only way it can determine how to send responses would be via the routing table, ie. if the destination is not a local network and there's no route, it must use the default gateway.  And, further, that there's no point to having more than one default route, and that specifying several could only gum up the works.

I need some Advil and a drink...

[cofee]

Hi there, 

Actually what you are describing could be a common practice in many networks. A physical server that's hosting multiple VMs can have many physical ports and those ports could be used for different functions. For example you can use 1 port for management vlan another for vmotion and a couple of them for data/storage.

Now all these networks can have their own vlans/subnets and gateways so they can be reached from other subnets inside your LAN. ESXI/ virtual hosts have virtual switches just like physical switches and they have vlans configured, system engineers then bind physical ports on the server with virtual NICs. Usually active standby configuration works well when you have virtual hosts up linked to a pair of server access switches.

You mentioned ip layer having information about source and destination but then you have upper layer protocols for necessary information. It really depends how your network is set up and what the requirements are. I have seen some networks where everything that I mentioned ( vmotion, mgmt, data) all carried by 1 vlan. In my opinion this is a bad design, but I am not a system engineer. 

I hope it helps.

[#3]

I think you mean you have individual VMs with multiple virtual nics, (if I'm reading your message right), I also have the same and what I do is I have one nic that is basically my primary.  all my secondary networks do not have gateways assigned to prevent the system from sending packets out the wrong interface that is destined for the outside world.

I used to have issues with my core firewall dropping packets because of replies coming back on the wrong interfaces when I had multiple gateways on a single machine.

-Mike