Reason for the switch uplink and the network design - really wanted to know ;)

[SJ K]

Hi all,

I am involved in a project whereby there is a L2 switch which is uplink to L3.

The L2 are where all the end-user/edge devices are connected to.
The L3 are where all the servers devices are connected to.

Q1) Sorry for my ignorance, but can someone explain further on why the edge devices are connected to the L2 switches ?
In any cases, I believe these devices will still need certain level of routing. Why not just put them all to the L3 switches ?

Is it because the devices are more likely to be in a flat network, with less routing required + to not waste ports on the L3 + to save cost + the theory of access , distribution and core. ?
 

=================================================

The L2 devices are designed to be in 1 VLAN  (VLAN02) and the L3 servers in another VLAN (VLAN03).
Both the L2 and L3 devices need to interact with one another.
Both the L2 and L3 devices are required to access the Internet.
 

Q2) Can I say the main reason for the uplink from L2 switch to L3 switch is to enable routing between the L2 devices and the L3 servers ? Is there any other reason why ?

Q3) Can the uplink be a trunk uplink as well ?

==================================================

Assuming
VLAN02 = 192.168.2.0/24
VLAN03 = 192.168.3.0/24

Q4) I have never try on L3 device before. In the setup above, how do I enable routing between the L2 and L3 devices ? What should i set as the gateway for the L2 devices.

Do i need to assign a port/interface in the L3 switch to be part of VLAN02 and set it as the gateway ? Or is there something like a virtual port setup in the L3 switch. Then I will assign it as part of VLAN02 and assign an IP to it.  Then I will set it as the gateway for the L2 devices ?

Do the same for the L3 servers. Then how does VLAN02 and VLAN03 communicate to each other ? through a routing table in the L3 switch ? and how does the traffic from the L3 servers get routed down through the uplink to the L2 switches ?

Q5) Also, how do I enable routing to allow both the VLAN02 and VLAN03 to access the internet ?
 

I have try to draw a diagram as below to conceptualize as shown above, but fail to do so :(
Hope some gurus can shed some light (hopefully with some illustrations).

Thanks....
Regards,
Noob
 

[Hieu Cao · Accepted solution]

Q1 - Cost or L2 switch the only switch available at the time to extend your network connections to the clients

Q2 - Cost or L2 switch is the only switch available at the time to extend your network connections to the clients. Routing is done at L3, switching is done at L2

Q3 - The uplink is more than likely already setup as a trunk

Q4 - Routing is being on L3 switch since you've multiple VLANs in the same VTP domain.  See if the L3 switch has the "ip routing" command

       Gateway for L2 devices should be set to 192.168.2.3

Q5 - Set the default route in the L3 switch to point to its next hop, which is the router FastEthernet's IP address
        ip route 0.0.0.0 0.0.0.0 FastEthernet_IP_Address

Hope that helps.

[Jon Marshall]

By the way, I don't know why the connection to the router is shown as a trunk.

You would only need this if you were routing between the vlans on the router but you have a L3 switch so I assumed you were doing the routing there.

If so the connection to the router should just be a L3 link.

Jon

[Jon Marshall · Accepted solution]

q1) the access layer is primarily for end devices such as PCs etc.

In terms of routing it depends where the routing between vlans is being done. A common design is for L2 at the access layer and then L3 at the distribution layer so the uplinks from the access switches are usually trunks and the vlans are then routed on the distribution switches.

It's nothing to do with a flat network because you can have as many vlans as you like, it's just they are routed at the distribution layer.

That's not the only way to do it ie. you can have L3 access layer switches and route on them but that is a lot less common.

q2) The main reason for the uplink is because without it the devices on the access switch wouldn't be able to communicate with anything else in the network.

It is used for traffic to and from the devices on the access switch, that's it's purpose.

Whether that is to other clients in the same vlan, or a different vlan, on other access switches, servers in another vlan or the internet.

In your setup it looks like it is for traffic to the servers and the internet but, as I say, more generally the uplink is just for connectivity to the rest of the network.

q3) yes and if the uplinks are L2 they are often etherchannel trunks.

q4) On a L3 switch you create what are called SVIs (Switched Virtual Interfaces) eg.

int vlan <num>
ip address x.x.x.x <subnet mask>

and the clients in that vlan, whether they are PCs, servers etc. would use the IP address on the SVI as their default gateway.

Each vlan that you wanted to route to and from would need an SVI.

You also have just one SVI on a L2 switch for management but it should never be the default gateway for clients.

5) you add a default route to your L3 switch pointing to the firewall and you add routes to the firewall for the IP subnets on the L3 switch.

Jon

[SJ K]

Hi Jon and Hieu,
 

Sorry for the late reply and thanks for the responses.
 

q1) If there is only 1VLAN in the L2 switch, is it necessary for doing trunk then ? (or it has to be at least 2 VLANs) considering the default management VLAN1

q2) if we decide not to trunk,  does it means that for every VLAN, i will need to have waste 2 interfaces (1 on each switch) for the uplink from L2 to L3 ?
 

q3) in the earlier setup (with trunking and SVIs), you mention "5) you add a default route to your L3 switch pointing to the firewall and you add routes to the firewall for the IP subnets on the L3 switch"

Does the connection/interfaces between the switch and router need to be in any VLAN ?

**q4) Most importantly, can gurus here advise how the traffic to/fro from the client to the l2 switch to the l3 switch to router and vice versa will be ? is it as per below ?
 

(assuming all interface macs addresses are already known and populated in the mac address tables -- arp completed)

outgoing packet from end user device to internet

l2 device send a packet --> to l2 switch interface --> vlan information is tag -->  l2 switch check mac table --> saw that the frame is for the SVI's mac address --> go up to l3 switch via trunk port (how does it know it must go up via the uplink port ?) --> l3 switch receive the packet --> look at the vlan tag --> only send to the interfaces/port belonging to the vlan --> svi interface saw that the packet is for itself --> since it is a l3 switch --> strip off the vlan information and mac address --> look at the destination ip --> decided its not for itself --> check the routing table -->  saw the default gateway which exit by (e.g. interface fe0/5, next hop address = router's connected to the switch interface's ip address) --> rebuild the src and dest mac address, frame the packet --> send out to the router

how does a switch mac address table looks like when it has an uplink port ? when does a switch knows when it must send a frame up/down via the uplink port ?
 

incoming packet to end user device from internet

router internal facing interface  --> send to L3 switch interface --> L3 switch remove the frame --> check the ip address, decided its not itself --> look at the routing table --> saw that it is for the client/enduser network and need to exit via the client/enduser SVI (vlan02) --> reframe the packet with src(svi interface mac)/dest mac + vlan02 information --> and send out via SVI interface --> how does the L3 switch knows that it needs to go down the uplink trunk to the L2 switch and then to the designated device ??  (the l3 switch look at its mac address table and knows that the destination make need to exit via the trunk interface on L3 ? and thus the frame get send down via the trunk to the L2 switch ) ?

q5) can 2 vlans have the same private subnet addresses ? since theoretically they are connected to "separate" switches ?  It is just the 2 vlans will be non-routable, am i right ?

Regards,
Noob

[Jon Marshall]

q1) no but you usually have a separate management vlan

q2) if you have multiple vlans then use a trunk don't have separate connections.

q3) no, it should be a routed link

q4) the vlan tag is stripped by the trunk port not the SVI.

The switches know to use uplink port because the mac address tables of the switches record which mac addresses are reachable via this port.

Everything else you wrote is correct.

q5) no in the sense you cannot create SVIs on the L3 switch with IPs from the same subnet.

If you used a VRF you could use the same subnet for two vlans but then you couldn't route between VRFs for those subnets.

Jon

[SJ K]

Hi Jon,

Thanks for answering my queries. I am really glad as I don't think no one elsewhere would have gone through my questions. Truly appreciate it. Thank you.

 

if you have multiple vlans then use a trunk don't have separate connections

q1) Just out of curiousity, is it possible to not use a trunk, but have separate uplink connections for each vlan  ? (although I know no one would have done it to waste ports)
 

no, it should be a routed link

q2) What do you mean by a routed link ?  Is it basically just
ip route 0.0.0.0  0.0.0.0  192.168.5.1(=default gateway to router) ?

the vlan tag is stripped by the trunk port not the SVI.

q3) Do you mean that when the trunk port receive a frame with vlan tagged, it will look at the vlan tag, strip it off, and send to the associated ports ?

Can I check if each VLAN will have its own MAC table or the switch itself will have an overall MAC table ?
Is the VLAN information stored together in the MAC table also ? or it is store elsewhere ?

How does a trunk port strip off a frame's VLAN information and yet still know where/how to move the frame to the correct VLAN? It would be good if you can elaborate abit further more on what happen next when a trunk port receive a frame with a vlan tagged

======================================================================

q4) In a mac table on L2, does it contains all the MAC addresses of all the connected ports in the L3 switch ?

.e.g  a.b.c.d is a mac address of fe0/1 in the L3 switch &
        fe0/5 is the uplink port on L2

L2 Switch
MAC address  exit Port
a.b.c.d               fe0/5

========================================================================

q5) Lastly Jon, I really do not have a lab to test this out, is there anywhere with free simulator whereby I can try what I have learn here 
-- e.g. to create l2 vlan, uplink to l3 switch, route between l2 and l3 vlan, and to router etc..

Thank you!
Regards,
Noob

[Jon Marshall]

q1) it's possible yes but in practice it isn't done because as you say it is a waste of ports.

q2) on a L3 switch all ports by default are L2 switchports and you create SVIs to route between vlans. However you can make the port a L3 port eg.

int <x/y>
no switchport
ip address x.x.x.x <subnet mask>

so the port actually has an IP address assigned and the L3 switch can route between this and the SVIs.

q3) trunk ports add or remove tags for vlans (except the native vlan). So a trunk port sending a frame will add a vlan tag to the L2 header to identify which vlan it is in.

The receiving switch removes the tag to see what vlan the frame is in and then using the mac address table deliver it to the correct host (if the mac address is known).

As far as I know it is one mac address table although that question has come up before and I could never find a definitive answer.

q4) the mac address table is primarily concerned with learning the addresses of connected devices. That may be an end host but it could also be an uplink port to another switch where there will be multiple mac addresses recorded against that port.

It's doesn't record the mac addresses for the actual ports on the switch.

q5) there are two main simulators I know but both have issues in terms of availability -

Packet Tracer allows you to configure switches and routers but you need to a member of the Cisco Net academy I believe to be able to download it

The one most widely used by people on here, I suspect, is dynamips and GNS3 which is a graphical front end.

This allows you to run the actual IOS on your PC, laptop and you can build topologies of routers.

It doesn't do switches although it does do switch modules I believe. There was talk of it being able to support switches soon but I haven't kept up with it.

The problem is you need an IOS version to use and these are not free so you cannot just download one from Cisco.

There may be other simulators available but I'm not aware of any.

Jon

[SJ K]

Hi Jon,

Thanks for the reply

it's possible yes but in practice it isn't done because as you say it is a waste of ports..

q1) Noted. In the event above (uplink for every individual vlans), does the uplink ports itself, need to belong to that particular VLAN ? is it still call a access port ?

q2) can i confirm that uplink ports, even for usage of trunking  - will not have any IPs assigned to it - am i right ?
 

on a L3 switch all ports by default are L2 switchports and you create SVIs to route between vlans. However you can make the port a L3 port eg.

q3) Sorry Jon. I am lost on this one. I thought once the SVIs are created, we just need to route them as it they are interfaces on the routers.
Do you mean that there is another way to route beside creating SVIs and that is to create a L3 port ?

q4) However, in my previous question , it was to route the VLANs traffic on the L3 switch to the actual router.  So i thought it was to add an entry in the route table inside the L3 switch to point access to other networks to the gateway IP of the router's internal facing interface which is connected to the L3 switch.

 trunk ports add or remove tags for vlans (except the native vlan). So a trunk port sending a frame will add a vlan tag to the L2 header to identify which vlan it is in.

The receiving switch removes the tag to see what vlan the frame is in and then using the mac address table deliver it to the correct host (if the mac address is known).

As far as I know it is one mac address table although that question has come up before and I could never find a definitive answer.

q5) Can I say that the only reason why VLAN tagged is read is for broadcast frames only.  Because if a receiving switch know which mac addresses/hosts reside on which ports/interfaces with the use of a MAC address table, there is no reason that a frame will get send to a wrong VLAN, isn't it ?

the mac address table is primarily concerned with learning the addresses of connected devices. That may be an end host but it could also be an uplink port to another switch where there will be multiple mac addresses recorded against that port.

q6) Sorry Jon, my brain isn't thinking straight..  lets say in the example below, how will the mac table looks like ? Assuming a frame arrive on the L3 switch and needs to go to the designated hosts on the L2. Does the mac table below on the L3 switch looks right ?

q7) I am seeing fe x.y ; example fe0/1 fe0/2  - - what does "x"  actually represents ?  could there be a fe1/1 ?
 

Thanks!

Regards,
Noob

[Jon Marshall]

q1) yes it would need to belong to that vlan as an access port.

q2) L2 ports ie. access ports or trunk ports don't have IP address assigned to them

q3) it's not another way to route, it's more that you can have a different L3 interface type.

SVIs are used for vlans and this makes sense ie. imagine a L3 switch with a trunk connection for multiple vlans. The trunk port is L2 so you can't make that a L3 port with an IP address.

Technically you could make it a L3 port with subinterfaces as in routing on a stick but you don't do this because -

1) most L3 switches don't support subinterfaces on L3 ports

and

2) far more importantly your access switches connect to the L3 switch with trunks.

Now imagine if a client on one access switch needs to communicate with a client on another access switch but they are in the same vlan. The traffic has to go via the L3 switch but it isn't routed because it is in the same vlan. It works because the uplink is a trunk.

If it was a L3 port with subinterfaces it wouldn't work because the traffic between access switches could no longer be L2 switched ie. it would have to be L3 routed and you don't route within the same vlan.

To be more specific you don't route within the same IP subnet but it is usually a one to one mapping between vlan and IP subnet.

So that's why you use SVI for vlans.

For connected devices like routers though you don't have these issues so you can simply make the port a L3 port.

The switch will route between SVIs and L3 ports, you don't have to add any additional configuration.

You could, if you wanted connect the router in it's own vlan, assign the port on the switch where the router connects into that vlan and use an SVI.

But it's just a lot simpler if you use a L3 port and it also means because it is L3 that you aren't sending things like STP etc. to the router which doesn't need to receive them.

q4) you still add routes are you would whether using SVIs or L3 interfaces. As I say there is nothing extra needs to be done in terms of route entries.

So your default route simply uses the next hop IP of the router. Doesn't matter whether you are using an SVI or a L3 port on your L3 switch.

One thing design wise, even if you did use a vlan and SVI to connect to the router it should be a dedicated vlan ie. don't use an existing vlan with clients in it.

q5) good question and as I said before I don't have had a definitive answer but yes I suspect what you say is correct ie. broadcasts and potentially multicast are why the switch needs to know the vlan.

Must get round to looking into this :-)

q6) no need to apologise, all good questions. Yes, that is exactly what the switches mac address table would show.

Don't get too hung on uplinks ports and other ports. From a Cisco perspective, other than what we discussed in another thread, they behave the same way in terms of mac address learning etc.

q7) if I understand the question it is just the interface numbering. So for example on a stack of switches, or a modular switch like the 6500, you would see the x part increment for each switch in a stack or for each module in the chassis.

The y part is simply the port on that specific switch or module.

Jon

[SJ K]

Hi Jon,

Thanks for the reply :)!

q1) So can I confirm for trunk uplink, the uplink ports will not belong to any VLAN
     For individual uplink of each VLAN (if i do not want to use trunking),  the uplink ports will have to belong to that particular VLAN.

Am i right ?

q2) For the portion on q3 and q4 in the earlier thread -> I am sorry for the misunderstanding, as I am not trying to replace the SVIs as a L3 port.
Assuming

interface VLAN02 - SVI gateway IP = 192.168.2.3  (for 192.168.2.0/24 network)
interface VLAN03 - SVI gateway IP = 192.168.3.3  (for 192.168.3.0/24 network)
Router's interface IP - 200.1.1.2
 

Do I need to create/assigned an L3 interface in the L3 switch for interconnection with the router ?   (I guess this is what you meant by routed link earlier on ;))
(e.g. as below)

Switch(config)#interface FastEthernet 0/1
Switch(config-if)#no switchport
Switch(config-if)#ip address 200.1.1.1 255.255.255.0
Switch(config-if)#no shutdown

q3) In the event i need to create static routes between the VLANs and the Router, is the command below correct then ?

ip route 192.168.2.0 255.255.255.0 [???]
ip route 192.168.3.0 255.255.255.0 [???]
ip route 0.0.0.0 0.0.0.0 200.1.1.2 or fe0/1

But for the ??? what should I put ?
- I do not have the exit interface (is there an interface name for SVI ?) 
- Neither do I have the next hop IP addresses

q4) Lastly, as you mentioned fe x/y, the x means the switch module. For example in a stack switch of 3,  which switch in the stack  with fe0/y ? the 1st one from the top, or the last one from the bottom ?

Thanks! Jon.. I have only thanks to offer, please accept them.
Truly appreciate all the help these while. To be honest, I am abit dishearten in my journey of networking as I am all theory and no actual practical/work experience... lucky you are there to offer guidance. :)

Regards,
Noob

[Jon Marshall]

q1) yes, trunk ports don't belong to any specific vlan and access ports do.

q2) no I didn't think you were trying to replace the SVIs. We were only talking about the router connection.

Yes your configuration is correct although usually for a P2P link you use a subnet mask of 255.255.255.252 ie. two useable IPs.

You don't have to and your configuration will work it is just normal practice.

q3) using your configuration on the L3 port, for the first two routes they would go on the router and the next hop IP would be 200.1.1.1 ie. the IP of the L3 port on the switch.

Note these are added to the router so it knows how to send return traffic back to the L3 switch.

The third route would go on the L3 switch and you would use the next hop IP of 200.1.1.2 ie, the router interface.

That's all you need.

q4) With stacks I think, although not sure, it depends on the order of switches although I could be wrong there. Haven't used them in a while.

For a modular switch it increments from top to bottom in the chassis.

Don't get too disheartened, you are asking the right questions and you'll get the experience you need in time.

The key thing is to understand the concepts and how it all works and you seem to be doing fine with that.

Jon

[SJ K]

Hi Jon,

Thanks for the reply.
 

You don't have to and your configuration will work it is just normal practice.
Q1) Do you mean I dont have to create an L3 interface on the switch port ? (this below)

Switch(config)#interface FastEthernet 0/1
Switch(config-if)#no switchport
Switch(config-if)#ip address 200.1.1.1 255.255.255.0
Switch(config-if)#no shutdown

======================================================================

using your configuration on the L3 port, for the first two routes they would go on the router and the next hop IP would be 200.1.1.1 ie. the IP of the L3 port on the switch.
Note these are added to the router so it knows how to send return traffic back to the L3 switch.
The third route would go on the L3 switch and you would use the next hop IP of 200.1.1.2 ie, the router interface.

Q2) Jon, I am actually referencing to the creation of the statics/route table inside the L3 switch and not the router.

ip route 192.168.2.0 255.255.255.0 [???]
ip route 192.168.3.0 255.255.255.0 [???]
ip route 0.0.0.0 0.0.0.0 200.1.1.2 or fe0/1

Hence when you mention "the 1st 2 route will be using the next hop IP 200.1.1.1", I believe you are referencing to the routing inside the router. 

But how about routing on the L3 switch ? If I wanted VLAN02 and VLAN03 to be routable between each other.  On the L3 switch, what should the ??? be ? (e.g. the directly connected interface would be the SVI for VLAN02 for network 192.168.2.0)

ip route 192.168.2.0 255.255.255.0 [???]
ip route 192.168.3.0 255.255.255.0 [???]
 

Regards,
Noob

[Jon Marshall]

q1) sorry, I meant you don't have to use a 255.255.255.252 subnet mask, you can use the subnet mask you used instead.

You still need the L3 port.

q2) you don't need to add routes on the L3 switch for the directly connected networks. They will be automatically put into the routing table for you.

Any L3 device, as long as routing is enabled, will route between subnets which are directly connected ie. it has a L3 interface (SVI or port) with an IP from that subnet.

If you did a "sh ip route" on the L3 switch you would see them in the routing table.

Nothing else needs to be done for these subnets and routing will happen automatically.

But for networks not connected to the L3 switch or router you do need to add routes which is why the L3 switch needs a default route pointing to the router and the router needs routes for the L3 switch subnets.

You can use a dynamic routing protocol between the devices to exchange the routing information but in this case you are using static routes which works fine with the number of subnets you have but when you scale up adding statics everywhere becomes unmanageable so then you would use a routing protocol instead.

Jon

[SJ K]

Sorry for the late reply Jon and thank you so much for the answers!