cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
639
Views
0
Helpful
8
Replies

Virtualized Lab Infrastructure - 3560G connecting to a Nexus 7000 - Help!

mattmcspirit
Level 1
Level 1

Hi all,

I've been struggling with the configuration for my small environment for a week or so now, and being a Cisco beginner, I'm worried about going down the wrong path, so I'm hoping someone on here would be able to help with my lab configuration.

 

As you can see from the graphic, I have been allocated VLANs 16-22 for my use, on the Nexus 7000. There are lots of other VLANs in use on the Nexus, by other groups, most of which are routable between one another. VLAN 99 is used for switch management, and VLAN 11, is where the Domain Controller, DHCP and Windows Deployment Server reside for the lab domain. Servers across different VLANs use this DC/DHCP/WDS set of servers. These VLANS route out to the internet successfully.

I have been allocated eth 3/26 on the Nexus, as my uplink connection to my own ToR 3560G. All of my servers, of which there are around 8 in total, are connected to the 3560. I have enabled IP routing on the 3560, and created VLANs 18-22, providing an IP on each. This config has been assigned to all 48 gigabit ports on the 3560 (using the commands in the graphic), and each Windows Server 2012 R2 Hyper-V host connects to the 3560 via 4 x 1GbE connections. On each Hyper-V host, the 4 x 1GbE ports are teamed, and a Hyper-V vSwitch is bound to that team. I then assign the VLAN ID at the vNIC level.

Routing between the VLANs is currently working fine - As a test, i can put 2 of the servers on different VLANs, each with their respective VLAN default gateway, and they can ping between one another.

My challenge is, I'm not quite sure what i need to do for the following:

1) How should I configure the uplink gi 0/52 on the 3560 to enable my VLANs to reach the internet?
2) How should I configure eth 3/26 on the Nexus?
3) I need to ensure that the 3560 is also on the management VLAN 99 so it can be managed successfully.
4) I do not want to route to VLAN 11, as i intend to have my own domain (DC/DNS/DHCP/WDS)

Any help or guidance you can provide would be much appreciated!

Thanks!

Matt

8 Replies 8

Jon Marshall
Hall of Fame
Hall of Fame

Matt

To ensure isolation except internet ideally the link should be a routed link and then you would have a default route pointing to the Nexus end and the Nexus would need routes for your subnets.

I would suggest using static routes if this is your own setup although if the Nexus is using a routing protocol and your feature set supports it you could run that.

This way you simply don't advertise the vlan 11 subnet  to the Nexus or have a static route so there can be no conflict between your DHCP etc. and the others.

And because it is L3 then things like DHCP broadcasts couldn't go across the link even if you misconfigured it.

However the drawback is you can't then extend vlan 99 to the Nexus.

So if you want to have the 3560 in vlan 99 then you are going to need a trunk link to pass multiple vlan traffic.

So the easiest configuration then would be -

1) create a new vlan just for routing between the 3560 and the Nexus. This new vlan has no end devices in it.

You need a new IP subnet for this vlan but it only needs two useable IPs one for each end.

2) on the 3560 and the Nexus create an SVI for this vlan and allocate one of the IPs to each SVI.

3) make the link between the 3560 and the Nexus a trunk link and then you only allow the new vlan and vlan 99 on that trunk link ie. all your other vlans are excluded from the trunk.

Then you simply treat the link as a P2P link as above and add static routes or run a protocol using the SVI IPs as the next hops.

Again you wouldn't need to advertise vlan 11 and in addition you don't need to advertise vlan 99 because that vlan has been extended back to the Nexus switch.

So basically all your vlans  are routed locally on your 3560 switch and only go to the Nexus (with the exception of vlan 11)  for internet.

And vlan 99 is routed on the Nexus switch so you still need an SVI on the 3560 in vlan 99 but it is purely for management.

Jon

Jon - thank you so much for your detailed reply - i really do appreciate it. I apologize in advance if I don't fully grasp all of your instructions, but from a command perspective, would it look something like this? I'm going to use VLAN 16 as my 'way out to the internet'

3560:

interface vlan16
ip address 10.0.6.2 255.255.255.252
no shutdown

interface vlan99
ip address 10.0.99.100 255.255.255.0
no shutdown

interface gi0/52
switchport mode trunk
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 16,99
end

ip route 0.0.0.0 0.0.0.0 10.0.6.1

Nexus: (VLAN 99 already configured)

interface vlan16
ip address 10.0.6.1 255.255.255.252
no shutdown

interface ethe3/26
switchport mode trunk
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 16,99
end

2 things I'm not sure how to configure:

1) Treat the link as a P2P link as above and add static routes or run a protocol using the SVI IPs as the next hops

For example, if i have VLAN 18 configured as 10.0.8.1 255.255.255.0 on the 3560, what would i need to define as a static route to allow it to reach the internet? it already routes successfully to the other VLANs on the 3560 without adding any static route (and when i tried to add one, such as 10.0.8.0 255.255.255.0 10.0.9.1 (to route to VLAN 19) it gave me an error along the lines of 'it's on this router' so I'm presuming perhaps the static route should have been 10.0.8.1 255.255.255.0 10.0.6.1 (or 10.0.6.2, the nexus end of the 'internet' trunk)?

2) I'm not sure what 'advertising a VLAN' means: "you wouldn't need to advertise vlan 11 and in addition you don't need to advertise vlan 99 because that vlan has been extended back to the Nexus switch"

How does that look?  Thanks again, so much for the help so far!

Matt

Matt

Not that familiar with Nexus configuration but if that is how a trunk is configured then yes for both switches that is exactly how it would look.

If you aren't sure about the Nexus syntax I can always look at the command references if you need me to.

1) what I meant by that is pretty much what you have done in terms of the default route you have added to the 3560.

So the only thing missing is you also need routes on the Nexus switch for the IP subnets on your 3560 and the next hop IP would be 10.0.6.2 ie the vlan 16 SVI IP on the 3560.

You don't need to add a route for the vlan 99 IP subnet as the Nexus already knows about it.

2)  when I referred to advertising I was talking about not using statics and running a dynamic routing protocol between the Nexus and the 3560 so you would need to tell the routing protocol on the 3560 to advertise it's subnets to the Nexus ie. tell the Nexus about the IP subnets on your 3560.

And I simply meant you wouldn't advertise vlan 11 because you don't want the Nexus to know about it and you wouldn't advertise vlan 99 because the Nexus already knows about this network.

However I have been assuming vlan 11 doesn't need internet access.

If it does then you do need the Nexus to know about that IP subnet so if you are going to use statics you need to add it as route on the Nexus as above or include it in your dynamic routing protocol.

To be honest if it is lab environment and you are not going to be creating a lot more vlans/IP subnets then I would use statics because there is less to go wrong and you can't affect the main routing on the Nexus but it's entirely up to you.

Any more queries or anything I haven't explained properly just let me know.

Jon

Thanks again Jon, for both of your responses!

So, for VLAN 11, it's defined on the Nexus, and won't be defined on the 3560. VLAN 11 contains the DC, DHCP and DNS for servers on VLAN 11, and lots of other VLANs. VLAN 11 is also internet accessible already.

I don't want my 3560 to touch VLAN 11 (as I'll be running my own domain inside VLAN 18) so i haven't defined that as an SVI on the 3560).

I'll work on the static routes, and update the thread accordingly. I'm confident that the routing from Nexus on to the internet is all configured and setup, so we should be good to go there.

Thanks again for your help, and i'll update as I go forward!

Hi again Jon,

OK, been battling with it a little more.

Here's the config for the 3560:

Current configuration : 11643 bytes
!
version 12.2
no service pad
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname CSP_DX_Cluster
!
!
no aaa new-model
vtp mode transparent
ip subnet-zero
ip routing
!
!
!
!
no file verify auto
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
vlan 16,18-23,99
!
interface GigabitEthernet0/1
switchport trunk encapsulation dot1q
switchport trunk native vlan 18
switchport trunk allowed vlan 18-22
switchport mode trunk
spanning-tree portfast trunk

<same through interface GigabitEthernet0/48>

interface GigabitEthernet0/52
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 16,99
switchport mode trunk
!
interface Vlan1
no ip address
!
interface Vlan16
ip address 10.0.6.2 255.255.255.252
!
interface Vlan18
ip address 10.0.8.1 255.255.255.0
!
interface Vlan19
ip address 10.0.9.1 255.255.255.0
!
interface Vlan20
ip address 10.0.12.1 255.255.255.0
!
interface Vlan21
no ip address
!
interface Vlan22
ip address 10.0.14.1 255.255.255.0
!
interface Vlan99
ip address 10.0.99.87 255.255.255.0
!
ip classless
ip route 0.0.0.0 0.0.0.0 10.0.6.1
ip http server
!
!
control-plane
!
!
l
end

At the Nexus end, the port connecting to the 3560 is configured as:

interface Ethernet3/26
  description DX_3560_uplink
  switchport
  switchport mode trunk
  switchport trunk allowed vlan 16,99
  no shutdown

Now, the problem I'm currently having, is that on the 3560, things route fine, between VLANs. However, from on a server within one of the VLANs, say, 18, trying to ping the default gateway of the 3560 fails. I can ping 10.0.6.2 which is the 3560-end of VLAN 16, but i can't get over to 10.0.6.1 and beyond. I suspect, it's relating to what you said about "the only thing missing is you also need routes on the Nexus switch for the IP subnets on your 3560 and the next hop IP would be 10.0.6.2 ie the vlan 16 SVI IP on the 3560"

I suspect that, in layman's (my terms!) terms, the Nexus simply doesn't know about the networks 10.0.8.1 (VLAN 18), 10.0.9.1 (VLAN 19) and so on.

So, i need routes on my Nexus to fix this. The problem is, I'm not quite sure what that looks like.

Would it be:

ip route 10.0.8.0 255.255.255.0 10.0.6.2
ip route 10.0.9.0 255.255.255.0 10.0.6.2 and so on?

To give a bit of history, prior to me creating VLANs 18-22 on the 3560, all VLANs originally existing on the Nexus. Everything routed fine out to the internet, for all of the VLANs (with the same subnet settings that i have configured, i.e. 10.0.8.x for VLAN 18 etc), so i'm presuming once I get the Nexus to understand that the IP subnets live on the 3560, traffic should flow successfully to the internet.

Should.... :-)

Matt

You just caught me before logging off.

The syntax for adding the routes is correct as far as I can see but if they don't work then you can also try using -

ip route 10.0.8.0/2410.0.6.2

ie.instead of 255.255.255.0 you use the / notation but I believe either will work.

However if the vlans/IP subnets used to exist on the Nexus you can't assume it will work because the Nexus may have been using a dynamic routing protocol to advertise those routes.

It all depends on what other L3 devices the Nexus has to go through to get out to the internet.

I don't know how that has been setup so you may need to use a dynamic routing protocol or you may need to redistribute those static routes on the Nexus but you need to talk to the admins of the Nexus if internet isn't working.

They may have a preference as to how they want it done.

So you can try adding statics but it may not work. And they may say they don't want to have redistribute the statics and they would rather use a routing protocol but if the statics don't work I can't tell you what to do because what you do there could affect the rest of the network.

But it may work with just statics.

Jon

Thanks Jon - much appreciated. I'll try that out on Monday!

Thanks for your help, and have a good weekend.

Matt

Matt

One thought that has just occurred to me.

You say the Nexus handles routing to the internet.

If it sends traffic a firewall or there are other L3 devices between it and the firewall all those devices will also need to know about the IP subnets you are using.

So using statics may only get you connectivity to the Nexus and not beyond.

I can't be more specific because I don't know how the routing is setup in your environment so you would need to talk to the Nexus administrators as to which is the best way to get those routes known beyond the Nexus.

You will also need NAT setup on the firewall for your IP subnets and perhaps updates to the firewall rule base to allow your traffic access although both may already be in place.

Sorry I can't be more specific but I would need more information to be precise.

Jon

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Innovations in Cisco Full Stack Observability - A new webinar from Cisco