04-30-2013 03:12 PM - edited 03-07-2019 01:07 PM
Hey all,
So .. as of right now I have a pair of N5K's, down stream from them are from Fabric Interconnects and a UCS chassis. Upstream is a stack of 3750's then ASA5510's.
I am trying to backup the config to our TFTP server and I am getting 'no route to host'.. I tried to add a route, and found that N5K uses VRF's for routing?? .. After some looking I see there are two base VRF's 'management' and 'default'.. the management VRF has a default gateway entry and a single interface member (mgmt0).. when I look at the default VRF .. there are no interface members or routing entries.. Ok, I can handle that just add some interfaces and add a default gateway. Then I get lost:
I'm able to access the UCS manager..... so how the heck is that even possible if there's no gateway defined anywhere (or maybe I'm missing something?). My theory was: add all other ports but mgmt0 to the default VRF, and have the default gateway point out of the uplinks (a vPC).. but wasn't sure how that would affect anything and mainly just wanted to know how I was able to access the UCS manager in light of the fact that there is no default gateway anywhere that I could see...
Thanks so much in advance ! This NX-OS is really something else...
Kindest Regards,
ALAN
Solved! Go to Solution.
05-01-2013 03:59 PM
Alan,
You not being able to ping the tftp server from the default vrf (global) is a correct behavior, because the tftp as you already know is part of vrf management and you can't ping it from default. Remember, vrf management and vrf default have different routing tables and you can not mix them together. Think of these 2 vrfs as 2 different devices, but in this case it is not a physical separation rather logical.
It that more clear now?
Reza
04-30-2013 04:08 PM
Hi Alan,
The Nexus-OS is pretty cool. Once you get use to it, you don't want to use IOS anymore.
Ok, so you understand the difference between the management VRF and default. Default is the equivalent of global routing table and the mgmt0 is in management VRF
Are the 5K layer-2 switches only or also layer-3?
If you are running them as layer-2 then all you need is default routes towards the 3750
If they are layer-3, are you running a routing protocol with the 3750?
Can you post your config?
HTH
05-01-2013 08:17 AM
Hey Reza!
Absolutely - just from scratching the surface of the documentation I think it's a very intelligent OS/design and am looking forward to implementing it properly/fully.
The 5k's are Layer 2 right now (I'm not 100% on this but a 'sh license usage' shows LAN_BASE_SERVICES_PKG as a 'no') afaik. Again a bit sketchy on this.. from a logical angle, there are no layer 3 protocols running on the device so far. Our management VRF does have a default route and shows a few directly connected entries though.
So - that makes sense that default is the equivalent of global routing - my initial thought was: Ok, add the uplink port-channel/vPC to the 3750's to the default VRF (along with all the other ports save mgmt0) and set the uplinks as the default route (interface destination not IP). What do you think?
My only question in all that was: well then how can we access the UCS manager now - if there are no interfaces in the default VRF and there are no routes in the default VRF?? It just made me pause and think before changing stuff. Very excited to hear your response!! I'm posting the running config now:
SGCN5K01# sh run
!Command: show running-config
!Time: Fri May 15 15:11:04 2009
version 5.2(1)N1(1b)
hostname SGCN5K01
no feature telnet
cfs eth distribute
feature lacp
feature vpc
feature lldp
username _____ password 5 $1$Bczq.c99$xbC4VG18Xg8M5knPFn2lG0 role network-admin
banner motd #Nexus 5000 Switch
#
ip domain-lookup
logging event link-status default
ip access-list classify_COS_4
10 permit ip 10.132.13.0/24 any
20 permit ip any 10.132.13.0/24
ip access-list classify_COS_5
10 permit ip 10.132.5.128/25 any
20 permit ip any 10.132.5.128/25
class-map type qos class-fcoe
class-map type qos match-all Silver_Traffic
match access-group name classify_COS_4
class-map type qos match-all Platinum_Traffic
match access-group name classify_COS_5
class-map type queuing class-fcoe
match qos-group 1
class-map type queuing class-all-flood
match qos-group 2
class-map type queuing class-ip-multicast
match qos-group 2
policy-map type qos Global_Classify
class Platinum_Traffic
set qos-group 2
class Silver_Traffic
set qos-group 4
class class-default
class-map type network-qos class-fcoe
match qos-group 1
class-map type network-qos class-all-flood
match qos-group 2
class-map type network-qos Silver_Traffic_NQ
match qos-group 4
class-map type network-qos class-ip-multicast
match qos-group 2
class-map type network-qos Platinum_Traffic_NQ
match qos-group 2
policy-map type network-qos jumbo
class type network-qos class-default
mtu 9216
multicast-optimize
policy-map type network-qos Setup_QOS
class type network-qos Platinum_Traffic_NQ
set cos 5
mtu 9000
class type network-qos Silver_Traffic_NQ
set cos 4
mtu 9000
class type network-qos class-default
multicast-optimize
system qos
service-policy type qos input Global_Classify
service-policy type network-qos jumbo
snmp-server contact Planview Infrastructure Dept.
snmp-server location SunGard Austin
snmp-server user admin network-admin auth md5 0x40a021c27c5d155338065b7a16f8b0b9 priv 0x40a021c27c5d155338065b7a16f8b0b9 localizedkey
snmp-server community pvcorporate group network-operator
vrf context management
ip route 0.0.0.0/0 10.132.5.1
vlan 1
vlan 13
name vMotion-VLAN
vlan 16,20
vlan 51
name MGMT-VLAN
vlan 52
name NFS-VLAN
vlan 61-67
spanning-tree port type edge bpduguard default
spanning-tree port type edge bpdufilter default
spanning-tree port type network default
vpc domain 1
role priority 10
peer-keepalive destination 10.132.5.33
port-profile default max-ports 512
interface port-channel4
switchport mode trunk
speed 10000
vpc 4
interface port-channel10
description vPC peer-link
switchport mode trunk
switchport trunk native vlan 999
switchport trunk allowed vlan 13,16,20,51-52,61-67
spanning-tree port type network
vpc peer-link
interface port-channel11
description sgcnscnt04
switchport mode trunk
switchport trunk native vlan 999
switchport trunk allowed vlan 52
spanning-tree port type edge trunk
vpc 11
interface port-channel12
description sgcnscnt05
switchport mode trunk
switchport trunk native vlan 999
switchport trunk allowed vlan 52
spanning-tree port type edge trunk
vpc 12
interface port-channel13
description SGCFIC01
switchport mode trunk
switchport trunk native vlan 999
switchport trunk allowed vlan 13,16,20,51-52,61-67
spanning-tree port type edge trunk
vpc 13
interface port-channel14
description SGCFIC02
switchport mode trunk
switchport trunk native vlan 999
switchport trunk allowed vlan 13,16,20,51-52,61-67
spanning-tree port type edge trunk
vpc 14
interface Ethernet1/1
description itype:trunk|dtype:switch|rdevice:SGCN5K02|rport:Eth1/1|notes:cisco_5548
switchport mode trunk
switchport trunk native vlan 999
switchport trunk allowed vlan 13,16,20,51-52,61-67
channel-group 10 mode active
interface Ethernet1/2
description itype:trunk|dtype:switch|rdevice:SGCN5K02|rport:Eth1/2|notes:cisco_5548
switchport mode trunk
switchport trunk native vlan 999
switchport trunk allowed vlan 13,16,20,51-52,61-67
channel-group 10 mode active
interface Ethernet1/3
description itype:trunk|dtype:switch|rdevice:SGCN5K02|rport:Eth1/3|notes:cisco_5548
switchport mode trunk
switchport trunk native vlan 999
switchport trunk allowed vlan 13,16,20,51-52,61-67
channel-group 10 mode active
interface Ethernet1/4
description itype:access|dtype:storage_head|rdevice:SGCNSCNT04|rport:e1A|notes:netapp_3220
switchport mode trunk
switchport trunk native vlan 999
switchport trunk allowed vlan 52
channel-group 11 mode active
interface Ethernet1/5
description itype:access|dtype:storage_head|rdevice:SGCNSCNT05|rport:e1A|notes:netapp_3220
switchport mode trunk
switchport trunk native vlan 999
switchport trunk allowed vlan 52
channel-group 12 mode active
interface Ethernet1/6
description itype:trunk|dtype:fabric_iconnect|rdevice:SGCFIC01|rport:Eth1/1|notes:cisco_6248
switchport mode trunk
switchport trunk native vlan 999
switchport trunk allowed vlan 13,16,20,51-52,61-67
channel-group 13 mode active
interface Ethernet1/7
description itype:trunk|dtype:fabric_iconnect|rdevice:SGCFIC01|rport:Eth1/2|notes:cisco_6248
switchport mode trunk
switchport trunk native vlan 999
switchport trunk allowed vlan 13,16,20,51-52,61-67
channel-group 13 mode active
interface Ethernet1/8
description itype:trunk|dtype:fabric_iconnect|rdevice:SGCFIC02|rport:Eth1/1|notes:cisco_6248
switchport mode trunk
switchport trunk native vlan 999
switchport trunk allowed vlan 13,16,20,51-52,61-67
channel-group 14 mode active
interface Ethernet1/9
description itype:trunk|dtype:fabric_iconnect|rdevice:SGCFIC02|rport:Eth1/2|notes:cisco_6248
switchport mode trunk
switchport trunk native vlan 999
switchport trunk allowed vlan 13,16,20,51-52,61-67
channel-group 14 mode active
interface Ethernet1/10
interface Ethernet1/11
interface Ethernet1/12
interface Ethernet1/13
interface Ethernet1/14
switchport mode trunk
channel-group 4 mode active
interface Ethernet1/15
switchport mode trunk
channel-group 4 mode active
interface Ethernet1/16
interface Ethernet1/17
interface Ethernet1/18
interface Ethernet1/19
interface Ethernet1/20
interface Ethernet1/21
interface Ethernet1/22
interface Ethernet1/23
interface Ethernet1/24
interface Ethernet1/25
interface Ethernet1/26
interface Ethernet1/27
interface Ethernet1/28
interface Ethernet1/29
interface Ethernet1/30
interface Ethernet1/31
interface Ethernet1/32
interface mgmt0
ip address 10.132.5.37/24
line console
line vty
boot kickstart bootflash:/n5000-uk9-kickstart.5.2.1.N1.1b.bin
boot system bootflash:/n5000-uk9.5.2.1.N1.1b.bin
ip route 0.0.0.0/0 10.132.5.1
Kindest Regards,
ALAN
05-01-2013 02:17 PM
Hi Alan,
Looking at your conifg, since the 5Ks are utilized as layer-2 devices, the only default route you need is the one you already have under vrf context management (mgmt0) The purpose of this default route is just for you to be able to manage the switches remotely. So, you don't need the global default route (below). Also, if you notice, it is pointing to the same ip as the one for mgmt0.
boot system bootflash:/n5000-uk9.5.2.1.N1.1b.bin
ip route 0.0.0.0/0 10.132.5.1
Also, I am assuming all the SVIs for all your vlans are on the 3750, it that correct?
HTH
05-01-2013 02:35 PM
Hey Reza,
Right on, the SVI's are defined in the 3750. Here's something interesting, if I ping from the management vrf, I can hit my tftp server: 10.132.6.144. If I ping from the default vrf, with the configuration as you see above, there is no response. That's why I added that global static route that shows up the same as the default gateway for the management vrf.
I figured as much about the default route for the mgmt0 interface, good to know that assumption was correct. Is the UCS manager served from the mgmt0 interface then?? Anywho, understanding is increasing but still I cannot ping the TFTP server.. Looking forward to getting this resolved and pending resolution I will grade and award correct answer here!
Thanks so much,
ALAN
05-01-2013 03:59 PM
Alan,
You not being able to ping the tftp server from the default vrf (global) is a correct behavior, because the tftp as you already know is part of vrf management and you can't ping it from default. Remember, vrf management and vrf default have different routing tables and you can not mix them together. Think of these 2 vrfs as 2 different devices, but in this case it is not a physical separation rather logical.
It that more clear now?
Reza
05-02-2013 11:39 AM
Hey Reza,
Awesome! You helped me connect the dots! When I copied the config to tftp this time I specific the management vrf and voila it worked. Still lots to learn but you've put me on the right path. Thanks so much, marking as correct answer and rating other posts!
Kindest Regards,
ALAN
05-02-2013 11:57 AM
Glad to help Alan.
Thanks for the rating and good luck!
Reza
05-28-2016 10:30 PM
I am having the same issue with management VRF , the stupidest thing ever added to multilayer switches. My environment was even more complicated by adding every device management port to the same vlan attached to the same vlan (10). I am trying to undo this mess . Now I need to buy a stupid switch(layer 2 1 gig ports) to physically separate the management network from the default VRF and access through firewall. I think this is a consequence of not spending the big bucks for 7K for layer 3. Especially if look at design guides and marketing material 5K are not the best routers yet alone layer switch.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide